Why plan your Usability Engineering activities?
What is User Interface Evaluation planning and why should you make an evaluation plan in the beginning of your project?
Most larger projects begin with a detailed plan, which is then updated as the project progresses. This is an important tool to ensure progress and movement in the right direction throughout the project. Similarly, it is very valuable to make a User Interface Evaluation plan, which gives an overview of the activities related to Usability Engineering throughout the project.
Naturally, this can be integrated in the project plan, but more details can be put in a separate plan. This plan can then enable execution of user research and user interface evaluation activities and ensure agreement on activities and dependencies involving potential usability issues across the R&D disciplines.
The documentation of User Interface Evaluation planning can be structured as illustrated below:
The User Interface Evaluation Plan is the top level plan, complimented by the specific formative and summative evaluation protocols.
What a User Interface Evaluation Plan must contain
The User Interface Evaluation Plan should include the following:
- A brief description of the intended device and User Interface, including which parts are to be evaluated. This will frame the evaluation approach and activities needed. This description will change as the design matures. In the early development stages, preliminary sketches and descriptions will suffice to define and plan the Usability Engineering activities.
- An evaluation of the overall product risk level. High-risk products, like implants, most likely requires a larger Usability Engineering effort. It is important to capture these considerations early on, as the forthcoming evaluation activities will depend hereof.
- A plan of the different activities and a rationale and purpose of each (include reference to the overall project plan). This is important since it outlines the sequence and importance of each activity and is part of documenting the evaluation process.
- Project dependencies, ex-state of prototypes and IFU to be ready for each test and a pilot summative test to be conducted prior to design freeze.
- Who is responsible for each activity? This will help ensure that the plan is followed and updated but also that capable resources are appointed each activity.
- How the result of each activity will be integrated into the development project, to make sure the Usability Engineering efforts create value in the product and project
The plan should be updated throughout the project so that it reflects the current conducted and planned activities, ex if a new test iteration is required because a test has revealed new use errors, which needs to be mitigated, which will lead to new tests to evaluate the effectiveness of these mitigations.
Different strategic considerations should be made to optimize the Usability Engineering process, including, among others, launch strategy and which formative test strategy to apply. These aspects are discussed below.
An important input to the User Interface Evaluation plan is which countries the product will be marketed in, an in what sequence. For instance, if the product is to be launched in the US, the summative usability evaluation must be performed in the US as the participants should reside in the US. This is often very time consuming and expensive, but the test will be valid both in the US and the rest of the world.
However, if the product will not be put on the US market, it often makes sense to conduct the summative test in Denmark (i.e. the country of residence of the legal manufacturer). This will lower the cost and be both easier and faster for the Usability Engineering team to set up and conduct the test. In addition to this, the US FDA’s expectation of 15 test participant per user group, even though reasonable, is not a requirement in the EU. For this reason, to get quickly on the market in the EU, it can sometimes make sense first to conduct the summative test in the EU and postpone the summative test in the US to when the product is to be marketed there.
It is important to note that a summative validation test conducted outside the US, is not likely to be accepted by the FDA.
Integrating User Interface evaluation early on
There are multiple reasons to prioritize formative UI evaluation activities in the early development stages. A few are described below:
The formative UE activities performed can and should support the final summative evaluation documentation. IEC 62366-1:2015 is a process standard, requiring you to show the process of how you have optimized the user interface and mitigated the use related risks.
It makes sense to front-load the UE activities to get valuable design input in the early development stages. This will allow for much quicker and cheaper design changes as illustrated in the graph below. For instance, it is crucial to select the right User Interface design concept from the beginning, as this can be almost impossible and at least expensive and time-consuming to change at a late stage. This is when the product gets more defined and multiple dependencies start to shape the final product.
We find that it is often more time consuming to patch a flawed UI design concept late in the development process than to spend time selecting the right one from the beginning. This includes defining the division of tasks between the user and the device. Careful considerations can save a lot in the later Usability Engineering work and validation activities. Remember, if you can verify that a use risk is eliminated by design, you do not have to validate it.
Recommendations for your formative evaluation strategy
- There are no requirements as to how many formative evaluations to perform. At Technolution we recommend more iterations with fewer test participants vs. fewer test iterations with more test participants. This allows for multiple design iterations, which is beneficial to obtain an optimal UI design.
- Remember that internal reviews or expert UI evaluations are also effective and cost-efficient ways of evaluating your User Interface. Apply and document them to get valuable design input and credit for the work. However, be careful to consider the pitfalls by not testing on representative users. This can potentially give misleading data if not handled correctly since non-representative test participants might have different capabilities and insights for using the product.
- Technolution also recommends conducting a final formative test mimicking the summative test before Design Freeze. This will help prepare the summative test and verify that the test setup works. If testing multiple similar user groups, formative tests can show if they perform equally. This might then be used to argue that two similar user groups can be pooled into one in the summative test, hence reducing the number of test participants, time and resources spent. This needs to be carefully evaluated as it should always be the first priority to test as many user groups as needed to cover ALL intended users. We always recommend having the Human Factors validation protocol reviewed by the FDA, since this will minimize the risk of having to redo the summative test if the FDA disagrees with the test setup.
Get ready, set, plan!
We hope that the above pointers have given you input to your project planning and how to reduce the risk of rework late in your project. Remember to dedicate enough resources to Usability Engineering and you will not regret it. If you have any questions, please don’t hesitate to contact us at Technolution.
Jon Storm Madsen
Partner / Head of Usability Engineering