Why Human Factors Formative and Summative Usability Testing Should be Conducted Externally.
Formative and Summative usability testing plays a pivotal role in the product design and development process, ensuring that the device is designed for safe and effective use and will achieve regulatory and commercial success. However, a critical question arises: who should conduct these crucial evaluations? While there is an intrinsic appeal to performing these studies in-house, there is a growing consensus that external execution by a human factors company mitigates the risk of an unsuccessful FDA submission. In this article, we will explain why we recommend conducting formative and summative usability testing externally.
The Benefits of Conducting Formative and Summative Usability Testing Externally
Human Factors Engineers (HFE) who have not been involved in the design and development of the product should be brought in to evaluate the product for a couple of reasons—first, bias. People who have been part of a product’s design and development may have a personal attachment to the design of the product or its success and not effectively capture usability issues or use errors. People with a stake in the product’s success may (consciously or subconsciously) fail to identify use errors or difficulties—second, fresh eyes. A second set of eyes may catch a use error or difficulty that has gone unnoticed by people who have been so embedded in the design and development of the product. Our brains tend to auto-correct before we even realize it, which is why we are no longer able to catch typos after having read a document so many times.
From a psychological perspective, prolonged exposure and familiarity with a project can cause even the most meticulous product development teams to overlook obvious errors. This phenomenon, known as ‘perceptual adaptation,’ suggests that our brains tend to normalize repeated stimuli, making it increasingly challenging to identify anomalies over time. In the context of summative evaluation, this implies that teams deeply involved in the design process may unconsciously disregard issues or design flaws simply because they have become accustomed to them.
External Human Factors Engineers can provide input on conducting formative and summative usability testing, test protocols, and the necessary documentation required. Usability studies typically include task analysis, use-related risk analysis, protocol development, moderator script/discussion guide development, and study execution. As human factors consultants, our responsibility is to position our clients best to succeed. Sometimes, this means expanding the scope of our work to include educating a broader team of client stakeholders on the FDA’s expectations when it comes to documentation in the usability file. This has included the strategic use or omission of particular language in the documents, appropriately performing a use-related risk analysis of a product, and adequately designing a summative evaluation to yield the best data.
Planning, executing, and reporting on formative and summative usability testing is very time- and labor-intensive. It requires significant HFE resources. In some cases, four HFEs may be needed to execute a successful usability evaluation. This type of testing with an extended HFE team allows for simultaneous sessions and efficient data collection, enabling the study to be completed in less time. Resourcing a summative study with multiple HFEs also ensures that the user testing data is analyzed from different perspectives to tell a richer story to the FDA about the safety, effectiveness, and usability of the medical device. In addition, an external Human Factors company can take on the burden of interacting with the FDA in response to Human Factors-related comments and inquiries to free up your internal HFEs to support other projects.
Engaging an external Human Factors company for summative studies, despite appearing as an initial expense, leads to significant long-term economic benefits and time efficiencies. One of the primary advantages lies in the early identification and rectification of overlooked usability engineering issues. When internal development teams overlook or underestimate certain design flaws or usability problems, these issues can escalate into more complex problems during the product’s development as you head toward launch. An external review, providing a fresh, objective perspective of the user experience, can detect these ease-of-use irregularities in their early stages, thereby saving considerable time and resources that would otherwise be spent on post-launch troubleshooting or modifications.
Moreover, the identification and resolution of potential issues during the summative study phase significantly mitigate the risk of costly redesigns or product recalls post-market launch. Such incidents, besides their direct financial implications, have the potential to damage a brand’s reputation, resulting in long-term economic consequences, as well as eroding trust and user satisfaction in the product and brand. External evaluations serve as a proactive measure, ensuring the readiness of the product for the market while minimizing post-launch complications. Furthermore, a product that undergoes rigorous validation by an impartial external entity carries a seal of reliability. This strengthens its market positioning and fosters user trust, a crucial differentiating factor in competitive markets. Ultimately, the initial investment in external formative and summative usability testing can yield substantial returns regarding cost savings, time efficiencies, and enhanced market credibility.
Rich knowledge base. An external Human Factors company consists of HFEs who have experience working with other clients across various medical devices, therapy areas, and intended users. When a client hires an external firm to conduct HF activities, they are also getting the consultant’s knowledge and experience they’ve acquired.
External Human Factors companies bring diverse perspectives to formative and summative usability testing due to their varied backgrounds and experiences. Internal development teams often have a relatively homogeneous understanding shaped by a singular organizational culture and mission. On the other hand, external evaluators offer insights gleaned from numerous projects across different sectors. This extensive experience equips them to approach a design concept from multiple angles, anticipating a broader range of user needs, expectations, and potential challenges. Their professional expertise, informed by diverse contexts, enhances the evaluation process and ensures comprehensive results.
Considering the number of user groups and scenarios is crucial for the success of any product. Products do not exist in isolation; users with diverse cultural backgrounds, physical abilities, and technological proficiencies may all access them. External teams, with their extensive experiences, possess the necessary sensitivity to identify and recommend adjustments that resonate with diverse and representative users. In today’s market, inclusivity and accessibility are essential requirements. Diverse viewpoints play a critical role in ensuring that a product is not only functional but also universally usable, reducing barriers and maximizing its reach.
With product success relying on innovation, usability, and user trust, the role of external formative and summative usability testing is of paramount importance. For companies on the verge of launching new products or refining existing ones, the message is crystal clear: prioritize external evaluations. By doing so, you safeguard your product’s integrity and strengthen its position in the market, ensuring that it resonates genuinely and effectively with its intended users. Embrace the power of external insights and let them guide your product development process.
______________________________________________________________________________
HUMAN FACTORS ENGINEERING AT THRIVE
THRIVE’s Human Factors professionals have decades of experience applying Human Factors to products ranging from medical devices used by specialized healthcare professionals in clinical environments to combination products used by laypeople in the home.
If it’s your first time applying to Human Factors and you need a comprehensive end-to-end Human Factors program, we’ll scope out the program and conduct the activities on your behalf, leaving you time and resources to focus elsewhere. If you’re a resource-constrained HFE professional, we’ll provide the teamwork, collaboration, and support to help you meet your goals. Or, if you want a final sanity check to ensure you’ve met the latest and greatest expectations, we’ll do that, too.
Contact us today at (404) 228 7342 or email [email protected] to get the conversation started.
Atlanta | Chicago
