By Eric Andrew Kristof, RN, CAHIMS | Healthcare IT Professional | HIMSS Arkansas Chapter Member
About This Series: The HIMSS Word of the Day is a running blog series drawing from the HIMSS Dictionary of Health Information and Technology Terms, Acronyms, and Organizations, 6th Edition (2025) — the authoritative reference for healthcare IT professionals and the primary study resource for CAHIMS and CPHIMS certification. Each post paraphrases the term in plain language, grounds it in real-world healthcare IT context, and adds a bedside nursing perspective you won’t find in a textbook.
📖 The Term: Usability Testing
Source: HIMSS Dictionary of Health Information and Technology Terms, Acronyms, and Organizations, 6th Edition (2025)
🔍 Plain-Language Definition
Usability testing is a way of evaluating a product or service by having representative users try it. In a typical session, participants work through realistic tasks as observers look on, listen, and record what happens. According to the HIMSS Dictionary, the aims are to find usability problems, gather qualitative and quantitative data, and learn how satisfied participants are. In health IT, the product might be an EHR order screen, a patient portal, a discharge summary, or a medical device interface.
Usability testing is one of several evaluation methods. Heuristic evaluation has experts inspect an interface against recognized principles. The think-aloud method asks users to say what they are thinking as they work. Scenario-based testing gives users realistic cases to complete. Questionnaires and interviews capture satisfaction and opinions. The standard usability measures are effectiveness (can users complete the task), efficiency (how much time and effort), and satisfaction (how users feel about the experience).
| Method | Who Takes Part | What It Finds |
|---|---|---|
| Heuristic evaluation | Experts | Violations of design principles |
| Think-aloud testing | Representative users | Confusion, wrong assumptions, workarounds |
| Scenario-based testing | Users completing realistic cases | Barriers within the workflow |
| Laboratory testing | Users in a controlled setting | Timing, errors, and success rates |
| Questionnaire or interview | Users | Satisfaction and perceptions |
⚠️ Exam Alert: Usability testing involves users performing tasks; a heuristic evaluation is an expert inspection without users; usability is the quality being measured; human factors engineering is the broader discipline. Testing during design is formative; testing to confirm that a finished product meets criteria is summative.
🌐 Why It Matters: Real-World Healthcare IT Context
Poor EHR usability is linked with patient safety concerns, user dissatisfaction, and clinician burnout. Vendors perform user testing as part of certification, but organizations customize their systems, and those local changes can create new problems that vendor testing never saw. Usability testing at the local level is how organizations catch them before go-live.
A 2024 literature review in the Journal of Medical Internet Research looked at which methods researchers use to evaluate electronic discharge summaries. Searching from inception to July 2023, it included 12 records (11 studies and one thesis). Heuristic evaluation was the most used method (seven records), followed by think-aloud (five) and laboratory testing (three), often combined with questionnaires (three) or semistructured interviews (four). The studies measured user performance, satisfaction, efficiency, and impact. The authors concluded that few studies examined discharge summary usability and that more research is needed to identify the most appropriate method for judging whether summaries are fit for purpose. The practical message is that no single method suffices: expert review is quick and finds many issues, and user testing finds how real people behave.
A 2022 report in Applied Clinical Informatics shows what a local program can accomplish. An academic medical center launched site-specific “discount” user testing, a lightweight approach that does not require a dedicated usability lab or specialists. Thirteen new EHR features, all of which had already received heuristic evaluation and iterative design, were tested in three events with five or six users each, using think-aloud and measures of efficiency, effectiveness, and satisfaction. Usability concerns arose at a rate of 2.5 per feature, and 70 percent were judged correctable before implementation. One feature moved to production despite low scores on the single ease question and was later withdrawn after user feedback; another was rebuilt, retested, and then released. The authors concluded that simplified testing with few participants finds correctable problems even after heuristic evaluation. It is a single-center experience report and cannot show effects on outcomes, but it demonstrates a feasible model.
Nursing documentation provides a third example. A 2013 study in the International Journal of Medical Informatics evaluated a nursing information system, a module of the EHR, in two hospitals of a three-hospital system (760 and 300 beds) shortly after implementation. Twelve nurses completed standardized scenarios while thinking aloud. The team found that documentation was hindered by several aspects of the interface and identified facilitators that could be strengthened. The study illustrates how scenario-based testing with the actual users reveals problems that requirements documents do not.
Good practice follows a few principles. Test early and repeat, since cheap changes are easiest before build is final. Recruit participants who resemble real users in role, experience, and unit. Use realistic tasks and data, including high-risk and unusual cases. Capture quantitative measures, such as completion, time, errors, and a simple rating, together with the observers’ notes. Prioritize findings by severity and the risk of harm, assign owners, and retest the fixes. The National Institute of Standards and Technology maintains resources on health IT usability, and the federal government’s Digital.gov site has guidance on usability methods.
Common usability testing problems:
- Testing only after the build is complete and changes are costly
- Using staff who are not representative of end users
- Unrealistic tasks or test data
- Reporting opinions without measures
- Findings with no owner or no retest
- Treating vendor certification testing as proof that the local build is usable
What supports effective usability testing:
- Early and repeated testing, including small sessions
- Participants drawn from the actual roles and units
- Realistic scenarios with clinical risk in mind
- Combined methods: expert review, think-aloud, measures, and interviews
- Severity ratings and tracked fixes
- Sharing results with governance and build teams
🏥 The Nurse’s Perspective: Clinical, Bedside, & Workflow
Nurses are among the heaviest users of the EHR, and their participation determines whether testing reflects reality.
At the Bedside
Bedside nurses see usability problems daily: a medication screen that takes too many clicks, a flowsheet row that is hard to find, an alert that interrupts at the wrong time. Reporting these specifically, including what the nurse was trying to do and what happened, gives a testing team material to work with.
In Clinical Workflow
Nurses serve as participants in think-aloud sessions and scenario tests, bringing the realistic cases that matter, such as a busy admission or a rapid change in a patient’s condition. Nurse managers release staff for sessions and help choose scenarios.
Nurse informaticists plan and run tests, write scenarios, observe sessions, rate severity, and make sure that findings return to the build team. They also help distinguish a usability problem from a training problem, which call for different fixes.
In EHR Implementation (Go-Live Perspective)
An Epic Go-Live moves the results of usability testing from the test environment to the unit. At-the-elbow support analysts see what testing missed, such as workarounds on night shift or a screen that behaves differently with real patient volumes, and they feed those observations back as new test cases for later releases.
🎓 CAHIMS / CPHIMS & HIMSS Perspective
CAHIMS Exam Domain Mapping
Usability testing appears in design, evaluation, and implementation questions:
| CAHIMS Domain | How Usability Testing Appears |
|---|---|
| Healthcare & Technology Environments | Certification testing, patient safety, human factors |
| Clinical Informatics | Workflow design, decision support design, user-centered design |
| Healthcare Information & Systems Management | System testing, implementation, optimization |
| Management & Leadership | Governance of changes, vendor management, change management |
Key Terms to Know Alongside Usability Testing
- Heuristic Evaluation — expert inspection of an interface against usability principles
- Think-Aloud Protocol — method in which users voice their thoughts while performing tasks
- User-Centered Design — design approach that involves users throughout development
- Human Factors Engineering — discipline applying knowledge of human capabilities to system design
- Formative and Summative Testing — testing to improve a design versus testing to confirm it meets criteria
- Single Ease Question (SEQ) — one-item rating of how easy a task was
HIMSS Organizational Position
HIMSS supports usable, safe health information systems through its research, education, and community work, including its Nursing Informatics Community. Members can find related resources through the HIMSS Nursing Informatics Community page.
🔗 Explore Further: External References
- Tesfaye et al. (2024), Journal of Medical Internet Research — Usability Evaluation Methods Used in Electronic Discharge Summaries: Literature Review https://pubmed.ncbi.nlm.nih.gov/39264712/
- Pierce et al. (2022), Applied Clinical Informatics — Improving the User Experience With Discount Site-Specific User Testing https://pubmed.ncbi.nlm.nih.gov/36323335/
- Rogers et al. (2013), International Journal of Medical Informatics — Use of a Human Factors Approach to Uncover Informatics Needs of Nurses in Documentation of Care https://pubmed.ncbi.nlm.nih.gov/24008175/
- NIST — Health Information Technology Usability https://www.nist.gov/programs-projects/health-information-technology-usability
- Digital.gov — Usability https://digital.gov/topics/usability/
Term source: HIMSS Dictionary of Health Information and Technology Terms, Acronyms, and Organizations, 6th Edition (2025). All definitions in this series are paraphrased for editorial purposes. Readers are encouraged to consult the primary source for exact language.
Eric Andrew Kristof, RN, CAHIMS is a Healthcare IT professional and HIMSS Arkansas Chapter member based in Hot Springs Village, AR, with hands-on Epic Go-Live experience and a diverse healthcare and IT background. He writes at kristof.org at the intersection of clinical care and healthcare technology.
Leave a Reply