From Simulation to the Real World

Simulated use studies are vital in medical device development, but do they truly reflect real-world behaviour? This article explores how behavioural insights can help close that gap and bring greater authenticity to human factors research

Simulated use studies are a crucial component of usability evaluations in medical device development. They enable identification of use-related risks, inform device development, and provide critical evidence that the device’s user interface mitigates risks to an acceptable level, whilst supporting regulatory expectations. But what happens when the behaviour we observe in these studies doesn’t reflect how people act in the real world?

When conducting simulated use studies, it’s common to observe discrepancies. In some cases, participants behave more cautiously or attentively, aware that they are being observed or recorded. This can result in overperformance compared to typical home use. Conversely, unfamiliar settings, test-related anxiety, or awareness of surveillance may induce errors that wouldn’t otherwise occur. While a few participants openly acknowledge this shift in behaviour, many do not, and some may not even realise their responses are uncharacteristic.

In a recent study, a participant commented, “The cameras put me off!” Nerves from being recorded and in the presence of cameras led to potential use-related issues outside the simulated environment, indicating a potential false negative. When a participant, on the other hand, said, “I read the instructions today because you were watching me, but I probably wouldn’t if I was at home”, it signals that they have become more diligent under observation: a potential false positive, giving the impression that the device is easier to use than it really is. In both examples, the observed behaviour of the participants was potentially not realistic.

The behavioural gap is significant for various reasons. First, it impacts the validity of results, as over- or under-performance can conceal or exacerbate usability issues, which may lead to poor development decisions. Additionally, safety concerns arise when critical use errors go undetected due to inauthentic testing behaviour. This gap also affects regulatory compliance, as unrealistic data can lead to delays, denials, or problems after a product reaches the market. Finally, misunderstandings about user needs can obstruct satisfaction and hinder the adoption of new devices.

What Drives Behavioural Differences?

Several factors that can influence how users behave in simulated use studies:

• Perceived risk: Without real-world consequences  (e.g., wrong medication dose), participants may not apply the same level of caution, leading to missed safety checks or shortcuts that do not reflect real-world vigilance.
• Confirmation bias: Participants may interpret new information in a way that fits their existing beliefs, overlooking differences in the device or IFUs. When faced with serious medical situations in life, users are typically more alert to what’s unfamiliar or critical, rather than relying solely on prior assumptions.
• Confidence or anxiety: Overconfidence may cause participants to skip steps or ignore difficulties, while anxiety may introduce atypical errors, distorting performance data.
• Artificial environments: Controlled, quiet study settings can remove real-life distractions, resulting in a hyper-focused evaluation that limits the ability to observe how users perform when managing interruptions or competing tasks.
• Moderator presence: Being observed often changes the behaviour of participants. They may try harder to perform well or behave unnaturally, reducing the authenticity of their actions.

These discrepancies highlight the importance of designing studies that elicit authentic behaviour.

Why behavioural insights matter

To reduce the gap between real-world behaviour and simulated use and mitigate the associated risks we first need to understand the factors driving these discrepancies. Behavioural insights offer a valuable approach.

Behavioural insights draw from psychology, behavioural economics, and social sciences to explore how people make decisions and act in practice. These decisions are often influenced by mental shortcuts, emotional reactions, and the surrounding environment, rather than deliberate, rational thinking. (OECD Behavioural Insights Guide)

By observing behaviour in both controlled and real-world settings, we can uncover key differences and refine study designs to better reflect actual use. Importantly, this isn’t just about recreating the external environment, but considering the mindset, motivations, and habits users bring to real-world use, because without representative behaviour, we cannot confidently assess a device’s safety,or effectiveness.

Encouraging Authentic Behaviour: Behavioural Approaches

Incorporating behavioural principles into study design can encourage more authentic user engagement in a subtle, respectful way. The following strategies can help:

• Priming: Introduce cues that activate relevant mental pathways and mindsets, such as urgency or first-time use. You could, for example, provide brief, context-setting scenarios or visuals to prompt first-time use thinking or emergency context.
• Anchoring: Shape participant expectations and motivation through meaningful context. Consider linking the activity to personal goals or outcomes, such as improving patient safety or helping others.
• Framing: Choose neutral, non-evaluative language to avoid creating artificial expectation and performance pressure. In other words, frame the task as an everyday activity, not a test.

Balancing Insight and Rigour

While behavioural techniques enhance realism, they must be balanced with methodological rigour.

To ensure that behavioural techniques enhance rather than compromise the integrity of a study, they must be implemented within a robust methodological framework. This includes maintaining non-leading discussion guides so that any cues or prompts do not bias how participants complete tasks. Ethical boundaries must also be respected at all times, participants should not be manipulated, and their autonomy must be preserved. Finally, transparency is critical: all behavioural methods and their justifications should be clearly documented to meet regulatory expectations and maintain scientific integrity.

Human behaviour in simulated use is shaped by more than the test environment. Recognising the mindsets and contexts that influence real use helps ensure studies deliver reliable, representative evidence. If you’d like to discuss how these principles can strengthen your studies, we would be happy to talk.