Question 4 of 14: Same Users vs. New Users Across Iterative Design Cycles

Question 4: Same Users vs. New Users Across Iterative Design Cycles (10 marks)

Question text not reproduced: the examination questions are © Engineers and Geoscientists BC. Open the official past paper (linked at the top of this page) to read the question, then follow the worked solution below.

Both choices are defensible, and in practice most iterative design programmes use a mix; the trade-off is between learning depth and validity of the usability measurements. Reasons to reuse the same users across iterations: continuity — participants already familiar with the domain (pharmacy drug-library management) do not need to be re-screened or re-trained on background context each round, which saves recruiting effort and session time; longitudinal insight — the same participants can meaningfully comment on whether a specific problem they raised in round 1 was actually fixed in round 2, giving direct before/after feedback that new participants cannot provide; and rapport — returning participants are often more candid once they trust the evaluator and understand the process is genuinely iterative, not a one-shot judgement of them. Reasons to recruit new users each iteration: learning effects — a returning participant remembers the previous version's workflow and workarounds, so their performance and even their opinions are contaminated by that memory rather than reflecting how a first-time user experiences the current design, which is what usability testing is meant to measure; novelty bias fades — a returning user may rate the interface more favourably simply because they recognise improvements relative to what they remember, inflating perceived usability; and generalisability — testing with fresh users on each iteration gives a truer read of whether the design would be usable by the actual (mostly first-time) population of hospital pharmacists it will be deployed to, rather than by a small panel that has effectively become expert users of the prototype itself.

A common resolution is a rotating or partially overlapping panel: keep a subset of returning users to track whether specific known issues were fixed, while introducing new users each round to get an unbiased first-impression read on the design as a whole — and always use entirely fresh users for the final validation/summative test, since that test's purpose is to certify usability for the real, mostly first-time population.