CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/no-exit/

Buy a bag of decaffeinated coffee.

Planted, and hidden from the system: checkout removes the navigation and offers no way back to the basket
Taken from Nielsen heuristic #3 User Control and Freedom. Nothing in the prompt mentions it — read the findings below and judge whether the flock landed on it.

15people
12finished
0gave up
3ran out of steps

Where people struggled

1

127.0.0.1:61618/a/no-exit/checkout

click e3

3 of 15 people (20%) struggled here

  • frustration jumped +0.26 (×2)
  • frustration jumped +0.21
Hit hardestage: 35-44 (33% vs 13% of cohort)
In their wordsThe postcode format is wrong — it needs to be like SW1A 1AA instead of 12345. I'll click on the postcode field and fix it with a proper UK format to get past this error.The postcode format is wrong — it needs to be like 'SW1A 1AA' not just numbers. I'll click on the postcode field to clear it and enter a proper format.The postcode field still shows the error. I need to click on it first to focus and clear it before typing the correct UK format 'SW1A 1AA'.
2

127.0.0.1:61618/a/no-exit/checkout

type e2

2 of 15 people (13%) struggled here

  • action failed: type needs a value — say what to type (×2)
  • frustration jumped +0.15
Hit hardestage: 55-64 (100% vs 20% of cohort)
In their wordsI'm at the checkout page and need to fill in the Phone field, which is marked as required. I'll enter a phone number to continue with the checkout process.I'm filling out the checkout form to complete my coffee purchase. The Name and Email fields are already done, so now I need to enter my phone number in the Phone field to continue.
3

127.0.0.1:61618/a/no-exit/checkout

type e3

2 of 15 people (13%) struggled here

  • action failed: type needs a value — say what to type
  • frustration jumped +0.26
Hit hardesttech literacy: medium (50% vs 7% of cohort)age: 25-34 (50% vs 27% of cohort)age: 45-54 (50% vs 27% of cohort)
In their wordsI need to fill in the Postcode field to complete the checkout form. The field is required and empty, so I'll enter a postcode to proceed with placing my order.The system is telling me the postcode format is wrong. It wants it to look like 'SW1A 1AA' with a space. I'll clear the current entry and type it in the correct format so I can finally place the order.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

age
25-34 n=4 75%
45-54 n=4 75%
55-64 n=3 100%
income
low n=3 33%
middle n=10 100%
patience
high n=7 86%
medium n=6 67%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.