CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/inconsistent-naming/

Buy a bag of decaffeinated coffee.

Planted, and hidden from the system: the same container is called Basket, Cart and Bag on different pages and controls
Taken from WCAG 2.2 SC 3.2.4 Consistent Identification (Level AA); Nielsen heuristic #4. Nothing in the prompt mentions it — read the findings below and judge whether the flock landed on it.

15people
10finished
0gave up
5ran out of steps

Where people struggled

1

127.0.0.1:61618/a/inconsistent-naming/checkout

click e6

4 of 15 people (27%) struggled here

  • frustration jumped +0.31 (×2)
  • frustration jumped +0.26 (×2)
Hit hardestage: 16-24 (25% vs 13% of cohort)age: 35-44 (25% vs 13% of cohort)device: desktop (50% vs 27% of cohort)
In their wordsI see there's an error message about the postcode format — it needs to look like SW1A 1AA with letters and numbers in a specific pattern. My French postcode '75001' is not in the right format for this system. I'll click on the postcode field again to correct it with a UK-style postcode format that matches their example.The page is telling me my postcode format is wrong. It wants something like SW1A 1AA format. I need to click on the Postcode field again and fix it with the correct format.The page is telling me my postcode format is wrong — it needs to look like 'SW1A 1AA' with letters and spaces. I'll click on the postcode field and enter a properly formatted UK postcode this time.
2

127.0.0.1:61618/a/inconsistent-naming/checkout

type e6

2 of 15 people (13%) struggled here

  • frustration jumped +0.31
  • action failed: type needs a value — say what to type
Hit hardestage: 45-54 (50% vs 27% of cohort)age: 25-34 (50% vs 27% of cohort)device: desktop (50% vs 27% of cohort)
In their wordsThe system is telling me the postcode format is wrong. It wants the British format SW1A 1AA. I'll clear my entry and use the format it's asking for so I can finally place this order.I'm on the checkout page and have filled in name, email, and phone. Now I need to enter the postcode in the Postcode field to complete all required information before I can place the order.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

device
desktop n=4 50%
mobile n=11 73%
price sensitivity
high n=4 75%
low n=7 43%
medium n=4 100%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.