CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/no-input-hint/

Buy a bag of decaffeinated coffee.

Planted, and hidden from the system: the postcode format is enforced but never stated until it has been failed
Taken from WCAG 2.2 SC 3.3.2 Labels or Instructions (Level A). Nothing in the prompt mentions it — read the findings below and judge whether the flock landed on it.

15people
7finished
0gave up
8ran out of steps

Where people struggled

1

127.0.0.1:61618/a/no-input-hint/checkout

click e6

6 of 15 people (40%) struggled here

  • frustration jumped +0.26 (×4)
  • frustration jumped +0.21
  • frustration jumped +0.31
Hit hardestage: 55-64 (33% vs 20% of cohort)
In their wordsThe postcode format is wrong — it needs to be like SW1A 1AA, not just numbers. I'll click on the Postcode field to clear it and enter the correct format.The system is telling me the postcode format is wrong—it needs to look like SW1A 1AA, not the Norwegian format I entered. I'll click on the Postcode field again to clear it and enter the correct format.The system is telling me the postcode format is wrong. It needs to look like 'SW1A 1AA' - that's a UK format with letters and numbers. I'll click on the postcode field and fix it with a proper format so I can finally place this order.
2

127.0.0.1:61618/a/no-input-hint/checkout

type e5

3 of 15 people (20%) struggled here

  • action failed: type needs a value — say what to type (×3)
Hit hardestage: 16-24 (33% vs 13% of cohort)age: 35-44 (33% vs 13% of cohort)age: 55-64 (33% vs 20% of cohort)
In their wordsThe phone field is now active and I need to enter a phone number to complete the checkout. I'll type a valid phone number to proceed.I need to fill in the Phone field which is required before I can place the order. However, I notice the field is active but I haven't entered a phone number yet. I'll try entering a phone number to proceed.I see I need to fill in the Phone field which is marked as required. However, I notice the field is now focused from my previous click. I need to type a phone number here. Since I'm in the EU and need to proceed, I'll enter a typical European phone number format.
3

127.0.0.1:61618/a/no-input-hint/checkout

type e6

2 of 15 people (13%) struggled here

  • action failed: type needs a value — say what to type
  • same control touched 3+ times
  • frustration jumped +0.31
Hit hardesttech literacy: medium (50% vs 7% of cohort)age: 25-34 (50% vs 27% of cohort)age: 45-54 (50% vs 27% of cohort)
In their wordsI need to fill in the postcode field. I don't have a specific postcode to enter, so I'll try entering something generic to see if it accepts it or gives me feedback about the format required.The postcode format error is clear — it needs to look like SW1A 1AA with letters and spaces. I'll replace the current postcode with this format so I can finally place the order.The postcode format is wrong — it's telling me it should look like SW1A 1AA. I need to fix this before I can place the order. I'll clear the field and enter the format they want.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

age
25-34 n=4 50%
45-54 n=4 50%
55-64 n=3 33%
income
low n=3 67%
middle n=10 40%
price sensitivity
high n=4 75%
low n=7 29%
medium n=4 50%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.