CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/image-only-link/

Buy a bag of decaffeinated coffee.

Planted, and hidden from the system: the only route from the shop to a product is an image link with no alt text, no aria-label and no accompanying text link
Taken from WCAG 2.2 SC 1.1.1 Non-text Content (Level A). Nothing in the prompt mentions it — read the findings below and judge whether the flock landed on it.

15people
13finished
0gave up
2ran out of steps

Where people struggled

1

127.0.0.1:61618/a/image-only-link/checkout

type e5

5 of 15 people (33%) struggled here

  • action failed: type needs a value — say what to type (×5)
  • frustration jumped +0.15 (×2)
  • same control touched 3+ times
Hit hardestage: 45-54 (60% vs 27% of cohort)device: desktop (60% vs 27% of cohort)age: 25-34 (40% vs 27% of cohort)
In their wordsI need to fill in the Phone field which is required for checkout. I'll enter a phone number to complete the form.I need to fill in the Phone field which is marked as required. I'll enter a phone number to continue completing the checkout form.I need to fill in the phone number field which is marked as required. I'll enter a phone number to continue with the checkout process.
2

127.0.0.1:61618/a/image-only-link/checkout

type e6

4 of 15 people (27%) struggled here

  • action failed: type needs a value — say what to type (×3)
  • same control touched 3+ times
  • frustration jumped +0.26
Hit hardesttech literacy: medium (25% vs 7% of cohort)age: 45-54 (50% vs 27% of cohort)
In their wordsI can see the Postcode field [e6] is still empty and required. I need to fill this in before I can place the order. I'll enter a postcode to complete all required fields.The postcode field needs to be in the correct format like 'SW1A 1AA' according to the error message. I'll enter this format example to fix the validation error so I can finally place my order.I need to fill in the postcode field which is the last required field before I can place the order. The form is almost complete.
3

127.0.0.1:61618/a/image-only-link/checkout

click e6

3 of 15 people (20%) struggled here

  • frustration jumped +0.31 (×2)
  • frustration jumped +0.26
  • confusion jumped +0.18
Hit hardestage: 35-44 (33% vs 13% of cohort)age: 16-24 (33% vs 13% of cohort)
In their wordsThe postcode format error is clear — it needs to be like 'SW1A 1AA' (UK format), not the French format I entered. I'll click on the Postcode field to correct it with a proper UK-style postcode before trying to place the order again.The postcode format is wrong — it needs to be like 'SW1A 1AA' not '12345'. I'll click the postcode field and fix it with a properly formatted postcode.The postcode format is wrong — I need to fix it. The error message shows it should look like 'SW1A 1AA', so I'll click on the postcode field again and enter a properly formatted postcode.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

age
25-34 n=4 100%
45-54 n=4 75%
55-64 n=3 100%
device
desktop n=4 50%
mobile n=11 100%
income
low n=3 100%
middle n=10 80%
price sensitivity
high n=4 75%
low n=7 86%
medium n=4 100%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.