CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/vague-error/

Buy a bag of decaffeinated coffee.

Planted, and hidden from the system: the phone field rejects spaces and says only 'Invalid entry.' — no field named, no fix offered
Taken from WCAG 2.2 SC 3.3.1 Error Identification and SC 3.3.3 Error Suggestion (Level A); Nielsen heuristic #9. Nothing in the prompt mentions it — read the findings below and judge whether the flock landed on it.

15people
4finished
3gave up
8ran out of steps

Where people struggled

1

127.0.0.1:61618/a/vague-error/checkout

click e8

7 of 15 people (47%) struggled here

  • frustration jumped +0.31 (×6)
  • confusion jumped +0.18 (×5)
  • confusion jumped +0.23
  • frustration jumped +0.26
Hit hardestage: 55-64 (29% vs 20% of cohort)
In their wordsI'm getting an 'Invalid entry' error but I don't know which field is wrong. Rather than guess, I'll go back to the basket to review what I selected and start fresh with the checkout process.I got an 'Invalid entry' error after clicking Place order. One of my filled fields must be wrong. Let me go back to the basket to review what I'm trying to buy and start fresh with the checkout — maybe I'll catch what went wrong.I got an 'Invalid entry' error when trying to place the order, but I'm not sure which field is actually wrong. Rather than guess and keep hitting the same error, I'll go back to the basket to review my order and see if there's an issue with what I'm trying to buy, or if I can see more details about the items.
2

127.0.0.1:61618/a/vague-error/checkout

click e6

4 of 15 people (27%) struggled here

  • frustration jumped +0.31 (×3)
  • confusion jumped +0.18 (×3)
  • confusion jumped +0.23
  • frustration jumped +0.26
Hit hardestage: 35-44 (25% vs 13% of cohort)age: 16-24 (25% vs 13% of cohort)
In their wordsI got an 'Invalid entry' error after clicking Place order. The postcode field is most likely the issue — French postcodes should be 5 digits, and 75001 is correct, but maybe this system wants a different format or there's a validation issue. Let me click on the postcode field again to check what I entered and see if I need to correct it.I got an 'Invalid entry' error after clicking Place order. The postcode field must have been rejected. Let me click on the postcode field again to re-enter it more carefully — maybe the format wasn't right.I see an 'Invalid entry' error message, which likely refers to the postcode field I just filled. The postcode '10115' might not be in the correct format or might be invalid for this system. Let me click on the postcode field again to check what I entered and potentially correct it.
3

127.0.0.1:61618/a/vague-error/checkout

type e6

3 of 15 people (20%) struggled here

  • frustration jumped +0.26 (×2)
  • frustration jumped +0.21
  • confusion jumped +0.18
Hit hardestage: 35-44 (33% vs 13% of cohort)age: 55-64 (33% vs 20% of cohort)
In their wordsI got an 'Invalid entry' error, and the postcode field seems to be the problem. I tried 75001, but maybe this specific postcode isn't accepted by their system. I'll try a different valid Paris postcode (75002) to see if that works better.I'm getting an 'Invalid entry' error on the postcode field. I've clicked on it again, so now I'll clear it and retype the postcode. Maybe I need to enter it differently or there's a formatting issue.The postcode field shows an 'Invalid entry' error. I've clicked on it again, so now I should clear and re-enter the postcode. Maybe I need to try a different format or the field has specific requirements. Let me type the postcode again—this time I'll make sure it's entered correctly.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

device
desktop n=4 50%
mobile n=11 18%
income
low n=3 0%
middle n=10 40%
price sensitivity
high n=4 50%
low n=7 14%
medium n=4 25%
patience
high n=7 14%
medium n=6 50%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.