CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/baseline/

Buy a bag of decaffeinated coffee.

This shop has no planted defect. It is published for the same reason the others are: what a tool reports when nothing is wrong is the half nobody shows.

15people
9finished
1gave up
5ran out of steps

Where people struggled

1

127.0.0.1:61618/a/baseline/checkout

type e5

4 of 15 people (27%) struggled here

  • action failed: type needs a value — say what to type (×4)
  • frustration jumped +0.15 (×2)
  • frustration jumped +0.2
Hit hardestage: 55-64 (50% vs 20% of cohort)
In their wordsI need to fill in the Phone field which is required. I'll enter a phone number to complete the checkout form.I need to fill in the Phone field which is required. I'll enter a phone number to complete this mandatory field before I can proceed to place the order.I need to fill in the Phone field, but I notice it's required. However, I haven't been given a phone number for John Smith. I'll enter something reasonable - a typical EU phone number format. Let me try a generic placeholder since the instructions don't specify what number to use.
2

127.0.0.1:61618/a/baseline/checkout

click e6

4 of 15 people (27%) struggled here

  • frustration jumped +0.26 (×3)
  • frustration jumped +0.31
  • confusion jumped +0.18
Hit hardestage: 16-24 (25% vs 13% of cohort)
In their wordsThe system is telling me my postcode format is wrong — it needs to look like 'SW1A 1AA' (UK format), not the French format I entered. I need to click on the postcode field and correct it. Let me try a proper UK postcode format since that's what the system expects.The system is telling me the postcode format is wrong. It wants something like 'SW1A 1AA' which looks like a UK format. I need to click back into the postcode field and enter a properly formatted postcode. I'll try a UK format since that's what the example shows.The postcode format is wrong — it needs to be like SW1A 1AA, not just numbers. I'll click on the postcode field and fix it with a proper UK format postcode.
3

127.0.0.1:61618/a/baseline/checkout

type e6

2 of 15 people (13%) struggled here

  • frustration jumped +0.26
  • action failed: type needs a value — say what to type
  • same control touched 3+ times
Hit hardesttech literacy: medium (50% vs 7% of cohort)age: 55-64 (50% vs 20% of cohort)age: 45-54 (50% vs 27% of cohort)
In their wordsThe system rejected my French postcode and specifically asked for a format like 'SW1A 1AA'. Even though I'm EU-based, I need to match what the form expects. I'll enter this example format to proceed with the order.I need to fill in the postcode field which is required before I can place my order. I'll click on it and enter a postcode to complete all required fields.The error message is clear — the postcode needs to be in UK format like 'SW1A 1AA'. I'll clear the field and enter a proper UK postcode format so I can finally place my order.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

age
25-34 n=4 25%
45-54 n=4 75%
55-64 n=3 67%
device
desktop n=4 75%
mobile n=11 55%
income
low n=3 67%
middle n=10 50%
price sensitivity
high n=4 75%
low n=7 57%
medium n=4 50%
patience
high n=7 71%
medium n=6 50%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.