CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/silent-submit/

Buy a bag of decaffeinated coffee.

Planted, and hidden from the system: adding to the basket works but produces no confirmation, no count change and no navigation
Taken from Nielsen heuristic #1 Visibility of System Status; WCAG 2.2 SC 4.1.3 Status Messages (Level AA). Nothing in the prompt mentions it — read the findings below and judge whether the flock landed on it.

15people
10finished
1gave up
4ran out of steps

Where people struggled

1

127.0.0.1:61618/a/silent-submit/product/decaf

click e3

6 of 15 people (40%) struggled here

  • frustration jumped +0.19 (×6)
  • same control touched 3+ times
In their wordsI clicked 'Add to basket' in the previous step, but the page shows the same content — nothing appears to have changed. Let me try clicking the button again to confirm the item is actually being added, or to see if it progresses to the next stage like a basket view or checkout.The 'Add to basket' button still shows on the page, and I haven't seen any confirmation that my item was added. The page says nothing has changed since my last attempt. I'll click it once more — sometimes these buttons need a moment or a firm click to register, especially on slower connections.I'm clicking 'Add to basket' again because the page hasn't changed since my last action - it seems like the first click might not have registered. I want to make sure the coffee is added to my basket so I can proceed to checkout.
2

127.0.0.1:61618/a/silent-submit/checkout

type e5

5 of 15 people (33%) struggled here

  • action failed: type needs a value — say what to type (×5)
Hit hardestage: 55-64 (40% vs 20% of cohort)age: 35-44 (20% vs 13% of cohort)
In their wordsI need to fill in the Phone field which is required for checkout. I'll enter a placeholder phone number to proceed with the order.I'm on the checkout page and I've filled in my name and email. Now I need to complete the remaining required fields - phone and postcode. I'll click on the phone field to enter a phone number so I can proceed to place the order.I'm on the checkout page and I've filled in the name and email. Now I need to complete the Phone field which is marked as required. I'll click on it and enter a phone number to move forward with the checkout.
3

127.0.0.1:61618/a/silent-submit/checkout

click e6

3 of 15 people (20%) struggled here

  • frustration jumped +0.31 (×2)
  • confusion jumped +0.18 (×2)
  • frustration jumped +0.26
Hit hardesttech literacy: medium (33% vs 7% of cohort)age: 45-54 (67% vs 27% of cohort)age: 35-44 (33% vs 13% of cohort)
In their wordsThe system is telling me the postcode format is wrong—it needs to look like 'SW1A 1AA', not '1234'. I need to fix this field with the correct format before I can place the order. Let me click on the postcode field to clear it and enter a properly formatted UK postcode.The postcode field is showing an error—it needs to be in UK format like 'SW1A 1AA', not just numbers. I'll click on the field and clear it, then enter a properly formatted postcode so I can finally place this order.The postcode field is showing an error message that it needs to be in the format SW1A 1AA. I entered a French postcode but the system expects a UK format. I'll click on the postcode field to clear it and enter a properly formatted UK postcode.
4

127.0.0.1:61618/a/silent-submit/checkout

type e6

3 of 15 people (20%) struggled here

  • action failed: type needs a value — say what to type (×2)
  • frustration jumped +0.21
  • same control touched 3+ times
Hit hardesttech literacy: medium (33% vs 7% of cohort)age: 55-64 (33% vs 20% of cohort)
In their wordsThe postcode field is rejecting my previous entry because it needs UK format like 'SW1A 1AA'. I'll type this properly formatted example postcode so I can finally place the order. I'm frustrated by this extra step but determined to complete the purchase.I need to fill in the Postcode field which is the last required field before I can place the order. Looking at the form, this is the only empty required field left.I've filled in Name, Email, and Phone. Now I need to complete the Postcode field which is the last required field before I can place the order. I'll enter a postcode to finish the form.
5

127.0.0.1:61618/a/silent-submit/product/decaf

click e2

2 of 15 people (13%) struggled here

  • frustration jumped +0.19 (×2)
Hit hardestage: 25-34 (100% vs 27% of cohort)
In their wordsThe 'Add to basket' button doesn't seem to be working - I've clicked it twice and nothing changed. Let me try clicking the Basket link to see if the item was actually added, or if I need to try a different approach.I've clicked 'Add to basket' twice on this product page, but nothing has changed — no confirmation, no navigation to checkout. The page still looks the same. Let me check my basket to see if the item was actually added. I'll click the Basket link to proceed.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

age
25-34 n=4 100%
45-54 n=4 25%
55-64 n=3 100%
price sensitivity
high n=4 75%
low n=7 71%
medium n=4 50%
patience
high n=7 43%
medium n=6 83%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.