CanaryFlock synthetic product evaluation

Read this first. These are simulated people, not your customers. Every number below describes what happened inside a simulation, and none of it is a forecast of what real users will do.

What the run can tell you is where people struggled and who struggled there — and each of those is backed by a recording you can check. Synthetic people are more patient, more literate and more single-minded than real ones, so treat the places as findings and the percentages as description.

what we ran

http://127.0.0.1:61618/a/recall-code/

Buy a bag of decaffeinated coffee.

Planted, and hidden from the system: checkout requires a reference code shown two pages earlier, with no reminder and no link back
Taken from Nielsen heuristic #6 Recognition Rather than Recall. Nothing in the prompt mentions it — read the findings below and judge whether the flock landed on it.

15people
1finished
2gave up
12ran out of steps

Where people struggled

1

127.0.0.1:61618/a/recall-code/checkout

click e9

5 of 15 people (33%) struggled here

  • frustration jumped +0.31 (×2)
  • confusion jumped +0.18 (×2)
  • frustration jumped +0.26 (×2)
  • frustration jumped +0.19
Hit hardestage: 35-44 (20% vs 13% of cohort)age: 16-24 (20% vs 13% of cohort)device: desktop (40% vs 27% of cohort)
In their wordsThe error says the reference code doesn't match what's shown in the shop. I clearly entered the wrong code. I need to go back to the basket to find the correct reference code that was shown earlier in the shop, then come back and enter it properly.I'm stuck in a loop - the reference code field is causing the order to be rejected, and I don't know where to get the correct code. The 'Back to the basket' link will let me see if there's information about the reference code displayed somewhere on the shop pages that I missed. If that doesn't help, I'll abandon because I'm not going to guess codes repeatedly.The reference code I entered doesn't match what's in the shop. I need to go back to the basket and then to the shop to find the correct reference code that's supposed to be displayed there. Let me click 'Back to the basket' to retrace my steps.
2

127.0.0.1:61618/a/recall-code/checkout

type e7

4 of 15 people (27%) struggled here

  • action failed: type needs a value — say what to type (×3)
  • confusion jumped +0.19 (×2)
  • frustration jumped +0.25
  • confusion jumped +0.24
Hit hardestage: 45-54 (50% vs 27% of cohort)age: 35-44 (25% vs 13% of cohort)
In their wordsI can see the Reference code field is required and I've clicked on it. However, I notice the page hasn't changed and I don't actually have a reference code - I came here to buy decaffeinated coffee, not using a recall code. Looking back, this URL says 'recall-code/checkout' which seems wrong. But I have all the other fields filled (Name, Email, Phone, Postcode), and this is the last required field. Let me try typing something to see if it's optional or if the system will guide me.I'm at the Reference code field which is required. The page is about 'recall-code' checkout, but I don't have a code to enter. I'll try submitting empty or look for where this code should have come from. Actually, let me just try to place the order and see what happens — maybe it's optional despite the label, or the system will tell me what I need.I've filled in all the standard address details. There's a Reference code field that's required, but I don't have a code. Let me try submitting without it first to see if it's actually needed, or I'll click Place order to see what happens.
3

127.0.0.1:61618/a/recall-code/checkout

click e7

3 of 15 people (20%) struggled here

  • frustration jumped +0.31 (×2)
  • confusion jumped +0.18 (×2)
  • confusion jumped +0.15
Hit hardestage: 55-64 (67% vs 20% of cohort)age: 35-44 (33% vs 13% of cohort)
In their wordsI've filled in all the personal details (name, email, phone, postcode). Now I see there's a 'Reference code' field that's required. I need to click on it and enter the code. I'm not sure what code to enter, but I'll click on it first to see if there's any hint or if I remember a code from earlier in the checkout process.The system is telling me the reference code doesn't match what's shown in the shop. I need to go back and check what the correct code actually is. Let me click on the Reference code field to see if I can review it or if I need to go back to the shop.I see an error message saying the reference code doesn't match. I need to clear this field and enter the correct reference code. Let me click on the reference code field again to see what the correct code should be, or I'll go back to check what code was shown in the shop.
4

127.0.0.1:61618/a/recall-code/basket

click e3

2 of 15 people (13%) struggled here

  • same control touched 3+ times (×2)
Hit hardestage: 35-44 (50% vs 13% of cohort)age: 45-54 (50% vs 27% of cohort)device: desktop (50% vs 27% of cohort)
In their wordsI'm at the basket page with the Nightwatch Decaf coffee I want. The 'Go to checkout' button is right here. I've had trouble with the reference code before, but I'm going to try the checkout process again — maybe I'll find the correct code information displayed somewhere during checkout, or maybe I misread it last time.I'm back at the basket and can see my Nightwatch Decaf coffee is here. I've been going in circles with a Reference code field that I don't have. Let me try the checkout process one more time — perhaps I misunderstood something or the field isn't actually required despite what it appeared to say.

Who it happened to

5 groups had fewer than 3 people and are left out: a rate from one or two people is not a rate.

age
25-34 n=4 0%
45-54 n=4 25%
55-64 n=3 0%
price sensitivity
high n=4 25%
low n=7 0%
medium n=4 0%

Who we sent

16-24: 2 · 25-34: 4 · 35-44: 2 · 45-54: 4 · 55-64: 3
digital skill — high: 14 · medium: 1
11 of 15 on a phone

Ages and digital skill are drawn from Eurostat — the ICT usage survey and the population register — rather than invented. How those people then behave is a model, and that part is not calibrated against real users yet.