The phishing training everyone passed, and still clicked

The problem

A firm did everything the compliance calendar asked of it. A mandatory phishing-awareness session, once a year, every year. And every year the phishing-simulation click rate sat at 18%, flat across three cycles, refusing to move.

Then a senior partner clicked a spoofed invoice email. High inbox volume, very little time, exactly the person the annual session was least built to reach. When I looked at what the training had actually taught, the picture was uncomfortable but clear. People could tell you what phishing was. They could define it, explain it, pass the quiz. They just could not catch it on a Tuesday, mid-task, with a plausible invoice and a payment due that afternoon. Knowing and catching are not the same skill, and the training only ever built the first one.

The learner and the constraints

The people at the end of this are busy, senior and email-heavy. They are not careless. They are interrupted. The moment that matters is not a training room. It is fifteen seconds between two meetings, when an email that looks right asks for something ordinary. Whatever I built had to live in that moment, not in a slide deck the week before, and it had to respect that these are competent professionals who will not sit through being told, again, what a phishing email is.

The question I set myself

The approach, and why

I did not write another awareness module. I built the training around the decision itself.

• Find out who actually clicks, and when. I started from the click-pattern data rather than assumptions, and grouped staff by the shape of their risk, high-volume senior inboxes with low day-to-day IT contact, instead of treating everyone as one audience. The figures are illustrative, but the method is the point: design for the people actually getting caught.
• Rehearse the moment, not the definition. The core is a short branching scenario where the learner makes the real call under a gentle clock. This is the "apply" moment in the 5 Moments of Need, the help you need at the point of performance, not the day before. The countdown is a rehearsal cue, never a fail state, so the pressure feels real without punishing anyone.
• Let the wrong call play out. Every branch has a consequence you watch unfold, not a red cross. People reach the lesson by living the near-miss, which is what real judgement practice looks like rather than a knowledge check in disguise.
• Keep the nudge going after the module closes. A light reinforcement campaign spaces the practice over the following weeks, because one sitting does not change a reflex.
• Measure whether behaviour moved, not whether people finished. The evaluation tracks the click rate and reporting behaviour at Kirkpatrick Levels 3 and 4 with xAPI, so the question is always whether people are catching more and reporting faster, never whether they completed it. Built to WCAG 2.2 AA throughout, so the clock, the audio and the feedback work for everyone.

Artefacts

The outcome

The honest number here is the one we started with. The 18% click rate, three years running, is the only real figure. Everything the redesign is built to move is a design target, marked in amber until a real cohort has run it.

What I took from it

Awareness was never the gap. Everyone was aware. The gap was that nobody had ever practised the decision at the speed it happens in real life. You do not fix a reflex with a definition. You fix it by rehearsing the moment, again and again, until catching it is the reflex. That is the difference between training people can pass and training that changes what they do when the invoice lands.

Further reading: why AI should support, not replace, the designer, measuring behaviour, not completion.