The Japanese fix for MALFUNCTION 54

MALFUNCTION 54 flashed forty times a day, so the operators ignored it. Until people started dying.

In the mid-80s, the Therac-25 radiation therapy machine threw a cryptic error dozens of times a day. But it wasn't in the manual. So operators built a reflex: press P to Proceed, and move on.

Something scary hid behind that reflex. When the machine overdosed a patient, the readout showed a low dose number because the meter had overloaded and rolled over. The machine delivered up to 100 times the prescribed amount. One patient said, "you burned me." But they just trusted the screen.

Six patients got massive overdoses.

At least three died.

The machine wasn't broken.

It reported every error.

The humans had stopped looking.

Japan Railway solved that exact problem a century ago, and the fix looks ridiculous.

It's called shisa kanko. Pointing and calling. A driver doesn't just glance at the speedometer; they point at it, say the number out loud, and hear their own voice say it.

A small ritual performed in an empty cab.

In 1994, the Railway Technical Research Institute measured it. People doing a task silently made 2.38 errors per 100 actions. Pointing and calling dropped that to 0.38. An 85% cut. It was the looking plus the pointing plus the voice that did the work.

Last year Microsoft and Carnegie Mellon University surveyed 319 knowledge workers: The more someone trusted AI, the less they questioned it. When Anthropic found users automatically accepting 93% of Claude’s suggestions, they built Auto Mode, a plain-language set of rules to ensure risky actions like changes to a repo’s .git or Claude’s own config are never auto-approved.

We keep stripping friction out of workflows, when friction is the only thing keeping us awake.

The fix for falling asleep at the wheel isn’t just a better autopilot.

It's a job designed so no one can proceed without looking.

(With props to Bob Sutton and Hayagreeva Rao, who wrote a whole book on keeping good friction.)

—

Sources:
"An Investigation of the Therac-25 Accidents," Leveson & Turner, 1993
"The effects of 'finger pointing and calling' on cognitive control processes in the task-switching paradigm," Shinohara, 2013
"The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers," Lee et al, 2025