Why Real-World Testing Beats Lab Simulations

Why Real-World Testing Beats Lab Simulations(From Someone Who’s Watched Million-Pound Projects Nearly Die in a Customer Car Park)

I’m Nick Thorpe.
I’ve spent 25 years making sure software doesn’t kill people — literally — in cars, trains, satellites, sonar systems, and now secure government networks. I’ve sat in freezing Jaguar Land Rover prototype vehicles at 4 a.m. in minus-10°C while the ECU refused to start because the lab “-40°C to +85°C” thermal chamber never replicated the exact condensation pattern on a real windscreen.
I’ve watched a £42 million defence communications system pass every single Factory Acceptance Test (FAT) with flying colours… only to blue-screen the first time a real soldier plugged in a tactical radio on a Welsh hillside in the rain. That’s the difference between ticking boxes in a lab and real-world Site Acceptance Testing (SAT).

FAT: The Comfortable Lie We All Agree To Tell

Factory Acceptance Testing is theatre. Beautiful, expensive theatre.

• Clean lab, perfect power supplies, zero RF interference

• Customer turns up in a suit, we run the pre-agreed script for two days

• Everyone signs the FAT certificate, champagne is metaphorically popped

• Project “de-risked”, bonus unlocked, lessons-learned meeting scheduled

Total time: 2–5 days
Total cost: £50k–£250k (travel, test rigs, engineers on overtime)What actually gets proven?
That the system works in one very specific, very artificial environment that will never, ever exist again.

SAT: The Moment the Universe Laughs at Your Test

PlanSite Acceptance Testing is when reality bites.

• Real hardware, real power (brown-outs, spikes, shared generators with welders)

• Real environment (vibration, temperature cycling, dust, rain, EMC from a thousand other systems)

• Real users (who immediately do the thing you never thought anyone would do)

I’ve seen:

• A transmission controller that passed 10,000 lab cycles fail on the first real gear change because the production CAN bus had 8 ns more propagation delay than our “reference” bus

• A railway signalling system that was 100% in the lab… until the train driver rested his laptop on the desk and the Wi-Fi channel overlapped with the safety comms

• A vacuum-pump controller that survived every thermal test known to man… until it was mounted vertically in the real machine and convection changed completely

Every single one of these issues would have cost 10–100× more to fix after delivery.The Numbers Don’t Lie (I’ve Got the Scars and the Spreadsheets)

Yes, SAT is more expensive and slower — up front.
No, you do not want to skip it or fake it with “extended FAT” or “virtual SAT” (I’ve seen both tried. Both ended in tears and lawyers).

The Stories That Still Wake Me Up

1. The £750,000 satellite comms box that passed FAT perfectly… until it was bolted to a vibrating ship and the crystal oscillator drifted 0.02 ppm. Six-month delay, £4m overrun.

2. The autonomous pod vehicle that drove flawlessly for 4,000 simulated miles in the lab… then emergency-braked the first time a real plastic bag blew across the test track. We added one extra sensor and re-wrote the perception stack in three weeks on-site. In the lab it would have taken months to reproduce.

3. The secure payment terminal that passed every EMI test in an anechoic chamber… until it was installed next to a real shop’s fluorescent lights and the touch screen went berserk. Fixed in two days on-site. Would have been a recall nightmare.

The Hard Truth for 2025

Simulations, emulators, virtual environments, digital twins — they are amazing.
They will find 80–90% of your bugs faster and cheaper than ever before, but the last 10% — the gnarly, environment-specific, timing-dependent, user-induced failures — still only show up when real photons hit real sensors, real electrons fight through real cables, and real humans do real stupid things.

My Rule of Thumb After 25 Years

• Spend 60–70% of your test budget on automated lab/regression/CI testing

• Spend 30–40% on real-world, on-site, “let’s break it with the customer watching” testing

Skip the second part and you’re just gambling with someone else’s money (and usually your reputation).Real-world testing isn’t sexy.
It’s cold, stressful, expensive, and ego-bruising.It’s also the only thing that has ever, in my entire career, actually proven a system is ready for the real world.Everything else is just PowerPoint.– Nick Thorpe
The guy who still has nightmares about a Welsh hillside in the rain
Lead Technical Security Tester | Former automotive/defence/rail test dinosaur | Still believes hardware should be scared of software
November 20, 2025