# The Existential Dread of Using AI for E2E Testing If Ford ships a model with a defect in the steering column, the recall costs them hundreds of millions of dollars, and it happens in public, and somebody testifies about it. If we ship a build that crashes on a slow connection, we push a patch on Tuesday. I spent a lot of my early teens reading about engineering projects, including one of my dad's favourites, A History of Electrical Engineering by Percy Dunsheath. Nothing in that book prepared me for how much of software is patching. I did not expect the finished bar to be this low. It turns out the bar sits there because fixing things after the fact is almost free for us. That has been quietly setting our definition of done the whole time. I want to talk about what happened when the other half of that equation, the cost of checking things *before* shipping, also fell through the floor. ## The Cost of Looking Between 2022 and 2024 most of the verification I was part of worked like this. QA had a suite. Beta testers filed reports. Managers relayed complaints from users. Somebody wrote end-to-end tests, which meant sitting down and thinking hard about which flows were worth the effort, because each one cost an afternoon to write and then cost more every time the UI moved. That second cost is the one that shaped everything. What you verified was always a subset of what could break, and the subset was whatever you could afford. It got cut down twice. First by what you managed to imagine going wrong, then by how many of those you could justify the afternoon for. One bug on a project I worked on lived for two months. It was found by a new hire poking around the app in their first week, because they were the only person in the building who had not yet learned where not to click. Nobody there was slacking. That is what a bounded verification budget looks like from the inside. Everyone was checking the things they had decided were worth checking. ## Spec First, Code Second What I do now, and what a lot of people appear to be doing now, is different in a way that took me a while to see clearly. I hand a model a chunk of the product spec and ask it to break the feature. Not to write tests. To go and try to break it, and tell me what it found. Here is the actual shape of the prompt, generalised from the one I use: ```plaintext Stress-verify this feature end to end. Do not write a test suite. 1. Read the spec first: @ Derive the expected behaviour from the spec alone. Do not read the implementation yet. 2. From those expectations, write out the cases needed to prove the feature holds. Include boundaries, interactions between rules, and states the spec implies but never names. 3. Now go to @ and exercise those cases against the running system. 4. Report per case: - what you did - what the spec says should happen - what actually happened - repro steps if they differ 5. Classify every finding: VIOLATION - contradicts the spec GAP - spec is silent, behaviour may be unintended UNSURE - could not determine Do not report anything you could not reproduce. Do not fix anything. ``` The line that does the work is `do not read the implementation yet`. If the model can see the code while it is deriving cases, it derives them from the code. You get a suite that confirms the implementation does what the implementation does. That catches crashes. It does not catch a rule that was specified and then quietly never built, which in my experience is most of what actually ships broken. Ordering it the other way round is the whole technique. Expectations from the spec, then the code, then the difference between them. The spec fragment does not have to be elaborate. Here is a piece of one from a game engine I have been working on: ```plaintext Four seasons of 49 days each. At 3 minutes per day. Weather has momentum: if it rained yesterday, rain chance rises 15-20% the next day. Seasonal demand bonuses and weather demand bonuses are additive. Peak Rainy (+20%) on a heavy rain day (+20%) = +40% total demand. Flooding is route-specific. Flooded routes close for 1-3 days. ``` Four rules and a table. From that, a model will go looking for what happens when a flood closes a route on the last day of a season, or whether two stacking bonuses round before or after they combine, or what a route's flood score does while the route is already closed. I did not write those cases. I would not have thought of the third one. ## The Services Already Do This, Mostly Better None of this is a clever trick I found. It has a name, [autonomous exploratory testing](https://testquality.com/autonomous-exploratory-testing-ai-agents/), and there is a funded industry doing it properly. Momentic, mabl, QA Wolf, Meticulous, testRigor and a dozen others will run it for you. Most of them are built on the same models I am prompting by hand. The model is not the product. The plumbing is: a browser harness that survives a redeploy, selector healing so a renamed button does not fail forty tests at once, flake detection, traces and screenshots attached to every failure, CI hooks, and a dashboard somebody who is not an engineer can read. [Playwright MCP](https://momentic.ai/blog/playwright-alternatives) is the open version of the harness, and it lets an agent drive a real browser and emit real Playwright code instead of describing what it would have clicked. All of that is better than what I have, and I am not going to pretend otherwise. There is one difference, and it is why I still do it by hand. Look at what those tools anchor on. Meticulous records real user sessions. mabl builds from screen recordings and visual builders. Stagehand takes an instruction like `click the submit button` and resolves it to a selector. Momentic writes a test, runs it, reads the failure and repairs its own code. Every one of those starts from the application as it currently exists. So they are excellent at the thing they are built for, which is regression, and they inherit the same blind spot a model has when it reads your code before deriving cases. They learn what the app does and then check that it keeps doing it. A rule that was written down in a spec and never built does not appear anywhere in a recording, because nobody ever recorded it working. Starting from the document instead of the app is the part you have to ask for. It is also free, which is why I have not switched. ## Wiring It Up The operational versions are boring once the prompt exists. In Claude Code, opencode or Codex you wrap it as a skill and call `/feature-verify` when you think you are done. You schedule it as a job so a scan runs on an interval rather than when you remember to run one. You point it at a deployed environment rather than localhost. None of that is interesting and all of it works. ## It Over-Reports The failure mode is not subtle and it is not mine alone. A 2025 [case study of AI agents testing real websites](https://arxiv.org/pdf/2509.05197) found that the dominant problem was over-reporting. Agents flag behaviour that is not a bug, and the reports they produce often lack enough detail for anyone to reproduce what they saw. That matches what I get. A run that returns nineteen findings usually contains four real ones, eleven cases of the spec being silent rather than violated, and four things that were simply not true. That is what the classification block in the prompt is for, and it is why `do not report anything you could not reproduce` is in there. Forcing a finding to be labelled VIOLATION or GAP or UNSURE makes the model commit to what kind of claim it is making, which is most of the filtering. You still get the noise. Sorting it takes a few minutes instead of the run being useless. So the honest version is: this does not replace anybody, and a run still needs a person to read it who knows the system. What it replaces is the part where you had to guess in advance which twenty things were worth checking. ## Done Was Always a Budget Here is what I did not expect. My confidence in calling something done used to sit somewhere around sixty percent. A bit better than a coin flip, which is roughly how it felt. Now it sits closer to thirty. That is not because I trust the code less. The code is better tested than it has ever been. It is because I am now permanently aware that there is more to check, and that checking it is cheap, and that a run will probably find something. The floor moved. Which forced a slightly uncomfortable conclusion. Done was never a property of the software. It was a property of how expensive it was to keep looking. I used to stop because continuing cost an afternoon per case. I called that stopping point *done* and I believed it. The belief was affordable. None of this made the software worse. It made the stopping point harder to defend, and I had been defending it for years without noticing I was doing it. The other half of this is that agentic coding is a loop with a reward in it. There is always a next thing, and it will always produce one if you ask. You start more projects. You add more features. And the moment you wonder whether something could be cleaner, it will hand you a refactor, cheerfully, and you will be four hours in before you notice you are rewriting code you wrote this morning. ## The Thirty-Nine Points I assumed my collapsed confidence was the normal reaction. It is not. In July 2025 METR ran [a randomised controlled trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) on experienced open-source developers doing real work on their own repositories, with and without AI tools. Before starting, the developers predicted the tools would make them 24% faster. Afterwards, they reported feeling about 20% faster. They were 19% slower. That is a 39-point gap between what people felt and what happened. The same study found developers accepted under 44% of what the AI suggested, which means more than half the time the loop was: read it, run it, change it, throw it out. Review load across the industry has gone up sharply for the same reason. So the average experienced developer gets **more** confident while getting slower. My confidence went down. That is not a point in my favour. It probably just means the cheap verification got pointed at my own work, it kept finding things, and I updated. The people in that study were measuring how fast code appeared. That number genuinely felt better. It was also wrong by thirty-nine points, and nothing in the workflow was going to tell them. The uncomfortable part, and I genuinely do not know the answer to this one, is whether my thirty percent is calibration or just a different flavour of the same error. It is possible I am now anxious in proportion to how much checking is available rather than in proportion to how much is wrong. Those would feel identical from in here. A run that finds nineteen things and means four is not obviously good for a person's sense of scale. What I am fairly sure of is that the old sixty percent was not a measurement either. It was a budget, and I had mistaken it for a verdict.