A test user broke my app with her thumb, and it’s the best argument I have for why software engineers still matter as AI coding agents get better.
When I first launched VersoID, my tester tapped the button that creates a persona twice, fast. Whether she was impatient or her thumb slipped doesn’t really matter. The app saved two identical persona cards.
My test cases never did that. Neither did my own physical testing. I only found out because my very confused tester told me.
That duplicate is the skill in practice: finding a problem nobody has described yet. It’s one half of what keeps engineers relevant.
The other half is checking that a fix is actually correct.
Both depend on knowing how software behaves once a real person gets their hands on it, and my tester’s thumb taught me that better than any test I wrote 😓
The first half is the harder one for an AI to do, and you can see how hard. In SWE-sweep, a Meta FAIR benchmark, the same model fixed 75.3% of bugs when it had the original bug reports. Give it nothing but the code, and that dropped to 4.8%.
Finding the bug is where the difficulty lives, and my double-click is exactly that kind of bug.
How Does a Double-Click Bug Slip Past Your Tests?
Nobody had reported that duplicate persona, because nobody knew it existed. (By nobody, I mean no person and no AI agent.)
It took someone using VersoID in a way I hadn’t thought to try, and then telling me what they saw.
A double-click is two presses a fraction of a second apart, and the app treats each one as its own. If the button saves a new persona every time, the second press saves a second one, because nothing in the app accounted for it.
That’s why nothing flagged it.
Nothing I wrote said what two presses in a row should do, so my test cases had nothing to check against.
My own physical testing didn’t either. The only way it could surface was if someone pressed the button impatiently or by accident.
It can happen to plenty of users, but only under very specific conditions. I couldn’t reproduce it easily until she pointed it out explicitly, screen recording and all 🫠
Once she had, it was a problem somebody had described, and I could finally hand it to my AI agent in plain words.
Tip: Double-click your buttons if they create something. It takes five seconds, and none of my test cases ever made that check. Small quirks like this only show up when real people poke at your app, and they quickly humble you.
Related: Lessons Learned From Shipping An App For The First Time
Is Finding a Bug Harder Than Fixing One?
Most coding benchmarks hand the agent a codebase and a written bug report. That’s a bit like giving a plumber the address and telling them which pipe is leaking.
In SWE-sweep, the agent gets only the code: 4,068 bugs hiding across 100 repositories in 22 programming languages, with no hint of where any of them live. The best any model managed was GPT-5.6 Sol, which fixed 4.7% of them.
Then the same experiment ran with one change: written bug reports.
On the first 20 repositories, the same model fixed 4.8% of bugs with no information and 75.3% once it had those reports. The only thing that changed was whether anyone had described the bug.
Fixing a described bug is about as hard as the SWE-bench tasks, which agents already handle well. Finding the bug is the hard part.
Interactive apps show the same split, which matters for my case. In one benchmark, agents with a detailed bug description passed about 96% of the tasks. Those bugs are hard to find, obvious once someone points at them, and reproducible through normal use.
That’s my double-click exactly 💁♀️
Note: Only bugs that someone reported and a maintainer fixed make the SWE-sweep list, so it undercounts the bugs in any codebase. A duplicate persona like mine would sit outside it too.
Explore: I Easily Find Bugs In AI Code Every Single Time
Which Skills Matter as AI Writes More Code?
So which skills matter when finding the bug is the hard part? Two stand out to me.
- Noticing that something is wrong before anyone files a report.
- Checking that a fix or a feature does what it’s supposed to do.
The second one has a formal name, verification and validation, which sounds heavier than it is.
- Verification asks whether the code matches the plan.
- Validation asks whether the software does what people need.
Both skills depend on knowing how a system behaves, and my persona button shows why. To check it, somebody has to think of pressing it twice in a row. An agent that writes the button’s code can write tests for the cases it was told about. The case nobody mentioned needs a person who knows what to ask.
I’m not the only one who sees it. Other people have reached the same conclusion, including a group of researchers and practitioners in New York and Singapore, who argue that verification and validation matter more as agents take over implementation.
Developers already feel it. 66% use coding assistants or agents at work, and nearly half trust AI output when they can verify it easily. That’s only 6.6% who will trust it with an important decision.
Why does that matter? Reading the right files and finding the bug are different jobs.
By its 140th action, GPT-5.6 Sol had read 18.5% of the files that known fixes touch, and every bug tied to only 1.7% of those files was resolved.
An agent can spend a long time in the right file and still walk away without the bug.
Note 💬
There’s also the orchestrator idea. Addy Osmani calls the emerging role the orchestrator: an engineer overseeing many agents working in parallel on different parts of a project. That role needs the checking skill above, because someone has to judge what the agents hand back. (It also starts from a list of tasks, and nobody had put “double-click the create button” on mine 😒)
How Do You Practice Finding Bugs Nobody Has Reported?
The half you can actually practice is noticing, and the easiest way I know is to write down what your code assumes about the person using it.
Say you’ve built a Pokémon search box that calls the PokéAPI with the Fetch API, the browser’s built-in way to request data. Your test types “pikachu,” gets Pikachu back, and passes.
Underneath it sit four assumptions:
- person types a whole name
- spells it right
- presses enter once
- waits for the answer
Each one is a test case you haven’t written. Your test covers the one path where everyone behaves, but real people rarely do.
A dropdown works the same way. Your test picks the second option and checks the page. A person opens the menu, picks one, changes their mind, and picks another before the page finishes updating.
Another way is to hand your project to someone who didn’t build it and watch their hands.
My tester found the duplicate persona by tapping a button in a way I hadn’t thought to try. Watching someone else use the app shows you the paths your own habits skip. Ask them to create a few things in a row, with no instructions from you, and write down every moment they hesitate or press something twice.
It also helps when the app records what happened, so a problem like mine leaves something to find. 44% of investigations by AI incident tools and monitoring software fail because the data needed to explain the failure was never captured.
For example, if your app talks to a server, a request log would show two create requests arriving a fraction of a second apart.
Once you find a bug like mine, a test keeps it found. It presses the button twice and checks that only one persona exists, and from then on the bug gets checked automatically.
Will AI Agents Get Better At Finding Bugs On Their Own?
Probably, and the SWE-sweep authors think so too. Agents fixed more bugs the more turns they got, which hints that more capable models will find bugs better.
The gains flatten out quickly, though. On the five repositories with the most bugs, one model went from 1.6% to 3.5% after ten agents took turns, and ten independent attempts together reached 4.1%.
Agents also undo their own work along the way. In the SWE-sweep runs, 28 bug repairs passed at an intermediate checkpoint and failed in the final patch because later edits broke them.
Size gets in the way too. On repositories under 100,000 lines of code, the top model fixed 13.4% of bugs. Above 500,000 lines, it fixed 1.2%. Searching a huge codebase for something wrong is where most of the difficulty sits, and it’s a pricey way to search. The top result took about $7,000 in total, while the second-place model fixed about half as many bugs for $224 💸
There are no human scores on these bugs, so I can’t tell you how a person would do, and it leaves me betting on a skill that may shrink.
If agents get good at sweeping a codebase, what stays with engineers moves toward bugs that live in how people behave.
Related: The Uncomfortable Truth About Developer Value in the AI Era
It’s a Wrap
My tester found the double-click with a thumb, and every other check I had came from someone who knew exactly what the button was supposed to do.
For anyone with a work-in-progress project, I encourage you to try this. Press a create button twice fast and see what you get.
If it’s a web project, keep DevTools’ Network tab open while you do it and look at what it sent.
If you want to see where the persona cards come from, check out VersoID.
Thank you for reading this one. I’d love to know if a user has ever found a bug you couldn’t reproduce on your own and what that looked like to troubleshoot for you.
I’ll see ya next time.
Debug away ✌️
