Use the AI.Just Understand What It Shipped.

We build CodeSubmit with coding agents every day, and we have done AI-assisted assessment work since 2022, back when it meant pointing DaVinci at test-runner output. So when I say we want the engineers we hire to use agents too, I mean it as a practitioner, not a vendor. The only thing I care about is whether the person can explain what the agent handed them.

Those two things have quietly come apart. An agent can produce a working pull request in a few minutes, and working is not the same as understood. Understood is the part you are paying a senior engineer for, and it is the part that does not show up on a green CI run. The whole argument that follows is really just that one gap, traced through the research, through how people behave around good tools, and through what we decided to build because of it.

The output still needs a human who gets it

The pitch for coding agents is speed, and the speed is real most days. The data underneath it is messier than the pitch. A 2025 randomized controlled trial from METR put experienced open-source developers on their own repositories and found that AI tooling made them about nineteen percent slower, even though those same developers came away convinced it had sped them up by roughly twenty percent. I do not read that as people lying. I read it as people who genuinely could not tell, which is the whole problem shrunk down to one number.

That gap between feeling fast and being fast shows up in the code that lands. GitClear ran the numbers on 211 million changed lines and found that copy-pasted code overtook moved code for the first time in their records, duplicate blocks climbed sharply, and the share of lines tied to refactoring fell from about a quarter to under a tenth. The agents are not the villain in that story. The humans steering them stopped reading closely, and the codebase absorbed the difference without anyone deciding it should.

The blind spot gets sharper the moment the stakes get specific. Stanford researchers handed developers an AI assistant and a set of security-sensitive tasks, and the ones with the assistant wrote less secure code on most of them while being more likely to believe their code was safe. On a cryptographic signing task, barely three in a hundred of the assisted developers reached a secure solution, against roughly one in five of the people working without the help. A 2025 Veracode study found the pattern survives into today's models, with around forty-five percent of the AI-generated samples it tested across more than a hundred models introducing a known vulnerability. The tooling does not only make the mistakes easier to produce, it makes them easier to feel good about, and that second part is the one that should worry a reviewer.

Developers are not fooled about this in the aggregate. When Stack Overflow asked in its 2025 survey how much people trust the accuracy of what AI tools produce, more of them said they distrust it than trust it, and that trust had dropped sharply from the year before. The field already agrees with itself on the shape of the thing. The output is worth having, and it still needs a person who can stand behind it before it ships.

Skimming the diff is the default, not the exception

There is a name for what happens when a capable tool earns your trust a few times in a row. Researchers call it automation complacency, and the foundational work on it predates modern AI by more than a decade. The findings are not flattering to any of us. Complacency shows up in novices and experts alike, it does not wear off with practice, and telling people the system is fallible barely dents the errors they wave through. Thoughtworks gave the software-specific version of this a name on its Technology Radar, complacency with AI-generated code, and put it on Hold, warning that it is all too tempting to be less vigilant about AI suggestions once a few of them have worked out.

If that sounds like a software problem, it is not. The same reflex turns up wherever a confident machine sits next to a skilled person. When researchers had radiologists read mammograms alongside an AI tool that sometimes gave the wrong call, reading accuracy dropped across every level of experience, and the least experienced readers went from getting about four in five cases right to fewer than one in five once the tool steered them wrong. The senior readers held up better and still landed well below their own unaided baseline. Expertise does not buy immunity here, it buys a slightly smaller collapse.

So the failure mode I worry about is not a candidate maliciously gaming an assessment. It is an ordinary, well-meaning developer who let a good agent run, skimmed the diff, and shipped something they could not really defend in a review. Even with a strong harness, the model will sometimes take a shortcut, misread the instructions, or quietly invent an import, and someone has to catch it. That someone is the person you are about to hire.

Review was supposed to be the backstop

The usual answer to all of this is code review, and code review is exactly where it strains. A large, widely cited study of the practice, run on roughly 2,500 reviews and millions of lines at Cisco, found that a reviewer's ability to catch defects falls off as the change gets bigger. Past a few hundred lines, people stop giving every line the same attention, and the data is blunt about the ceiling. A reviewer will not really get through more than three or four hundred lines before the quality of the review drops, and once they push past that, the defects they wave through climb fast.

The teams who take review seriously already design around this. Google studied its own process and found the median change is about two dozen lines, with most touching only a handful of files, because small changes are the only kind a human can review well. That is the tension in one place. Agents are very good at producing large, plausible diffs, and people are reliably bad at reviewing them. The backstop we counted on was built for changes a fraction of the size of what an agent will hand you on a quiet afternoon.

A conversation, not a camera

The reflex in hiring has been to reach for surveillance. Lock the browser, watch the webcam, log the keystrokes. I think that is the wrong signal and the wrong message at the same time. Nobody works that way on the job, so measuring people that way tells you how they behave under a microscope rather than how they build software. I would rather ask the thing a decent teammate asks in the hallway: can you walk me through this?

That is what we built Sentinel to do. After a candidate submits a take-home, an AI interviewer has a short, push-to-talk conversation about the code they just wrote, puts their own files on screen, highlights the exact line it is curious about, and asks them to explain it. A real author reads their own code without missing a beat. Someone who pasted an agent and never looked back tends to stall on the first honest follow-up. The interview runs about five minutes and happens asynchronously, so nobody schedules another slot, and it produces an authorship and understanding score plus a plain-language summary for whoever reviews it. It is off by default, decided per assignment, and the hire decision stays with the human, which is where it belongs.

We deliberately do not record video, watch the screen, or log keystrokes. That is not how people work once they have the job, so it is a strange thing to grade them on while you are deciding whether to give it to them.

The version of this idea I find most interesting is not in hiring at all. Picture the same lightweight check sitting in front of a pull request. Before you ask a teammate to review four hundred lines an agent generated, right around the size at which the Cisco data says review quality starts to drop, you spend two minutes confirming you can defend them. That is not surveillance and it is not a gate against AI. It is a small, humane habit that keeps a person in the loop while the agents do more of the typing, and it fits the exact moment our industry is in.