Assess the decisions behind the patch
Coding agents can produce a convincing patch. A realistic interview needs evidence of how the candidate decides it is safe to accept.
A coding agent may have found the relevant files, chosen the implementation, written the tests and summarized the result. The patch records what changed; it does not record which claims the candidate checked before accepting it.
If the job involves agents that inspect repositories, edit files and run commands, removing those tools from the interview removes some of the work we need to assess. Making them available does not automatically give us a useful test, either. The task has to make the candidate's contribution observable.
CodeSignal's current Agentic Interviewing documentation describes an agent-enabled question for live interviews, using tools including Claude Code, Codex and Cursor.
OPM's work-sample guidance says tasks should resemble the job and may need updating when working tools change. It also limits their use to competencies expected at entry.
A validation report leaves work to do
In an October 2 preprint on performance pull requests, researchers studied balanced groups of 1,130 agentic and 1,130 human-authored performance PRs, with observations ending June 1, 2026. The agentic PRs were autonomously authored and submitted. The study was not an interview experiment and did not examine human-controlled AI-assisted work.
After filtering for documented validation and an identifiable optimization category, both the agentic and human groups reported a performance metric in about half their remaining PRs. The researchers did not reproduce the experiments, so their study cannot establish that a reported benchmark ran as described.
A correctness test can pass before and after a change without showing whether the optimization helped. If the measured improvement is fewer database reads, a claim about production latency still needs evidence from a representative workload.
Give the optimization a contract
Consider a hypothetical interview built around a small pricing service, using a fabricated repository and synthetic data. Different tenants can sell the same product at different prices. The brief states that responses must remain tenant-specific, and a new request must see a price update committed before that request begins. Within a request, the fixture uses a fixed price snapshot.
The task is to reduce repeated database reads within a request without changing those semantics. Supply the contract, documentation and fixtures before the candidate starts. Give candidates comparable access to the supported tools and a clear task budget, rather than making a personal paid subscription part of the qualification. Keep those tools available when the operating conditions change.
The candidate might batch lookups, memoize within the request or choose another implementation. The assessment need not prescribe the solution.
Evidence in the pricing exercise
Illustrative assessment with synthetic data. The hiring rubric remains unvalidated.
- Tenant-specific prices
Two tenants share a product identifier but retain their different prices.
Does not establish every authorization property.
- Fewer repeated reads
An instrumented workload compares reads with correctness checks enabled.
Does not establish production latency.
- Freshness between requests
A committed price update appears in the next request.
Does not establish arbitrary distributed consistency.
| Property | Evidence to inspect | Remaining limit |
|---|---|---|
| Tenant-specific prices | Two tenants share a product identifier but retain their different prices. | Does not establish every authorization property. |
| Fewer repeated reads | An instrumented workload compares reads with correctness checks enabled. | Does not establish production latency. |
| Freshness between requests | A committed price update appears in the next request. | Does not establish arbitrary distributed consistency. |
Change the cache lifetime
The interviewer adds a request to retain cached prices across requests, reducing reads further. The candidate receives this change explicitly; the original freshness contract still applies.
A naive process-wide cache can return yesterday's stored price after the database has changed. The candidate can demonstrate that with a test that updates a price, starts another request and checks its response. A key containing the tenant identifier protects one property while leaving freshness unresolved.
A revised design might require invalidation or versioned entries. Those mechanisms bring assumptions that also need checking, including how a read remains correct when a price write commits before its cache entry is invalidated. If the requested cache cannot satisfy the contract within the exercise's limits, a justified recommendation against shipping it is a useful outcome.
Anthropic's October 9 report of unintended agent actions describes cases in which blocked or ambiguous tasks led agents to work around restrictions. Those cases are not a hiring study. For the cache task, state permitted actions and keep price checks within the supplied fixtures and synthetic data.
A capable agent can pass this too
In Anthropic's January 2026 account of redesigning its performance-engineering take-home, AI was explicitly permitted, yet the author reported that Opus 4.5 matched the strongest time-limited human output. The replacement gave up some realism for unusual constrained puzzles.
An agent could also read the pricing contract, construct the adversarial tests and reject the incompatible cache request. A candidate might relay instructions while contributing little. Watching the process does not resolve that problem by itself.
Record the proposed change, its supporting evidence and any remaining limitation. Rate the evidence for each property as absent, asserted, or demonstrated under the supplied conditions. Fluent narration alone stays an assertion, and an outage that prevents a check needs to be recorded as a cause of the missing observation.
Before using this exercise as a filter, pilot it with engineers who have the target competency and with agent-only runs under comparable conditions. Have reviewers assess the records without knowing their origin. If agent-only runs or simple instruction relays receive the same ratings, the exercise needs revision. Check whether ratings track tool access or interface familiarity.
This is a proposed design, not a validated predictor of job performance. A focused unaided check can still be appropriate when the role requires a particular skill without assistance, including troubleshooting when an external service is unavailable. State that competency and condition explicitly.
CodeSubmit's take-homes have long let candidates clone a repository and work with their own tools. We are testing native coding harnesses inside the CodePair v3 project workspace, where real agent executables can inspect and modify the project.
Start with one task already in your hiring loop. Name the competency it is supposed to reveal, state the invariant a change must preserve, and identify the evidence a reviewer can actually examine. Then change one consequential condition and test whether the rubric still distinguishes the skill you meant to assess.