Vendors sell tools that promise to point an LLM at your traces, find every bug, and fix your product end-to-end. Shreya Shankar’s argument is that this can’t work, and the reason is epistemic rather than technical: what “good” means for your product lives in your head, not in the traces, so AI’s real job is to help you apply your judgment faster, not to replace it.
Why evals can’t be fully automated
Evals are systematic ways to measure the quality of an AI application, and output quality is repeatedly cited as the biggest barrier to putting agents into production. That demand is why LangChain, Braintrust, and Arize all ship automated eval tools whose vision is to read your agent traces, tell you where the app is broken, and fix it for you.
Shankar’s core claim is that this vision is unattainable, and the reason is not that the models are too weak. It is that the definition of “good” for your product is not externalized anywhere the tool can read. If a tool genuinely could build and fix your product from traces alone, it could do the same for every competitor, and there would be nothing left to differentiate your product. Your judgment is the differentiating factor, so the right framing for AI tooling is that it helps developers express and apply the taste in their head faster across the product lifecycle, not that it discovers taste on its own.
The analyze, measure, improve lifecycle
The course teaches a three-step, iterative lifecycle. Error analysis (analyze) means taking all the traces or observed agent behavior and figuring out what the failure modes actually are. Measurement means quantifying how prevalent each failure mode is, so you can prioritize; in practice a Pareto pattern holds, where roughly 80% of the issues come from 20% of the failure modes, so finding the common ones matters most. Improvement means going into the product and fixing it, whether by adding a prompt instruction, switching models, or fine-tuning.
Where AI fits changes along this lifecycle. The early error-analysis step is the hardest and the most taste-specific, and AI is weak here because so much of what counts as a failure is never written down. The later steps automate far better: measuring prevalence at scale, hill-climbing on prompts, and prompt optimization are all things AI does well. Shankar illustrates this with an AI writing assistant whose goal is to avoid “AI slop,” a deliberately fuzzy, know-it-when-I-see-it target that has no clean label the way supervised ML did, which is exactly why the analyze step involves so much open-ended discovery.
Mistake one: farming error analysis out to a coding agent
The most common mistake is opening Claude Code, pointing it at an S3 bucket or a file of outputs, and saying “evaluate my app.” The agent will do something reasonable-looking and, minutes later, produce a list of problems. Three things go wrong. First, the agent only finds what is externalized in the traces or prompt, so every taste-specific issue living in your head is invisible to it. Second, it does not reliably surface the highest-priority issues and will confidently fabricate a “biggest issue” (in her example, a vague “extremely repetitive voice”) that may not be what actually matters most. Third, there is no reuse: each run rereads the data and reinvents failure modes from scratch, with nothing persisted across batches, which is bad for both cost and cumulative learning.
The better pattern is to keep your own workflow and ask AI to help with specific parts of it, persisting the intermediates (failure modes and their definitions) in memory or code so they carry forward. Shankar’s open-source error-analysis skill does this by having the agent (1) read a sample of the data and infer its structure and repeated content, (2) design a visual encoding to represent traces in a UI, for example coloring different message roles, (3) build a review app with a one-by-one trace viewer, a map view of all traces, and a progress view showing a tree map of discovered failures, (4) cluster and sample representative traces so the human need not review everything, and (5) run a live human-in-the-loop loop. Notably the skill does not prescribe how to cluster; it just tells the agent to find useful features and cluster, trusting the agent to reason about the data (an earlier version hard-coded K-means over embeddings, which she dropped because embeddings are not always the right representation, especially when you are analyzing a final essay rather than a trace).
The human-in-the-loop annotation workflow
The demo shows the payoff of keeping a human in the loop rather than one-shotting. The human reviews the sampled traces and gives open-ended, in-situ feedback by highlighting text directly in the article and writing what is wrong (“I don’t like colons in every sentence,” “I want complete sentences,” “I don’t like the ‘it’s not just X, it’s Y’ framing”). Meanwhile the agent runs in the background, monitoring every annotation and continuously building a taxonomy of failure modes, which then populates the progress view (for example “informal fragments and colon interjections” and “contrastive framing”).
The division of labor is the point: the human supplies raw judgment without overthinking, and the tedious work of categorizing, taxonomizing, and visualizing progress is delegated to the agent. Two annotation principles guide the human: go for breadth by covering as many distinct failure modes as possible, and go for depth by collecting multiple examples of each one. Shankar is candid that even a state-of-the-art agent misbehaves live (it initially failed to fire its monitor tool, and it generated redundant UI chrome and reused a color), so you coax and cajole and keep iterating the skill itself as you spot rough edges. A telling insight from the demo: she only realized her writing preferences should have been encoded in the system prompt because error analysis surfaced them, which feeds directly into the improve step.
Mistake two: only looking at each trace once
Reviewing every output exactly once is a mistake, because looking again at traces you have already analyzed reliably surfaces new failure modes. The mechanism is that once more traces are “living in your head,” new combinations spark patterns you could not see before. Her concrete example is the overuse of the word “matters”: she only noticed it deep into the process, after seeing it many times, and once seen it was visible retroactively in essays she had already reviewed.
She frames this as an outer loop that iterates over the dataset and an inner loop that iterates over hypotheses within a single data point. The AI accelerates it: when she gives a new annotation, she has the agent scan both previously reviewed and unreviewed traces for other instances of that same failure, surfaced in an “agent suggestions” tab she can accept or reject. The boundary is deliberate. The agent applies and scales out her taxonomy, but it does not invent new failure-mode categories; earlier versions that let the agent propose new taste were dropped because validating the agent’s taste is a worse experience than stating her own and having the agent scale it. She also notes the agent flags some instances of a known failure but is not exhaustive, so it accelerates rather than replaces the human pass.
Mistake three: a uniform accuracy bar for every application
The last mistake is investing equally in evals across every AI product. The right bar depends on context: internal tools like a Slack-thread summarizer do not need to be near-perfect and warrant little eval effort, whereas anything customer-facing demands a high accuracy bar and much more investment.
To set that bar, Shankar reasons about worst-case scenarios up front, and this is a task she is happy to hand to Claude Code or Codex. She gives the agent a sample of traces, a description of the app, or even the codebase, and asks it to put itself in the user’s shoes and enumerate the worst things that could happen (for the writing assistant, sabotaging a citation, or leaking private information a journalist had collected into the published article). Those hypothetical worst cases work backwards into which evals and guardrails are actually worth building.
Why vendor eval tools may not beat a coding agent
An early result from ongoing benchmarking (led by Hamel and Antariksha Dasgupta, on real data) is that general-purpose coding agents like Claude Code and Codex often find failure modes more exhaustively than dedicated eval-discovery platforms. Hamel’s intuition for why: at the end of the day these vendor tools are mostly someone else’s prompt plus a bit of harness, so you can close the gap by injecting your own domain expertise into a prompt and customizing a coding agent to your data. The consistent bottom line is that no eval tool finds all your failure modes with good precision, and there will always be a human element, because the whole task is externalizing the taste hidden in your brain into your development workflow, and looking at the data is the best trigger for surfacing it.
Key takeaways
- AI cannot fully automate evals because “good” lives in your head, not the traces; use AI to apply your judgment faster, not to supply it.
- Follow the analyze, measure, improve lifecycle; AI is weakest at the taste-heavy analyze step and strongest at measuring prevalence and optimizing prompts.
- Don’t tell a coding agent to “evaluate my app”: it misses taste-specific issues, fabricates priorities, and reuses nothing. Split the workflow and persist failure modes.
- Keep a human in the loop giving raw, in-situ annotations while the agent builds the taxonomy in the background; aim for breadth (many failure modes) and depth (many examples each).
- Loop over traces more than once (outer loop over data, inner loop over hypotheses); re-reading surfaces failure modes like the overused word “matters” that a single pass misses.
- Let the agent scale your taxonomy across old and new traces, but keep category creation human, it is not exhaustive.
- Set eval investment by context, not a uniform bar; reason about worst-case scenarios (using an AI assistant) to decide what guardrails customer-facing apps need.
- Vendor eval tools are largely someone else’s prompt; a coding agent customized to your data can match or beat them.
Sources
- Watch the video: youtube.com/watch?v=tqUDjc1HzO4
- Full transcript: automate-ai-evals
- Related: build-ai-that-earns-user-trust, what-are-ai-agents