Evals matter, but a bigger question comes first: are you building the right thing? Hamel argues that most AI products fail not on model quality but on trust, because they hand users a one-shot answer that is impossible to verify.
The verification bottleneck
The typical AI product takes a question (“what was revenue for product A last quarter?”), queries a database, thinks, and returns a number like 4.21 million. It looks like a win: no data scientist needed, faster answers. But the user has no way to know if the number is right, and putting an unverified figure on a board deck is how you get in trouble. To trust it, they would have to redo all the work themselves, which means the product saved almost no time.
Hamel ties this to a tweet from Lenny: a data scientist now spends most of his time reviewing half-baked AI analyses from PMs and engineers, and roughly 50% of them are wrong. Producing output has become trivially cheap; the real bottleneck has shifted to verification. His diagnostic test is a “product smell”: if you as the builder can’t easily tell how you would eval whether an answer is correct, then your users can’t check it either. The fix is to design the product so it collects and surfaces the evidence that supports its answer.
Show your work: provenance and intermediate steps
The reimagined data-analysis product returns the same answer but wraps it in supporting evidence. It bases the analysis on a vetted notebook someone already signed off on, states its assumptions by confirming the metric definition (pulled from the company’s semantic or knowledge layer and made clickable), and exposes intermediate calculations like returns and customer counts, flagging issues along the way. The point is not the specific layout but the principle: back the number with information so the user has something to stand on instead of starting from scratch.
Behind an “open as notebook” button sits a familiar data-analysis surface (Jupyter, Marimo) with prose narrative, queries, and definitions. The notebook can be fully AI-generated, yet a data scientist can step through it, edit it, ask the AI questions inside it, and confirm the cells check out; anything the AI could not verify, it flags explicitly. This is a real, shipping pattern, not science fiction: Hamel points to Hex, a chat interface that drops you into a notebook.
Constrain and ground the output with social proof
The PE lesson-plan assistant shows the same lesson. A teacher enters grade, class length, location, and equipment, and the tool generates a polished plan, but neither the developer nor the teacher can tell if it is actually any good. The better version grounds the generated plan on established curriculum and shows what trusted colleagues elsewhere are doing: this plan is used at 14 schools, has been run 30 times, and you can open the original. Seeing peers use something builds far more confidence than a plan conjured from nothing, and the teacher learns from it too.
When the AI adapts an existing plan it presents the changes as an accept/reject diff (“shortened to a 45-minute class, matched to your equipment”). On the cold-start worry (no existing plans), the answer is to seed a handful of expert-vetted plans up front; that investment makes the product both easier to eval and easier for users to trust.
Act as a research assistant, not a report machine
The worker’s-comp report generator ingests a patient’s entire claim (thousands of documents) and produces a report, often 52 pages, that a doctor uses to support or deny the claim. A one-shot 52-page document is unverifiable: the doctor would have to re-read every source to confirm it is represented correctly, which is no help at all.
Instead, the product should mirror the work a doctor actually does, which starts with research: understanding the documents, how they relate, the key facts, contradictions, and gaps. The redesign surfaces atomic elements the doctor can interrogate: open both pages behind a contradiction and resolve it, agree with or dismiss a key fact, add a note to an open question the AI flagged but couldn’t answer. Building that shared understanding is valuable even if the system never emits a final report, because the goal is to make the human understand, not merely to generate a document.
The four trust principles
Hamel distills the examples into four design principles. Provenance means showing where information came from (a prior analysis, source documents, someone’s lesson plan); the more curated and trustworthy the source, the more trust it earns. Progressive disclosure means revealing detail at the right moment so you don’t overwhelm the user (the notebook opened in a second step, the diff explained why a change was made), and the right amount shifts as the product and user mature. Signals and heuristics are the quick checks experts already use (a smell test against a related metric, whether a trusted colleague uses this plan, social proof and supporting evidence) and the product should surface them. Keeping things modular means breaking work into small, verifiable steps and showing your work along the way; sometimes you deliberately force the human into the loop, as with making the doctor step through the evidence before a report is generated.
Key takeaways
- If you can’t easily eval whether an answer is correct, neither can your users; that difficulty is a product smell, not just an eval problem.
- The bottleneck has moved from producing output to verifying it, so design products around verification.
- Ground AI output in vetted, provenance-rich sources rather than generating one-shot from nothing.
- Use progressive disclosure to reveal intermediate steps and reasoning without overwhelming the user.
- Surface the signals and heuristics experts already use to sanity-check answers, including social proof.
- Break work into small verifiable steps; the goal is to make the human understand, not just to emit a work product.
Sources
- Watch the video: youtube.com/watch?v=0ASUz7gFRcw
- Full transcript: build-ai-that-earns-user-trust
- Related: automate-ai-evals