Full transcript. Source: https://www.youtube.com/watch?v=tqUDjc1HzO4
This is the first part of a 12-part series, a mini course that we’re hosting on AI product engineering, and we’re going to kick it off with a bang. Shreya is going to go over how to use LLMs to help you with evals and do so in the right way, cuz a lot of people are getting stuck on this exact topic.
Thanks, Hamel. Really excited to kick off the series. So, I’m Shreya Shankar. I am a computer science professor, and I also co-teach, co-created the AI evals course with Hamel. And today, I’m going to talk about how to automate some of the tricky parts of evals effectively. It’s not automating AI evals with AI is not very easy, but I mean, it’s not stopping many vendors from trying to sell you automated eval tools. Goal of this talk is not to tear down any particular vendor. In fact, I won’t really get into specific vendors in this talk. The goal of this talk is to convince you that you can’t just farm out evals to your AI coding agent, but you can still use AI to help you. The goal is to inspire more thoughtful uses of AI when evaluating AI. So, what are evals? Evals are, in the abstract, systematic ways to measure the quality of your application. And building AI products requires evals because the quality of AI outputs has to be good. In fact, according to one of LangChain Smith’s recent annual reports, output quality is the biggest barrier to productionizing agents. And according to lots of Reddit posts out there, figuring out how to evaluate models is actually super hard. It’s quite natural that vendors want to automate evals for you. So, LangChain, Braintrust, and Arize all has automated evals tools, and their vision is to basically do evals end-to-end, it seems. So, point their tool, which is powered by an LLM, at your agent traces, and then the tool will tell you where your application is broken and fix it for you. So, this is a very ambitious vision, and I would love for it to be true. And while these tools can find some errors, they’re not a cure-all. And I found that you can’t really fully automate evals. And the reason for this is epistemic. What good means for your product is living in your head. It’s not in the traces. And if you think about it, if a tool could fully build and fix your product for you, it could do the same for everyone else, all of your competitors, and there would be nothing left to set your product apart from others. So, in other words, your judgment is the only differentiating factor of your product. So, this begs the question of what can AI do for evals if it can’t automate evals end to end? AI tooling, we should think about it as trying to help us developers express and apply our judgment that lives inside our head, apply that judgment faster to the product life cycle process. So, today we’re going to go over how to use AI effectively for evals. I’ll first talk about the evals framework we teach in our course, this life cycle of evals, and then some common mistakes that we’ve seen folks making when using AI to automate evals and how to avoid making these mistakes. And I’ll go through some demos as well with my own work on how I use AI to do evals. So, let’s start with a framework. Many of you might already know this, but we teach this three-step life cycle for evals, error analysis, measure, and then improve. So, the error analysis part is about how to take all of the traces or take the behavior that your agent is exhibiting in traces, and figure out what are the failure modes in there. Then, the second measurement step is how to measure the prevalence of each failure mode. So, if the failure mode is more prevalent, you want to prioritize fixing that. And then, the third step is, okay, actually going deep into the product and improving it. So, fixing your product so it doesn’t make those mistakes. And this is an iterative life cycle as your product evolves. So, you don’t just do all of these three things and then stop. So, I’ll illustrate some of these eval steps with a tool that we’re actually building in my research, an AI writing assistant. So, imagine we’re building an AI writing assistant, and form factor of it is not super necessary right now, but the idea is that the AI can help users write reports or essays or articles that they need to do. And our users don’t want to use the AI writing and see that it’s sloppy. And actually, AI slop is a real problem today. So, some recent statistics from Grass site show that over half of articles on the internet today sound noticeably AI generated. So, we want our assistant to not generate the slop, which is a very vague thing and very hard to do. So, how might we think about evals for such an app? First step would be to analyze AI writing for signs of mistakes or signs of bad quality. And analyze is actually the hardest step we found in this whole evals life cycle because there’s no perfect definition of a mistake here. So, for example, in writing, I can’t perfectly define what slop is. There is this know-it-when-I-see-it flavor to the problem and it makes it harder than found in traditional machine learning in supervised machine learning where we had labels for all of our data and we could simply define the failure as the prediction of the model doesn’t equal the label. So, in this case, we’re in a lot of discovery involved. Now, maybe after error analysis, we have found a bunch of different failure modes. The second step is measuring how prevalent these failure modes are. So, maybe we have all these failure modes of AI has overused a lot of the emojis or it’s creating these overusing some slogans or phrases like it’s not X, it’s just Y. I’m sure you guys have heard of this. What we found in eval is a lot is there’s this Pareto principle here. So, 80% of issues in the data are often caused by 20% of failure modes. So, we really want to find those most common failure modes and prioritize fixing those. Then, once we’ve measured how prevalent the failure modes are, then we can go and improve our AI writing assistant. And there are lots of different ways we can improve it for fixing a failure mode. For example, we could add an instruction to the prompt, we could switch models, we could fine-tune models. I won’t get into all of those details. But hopefully, this gives you a sense for what the evals life cycle is in the whole process of building a product. And in terms of how we might use AI effectively here, we found that AI is not very good at automating the early parts, the error analysis parts. These are very taste-specific. But the more we go in the life cycle, AI can automate and do much more of that for us. So, some of this measurement step and of the hill climbing and improving the prompts or doing prompt optimization, AI can be really helpful at that. So, this is gives you a sense for how to think about where AI sits into your life cycle of doing emails. So, now let’s talk about some of the common mistakes people make and good ways that we can actually use or infuse AI into our workflows. Let’s start with mistake one, which is opening up a coding agent and simply asking AI to evaluate the application for me. And often I’ll see prompts or people will tell me things like they just opened up Claude Code and then they said, “Check out my traces. Can you evaluate the app? Here is a pointer to Here’s the S3 bucket or here’s the file of all of the outputs. Go evaluate.” And if you look at the reading traces, the agent will do something reasonable. The agent will say, “Okay, let me read the data. Let me understand what we’re working with.” I promise that this is a real screenshot. I didn’t just doctor it. And then 5 minutes later after thinking, AI will come up with something. Here’s what Claude said after reading all the traces. It found some problems. They look fine at a glance. I’m sure many of these are issues we want to fix. But the real issues here are one, agents can discover some issues, but they will miss all of the taste specific issues. So, going back to what I said earlier, anything that lives in your head that’s not externalized in the traces or in the prompt, you’re never going to find it. The agent is never going to find it in the error analysis. The second point is that agents typically don’t find the highest priority issues and they’ll even fabricate what they think is the highest priority. Like in this screenshot here, extremely repetitive voice, the biggest issue. The first of all, extremely repetitive is pretty vague. But then second, it might not be the highest priority issue to fix. There might be other really weird phrases that the AI says or hallucinations that the AI makes. Who knows, right? That might actually be more high priority for you to fix. So, you can’t just rely on what the agent said. And then the third thing, and this is more of an efficiency and a cost standpoint, is that there’s no reuse in this workflow. So, when you ask AI to find issues in your data, you’re asking it to reread the data, come up with new failure modes, do the life cycle from scratch. And if you want to do it again on a new batch of traces or more traces, there’s nothing that’s being shared across these different runs, which is a very suboptimal from an efficiency as well as accuracy in learning standpoint. What we really want in our workflow is to persist intermediates of the life cycle. So, once we come up with good failure modes and definitions of that, we want to be able to persist them somewhere in the memory or persist them in code somewhere. We want constant visibility and updates into the process. So, that aside, can we ask AI to help us do evals better? One thing I found that’s helpful for me is just to stick to my workflow and ask AI to help in very specific parts of the workflow. So, instead of trying to automate end-to-end, maybe AI could help us in the error analysis part. It could help us design human-in-the-loop interfaces or dashboards, for example. I won’t get into the AI for measure and improve today. But, part of this talk I’ll share is the skill that I’ve developed to help me do AI-assisted error analysis. So, I’ll walk through it at a high level and then demonstrate it to you in action. So, first, the skill instructs the AI agent to look and try to understand the data. So, that might be load a sample of the data, which is in some JSON file, and then try to come up with different fields or structure or content that’s repetitive or shared across these traces. Then, the second thing the AI is instructed to do in this skill is to take this understanding of data and design a visual encoding. So, think of that as a way to represent all of this data in a UI for human to review. So, this is things like if you find things that vary within a trace, maybe assign different colors to those things that vary. So, a very a simple example of this is the role, if it’s an assistant or system or user that sent the message, maybe you can highlight those different dimensions with different colors. So, things like this where using visual encoding to help the human really understand the data better. It’s not really necessary that you know exactly what this means. It’s just a common thing in design that people do to help design good UIs. And the AI is instructed to do this in the skill. Then the meat uh of the building part is to design a review app to help the human or help myself go through these traces using this visual encoding. So there are three views that the AI agent is instructed to make. But this is a trace viewer, so one-by-one trace viewer. The second one is a map view, so you can see a map of all of the traces that need annotation. And then the third is a progress view, so the more that I have labeled or more progress that I’ve made more failure modes that I’ve found. tree map of all of the failures in that progress view. And the fourth part of this workflow is to figure out what samples or what traces to show in the UI. So the agent is instructed in this skill to cluster to sample representative traces from each cluster. So as a human I don’t have to review every single once. Then the last part is really the human-in-the-loop workflow that goes on with this review app. So the agent has built the app and then now the human-in-the-loop is to have a human review traces that were selected by the agent, of course. And as the human provides input or labels or annotations in the UI, the agent is instructed to always be monitoring, so you can see the AI agent box in that step five here. Always be monitoring, always try to build a taxonomy of failure mode, and then we instruct the agent to go for breadth and depth, which I’ll explain a little bit more. So the idea is go for breadth, try to help the user cover all of the failure mode possible, and then depth, which is for every single failure mode for them to go in and find many examples of that failure. I know this is a lot of it as a high level, and now let me get into actually showing you a demo. Okay, so to recap, what I’m trying to do is show you my error analysis skill in action. And here are some traces for my writing assistant. So what you These are not actually traces, these are just the actual essays that are finally generated by the LLM. There’s a lot going on here, so I don’t expect you to be able to read it, actually. Just know that there’s a bunch of different essays that are AI-generated and we want to analyze them for smells of bad AI writing. So, I’ll invoke my skill to help me construct this review interface. It’ll to help me to do analysis on the AI articles. And hopefully, as the custom review interface is being created, you’ll see what the AI is doing. So, I’ll start by reading the data set to understand the structure and the content and the step one that I mentioned. Okay, so it’s running some shell commands, so you’ll see that it’s slow cuz it’s actually trying to look at the data here. It’ll look at the structure of the data. Great. So, there’s 33 AI-generated blog articles. It’s not very many, but you could imagine that you have a lot more. So, now it’s going to cluster and select some samples and then build the review interface app. So, what I will say here it’s useful to talk about why we want to do some clustering and sampling. You can try to review every single AI-generated essay or trace. I found that I get very tired very fast, so it’s helpful to get a smarter sample. Unfortunately, sampling with the LLM is really hard in this skill, the LLM is instructed to find features that could explain how these articles or these traces vary, but of course, it’s it’s very much a guess. So, it’s useful to look at samples, but you can’t assume that the clustering or that the sample selection is perfect. Okay, so it’s preparing some data. I guess this is all the sampling stuff. Yeah, so it’s collected 21 samples for the human to review. Human means me or us. I wrote a server. Now building the HTML app. This is the main interface. It looks large, so I’ll write it in one go. So, in the first version of the skill, I actually instructed the AI to embed the traces and then cluster the embeddings with K-means. And then I realized that it’s not always the best idea to use embeddings. Embeddings may not be the best representation of the trace, or sometimes we might not even be analyzing the traces itself, like in this case, we’re just analyzing the final essay. So, the skill is actually very open and it just says find features that are useful in the data, and then cluster.” It doesn’t specify how to do clustering, and the hope is that the AI agent is smart enough to think about what the data is, and reason about what’s the best clustering to do there. So, I have done used this skill for error analysis for like data agent traces, think like data science, data that are helping with data science, and I’ve used it for this AI assisted writing. And I don’t have a very satisfying answer other than sometimes I’m just very much iterating on the skill. So, whenever I see the UI that’s created by the agent, sometimes I have feedback for how to make it better, and then I try to encode that back into the skill. So, it’s definitely very much a work in progress, and I don’t know of anyone who has a great error discovery skill. So, if you have any pointers to those, please send them to me. So, the skill comes from the human. This can be actual user queries. This could also be synthetic queries. It really is dependent on what you have your building. All right. So, we’re now having promising experiences. Oh, lovely. Okay, so the app has shown up. So, hopefully you got a sense for how long it also took. It took, I don’t know, a few 5 minutes, I would say, to come cooked for 4 minutes and 42 seconds to build this. So, it’s actually pretty fast in the grand scheme of things. I promise I didn’t pre-prepare anything for you. So, now let’s make this smaller, so that way we can see this UI that it generated. So, right off the bat, when I feel like UI generation with these agents, even though we’re using some of the best agents, I feel like we have a long ways to go for UI. Don’t judge this as like the UI needs to be perfect. It’s simply something rough for us to be able to look at data better. I often find it see like it generates a bunch of these labels. I don’t know why. Sometimes I get frustrated, and I tell it that my agent, “Please don’t generate all of these pills or these chips. It’s really not necessary.” But anyway, let’s go actually into the error analysis. So, I always like to start out with this map view. Now, again, someone said I would have done t-SNE instead of PC UMAP instead of PCA. I’ve totally used whatever clustering technique that you wanted. This the AI agent wanted to use this method, But, the idea here is that there are six different clusters and then some of these orange ones are annotated and some of them are Annotated means select or annotation. Maybe not. I don’t know why orange is said twice. So, again AI makes mistakes when generating this. So, there shouldn’t be the same color being used twice. That’s a little bit insane. But, anyways, you can kind of kind of get a sense for what these clusters are. So, maybe there’s different topics here that the AI tried to do and so forth. But, anyway, so next thing we do is go back to the reviewing. So, there’s 21 samples that are selected in this. And then I just actually just go through and like actually start trying to think about, okay, how is What are the errors? What are the mistakes in this essay? So, this is about container gardening for apartment dwellers. And the agent actually, when generating this article, also had access to the internet and could search for things. So, I’m going to just go through it. I’m not going into it too fast. But, I’m I’ve done some error analysis on writing before, so I already have an idea of what are AI writing mistakes. One thing is I don’t like having colons in each sentence. Or I don’t really like having colons. And I’ll just send that. And so, you’ll notice here that the way that I give feedback in this app is in situ. That is, I highlight the text and right so the text, I give feedback so that I can externalize it immediately. So, that’s going on. I keep can keep going. I want all my sentences to be complete sentences. I want sentences sentences to be complete sentences. I’m going to give it one more thing. So, I also good UI principles show progress as it’s going. So, maybe I’ll go to the next one here. And one thing that you should have seen, but for some reason we’re not seeing, is that the AI agent is instructed to monitor all the feedback that the human is giving. So, I don’t know why it’s not doing that. It’s supposed to use the monitor tool. So, why aren’t you using the monitor tool like I instructed in the skill? So, the skill is open source cuz all the annotations get saved to a file if somebody asked a question. So, yes, all the annotations get saved. I’m using one of the state-of-the-art agents. It’s not always perfect. You do need to coax and cajole as we go on. I know all of the bad things that you see in this app are probably they’re not hard-coded in the skill. I don’t like the smell of writing, which is it’s not just X, it’s Y. I don’t like the Y framing and phrasing. Great. All right, so that annotation went through. The monitor event was triggered. Three annotations just came in, actually just one. Yeah, so let me read them and start categorizing it. So, you can see the AI agent in the background is simply trying to build the taxonomy of all of these open-ended outputs. So, at a meta level, what’s happening to the human, I’m just getting my open-ended feedback. I’m not thinking too much. And then all of the tedious stuff around maybe taxonomizing it, categorizing it, and so forth, visualizing progress, all of that let’s delegate to the AI agent. Yeah, so taxonomy plus you can see it in the progress view now. So, if I go to progress view, I’ll see that there are two different failure modes. One is informal fragments and colon interjections. Another one is the contrasted framing. We don’t like that. And we could keep going even. So, you know what the great day was is whenever I built the first version of the app, I had no idea that these preferences should have been encoded in the system prompt. I thought the AI should simply know. But through doing this error analysis, I’m understanding that these are issues and then we can think about in the improved step of the life cycle going back and editing the system prompt. All right, so I’ll do one more and then let’s go back to the slides. This was the bulk of the demo that by the way that I wanted to show you. I just really wanted you to get a sense of what was happening. So, what are some takeaways from that experience? I know it was a little bit chaotic, but I love doing live demos. Anyways, point number one is to really build this interactive experience where you annotate data. So, it’s not just ask your AI assistant to build your application and then you sit and you annotate things in the application. You want to really have this iterative feedback loop where the AI agent is really reacting and responding to you in real time. And you can do that with the skill. The second point is trying to optimize for both breadth and depth, which I didn’t have a chance to annotate many examples, but as you’re annotating, think about covering adding as many failure modes as possible in that progress view is very helpful for that. And then as well as for every failure mode trying to have multiple examples of that failure mode. All right. So, the next mistake I want to talk about is not iterating over data. So, many people look at a trace only once when annotating or look at an output only once when annotating. And this is not ideal because research like my research as well has found that when you go back and you look at traces again, you’ve already analyzed, you actually end up find new failure modes. So, concretely, why is this the case? When you’ve seen more traces, you have more traces living in your head, then you can actually do better analysis. When you different combinations of traces in your head actually spark failure modes that you discover. So, let me actually show you an example of how that materialized for me in this AI writing. But one thing I noticed when I was reviewing essays one by one is that I found new trigger phrases for me that I hadn’t seen before. So, I already reviewed two. One trigger phrase that I found when reviewing the third one is this overuse of the word matters. So, every article for some reason has the word matters in it, including the ones that I already reviewed. So, here, such a matters. Up, this just see matters. Once I tell you this matters thing, you’re never going to unsee it. When you see the article on X, you’re going to be like, up, this break matters. First one, matters. Up, it matters in the hindsight. So, anyways, I didn’t like this phrasing of it matters, but this was something I only actually found very deep into my error analysis process when I’d seen the word matters many times. So, that’s kind of goes to show that you really do want to be going back and looking at things that you’ve already annotated for new emergence of new failure modes. And one thing I like to do is in this skill, I like to tell my AI agent to look at my annotations and then go back and look at previous traces I’ve annotated to find instances of those failures. So, this is exactly what happened here. The agent suggested new instances of failure modes of traces I’d I’ve reviewed. So, remember on one trace I said I don’t like sentence fragments. The agent went and looked at other traces for examples of sentence fragments, traces that are also reviewed, and then I can accept or reject them. So, here I want to accept that. Here’s another contrastive framing that I missed. Let’s accept that. And maybe there’s also more. Then my progress view I usually have an agent suggestions tab, which looks at again the things that I’ve already flagged and then can suggest to me. Yeah, so these are all informal fragments. Typically I’ve always found that I just still doll and accept these as examples of failures. But then I also note that the agent is not always exhaustive at coming up with these if they’re able to flag instances of the failures, but can’t flag all of them. Just something to note. Again, it’s for accelerating really your own review. So, one thing that’s interesting here is I don’t tell the agent to find me entirely new examples of failure modes. It’s not changing the taxonomy. It’s not adding to the taxonomy. That’s really left to me. I’ve asked in previous iterations of the skill for the agent to suggest new types of failure modes, but I’ve always found that some of them I agree, some of them I disagree, and I don’t love the experience of just trying to validate the agent’s taste. It’s much easier for me to say what I think and then have the agent scale that out to the applying that to many traces. Great, so what did we learn from this mistake number two, which is don’t try to one-shot discovering all the failures, just only look at your data once. You want to loop over your traces multiple times. So, one way to think about this is an outer loop and inner loop framing. So, you have an outer loop of iterating over the data, and you have an inner loop of kind of hypothesis iterating through different failure modes within a specific data point itself. And then there’s other thing that I mentioned is asking your AI agent to take every new annotation you give and try to apply it to previously labeled traces as well as unlabeled traces even. All right, so those were two mistakes. This last mistake that I see some people make, this is a very short I won’t spend too much time on it, is having a uniform accuracy bar for all of your AI applications. So, all All means is sometimes people think that they should equally invest in the emails process for all of their different AI products or AI applications in their company. That is not really ideal. How should you structure your emails? This really depends on the context of your application. So for some internal tools like folks like myself have AI tools for summarizing Slack threads or summarizing data. These things don’t need to be perfectly accurate. We don’t have to spend too much time investing in emails there. But for anything that’s customer user facing, you have to spend a lot more time having a high accuracy bar there. And one way I try to think about reasoning about what the accuracy requirements of an application are or eliciting some failure modes, hypothetical failure modes, is to reason about worst-case scenarios up front. And I actually like to do this with an AI assistant like Claude Code or Codex. I give Codex a sample of all my traces, a description of the application, or even my code base. And then I ask it to help me reason through if I were to put myself in the user’s shoes, like what are the worst things that could possibly happen. So for the writing assistant, maybe it’s like sabotaging the citation, or it could be leaking some information that was private. Like for example, if a journalist is trying to write a story, we don’t want all of the information that the journalist has collected often to be in the output of the article. So these kinds of worst-case scenarios can help reason about what kind of emails and guardrails to really look for. So these are just three mistakes that I had time to talk about today. Using AI to help with AI emails is a really fast-moving area. As you can see, it’s not perfect. We’re still working on it a lot. And we’re actually in the process of a more thorough deep dives and evaluation of the existing automated email tools out there. We’re benchmarking them on our data, on real data. So this work is actually led by Amel and Antariksha Dasgupta. I’m really excited for it to come out. Interestingly, we found that for general-purpose coding agents like Claude Code and Codex, what I showed you, often end up being more exhaustive and good at finding failure modes than some of these dedicated discovery platforms. So that was very surprising to me. I guess the agents are are pretty good. And this other saying is that no, there’s no such tool or no such evals tool that can find all of your failure modes with very good precision. So, stay tuned for this research to come out. Yeah, one thing I’ll say about it is might be surprising that these there’s like automated evals tools from vendors may not give you that much lift over Codex or Claude. And the reason that is is because at the end of the day it’s someone’s prompt. And really what these like eval tools are is like someone’s prompt. Maybe a little bit of a harness, but you can likely inject your own domain expertise and prompt it yourself. And then also you can also customize your coding agent to your data. And with that like you can really close the gap. That’s like the kind of some intuition behind this, but we’ll be sharing a lot more details as she mentioned. All right. So, to summarize my talk today, we discussed the analyze measure approve life cycle for evals. We talked about three mistakes that’s very easy to make when using AI to automate evals. So, don’t ask your AI to do evals. Split this your workflow into the steps. Engage in an interactive workflow with your AI. Second, don’t just do one pass over your traces. Loop over them to surface new failure modes. Third, don’t just have a universal accuracy or uniform accuracy bar across all of your applications. Really reason about what applications are customer facing, some worst case scenarios. Use those to work backwards into what evals and guardrails you want to have. The scale that I showed you is open source so you could check it out. Really in flux, I always edit it keep updating it. And last but not least, I’m teaching a new version of the evals course this fall. Himel and I are totally revamping it, super excited for that so you can sign up. I’m also hiring undergrads, masters, PhDs for my research lab. So, if you’re interested in doing research, I can be reached at this email address. This slide really captures how I think about using AI for evals, which is figure out how to express your taste that hidden in your brain and your judgement into your product development workflow. And this is pretty hard to do. So, I’m looking at the data as a best way to do that because it will trigger some concerns that you have about your application and then you can externalize it to the AI. I don’t know of a way for AI to probe you 20 questions style for how to externalize taste very good. So, there will be some human element in Evals always. Okay, we’ll figure it out later. So, thanks everyone for engaging and hope to see you next time. Thank you.