Chip Huyen: recommendation

Transcript context

…Awesome. That is a really pragmatic answer. What I'm hearing is evals are great, very important, especially if you're operating at scale, but pick your battles. You don't need to write evals for every little feature. Something that Hamel and Shreya shared is that people need just, I don't know, five or seven evals for the most important elements of their product. Is that what you see or do you see a lot more in production that people build and need? I don't think of just a fixed number on the evals. What was the goal of eval? The goal of eval is to guide the product development. So you see eval, because I think I'm a big fan of eval, is that it helps you uncover opportunities where the progress are doing well. So sometimes, we've seen a very obvious [inaudible 00:28:15] where you look at the eval and we realize it's like, okay, it performed really poorly on this specific segment of users and then we look into it's like, okay, what's wrong with it? And it turns out, it's like we just don't have a good messaging to it. So people should just focus on the things that we're doing poorly, can improve significantly. Yeah, so I kind of like the number of eval is really depends. We have seen product with hundreds of different metrics. Oh, wow.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.