Nick Turley: observation

Transcript context

…And there's also an element of the models are changing constantly and so you may not even realize what they're capable of, I imagine. Totally. The models are changing and the best way to improve them, whether or not you're a lab or actually just someone who's doing context engineering or fine-tuning a model maybe, you need failure cases, real failure cases, to make these things better. The benchmarks are increasingly saturated. So really you need real-world scenarios where your product or model is not actually doing the thing it was supposed to do, and the only way you get that is by shipping, because you get back to use case distribution and you can make those things good. And therefore, it's actually the best way to then go articulate to your team, especially your ML teams, what [inaudible 01:14:17] climb on? It's like, "Oh, people are trying to do X and the model's failing in ways. Why? Now let's make those things really good." This point about failure cases makes me think about something that both Kevin Weil and Mike Krieger shared, which is that evals are becoming a huge new skill that product people need to get good at because so much of product building is now writing evals. Is there something there you want to share?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.