Evidence receipt / recommendation
Published · transcript-backedShreya Shankar: recommendation
25 Sept 2025 Lenny's Podcast Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)
“If somebody is coming to you with a way to do something that's entirely new and not grounded in hundreds of years of theory and literature, then you should, I don't know, be a little bit wary of that.”
Source trail
Everything needed to verify it.
- Speaker
- Shreya Shankar
- Attribution
- Verified speaker
- Claim type
- recommendation
- Recorded
- 25 Sept 2025
- Publisher
- Lenny's Podcast
Transcript context
…And just to double-click on this point, this is not a thing everyone does or knows. This is something you two developed based on your experience doing data analysis and data science at other companies? Well, I want to caveat that we didn't invent error analysis. We don't actually want to invent things. That's bad signal. If somebody is coming to you with a way to do something that's entirely new and not grounded in hundreds of years of theory and literature, then you should, I don't know, be a little bit wary of that. But what we tried to do was distill, "Okay, what are the new tools and techniques that you need to make sense of the LLM error-out analysis?" And then we created a curriculum or structured way of doing this. So this is all very tailored to LLMs, but the terms open coding, axial coding, are grounded in social science. Amazing. Okay. What's funny about you guys doing this is I just want to go do this somewhere. I don't have any AI product to do this on, but it's just like, "Oh, this would be so fun." Just sit there and find all the problems I'm running into and categorize them and then try to fix them.…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.