Evidence receipt / evaluation
Published · transcript-backedHamel Husain: evaluation
25 Sept 2025 Lenny's Podcast Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)
“If you look at the eval products, let's say the ones up until recently that some of the big labs have, they don't have error analysis. They have a suite of generic tools, cosine similarity, hallucination score, whatever, and that doesn't work.”
Source trail
Everything needed to verify it.
- Speaker
- Hamel Husain
- Attribution
- Verified speaker
- Claim type
- evaluation
- Recorded
- 25 Sept 2025
- Publisher
- Lenny's Podcast
Transcript context
…Yeah, they don't correlate with math problem-solving, sorry to say. Exactly. If you look at the eval products, let's say the ones up until recently that some of the big labs have, they don't have error analysis. They have a suite of generic tools, cosine similarity, hallucination score, whatever, and that doesn't work. It's a good first stab at it. It's okay. At least you're doing something, getting people, maybe it's like getting people look at data. But eventually, what we hope to see is, okay, a bit more data science thinking in this eval process. That's hopefully the tools we'll get to. Yeah, Pamela and I should not be the only two people on the planet that are promoting a structured way of thinking about application specific evals. It's mind-boggling to me. Why are we the only two people doing this the whole world? What's wrong? I hope that we're not the only people and that more people catch on.…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.