Evidence receipt / uncertainty
Published · transcript-backedShreya Shankar: uncertainty
25 Sept 2025 Lenny's Podcast Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)
“Actually, I only need to do 15." I don't know. Depends on the application and depends on how savvy you are with error analysis for sure.”
Source trail
Everything needed to verify it.
- Speaker
- Shreya Shankar
- Attribution
- Verified speaker
- Claim type
- uncertainty
- Recorded
- 25 Sept 2025
- Publisher
- Lenny's Podcast
Transcript context
…... that. ... in data analysis and qualitative analysis called theoretical saturation. So what this means is when you do all of these processes of looking at your data, when do you stop? It's when you are theoretically saturating or you're not uncovering any new types of notes, new types of concepts, or nothing that will materially change the next part of your process. And this kind of takes a little bit of intuition to develop, so typically, people don't really know when they've reached theoretical saturation yet. That's totally fine. When you do two or three examples or rounds of this, you will develop the intuition. A lot of people realize, "Oh, okay. I only need to do 40, I only need to do 60. Actually, I only need to do 15." I don't know. Depends on the application and depends on how savvy you are with error analysis for sure. And your point about you're going to want to do a bunch. I imagine it's because you're just like, "Oh, I'm discovering all these problems. I got to see what else is going on here."…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.