High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Shreya Shankar

Published podcast speaker

Claims
15
Episodes
1
Shows
1
Named items
2

Books, apps, and tools

The evidenced stack.

Browse the grouped index →

book / uses

Pachinko

“I like to recommend a fiction book because life is about more than evals. Recently, I read Pachinko by Min Jin Lee.”

Lenny's Podcast · 25 Sept 2025

Evidence receipt · Source ↗

book / recommends

Pachinko

“I like to recommend a fiction book because life is about more than evals. Recently, I read Pachinko by Min Jin Lee. A really great book. Then, I also am currently reading Apple in China, which the name of the author is slipping my mind, but this is more of an exposition, written by a journalist on how Apple did a lot of manufacturing processes in Asia over the last couple, several decades. Very eye-opening.”

Lenny's Podcast · 25 Sept 2025

Evidence receipt · Source ↗

Claim ledger

What Shreya said.

15 transcript-backed records

01 / recommendation

Recommends Pachinko.

“I like to recommend a fiction book because life is about more than evals. Recently, I read Pachinko by Min Jin Lee. A really great book. Then, I also am currently reading Apple in China, which the name of the author is slipping my mind, but this is more of an exposition, written by a journalist on how Apple did a lot of manufacturing processes in Asia over the last couple, several decades. Very eye-opening.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

03 / belief

I think what's very empowering now is that product managers are doing this and can do this, and can really build very, very profitable products with this skill set.

“I think what's very empowering now is that product managers are doing this and can do this, and can really build very, very profitable products with this skill set.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

04 / belief

I think you can if you want to, but the whole game here is about prioritizing. You have finite resources and finite time, you can't write an eval for everything, so prioritize the ones that are the more pesky areas.

“I think you can if you want to, but the whole game here is about prioritizing. You have finite resources and finite time, you can't write an eval for everything, so prioritize the ones that are the more pesky areas.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

05 / belief

We guarantee you, no matter what you do, if you're doing parts of these process, you're going to find ways of actionable improvement, and then you're going to iterate on your own process from there. The other tip that I would say is, we are very pro-AI.

“We guarantee you, no matter what you do, if you're doing parts of these process, you're going to find ways of actionable improvement, and then you're going to iterate on your own process from there. The other tip that I would say is, we are very pro-AI.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

07 / belief

I think people just get overwhelmed by how much time they spend up front and then thinking that they have to keep doing this all the time.

“I think people just get overwhelmed by how much time they spend up front and then thinking that they have to keep doing this all the time.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

08 / uncertainty

Actually, I only need to do 15." I don't know. Depends on the application and depends on how savvy you are with error analysis for sure.

“Actually, I only need to do 15." I don't know. Depends on the application and depends on how savvy you are with error analysis for sure.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

09 / belief

I think that works. There's two things to that, right? One is they're standing on the shoulders of the evals that their colleagues are doing for coding.

“I think that works. There's two things to that, right? One is they're standing on the shoulders of the evals that their colleagues are doing for coding.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

10 / recommendation

If somebody is coming to you with a way to do something that's entirely new and not grounded in hundreds of years of theory and literature, then you should, I don't know, be a little bit wary of that.

“If somebody is coming to you with a way to do something that's entirely new and not grounded in hundreds of years of theory and literature, then you should, I don't know, be a little bit wary of that.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

11 / evaluation

When we're talking about LLM judge here, we're saying that this is a complex failure mode and we don't know how to evaluate in an automated way.

“When we're talking about LLM judge here, we're saying that this is a complex failure mode and we don't know how to evaluate in an automated way.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

12 / evaluation

I think everyone has a different mindset of evals going in, and the other thing I will say is that people have been burned by evals in the past.

“I think everyone has a different mindset of evals going in, and the other thing I will say is that people have been burned by evals in the past.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

14 / recommendation

Now, maybe you also, because these AI assistants are doing such open-ended tasks, you kind of also want to measure how good are they at very vague or ambiguous things like responding to new types of user requests or figuring out if there's new distributions of data like new users are coming and using your real estate agent that you didn't even know would use your product.

“Now, maybe you also, because these AI assistants are doing such open-ended tasks, you kind of also want to measure how good are they at very vague or ambiguous things like responding to new types of user requests or figuring out if there's new distributions of data like new users are coming and using your real estate agent that you didn't even know would use your product.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast

15 / evaluation

I think the products that are doing this, they have a very sharp sense of how well their application is performing, and people don't talk about it, because this is their moat.

“I think the products that are doing this, they have a very sharp sense of how well their application is performing, and people don't talk about it, because this is their moat.”
Speaker
Shreya Shankar
Publisher
Lenny's Podcast
Search evidence