book / uses
Pachinko
“I like to recommend a fiction book because life is about more than evals. Recently, I read Pachinko by Min Jin Lee.”
Public evidence record
Shreya Shankar
Published podcast speaker
- Claims
- 15
- Episodes
- 1
- Shows
- 1
- Named items
- 2
Books, apps, and tools
The evidenced stack.
book / recommends
Pachinko
“I like to recommend a fiction book because life is about more than evals. Recently, I read Pachinko by Min Jin Lee. A really great book. Then, I also am currently reading Apple in China, which the name of the author is slipping my mind, but this is more of an exposition, written by a journalist on how Apple did a lot of manufacturing processes in Asia over the last couple, several decades. Very eye-opening.”
Claim ledger
What Shreya said.
4 transcript-backed records
01 / evaluation
When we're talking about LLM judge here, we're saying that this is a complex failure mode and we don't know how to evaluate in an automated way.
“When we're talking about LLM judge here, we're saying that this is a complex failure mode and we don't know how to evaluate in an automated way.”
- Speaker
- Shreya Shankar
- Publisher
- Lenny's Podcast
02 / evaluation
I think everyone has a different mindset of evals going in, and the other thing I will say is that people have been burned by evals in the past.
“I think everyone has a different mindset of evals going in, and the other thing I will say is that people have been burned by evals in the past.”
- Speaker
- Shreya Shankar
- Publisher
- Lenny's Podcast
03 / evaluation
I think a lot of people prematurely do A-B tests, because they've never done any error analysis in the first place.
“I think a lot of people prematurely do A-B tests, because they've never done any error analysis in the first place.”
- Speaker
- Shreya Shankar
- Publisher
- Lenny's Podcast
04 / evaluation
I think the products that are doing this, they have a very sharp sense of how well their application is performing, and people don't talk about it, because this is their moat.
“I think the products that are doing this, they have a very sharp sense of how well their application is performing, and people don't talk about it, because this is their moat.”
- Speaker
- Shreya Shankar
- Publisher
- Lenny's Podcast