High Signal Podcasts Evidence ledger
Method
Browse
← All source episodes

Lenny's Podcast / episode intelligence

AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt)

19 Jun 2025 21 published claims 2 attributable people

Speakers in the public record

Claim mix

recommendation 7belief 5evaluation 4commitment 3prediction 1preference 1

Evidence policy

Every row below preserves an exact excerpt. Identified speakers are linked; unresolved voices are labeled and excluded from people counts.

Claim ledger

The useful parts, with receipts.

21 published records

01 / recommendation

Likes The River of Doubt.

“My favorite book is The River of Doubt, in which Theodore Roosevelt, after losing, I believe, the 1912 campaign, goes to Southern America and traverses a never before traversed river, and along the way gets all of these horrible infections, almost dies. They run out of food. They have to kill their cattle. I think half or more than half of their party died along the way. And it ended up just being this insane journey that really spoke to his mental fortitude. And one of my favorite anecdotes in that book was that he would do these point-to-point walks with people, where he'd look at a map and just kind of put two dots on the map and be like, "Okay, we're here. We're going to walk in a straight line to this other place." And straight line really meant straight line. I'm talking like climbing trees, bouldering, wading through rivers, apparently naked with foreign ambassadors. I feel like politics would be a lot better if our president would do that. It's only stories like those that are just core America to me. And I am actually entirely into bushwhacking and foraging. And if you had a plants podcast, that would be an episode. But I love that story. I love that book. It was entirely fascinating to me.”
Publisher
Lenny's Podcast

02 / recommendation

Likes The River of Doubt.

“My favorite book is The River of Doubt, in which Theodore Roosevelt, after losing, I believe, the 1912 campaign, goes to Southern America and traverses a never before traversed river, and along the way gets all of these horrible infections, almost dies.”
Publisher
Lenny's Podcast

04 / evaluation

There's no effect, no predictable effect of these roles." And so my thinking on this is that at some point with the GPT-3, early ChatGPT models, it might've been true that giving these roles provides a performance boost on accuracy-based tasks, but right now, it doesn't help at all.

“There's no effect, no predictable effect of these roles." And so my thinking on this is that at some point with the GPT-3, early ChatGPT models, it might've been true that giving these roles provides a performance boost on accuracy-based tasks, but right now, it doesn't help at all.”
Publisher
Lenny's Podcast

06 / belief

I think to a lot of people, they think of prompt engineering as just getting better at when you use Claude or ChatGPT, but there's actually more.

“I think to a lot of people, they think of prompt engineering as just getting better at when you use Claude or ChatGPT, but there's actually more.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

07 / commitment

There are cases that I will bring in those other techniques, but the most important places to use those techniques is the product-focused prompt engineering.

“There are cases that I will bring in those other techniques, but the most important places to use those techniques is the product-focused prompt engineering.”
Publisher
Lenny's Podcast

08 / belief

One that comes to mind, I think recently Anthropic released an example of where they were trying to shut it down and the LLM was attempting to blackmail one of the engineers into not shutting it down.

“One that comes to mind, I think recently Anthropic released an example of where they were trying to shut it down and the LLM was attempting to blackmail one of the engineers into not shutting it down.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

10 / belief

I actually was at the AI Engineer World's Fair yesterday, and there was somebody, I think before me, giving a talk that prompt engineering is dead.

“I actually was at the AI Engineer World's Fair yesterday, and there was somebody, I think before me, giving a talk that prompt engineering is dead.”
Publisher
Lenny's Podcast

12 / prediction

I also found this a little bit difficult to believe because that's not really how AI is supposed to work, but I don't know, we see all sorts of weird things from it.

“I also found this a little bit difficult to believe because that's not really how AI is supposed to work, but I don't know, we see all sorts of weird things from it.”
Publisher
Lenny's Podcast

15 / evaluation

I'll come back and again and again to things like math questions because a lot of these techniques are judged based off of data sets of math or reasoning questions simply because you're going to evaluate the accuracy programmatically as opposed to something like generating interview questions, which is no less valuable, but just very difficult to evaluate success for in an automated way.

“I'll come back and again and again to things like math questions because a lot of these techniques are judged based off of data sets of math or reasoning questions simply because you're going to evaluate the accuracy programmatically as opposed to something like generating interview questions, which is no less valuable, but just very difficult to evaluate success for in an automated way.”
Publisher
Lenny's Podcast

16 / recommendation

One, it can get cached. So subsequent calls to the LM with that same context at the top of the prompt are cheaper because the model provider stores that initial context for you as well as the embeddings for it.

“One, it can get cached. So subsequent calls to the LM with that same context at the top of the prompt are cheaper because the model provider stores that initial context for you as well as the embeddings for it.”
Publisher
Lenny's Podcast

17 / preference

Talk about maybe, as we go through them, which one this is most helpful for because it's not just like, "Oh, cool, I'm just going to get a better answer from ChatGPT.

“Talk about maybe, as we go through them, which one this is most helpful for because it's not just like, "Oh, cool, I'm just going to get a better answer from ChatGPT.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

18 / commitment

I'm very excited because I think I'm going to learn a ton in this conversation. What I want to do with this chat is essentially give people very tangible and also just very up-to-date prompt engineering techniques that they can start putting into practice immediately.

“I'm very excited because I think I'm going to learn a ton in this conversation. What I want to do with this chat is essentially give people very tangible and also just very up-to-date prompt engineering techniques that they can start putting into practice immediately.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

19 / recommendation

It's a more research-y approach. But just take some common format out there that the LLM is comfortable with, and I say that with air quotes because it's a bit of a strange thing to say the LLM is comfortable with something, but it actually comes empirically from studies that have shown that formats of questions that show up most commonly in the training data are the best formats of questions to actually use when you're prompting it.

“It's a more research-y approach. But just take some common format out there that the LLM is comfortable with, and I say that with air quotes because it's a bit of a strange thing to say the LLM is comfortable with something, but it actually comes empirically from studies that have shown that formats of questions that show up most commonly in the training data are the best formats of questions to actually use when you're prompting it.”
Publisher
Lenny's Podcast

21 / recommendation

Mentions personal use of Claude. Mentions personal use of ChatGPT.

“As an example, I use Claude and ChatGPT for coming up with title suggestions for these podcast episodes. And I give it examples of just examples of titles that have done well, and then it's 10 different examples, just bullet points.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast
Search evidence