Lenny's Podcast / episode intelligence
Speakers in the public record
Claim mix
Evidence policy
Every row below preserves an exact excerpt. Identified speakers are linked; unresolved voices are labeled and excluded from people counts.
Claim ledger
17 published records
01 / evaluation
“The hope the model just does a good enough job and not being tricked is fundamentally insufficient. And the only reason there hasn't been a massive attack yet is how early the adoption is, not because it's secured.”
02 / evaluation
“The way he put it, the only reason there hasn't been a massive attack yet is how early the adoption is, not because it's secured.”
03 / belief
“I think I have slides on this from probably two years ago and it's straightforward enough.”
04 / uncertainty
“I don't know if that's what killed them, but they don't seem to be in business anymore.”
05 / belief
“I think that's a really interesting point, even though it could... It's not great if you help support agents like Hitler is great, but your point is that that sucks.”
06 / belief
“How these get found, how do they get implemented at companies. And I think the easiest way of thinking about it is like, I'm a CSO at some company we are a large enterprise.”
07 / belief
“There are theoretically ways to train the eyes to be smarter, to be more adversarially robust, and we haven't really seen this yet, but there's this idea that if you start doing adversarial training in pre-training earlier in the training stack, so when the AI is a very, very small baby, you're being adversarial towards it and training it then, then it's more robust, but I think we haven't seen the resources really deployed to do that.”
08 / evaluation
“You'd never actually say this in practice because it's very difficult to estimate adversarial robustness because the search space here is massive, which we'll talk about soon.”
09 / evaluation
“CAMEL can't really help because it's like, "Okay, I'm going to give you read email permissions and also send email permissions," and now this is enough for an attack to occur.”
10 / prediction
“It's kind of funny, because AI researchers are the only people who can solve this stuff long-term, but cybersecurity professionals are, they're the only ones who can kind of solve it short term, largely in making sure we deploy properly permission systems and nothing that could possibly do something very, very bad.”
11 / evaluation
“I think another thing worth pointing out is looking at anthropic constitutional classifiers and other models, it does seem to be more difficult to elicit CBRN and other really harmful outputs from chatbots, but solving indirect prompt injection, which is basically prompt injection against agents done by external people on the internet is still very, very, very unsolved, and it's much more difficult to solve this problem than it is to stop CBRN elicitation, because with that kind of information, as one of my advisors just noted, it's easier to tell the model, "Never do this," than with emails and stuff, "Sometimes do this.”
12 / evaluation
“I found some major problems with the AI security industry. AI guardrails do not work.”
13 / evaluation
“Very interesting industry. And I'll quickly differentiate and separate out the Frontier Labs from the AI security industry because there's the Frontier Labs and some Frontier adjacent companies that are largely focused on research like pretty hardcore AI research.”
14 / evaluation
“Humans are adaptive attackers because they test stuff out and they see what works and they're like, "Okay, this prompt doesn't work, but this prompt does.”
15 / evaluation
“I think you gave the perfect example with ServiceNow, and that's the reason that this stuff is so important to talk about right now because with chatbots, as you said, very limited damage outcomes that could occur, assuming they don't invent a new bioweapon or something like that.”
16 / prediction
“When it comes to AI security, the AI security industry in particular, I think we're going to see a market correction in the next year, maybe in the next six months, where companies realize that these guardrails don't work.”
17 / evaluation
“If all you're doing is deploying chatbots that answer FAQs, help users to find stuff in your website, answer their questions with respect to some documents. It's not really an issue because your only concern there is a malicious user comes and, I don't know, maybe uses your chatbot to output hate speech or C-burn or say something bad, but they could go to ChatGPT or Claude or Gemini and do the exact same thing.”