Lenny Rachitsky: evaluation

Transcript context

…Look, there's so much talk. I think it depends on how much X or news you consume. So I think it's like what sort of our perspective. The general trend right now is going from models knowing things to models doing things. And we're pushing the boundaries of knowledge, like the benchmarks that we put out and that others put out are showing that the knowledge that these models have is getting, it's quite robust. And then, the next question becomes, well, what can it do for me? And as soon as you get into that world, that's where the environments we were talking about start to come into play. How do you navigate a Salesforce instance? How do you navigate a healthcare system? How do you navigate even a weather app on your phone, and how does the agent make decisions for you? We're just getting into the beginning of that. It'll be very interesting to see how quickly that happens. And I think that's where a lot of the speculation has a wide variance because we're at the beginning of it. People take different trajectories on how that's going to improve. And so, if you take a trajectory of the most aggressive trajectory, which is like, oh, it's actually going to be quite easy to train on these things, and then it's just a change management exercise in the economy, which by the way, change management exercises are not to be underestimated. There's still people in the world without an email address. And so, the adoption curve then becomes a human and policy issue, not a technological issue. We're not there from the technology standpoint, but I do think in the next two to three years, if I take the bait and have to make a guess is the technology will get to a point where it will push the change management and policy makers to say like, "Oh, what do we do with this because it's getting pretty close?" That's probably two or three years away. There's been a lot of talk these days about AI not delivering on the promise that we hear, especially at enterprises. There's this MIT study that just showed that there's all these pilots that people are excited about and then they don't work and companies aren't adopting these tools. There's data showing engineers are not actually as productive with tools. It actually slows them down sometimes. You work with a ton of companies implementing all kinds of AI. What are you seeing on the ground? What kind of gains are you seeing? Do you feel like it's overhyped, underhyped? There's a lot of hype out there, and our job is to actually build products that work, that deliver value for our customers and figure out where the rubber hits the road. And to get a sophisticated, my healthcare example is one, we do other sophisticated workflows, claims management for insurance companies. This is a financial decision that's happening, but it's an automatable process. But basically what happens is the POCs get to 60 or 70% of the way there, and the human mind goes, oh, the rest is no big deal. But it's like uptime in data centers where every nine is an order of magnitude investment in terms of reliability, backups, et cetera. One nine is basically a web server in a dorm room like we had at UCLA, and then five nines is this crazy high bar, but it just seems like a very small movement. So you have a similar dynamic going on here where you have a bunch of people, one of the reasons why the POCs have failed, one, there's a denominator effect because it's so easy to do, "Hey, I spun up a project, I spun up a project, I spun up a project." So it's really easy for people to try. So I don't necessarily know that the 95% number, I think is a bit of clickbait in a way. It tells the right story, but it is a little bit hyperbolic because if you take the efforts that happen in the company where they actually get a quality partner like we are, or if you do it yourself, if you have engineers who've worked with models before and they put in the time, and I'm talking about months, not like minutes like you see in these videos to actually get legal approval, policy approval, regulatory approval, change managements like an accuracy that everybody's comfortable with. If you actually do that, these things take 6 to 12 months to get them truly robust enough where an important process can be automated. So I think that's where the hype is right that when you do it, the impact is like, whoa, I never would've figured that out myself, and I'm one of the most educated doctors in the world as an example. But the time to get there is just longer than what people are selling.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.