Lenny Rachitsky: uncertainty

Transcript context

…g like, fuck you to the man. I think tokens are similar. Yeah, you used a lot of tokens, like good for you, did it produce a pull request that was good? And I think that's the whole point of all this. I think there's a huge difference between outcomes-based pricing and usage based pricing because especially in AI, they're not necessarily even correlated and you could have a long phone call and not solve the customer's problem and they give you a negative review online and call the call center again, all that effort was for nothing. In fact, you might've added negative value. And so, I am a huge believer in this. And what's fun about it is it really just aligns... I think every technology company aspires to be a partner, not a vendor. And I think at Sierra, we are truly a partner to every single one of our customers because we're all aligned on what we want to achieve. And I think that is really where the software industry should go. It requires a lot of different shape of a company. You have to be able to help your customers achieve those outcomes. You can't just throw software at the wall because you'll never get paid if it doesn't. Your orientation becomes so extremely customer-centric when you do this the right way. I think it's just a better version of the software industry. So I think it's right from first principles, it's right for procurement partners and I think it's right for the world. We've been chatting a little bit about productivity gains. There's a lot of skepticism in the headlines these days of just like what is AI actually doing? Is it actually helping people be more productive? There was a recent study actually, I don't know if you saw, where they showed engineers were less productive with AI because it was just putting them in different directions. They had to research all what's going wrong here? So I think CX is a really good example where you clearly are seeing gains. Are you seeing actual gains at your company or any other company you work with outside of CX in terms of productivity that is like clearly yes, this is working and a huge deal? I'm extremely bullish on the productivity gains from AI, but I do think the tools and products right now are somewhat immature and it's quite counterintuitive. So, for example, almost every software engineering firm I know uses something like Cursor to help their software engineers. Most people use Cursor right now as a coding autocomplete, though they have a lot of agentic solutions and there's a lot of... OpenAI has Codex and Cloud has... I can't remember the Anthropic products. So there's lots of agentic agents coming as well. One of the interesting things because the technology is immature, the code it produces often has problems. There's a lot of people approaching this to actually realize those productivity gains because as any engineer who's written a lot of code will tell you, it's pretty easy to look at and edit and fix code you wrote. Reviewing other people's code or particularly finding a subtle logical error in someone else's code is actually really hard. It's actually much harder than editing code that you wrote yourself. So if the code produced by a coding agent is often incorrect, it actually can take a lot of cognitive load and time to fix it. And in fact, if you end up producing lots of issues with your customers, you could end up producing a lot of features, but actually, is like mucking up the machine a little bit and having something that's not ideal. There's a couple of techniques that I think are interesting. First, I think there's a lot of AI starts now working on things like code reviews. I think this idea of self-reflection in agents is really important. Having AI supervise the AI is actually very effective. Just think about it this way, if you produce an AI agent that's right 90% of the time, that's not that great, but how hard would it be to make another AI agent to find the errors the other 10% of the time? That might be a tractable problem. And if that thing's right 90% of the time, just for argument's sake, you can wire those things together and have something that's right 99% of the time. So it's just a math problem and it turns out that you can make something to generate code, you can make something to review code and you're essentially using compute for cognitive capacity and you can layer on more layers of cognition and thinking and reasoning and produce things increasingly robust. So I'm very excited about that. The other thing though is root cause analysis. So we have an engineer at Sierra who exclusively focuses on the model context protocol server serving our cursor instance. And our whole philosophy is, if cursor generated something incorrect, rather than just fixing it, try to root cause it. Try to get it so the next time Cursor will produce the correct code and essentially, is context engineering. What context did Cursor not have that would've been necessary to produce the right outcome?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.