Lenny Rachitsky: evaluation

Transcript context

…The K through eight school our kid goes through had an AI and education expert come in and I had a very low bar or a very low expectation of what this conversation was going to be like. And actually I think it went over most of the people in the audience's heads because he was like, "All right, well let me take you all the way back to Claude Shannon in information theory." And I could see people's eyes going, "What did I sign up for and why am I hearing this school auditorium hearing about information theory?" But he did a really nice job I think of also just imagining there will be different jobs and we don't know what those jobs are going to be and so what are the skills and techniques and remain open mindedness around what the exact way we recombine those things. And even those will probably change three times between now and when they're 18. So we're talking about timelines and how things are changing. So I've seen these stats that you've shared, other folks at Anthropic have shared about how much of your code is now written by AI. So people have shared stats from 70% to 90%. There was an engineer lead that shared 90% of your code roughly is written by AI now, which first of all is just insane that it went from zero to 90%, I don't know, a few years, something like that. Yeah, basically. I don't think people are talking about this enough. That's just wild. You guys are basically at the bleeding edge. I've never heard a company that has this high a percentage of code being written by AI. So you guys are at the edge of where things are heading. I think most companies will get here. How has product development changed knowing so much of your code is now written by AI, so usually it's like PM, it's like here's what we're building, engineer builds it, it ships it. Is it still kind of roughly that or is it now PMs are just going straight to Claude, build this thing for me, engineers are doing different things? Just what looks different in a world where 90% of your code is written by AI? Yeah, it's really interesting because I think the role of engineering has changed a lot, but the suite of people that come together to produce a product hasn't yet. And I think for the worst in a lot of ways because I think we're still holding on some assumptions. So I think the roles are still fairly similar, although we'll now get in my favorite things that happen now are some nice PMs that have an idea that they want to express or designers that have an idea they want to express will use Claude and maybe even Artifacts to put together an actual functional demo. And that has been very, very helpful. No, no, this is what I mean that makes it tangible. That's probably the biggest role shift is prototyping happening earlier in the process via more of this code plus design piece. What I've learned though is the process of knowing what to ask the AI, how to compose the question, how to even think about structuring a change between the backend and the front end. Those are still very difficult and specialized skills and they still require the engineer to think about it. And we really rapidly became bottlenecked on other things like our merge queue, which is the get in line to get your change accepted by the system that then deploys into production. We had to completely re-architect it because so much more code was being written and so many more pull requests were being submitted that it just completely blew out the expectations of it. And so it's like, I don't know if you've ever read, is it the goal, the classic process optimization book, and you realize there's this critical path theory. I've just found all these new bottlenecks in our system, there's an upstream bottleneck, which is decision making and alignment. A lot of things that I'm thinking about right now is how do I provide the minimum viable strategy to let people feel empowered to go run and type and build and explore at the edge of model capabilities. I don't think I've gotten that right yet, but that's something I'm working on. And then once the building is happening, other bottlenecks emerge, let's make sure we don't step on each other's toes. Let's think through all the edge cases here ahead of time so that we're not blocked on the engineering side. And then when the work is complete and we're getting ready to ship it, what are all those bottlenecks as well? Let's do the air traffic control of landing the change. How do we figure out large strategy? So I think there hasn't been as much pressure on changing those until this year, but I would expect that a year from now the way that we are conceiving of building and shipping software just changes a lot because it's going to be very painful to do it the current way.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.