Boris Cherny: evaluation

Transcript context

…Have you noticed that it is getting much better at this? Because this is the holy grail. Right now it's, cool, building solved. Code review became the next bottleneck with all these PRs. Who is going to review them all? The next big open question is just, like, "Okay. Now humans are necessary for figuring out what to build, what to prioritize," and you're saying that's where Claude Code is starting to help you. Has it gotten a lot better with, like, Opus 4.6, or what's been the trajectory there? Yeah. Yeah. It's improved a lot. I think some of it is training that we do specific to coding. So, obviously, the best coding model in the world, and it's getting better and better. Like, 4.6 is just incredible. But also actually a lot of the training that we do outside of coding translates pretty well too. So, there is this transfer where you teach the model to do X, and it gets better at Y. Yeah. And the gains have just been insane. Like, at Anthropic, over the last year, like, since we introduced Claude Code, we probably ... I don't know the exact number. We probably 4X-ed the engineering team, or something like this. But productivity per engineer has increased 200% in terms of pull requests. And this number is just crazy for anyone that actually works in the space and works on dev productivity. Because back in a previous life, I was at Meta, and one of my responsibilities was code quality for the company. So, this is all of our code bases, that was my responsibility. Like, Facebook, Instagram, WhatsApp, all this stuff. And a lot of that was about productivity, because if you make the code higher quality, then engineers are more productive. And things that we saw is in a year with hundreds of engineers working on it, you would see a gain of a few percentage points of productivity, something like this. And so, nowadays, seeing these gains of just hundreds of percentage points is just absolutely insane. What's also insane is just how normalized this has all been. Like, we hear these numbers. Like, of course, AI is doing this to us. It's so unprecedented, the amount of change that is happening to software development, to building products, to just the world of tech. It's just so easy to get used to it, but it's important to recognize this is crazy.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.