Boris Cherny: evaluation

Transcript context

…These numbers are just totally crazy. Right? Like, 4% of all commits in the world is just way more than I imagined. And, like you said, it still feels like the starting point. These are also just public commits. So, we actually think if you look at private repositories it's quite a bit higher than that. And I think the crazy thing for me isn't even the number that we're at right now, but the pace at which we're growing, because if you look at Claude Code's growth rate across any metric it's continuing to accelerate. So, it's not just going up, it's going up faster and faster. When I first started Claude Code, it was just supposed to be a little hack. We broadly knew at Anthropic that we wanted to ship some kind of coding product. And for Anthropic for a long time, we were building the models in this way that fit our mental model of the way that we build safe AGI where the model starts by being really good at coding. Then it gets really good at tool use. Then it gets really good at computer use. Roughly, this is, like, the trajectory. And we've been working on this for a long time. And when you look at the team that I started on, it was called Anthropic Labs team, and actually Mike Krieger and Ben Mann they just kicked this team off again for round two. The team built some pretty cool stuff. So, we built Claude Code, we built MCP, we built the desktop app. So, you can see the seeds of this idea. It's coding, then it's tool use, then it's computer use. And the reason this matters for Anthropic is because of safety. It's, again, just back to that AI is getting more and more powerful, it's getting more and more capable. The thing that's happened in the last year is that, at least, for engineers, the AI doesn't just write the code. It's not just a conversation partner, but it actually uses tools. It acts in the world. And I think now with Cowork we're starting to see the transition for non-technical folks also. For a lot of people that use conversational AI, this might be the first time that they're using the thing that actually acts, it can actually use your Gmail, it can use your Slack. It can do all these things for you, and it's quite good at it. And it's only going to get better from here. So, I think for Anthropic for a long time, there was this feeling that we wanted to build something, but it wasn't obvious what. And so, when I joined Ant, I spent one month hacking, and built a bunch of weird prototypes. Most of them didn't ship, and weren't even close to shipping. It was just understanding the boundaries of what the model can do. Then I spent a month doing post-training. So, to understand the research side of it. And I think, honestly, that's just, for me, as an engineer, I find that to do good work you really have to understand the layer under the layer at which you work. o understand the research side of it. And I think, honestly, that's just, for me, as an engineer, I find that to do good work you really have to understand the layer under the layer at which you work. And with traditional engineering work, if you're working on product, you want to understand the infrastructure, the run time, the virtual machine, the language, whatever that is, the system that you're building on. But, yeah. If you work in AI, you just really have to understand the model to some degree to do good work. So, I took a little detour to do that, and then I came back and just started prototyping what eventually became Claude Code. In the very first version of it I have a ... There's, like, a video recording of this somewhere, because I recorded this demo, and I posted it. It was called Claude CLI back then. And I just showed off how it used a few tools, and the shocking thing for me was that I gave it a batch tool, and it just was able to use that to write code, to tell me what music I'm listening to when I asked it like, "What music am I listening to?" And this is the craziest thing. Right? Because it's, like, there's no ... I didn't instruct the model to say, "Use this tool for this," or do whatever. The model was given this tool, and it figured out how to use it to answer this question that I had that I wasn't even sure if it could answer, "What music am I listening to?" And so, I started prototyping this a little bit more. I made a post about it, and I announced it internally and it got two likes. That was the extent of the reaction at the time, because I think people internally ... When you think of coding tools, you think of IDEs, you think of all these pretty sophisticated environments. No one thought that this thing could be terminal-based. That's a weird way to design it, and that wasn't really the intention. But from the start I built it in a terminal, because for the first couple months it was just me. So, it was just the easiest way to build. And, for me, this was actually a pretty important product lesson. Right? This is, like, you want to under-resource things a little bit at the start. Then we started thinking about what other form factors we should build, and we actually decided to stick with the terminal for a while. And the biggest reason was the model is improving so quickly, we felt that there wasn't really another form factor that could keep up with it. And, honestly, this was just me struggling with, "What should we build?" For the last year, Claude Code has just been all I think about. And so, just late at night this is just something I was thinking about like, "Okay. The model is continuing to improve. What do we do? How can we possibly keep up?" And the terminal was, honestly, just the only idea that I had. And, yeah. It ended up catching on. After I released it, pretty quickly it became a hit at Anthropic, and the daily active users just went vertical, and it ... Really early on actually, before I launched it, Ben Mann nudged me to make a DAU chart. And I was like, "It's early. Should we really do it right now?" And he was like, "Yeah." And so, the chart just went vertical pretty immediately. I launched it, Ben Mann nudged me to make a DAU chart. And I was like, "It's early. Should we really do it right now?" And he was like, "Yeah." And so, the chart just went vertical pretty immediately. And then in February, we released it externally. Actually, something that people don't really remember is Claude Code was not initially a hit when we released it. It got a bunch of users. There was a lot of early adopters that got it immediately, but it actually took many months for everyone to really understand what this thing is. Again, it's just so different. And when I think about it, part of the reason Claude Code works is this idea of latent demand where we bring the tool to where people are, and it makes the existing workflows a little bit easier. But also because it's in the terminal, it's a little surprising, it's a little alien in this way. So, you have to be open-minded, and you had to learn to use it. And, of course, now Claude Code is available in the iOS and Android Claude app. It's available in the desktop app. It's available on the website. It's available as IDE extensions in Slack and GitHub. All of these places where engineers are it's a little more familiar, but that wasn't the starting point. So, yeah. At the beginning it was a surprise that this thing was even useful. And as the team grew, as the product grew, as it started to become more and more useful to people, just people around the world from small startups to the biggest [inaudible 00:13:05] companies started using it, and they started giving feedback. And I think just reflecting back it's been such a humbling experience, because we keep learning from our users and just the most exciting thing is none of us really know what we're doing, and we're just trying to figure out along with everyone else. And the single best signal for that is just feedback from users. So, that's just been the best. I've been surprised so many times.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.