Speaker unverified: belief

Transcript context

…e to figure out that yourself. You want a model to guide you through the thing. And is the step of creation going to be more and more natural language, is the goal versus with actual writing the book? I think sometimes. I don’t think it’s going to be the case that all of programming will be natural language, and the reason for that is if I’m pair programming with Sualeh and Sualeh is at the computer and the keyboard, and sometimes if I’m driving, I want to say to Sualeh, “Hey, implement this function,” and that works. And then sometimes it’s just so annoying to explain to Sualeh what I want him to do, and so I actually take over the keyboard and I show him. I write part of the example and then it makes sense and that’s the easiest way to communicate. And so I think that’s also the case for AI. Sometimes the easiest way to communicate with the AI will be to show an example and then it goes and does the thing everywhere else. Or sometimes if you’re making a website for example, the easiest way to show to the AI what you want is not to tell it what to do but drag things around or draw things, and maybe eventually we will get to brain machine interfaces or whatever and you can understand what you’re thinking. And so I think natural language will have a place. I think it will definitely not be the way most people program most of the time. I’m really feeling the AGI with this editor. It feels like there’s a lot of machine learning going on underneath. Tell me about some of the ML stuff that makes it all work? Where Cursor really works via this ensemble of custom models that we’ve trained alongside the frontier models that are fantastic at the reasoning intense things. And so Cursor Tab for example, is a great example of where you can specialize this model to be, even better than even frontier models if you look at evals on the task we set it at. The other domain, which it’s surprising that it requires custom models but it’s necessary and works quite well, is in Apply. So I think these models are… The frontier models are quite good at sketching out plans for code and generating rough sketches of the change, but actually, creating diffs is quite hard for frontier models, for your training models. You try to do this with Sonnet, with o1, any frontier model and it really messes up stupid things like counting line numbers, especially in super, super large files. And so what we’ve done to alleviate this is we let the model sketch out this rough code block that indicates what the change will be and we train a model to then Apply that change to the file. And we should say that Apply is the model looks at your code, it gives you a really damn good suggestion of what new things to do. And the seemingly for humans trivial step of combining the two, you’re saying is not so trivial. Contrary to popular perception, it is not a deterministic algorithm. ggestion of what new things to do. And the seemingly for humans trivial step of combining the two, you’re saying is not so trivial. Contrary to popular perception, it is not a deterministic algorithm. Yeah, I think you see shallow copies of apply elsewhere and it just breaks most of the time because you think you can try to do some deterministic matching and then it fails at least 40% of the time and that just results in a terrible product experience. I think in general, this regime of you are going to get smarter and smarter models. So one other thing that Apply lets you do is it lets you use fewer tokens with the most intelligent models. This is both expensive in terms of latency for generating all these tokens and cost. So you can give this very, very rough sketch and then have your model models go and implement it because it’s a much easier task to implement this very, very sketched out code. And I think that this regime will continue where you can use smarter and smarter models to do the planning and then maybe the implementation details can be handled by the less intelligent ones. Perhaps you’ll have maybe o1, maybe it’ll be even more capable models given an even higher level plan that is recursively applied by sauna and then the apply model. Maybe we should talk about how to make it fast if you like. Fast is always an interesting detail. Fast is good. Yeah, how do you make it fast? Yeah, so one big component of making it fast is speculative edits. So speculative edits are a variant of speculative decoding, and maybe it’d be helpful to briefly describe speculative decoding. With speculative decoding, what you do is you can take advantage of the fact that most of the time, and I’ll add the caveat that it would be when you’re memory bound in language model generation, if you process multiple tokens at once, it is faster than generating one token at a time. So this is the same reason why if you look at tokens per second with prompt tokens versus generated tokens, it’s much much faster for prompt tokens. So what we do is instead of using what speculative decoding normally does, which is using a really small model to predict these draft tokens that your larger model will then go in and verify, with code edits, we have a very strong prior of what the existing code will look like and that prior is literally the same exact code. So you can do is you can just feed chunks of the original code back into the model, and then the model will just pretty much agree most of the time that, “Okay, I’m just going to spit this code back out.” And so you can process all of those lines in parallel and you just do this with sufficiently many chunks. And then eventually you’ll reach a point of disagreement where the model will now predict text that is different from the ground truth original code. It’ll generate those tokens and then we will decide after enough tokens match the original code to re- start speculating in chunks of code. text that is different from the ground truth original code. It’ll generate those tokens and then we will decide after enough tokens match the original code to re- start speculating in chunks of code. What this actually ends up looking like is just a much faster version of normal editing code. So it looks like a much faster version of the model rewriting all the code. So we can use the same exact interface that we use for diffs, but it will just stream down a lot faster. And then the advantage is that while it’s streaming, you can just also start reviewing the code before it’s done so there’s no big loading screen. Maybe that is part of the advantage. So the human can start reading before the thing is done. I think the interesting riff here is something like… I feel like speculation is a fairly common idea nowadays. It’s not only in language models. There’s obviously speculation in CPUs and there’s speculation for databases and there’s speculation all over the place. Well, let me ask the ridiculous question of which LLM is better at coding? GPT, Claude, who wins in the context of programming? And I’m sure the answer is much more nuanced because it sounds like every single part of this involves a different model. I think there’s no model that Pareto dominates others, meaning it is better in all categories that we think matter, the categories being speed, ability to edit code, ability to process lots of code, long context, a couple of other things and coding capabilities. The one that I’d say right now is just net best is Sonnet. I think this is a consensus opinion. o1’s really interesting and it’s really good at reasoning. So if you give it really hard programming interview style problems or lead code problems, it can do quite well on them, but it doesn’t feel like it understands your rough intent as well as Sonnet does. If you look at a lot of the other frontier models, one qualm I have is it feels like they’re not necessarily over… I’m not saying they train on benchmarks, but they perform really well in benchmarks relative to everything that’s in the middle. So if you tried on all these benchmarks and things that are in the distribution of the benchmarks they’re evaluated on, they’ll do really well. But when you push them a little bit outside of that, Sonnet is I think the one that does best at maintaining that same capability. You have the same capability in the benchmark as when you try to instruct it to do anything with coding. Another ridiculous question is the difference between the normal programming experience versus what benchmarks represent? Where do benchmarks fall short, do you think, when we’re evaluating these models?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.