Speaker unverified: belief

Transcript context

…hat hidden chain of thought to replicate the technology, because pretty important data, like seeing the steps that the model took to get to the final results. So you could probably train on that also? And there was a mirror situation with this, with some of the large language model providers, and also this is speculation, but some of these APIs used to offer easy access to log probabilities for all the tokens that they’re generating and also log probabilities over the prompt tokens. And then some of these APIs took those away. Again, complete speculation, but one of the thoughts is that the reason those were taken away is if you have access to log probabilities similar to this hidden chain of thought, that can give you even more information to try and distill these capabilities out of the APIs, out of these biggest models and to models you control. As an asterisk on also the previous discussion about us integrating o1, I think that we’re still learning how to use this model. So we made o1 available in Cursor because when we got the model, we were really interested in trying it out. I think a lot of programmers are going to be interested in trying it out. o1 is not part of the default Cursor experience in any way up, and we still haven’t found a way to yet integrate it into the editor in a way that we reach for every hour, maybe even every day. So I think that the jury’s still out on how to use the model, and we haven’t seen examples yet of people releasing things where it seems really clear like, oh, that’s now the use case. The obvious one to turn to is maybe this can make it easier for you to have these background things running, to have these models and loops, to have these models be agentic. But we’re still discovering, To be clear, we have ideas. We just need to try and get something incredibly useful before we put it out there. But it has these significant limitations. Even barring capabilities, it does not stream. That means it’s really, really painful to use for things where you want to supervise the output. Instead, you’re just waiting for the wall text to show up. Also, it does feel like the early innings of test time, compute and search where it’s just a very, very much a v0, and there’s so many things that don’t feel quite right. I suspect in parallel to people increasing the amount of pre-training data and the size of the models and pre-training and finding tricks there, you’ll now have this other thread of getting search to work better and better. So let me ask you about strawberry tomorrow eyes. So it looks like GitHub Copilot might be integrating o1 in some kind of way, and I think some of the comments are saying, does this mean Cursor is done? I think I saw one comment saying that. It’s a time to shut down Cursor. Yeah. Time to shut down Cursor. [inaudible 01:58:38]. Thank you. So is it time to shut down Cursor? es this mean Cursor is done? I think I saw one comment saying that. It’s a time to shut down Cursor. Yeah. Time to shut down Cursor. [inaudible 01:58:38]. Thank you. So is it time to shut down Cursor? I think this space is a little bit different from past software spaces over the 2010s, where I think that the ceiling here is really, really, really incredibly high. So I think that the best product in three to four years will just be soon much more useful than the best product today. You can wax poetic about moats this and brand that and this is our advantage, but I think in the end, just if you stop innovating on the product, you will lose. That’s also great for startups, that’s great for people trying to enter this market because it means you have an opportunity to win against people who have lots of users already by just building something better. So I think over the next few years, it’s just about building the best product, building the best system. That both comes down to the modeling engine side of things, and it also comes down to the editing experience. Yeah, I think most of the additional value from Cursor versus everything else out there is not just integrating the new model fast like o1. It comes from all of the depth that goes into these custom models that you don’t realize are working for you in every facet of the product, as well as the really thoughtful UX with every single feature. All right. From that profound answer- All right, from that profound answer, let’s descend back down to the technical. You mentioned you have a taxonomy of synthetic data. Oh yeah. Can you please explain? Yeah, I think there are three main kinds of synthetic data. So what is synthetic data, first? So there’s normal data, like non-synthetic data, which is just data that’s naturally created, i.e. usually it’ll be from humans having done things. So from some human process you get this data. Synthetic data, the first one would be distillation. So having a language model, output tokens or probability distributions over tokens, and then you can train some less capable model on this. This approach is not going to get you a more capable model than the original one that has produced the tokens, but it’s really useful for if there’s some capability you want to elicit from some really expensive high-latency model. You can then distill that down into some smaller task-specific model. The second kind is when one direction of the problem is easier than the reverse. So a great example of this is bug detection, like we mentioned earlier, where it’s a lot easier to introduce reasonable-looking bugs than it is to actually detect them. And this is probably the case for humans too. And so what you can do, is you can get a model that’s not trained in that much data, that’s not that smart, to introduce a bunch of bugs and code. And then you can use that to then train… Use the synthetic data to train a model that can be really good at detecting bugs. hat much data, that’s not that smart, to introduce a bunch of bugs and code. And then you can use that to then train… Use the synthetic data to train a model that can be really good at detecting bugs. The last category I think is, I guess the main one that it feels like the big labs are doing for synthetic data, which is producing text with language models that can then be verified easily. So extreme example of this is if you have a verification system that can detect if language is Shakespeare level, and then you have a bunch of monkeys typing and typewriters. You can eventually get enough training data to train a Shakespeare-level language model. And I mean this is very much the case for math where verification is actually really, really easy for formal languages. And then what you can do, is you can have an okay model, generate a ton of rollouts, and then choose the ones that you know have actually proved the ground truth theorems, and train that further. There’s similar things you can do for code with lead code like problems, where if you have some set of tests that you know correspond to if something passes these tests, it actually solved problem. You could do the same thing where you verify that it’s passed the test and then train the model in the outputs that have passed the tests. I think it’s going to be a little tricky getting this to work in all domains, or just in general. Having the perfect verifier feels really, really hard to do with just open-ended miscellaneous tasks. You give the model or more long horizon tasks, even in coding. That’s because you’re not as optimistic as Arvid. But yeah, so yeah, that third category requires having a verifier. Verification, it feels like it’s best when you know for a fact that it’s correct. And then it wouldn’t be like using a language model to verify. It would be using tests or formal systems. Or running the thing too. Doing the human form of verification, where you just do manual quality control. Yeah. But the language model version of that, where it’s running the thing and it actually understands the output. Yeah. No, that’s- I’m sure it’s somewhere in between. Yeah. I think that’s the category that is most likely to result in massive gains. What about RL with feedback side RLHF versus RLAIF? What’s the role of that in getting better performance on the models? Yeah. So RLHF is when the reward model you use is trained from some labels you’ve collected from humans giving feedback. I think this works if you have the ability to get a ton of human feedback for this kind of task that you care about. RLAIF is interesting because you’re depending on… This is actually, it’s depending on the constraint that verification is actually a decent bit easier than generation. Because it feels like, okay, what are you doing? Are you using this language model to look at the language model outputs and then prove the language model? But no, it actually may work if the language model has a much easier time verifying some solution than it does generating it. Then you actually could perhaps get this kind of recursive loop. But I don’t think it’s going to look exactly like that.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.