Mike Krieger: evaluation

Transcript context

…I love that you still have such a deep product founder sense there as you're building products for this very large company now. On the flip side of this, people working with your models and API, so I imagine there's some companies that are finding ways to leverage your models and APIs to their max and are really good at maximizing the power of what you guys have built. And there's some companies that work with your APIs and models that haven't figured that out. What are those companies that are doing a really good job building on your stuff, doing differently that you think other companies should be thinking about? I think being willing to build more at the edge of the capabilities and basically break the model and then be surprised by the next model. I love that you cited the companies were like 3.5 was the one that finally made them possible. Those companies were trying it beforehand and then hitting a wall and being like, oh, the models are almost good enough or they're okay for this specific use case, but they're not generally usable and nobody's going to adopt them universally, but maybe these real power users are going to try it out. Those are the companies that I think continuously are the ones where I'm like, "Yep, they get it. They're really pushing forward. " We ran a much broader early access program with these models than we had in the past, and part of that was because there's this real, we can hill climb on these evaluations and talk about suite bench and towel bench and terminal bench, whatever, but customers ultimately know Cursor bench which doesn't exist other than in their usage and their own testing et cetera is the thing that we ultimately need to serve. Not just Cursor but Manus bench, right? If Manus is using our models and Harvey bench, those things and customers know way better than anybody. And so I would say there's two things. One is pushing the frontier of the models and then having a repeatable process. This actually goes back to our summit conversation repeatable way to evaluate how well your product is serving those use cases and how well if you drop a new model in, is it doing it better or worse? Some of it can be classic A/B testing, that's fine. Some of it may be internal evaluation, some of it may be capturing traces and being able to rerun them on with a new model. Some of it's vibes like we're still pretty early in this process and some of it is actually trying it and being one of my favorite early access quotes was the founder heard this engineer screaming next to him. He was like, "What? This model? I've never seen this before." This is like Opus 4. It was like, "Cool." We're going to engender that feeling and things, but you're not going to be able to feel that unless you have a really hard problem that you're asking the model repeatedly. So those are the things that I think kind of differentiate those companies that are maybe earlier in their journey of adoption versus the later ones. I can't help but ask about MCP, I feel like that's just so hot and just like Microsoft had their announcement recently where they're like, "That's part of the OS Window." Just what role do you think MCP was will play in the future of product going forward of AI?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.