Andrew Lee: belief

Transcript context

…How much do you think? So this is, I think, one of the more interesting debates right now in the AI builder community broadly. What matters more, model or harness? And I think you see pretty extreme positions on both ends where I see I get emails that are like, models don't matter anymore. It's all about the harness and vice versa. And Obviously, either of those like extreme positions is not going to be right. But I guess I have historically come down somewhat informed. I don't know if you've seen this graph from the UK AI security, I should say, Institute, where they do a capability plot over time with the minimalist harness, you know, whatever kind of basic vanilla thing, and then the best available harness. And of course, you know, both are going up. A year ago, though, the time delta between what level of capability you could get with the best available harness versus the vanilla harness was longer. And now it's gotten shorter. Some of that is maybe just due to more frequent model releases, which is like shortening every, you know, window of advantage. Some of it is maybe because the models are getting more deeply trained to use harnesses, and so You know, they're just good at it out-of-the-box. You don't have to compensate for their weaknesses so much. But I guess my overall summary would be, it seems like... I would say models seem to matter more and you can't get that... How much can I live in the future with the best available harness for any given model? It seems like it's not a huge amount, but it sounds like you maybe see that differently. So what's the case that that's... If you do, what's the case that that's wrong? I think... As models get better, they can replace good harnesses. A model today with a crappy harness is going to be better than a model from a year ago with a really good harness. I agree with that, and I think that trend is going to continue. But I also think the effects are multiplicative, and they're orthogonal disciplines, and there's no reason not to take the best model and put in the best harness, and I think we should. You might argue that like, oh, like given like the exponential that actually only buys us six months or something. And okay, fine, but it's six months. But I think more importantly, what you're getting, like the metric, the only metric that matter is not intelligence, right? Like in these real production systems, intelligence is one piece, but, you know, take TaskIt, for example. Much of what we do is automating specific workflows. Once the model plus Harness is smart enough to like, I don't know, order less lunch every day, which it does. And we've been able to do that for like six months. We're not going in there and messing with it very much. Incremental improvements to intelligence don't really matter, but performance and cost do. And so if you look at the harness and say, Hey, the only point is to make the thing smarter, fine, it buys you a fixed amount of time over the model exponential, which is maybe cool, but not amazing. But it might make a significant difference in cost and you know, other attributes, cost and reliability, and the ability to like do oversight and and and speed. And I think those things matter a ton for a commercial product. So like in our case, with our harness, like the benefits you get are, you have a nice UI, and the sidebar pops out at the right time to show you things, and you get nice indications of working states. You can see what it's doing at the time, and you get the ability to have things persisted across long periods of time, and you get nice performance trade-offs and cost trade-offs. I think those things should not be underestimated for a commercial product. Yeah, if you can make it work with Haiku instead of Opus, for example, that moves the needle quite a bit. For sure. Especially in a compute scarce world, which we increasingly seem to be in.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.