Andrew Lee: prediction

Transcript context

…So, okay, let's do this harness thing for a second. The word harness itself just always makes me think of trying to control and direct some sort of wild animal to get useful work out of it when it might rather be doing something else. I just took a long road trip with my kids in a Tesla FSD-enabled car over the last 10 days. We went to a lot of historical sites. The juxtaposition of horses and my FSD was pretty funny. But I kind of think of like, you know, trying to get this... you know, unruly animal that is a model to stay on track, right? And do what you want it to do. These days, as you said, like there's a lot more ability to give the model hints and say like, here's a file system and you can kind of go get what you need here. And I'm starting to feel like the concept of a harness is maybe a little anachronistic already. And maybe what we're doing now is more saying like, Here's the world you get to play in. It's not so much about trying to narrow what the model can do, but more about broadening what it can do. How do you think about that sort of like narrowing and focusing versus kind of broadening, giving access, you know, unlocking new possibilities, which, you know, might in some cases even surprise users given the model capabilities we have now? I hadn't actually thought of a harness as being a constraining thing, but yeah, you kind of make a good point of that would be the normal way you'd think about that word. I kind of think of it as a mecha suit. I agree with your thesis. The goal is to let that agent or let that LLM actually do things in the world. To do that, it's going to need storage, it needs compute, it needs to be able to reach out and connect to APIs, it needs to be able to talk to the user. There's a lot there. I think when I talk to people who are not deep into the harness world, I think most people assume when they play with an LLM product that it's a very raw thing on top of the model and the type of thing and they send it. What they see on the screen is just being sent to the model and the model's doing everything. That is becoming increasingly less true. The complexity of the code that's translating what you see for the LLM calls is getting more and more. I think that's going to keep going. sophistication of these harnesses is going to get just 10 times as complex. But I think there's going to be some pretty major breakthroughs here that increase the capabilities of these things pretty substantially in the way that we handle memories and the way that we handle oversight and control and the way we connect to other tools. So I'm very bullish on the opportunity here. And I think these things are just going to get more and more complicated. And yeah, I don't know, maybe we need a new name. Maybe it's like a mecha suit and it's not like a Like a harness. Yeah. How much do you think? So this is, I think, one of the more interesting debates right now in the AI builder community broadly. What matters more, model or harness? And I think you see pretty extreme positions on both ends where I see I get emails that are like, models don't matter anymore. It's all about the harness and vice versa. And Obviously, either of those like extreme positions is not going to be right. But I guess I have historically come down somewhat informed. I don't know if you've seen this graph from the UK AI security, I should say, Institute, where they do a capability plot over time with the minimalist harness, you know, whatever kind of basic vanilla thing, and then the best available harness. And of course, you know, both are going up. A year ago, though, the time delta between what level of capability you could get with the best available harness versus the vanilla harness was longer. And now it's gotten shorter. Some of that is maybe just due to more frequent model releases, which is like shortening every, you know, window of advantage. Some of it is maybe because the models are getting more deeply trained to use harnesses, and so You know, they're just good at it out-of-the-box. You don't have to compensate for their weaknesses so much. But I guess my overall summary would be, it seems like... I would say models seem to matter more and you can't get that... How much can I live in the future with the best available harness for any given model? It seems like it's not a huge amount, but it sounds like you maybe see that differently. So what's the case that that's... If you do, what's the case that that's wrong?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.