Andrew Lee: belief

Transcript context

…Yeah. Yeah, that makes sense. So when you think about the... best available harness and what that looks like, especially as you go to a multi-provider paradigm. How much do you think you're going to be building a harness per model versus trying to keep everything the same across models? Traditionally, one would think like, no way we can build this, you know, complicated product across you know, in a bespoke way for all these different models. We've got to keep it consistent, but obviously the old rules don't apply anymore. So what's your strategy? Like how much do you tailor the harness to each new model that you want to launch? Yeah, this is actually, this is very much in my mind right now. I think ideally as little as possible because we want to support a lot of models and like it's hard to have a thing that, you know, It's hard to maintain across both balls, but we also want to have the ability for these agents to switch between models. If you're like, Hey, you can run an Opus this way and it persists this state, but then you switch over the same agent to some other model and suddenly you have to find a way to translate things, it gets really complicated. We'd like to keep them as similar as possible. I think so far, we've been able to, and our approach has been, maybe we'll make some prompting tweaks that'll try to address issues in one model while not breaking it in the other model. And I think so far that's mostly worked. I think over time, the APIs of these things have converged and the basic capabilities of these things have converged. So my hope is that we'll get easier over time, not harder, but I could see us having some very model-specific harness things potentially, and then thinking about ways to do that in a really modular way so it's not a huge amount of overhead, but yeah, definitely something on my mind. Even beyond model capabilities, you alluded earlier to caching primitives being different across providers. So presumably on that level, at a minimum, you kind of have no choice but to, if not, I mean, you maybe could have the same context, but you're going to have some sort of different implementation, right, for certain things that are just different, that are inseparable from the models.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.