Flo Crivello: belief

Transcript context

…like, how broad do you think people should be thinking when they are approaching fine tuning? Obviously, the extreme would be one fine tune per task. You can definitely do multitask fine tunes. It's maybe getting into pretty challenging territory to say we wanna make our own general purpose model that's has the same breadth of action space as the the big ones, but it's like ours somehow. How would you guide people on, like, how big to think when they start to approach fine tuning? Most people should not fine tune. I think if you work at, like, at scale, you should you should fine tune. Then the more at scale you are, the more fancy you can be. But I generally one metaheuristic I've developed over the years is, like, you should place an enormous emphasis on simplicity. Enormous emphasis. And and I think it's just too complicated to start to have multiple fine tunes, so different multiple use cases and users and and model routers like you don't want. No. No. No. Just one model. You know, you don't like I said, you don't switch model mid midstream. And if you fine tune, you just fine tune into one model. Sure. Okay. Cool. Think I think one again, this goes to this metaheuristic I mentioned of extreme simplicity. But the thing I find myself going back to all all the time is, like, because I spent so much time thinking about context and memory, Right now, our memory is, like, these millions of tokens stored on on a file system and this memory agent that's really, really sophisticated. It does feel like memory belongs to the weights. This does feel like a little bit of a hack. And so there is something to be said about that. I think, eventually, with infinite resources, what I would like to do is I would like to do a LoRa per user because LoRa is actually pretty cheap to train. So now it's no longer on that being now you use it to be dreaming because you can't retrain the LoRa every fifteen minutes. You would have to do it every day or every week. You know? And and you would and and it's tremendous infrastructure challenge even to, like, so you gotta stool all of those LoRa's, which, like, the storage is not a problem. But, like, the inference time swapping in and out of the LoRa's is a huge pain in the ass. And the training pipeline and and all of that stuff said, good luck doing that. So but it would make sense because I think weights are are so much more compressive than than tokens. Yeah. That's I don't know if you have any thoughts on the great horizon scanning for continual learning, but this is, again, for quite some time, has been the thing that I'm like, boy, if that ever tips with and it could be as simple as one key insight, we could be in a very different world very quickly.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.