Tim Scarfe: prediction

Transcript context

…I tried to get an answer, I think, from Norm Brown at some point on Twitter about this because you see how better pre trained model you start with, and then they start reinforcement learning on it, and it seems like it gets up to some Like the reinforcement learning works better and better depending on how good the model was to begin with, in a sense. Yeah, it's super interesting. I mean, I agree with you. I I think it's wrong to say that LLMs are not intelligent. Right? Because there is some kind of a combinational closure, and that means from the primitives in an LLM, can do hill climbing, and we can build some computational structure to solve problems. But that's not quite it, and it can happen at different levels of abstraction. So if the LLMs have higher abstractions, then they can, you know, traverse the combinational closure of the higher abstractions, they can solve the problem. What seems to happen though is, I mean, there's no continual learning, so those don't get, you know, added to a library and reused later. But there's also no abstraction. So what humans would do is they would they would look at this computational graph, and they would say, ah, I see that that's just, an analog of this thing over here, and I and I'm now going to compress it into a new variable that screens off all of that complexity. Why do language models not do that? I think I think you're getting into like continual learning type thing which is is obviously a big thing we lack. I think, I mean, I do think language model do that during pre training. I do think like that's why for example, Methus is so good at at doing stuff in our code basis is because it has somehow seen a lot of code and it just seems to have this sort of intuition for bugs and problems and so on. And that must be kind of like that. It's seen the patents before, it has some kind of abstract. But it's true like when you actually are doing sort of running them more live, they don't do it very well. Like they don't, and I think a lot of people these days are thinking about this, how can we do better continual learning so it can just build all of these abstractions and these things really well on the fly. I mean I know some companies are actively against trying to do that also. Why? Like I think Anthropic, right? I think Dario said like he considers it like a big safety issue too because you're going to lose, you could easily lose all of the work that has gone into alignment if the agent is learning too much on the fly. Then it gets further and further away from the sort of safe checkpoint. But I think there are enough other people working on it that it will probably happen. But it's definitely like a big unknown, right? And I think it will also have to change a lot of things in how we serve models, for example, because suddenly you need to be able, like if we mean by continual learning that we actually update the weights live, it's going to, it causes a lot of problem for the current paradigm because then suddenly you can't use the same weights for all customers too. I mean, guess thinking machines has this thing where they have 1 shared model and then like some loras on top. So maybe something like that can work. But it's still like you have to somehow, yeah, keep doing this all the time. So we are also working in normal computing on like alternative forms of computation. Like these thermodynamic chips or just sort of unconventional computing. It actually becomes sometimes hard to keep the memories in these analog resistors unless you keep learning at the same time as you're doing inference. Like you have to use these fancy substrate like memory stickers or something if they have to be permanent. But if you're just using like sort of basic capacitors like you have in DRAM or something that require constant refreshing. So either you just have to spend a lot of energy on that or you just want to keep the learning going forever. That way they kind of automatically refresh. And that's probably more similar to also how brains work, right? Like they don't, they never, they never like just don't the same. Or Chumsky or somebody would say about it. If it ever just freezes. I think like the the synapses and stuff, they're always adapting.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.