Tim Scarfe: evaluation

Transcript context

…robot a model of the world that is like our model of the world. A model that is object centered. It's dynamic. It's it's largely causal. Right? It's you know, that's that's that's the big difference. And I think that the the the sort of sparse structured models is another sort of key differentiating component. Like when you think about how like a transformer and LLM work, a transformer takes every word in the document and says, now how does this word relate to every other word? It does it many many many many times. Right? It's a it it, you know, it's it's very much word. It's the same thing with like your your your generative vision language action models. They operate in pixel space. They are microscopic models. Now, yes. Do they have an implicit notion of sort of macroscopic? Yes. They must because they work. Right? But it's implicit and it's not implemented with the kind of sparse structure that actually exists in the real world and in our conceptualization of it. And that's the thing that we are gonna that we are we're saying, no, no. Look. Like, we want an AI that thinks like us, right, then we are gonna build models that are structured both like the real world is structured. They have this sparse causal macroscopic structure to it. And so should our models and so should and and and the only way to do that is not just to like put a robot in the real world, but to put a robot with a model that is structured in that fashion into the real world. No one's using the XLSTM. Not many people are using Mamba because why not? All you need to do is just scale the transformer as much as possible. So many people just really think you just magically get these things for free, right? So I think you could argue that with enough data that's the right kind of data. 1 of these, like, really big super scaled models will, obtain an implicit representation of the world that is more more or less correct. Now having an implicit representation is great if if your only goal is to just represent the world. If your only goal is to just predict what's going to happen. But it turns out people do something which is very different. People are creative. People can solve novel problems. They can't it's not just about mining old problems and figuring out where I can move some words around and get a and get an answer that looks more or less right. Right? We actually are capable of creating. We're capable of inventing new things. The way that we invent, I think, is exemplified by by like systems engineering. Right? How does systems engineering work? Well, I I, you know, I know how, you know, I I know repeat I know how an air air foil works to create lift. I know how a jet engine works to create thrust. Right? And I can take those 2 bits of information to invent something brand new, which is an airplane. Right? That kind of systems engineering was predicated upon having this sort of model of the world that was relational. Right? Here's the wing. I can put a jet on to it. I can, like I don't know. You don't staple it on. I'm sure you use rivets or something. Right? I know how to put things together. I know how to construct new relationships and new objects. An AI that that is designed for systems, that is designed to do systems engineering will have a object centered or system centered understanding of the world, and we'll know how those all of the objects relate so that it can sort of start experimenting with different ways to combine. That's absolutely, you know, without that, the only thing you will ever be able to do, right, is just retool old solutions for new purposes. And it won't even and even that is I think is a generous interpretation of what a purely predictive model is gonna do. Right? So this is how I like to think about like, you know, the the principal advantage of taking this object centered approach. Right? Is that it enables systems engineering.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.