Tim Scarfe: belief

Transcript context

…that. Yeah. So so what to do when that day comes for AI scientists is a is a is a broader discussion, think. I think now is a good segue to talk about this paper in a little bit more detail. So that this continuous thought machine, you were just pointing to it before. Look, first first of all first of all, mate, introduce yourself and set this thing up for us. Name's Luke. I am a research scientist at Sakana AI. And my primary sector of research is this continuous thought machines. It took us somewhere in the region of about 8 months working on this project with the whole team. I I did a lot of the work, but we also had a lot of people in different areas and doing different parts of it that I think an 8 month life cycle for a paper seems a bit long for AI research at the moment. But, yes, to the to the actual technical points of the paper. So we call it continuous thought machines. It originally had a different name. We called it asynchronous thought machines before. But every single time people asked us what's the asynchronous part, it became a bit confusing. So continuous thought machines basically depends on 3 novelties. The first 1 is having what we call a internal thoughts dimension. And this is not necessarily something new. It's related conceptually to the ideas of latent reasoning. And it's essentially applying compute in a sequential dimension. And when you start thinking about ideas and problems in this domain and in this framework, you start understanding that many problems that look like or solutions to problems that look intelligent are often solutions that have a sequential nature. So for instance, 1 of the primary tasks that we tested in the continuous thought machines was this maze solving task. And solving mazes for deep learning is is quite trivial. It's really easy to do if you make the task easy for machines. And 1 of the ways to do this is you give an image of a maze to a neural network, like a convolutional neural network, And it outputs a image. Same size of the maze and it's zeros where there isn't a path and there's ones where there is a path. There's some really brilliant work showing how you can train these in a careful way and scale them up essentially indefinitely. And this is fascinating and really interesting idea of how to solve this. However, when you take that approach out of the picture and you ask what is a more human way to solve this problem, it becomes a sequential problem. You have to say, well, go up, go right, go up, go left, whatever the case may be to trace a route from start to finish. And when you constrain that simple problem space and you ask a machine learning system to solve it like that, turns out to actually get much, much more challenging. So this became our hello world problem for the CTM. And applying an internal sequential thought dimension to this is how we went about solving this. 2 other novelties that we can touch on and talk about. We we sort of rethought the idea of what neurons should be. There's a lot of excellent research in this world in cognitive neuroscience particularly exploring how neurons work in biological systems. And then we get on the other side of the scale how deep learning neurons work which the quintessential example is a ReLU. It's off or on in a sense.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.