Adam Marblestone: belief

Transcript context

…Exactly. One way to think about it might be that test-time compute, inference-time compute is actually doing this sampling again. You literally read its chain of thought. It’s actually doing this toy example we’re talking about where it’s like, “Oh, can I solve this problem by doing X? Nah, I need a different approach.” This raises the question. I mean, over time it is the case that the capabilities which required inference-time compute to elicit, get distilled into the model. So you’re amortizing the thing which previously you needed to do these rollouts, these Monte Carlo rollouts, to figure out. In general, maybe there’s this principle that digital minds which can be copied, have different tradeoffs which are relevant, from biological minds which cannot. So in general, it should make sense to amortize more things because you can literally copy the amortization, or copy the things that you have sort of built in. This is a tangential question where it might be interesting to speculate about. In the future, as these things become more intelligent and the way we train them becomes more economically rational, what will make sense to amortize into these minds, which evolution did not think was worth amortizing into biological minds? You have to retrain every time. First of all, I think the probabilistic AI people would be like, of course you need test-time compute, because this inference problem is really hard and the only ways we know how to do it involve lots of test-time compute. Otherwise it’s just this crappy approximation that’s never going to… You have to do infinite data or something to make this. I think some of the probabilistic people will be like, “No, it’s inherently probabilistic and amortizing it in this way just doesn’t make sense.” They might then also point to the brain and say, “Okay, well the brain, the neurons are stochastic and they’re sampling and they’re doing things. So maybe the brain actually is doing more like the non-amortized inference, the real inference.” But it’s also strange how perception can work in just milliseconds or whatever. It doesn’t seem like it uses that much sampling. So it’s also clearly doing some baking things into approximate forward passes or something like that to do this. In the future, I don’t know. Is it already a trend to some degree that things that people were having to use test-time compute for, are getting used to train back the base model? Now it can do it in one pass. Maybe evolution did or didn’t do that. I think evolution still has to pass everything through the genome to build the network and the environment in which humans are living is very dynamic. So maybe, if we believe this is true, there’s a Learning Subsystem per Steve Byrnes, and a Steering Subsystem, that the Learning Subsystem doesn’t have a lot of pre-initialization or pretraining. It has a certain architecture, but then within lifetime it learns. Then evolution didn’t actually amortize that much into that network. It amortized it instead into a set of innate behaviors in a set of these bootstrapping cost functions, or ways of building up very particular reward signals. This framework helps explain this mystery that people have pointed out and I’ve asked a few guests about, which is that if you want to analogize evolution to pretraining, well how do you explain the fact that so little information is conveyed through the genome? So 3 gigabytes is the size of the total human genome. Obviously a small fraction of that is actually relevant to coding the brain. Previously people made this analogy, that actually evolution has found the hyperparameters of the model, the numbers which tell you how many layers there should be, the architecture, basically, how things should be wired together. But if a big part of the story is that increased sample efficiency aids learning, generally makes systems more performant, is the reward function, is the loss function—and if evolution found those loss functions that aid learning—then it actually makes sense how you can build an intelligence with so little information. Because the reward function, in Python the reward function is literally a line. So you just have a thousand lines like this, and that doesn’t take up that much space.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.