Dwarkesh Patel: evaluation

Transcript context

…You don’t have to evaluate all these energy values or whatever and sample around to make them higher and lower. You just say, approximately that process would result in this being the top one or something like that. Exactly. One way to think about it might be that test-time compute, inference-time compute is actually doing this sampling again. You literally read its chain of thought. It’s actually doing this toy example we’re talking about where it’s like, “Oh, can I solve this problem by doing X? Nah, I need a different approach.” This raises the question. I mean, over time it is the case that the capabilities which required inference-time compute to elicit, get distilled into the model. So you’re amortizing the thing which previously you needed to do these rollouts, these Monte Carlo rollouts, to figure out. In general, maybe there’s this principle that digital minds which can be copied, have different tradeoffs which are relevant, from biological minds which cannot. So in general, it should make sense to amortize more things because you can literally copy the amortization, or copy the things that you have sort of built in. This is a tangential question where it might be interesting to speculate about. In the future, as these things become more intelligent and the way we train them becomes more economically rational, what will make sense to amortize into these minds, which evolution did not think was worth amortizing into biological minds? You have to retrain every time. First of all, I think the probabilistic AI people would be like, of course you need test-time compute, because this inference problem is really hard and the only ways we know how to do it involve lots of test-time compute. Otherwise it’s just this crappy approximation that’s never going to… You have to do infinite data or something to make this. I think some of the probabilistic people will be like, “No, it’s inherently probabilistic and amortizing it in this way just doesn’t make sense.” They might then also point to the brain and say, “Okay, well the brain, the neurons are stochastic and they’re sampling and they’re doing things. So maybe the brain actually is doing more like the non-amortized inference, the real inference.” But it’s also strange how perception can work in just milliseconds or whatever. It doesn’t seem like it uses that much sampling. So it’s also clearly doing some baking things into approximate forward passes or something like that to do this. In the future, I don’t know. Is it already a trend to some degree that things that people were having to use test-time compute for, are getting used to train back the base model? Now it can do it in one pass. Maybe evolution did or didn’t do that. I think evolution still has to pass everything through the genome to build the network and the environment in which humans are living is very dynamic. So maybe, if we believe this is true, there’s a Learning Subsystem per Steve Byrnes, and a Steering Subsystem, that the Learning Subsystem doesn’t have a lot of pre-initialization or pretraining. It has a certain architecture, but then within lifetime it learns. Then evolution didn’t actually amortize that much into that network. It amortized it instead into a set of innate behaviors in a set of these bootstrapping cost functions, or ways of building up very particular reward signals.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.