Dwarkesh Patel: uncertainty

Transcript context

…I think in the end we will get the best of both worlds somehow. I think an obvious downside of the brain is it cannot be copied. You don’t have external read-write access to every neuron and synapse, whereas you do. I can just edit something in the weight matrix in Python or whatever and load that up and copy that. In principle. So the fact that it can’t be copied and random-accessed is very annoying. But otherwise maybe it has a lot of advantages. It also tells you that you want to somehow do the co-design of the algorithm. It maybe even doesn’t change it that much from all of what we discussed, but you want to somehow do this co-design. So yeah, how do you do it with really slow low-voltage switches? That’s going to be really important for energy consumption. Co-locating memory and compute. I think that hardware companies will probably just try to co-locate memory and compute. They will try to use lower voltages, allow some stochastic stuff. There are some people that think that all this probabilistic stuff that we were talking about—“Oh, it’s actually energy-based models, so on”—it is doing lots of sampling. It’s not just amortizing everything. The neurons are also very natural for that because they’re naturally stochastic. So you don’t have to do a random number generator in a bunch of Python code basically to generate a sample. The neuron just generates samples and it can tune what the different probabilities are and learn those tunings. So it could be that it’s very co-designed with some kind of inference method or something. It’d be hilarious…. I mean the message I’m taking from this interview is that like all these people that folks make fun of on Twitter, Yann LeCun and Beff Jezos and whatever, I don’t know maybe they got it right. That is actually one read of it. Granted, I haven’t really worked on AI at all since LLMs took off, so I’m just out of the loop. But I’m surprised and I think it’s amazing how the scaling is working and everything. But yeah, I think Yann LeCun and Beff Jezos are kind of onto something about the probabilistic models or at least possibly. In fact that’s what all the neuroscientists and all the AI people thought until 2021 or something.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.