Eric Jang: belief

Transcript context

…That is a very interesting insight, that a lot of problems which are proven to be NP-hard—I don’t know if Go is proven to be NP-hard, but protein folding, et cetera—neural networks can solve. They’re NP-hard in the worst case, but we’re usually not concerned about the worst case. These problems usually have a lot of structure to them. I think the question we should be asking ourselves is about how we’ve been formulating solutions to NP-hard problems in worst-case complexity. I wouldn’t say this solves Go. It doesn’t give us an exact solution of the optimum, but in practice, it is extremely useful. The same thing has been shown in AlphaTensor and AlphaFold. Yes, there is a very hard problem that, in the worst case, seems intractable, and yet we’re able to make almost arbitrary amounts of progress. In the limit, what might this look like? If you want to simulate something very complex like weather, or predict the future—do we live in a simulation or not—the computing resources you need to build a very complex simulation might be much smaller than you think, based on our ability to amortize a lot of that computation into the forward pass of a single network. To me, AlphaGo was the first paper that really showed this profound level of simulation being compressed into a small amount of compute. I feel totally not qualified on the computational complexity or the math to comment on this, but I wonder if there’s an important role of chaos here. What is the problem with weather, and why does it take 10x the amount of resources to predict weather a day out, and continually so for every additional day out? It’s because it’s a chaotic system, so small perturbations can totally change the final estimate as time goes on. I guess you would expect that for Go and protein folding as well.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.