Terence Tao: prediction

Transcript context

…An example of a conjecture: Gauss was interested in the prime numbers and created one of the first mathematical datasets. He just computed the first 100,000 prime numbers or so, hoping to find patterns. He did find a pattern, but maybe not the pattern he was expecting. He found a statistical pattern in the primes that if you count how many primes there are up to 100, 1,000, one million, and so forth, they get sparser and sparser, but the drop-off in the density was inversely proportional to the natural logarithm of the range of numbers. So he conjectured what we now call the prime number theorem: the number of primes up to X is X divided by the natural log of X. He had no way to prove this. It was data-driven. This was a conjecture. It was revolutionary for its time because it was maybe the first really important conjecture of math that was statistical in nature. Normally you’re talking about a pattern, like maybe the spacing between the primes has a certain regularity. But this didn’t tell you exactly how many primes there were in any given range. It just gave you an approximation that got better and better as you went further and further out. It started the field of what we call analytic number theory. It was the first in many conjectures like this, many of which got proved, which started consolidating the idea that the prime numbers didn’t really have a pattern, that they behaved like random sets of numbers with a certain density. They had some patterns, like they’re almost all odd. They’re also not actually random, they’re what’s called pseudo-random. There’s no random number generation involved in creating the prime numbers. But over time, it became more and more productive to think of the primes as if they were just generated by some god rolling dice all the time and creating this random set. This allowed us to make all these other predictions. There’s a still-open conjecture in number theory called the twin prime conjecture, that there should be infinitely many pairs of primes that are twins just two apart, like 11 and 13. We can’t prove that, and there are good reasons why we can’t prove it. But because of this statistical random model of the primes, we are absolutely convinced it’s true. We know that if the primes were generated by flipping coins, we would just—by random chance like infinite monkeys at a typewriter—see twin primes appear over and over again. We have over time developed this very accurate conceptual model of what the primes should behave like based on statistics and probability. It’s mostly heuristic and non-rigorous, but extremely accurate. The few times when we actually can prove things about the primes, it has matched up with the predictions of what we call the random model of the primes. We have this conjectural concept framework for understanding the primes that everyone believes in. ut the primes, it has matched up with the predictions of what we call the random model of the primes. We have this conjectural concept framework for understanding the primes that everyone believes in. It’s the same reason why we believe the Riemann hypothesis is true, and why we believe that cryptography based on the primes is mathematically secure. It’s all part of this belief. In fact, one reason why we care about the Riemann hypothesis is that if the Riemann hypothesis failed, if we knew it was false, it would be a serious blow to this model. It would mean there’s a secret pattern to the primes that we were not aware of. I think we would very rapidly abandon any cryptography based on the primes, because if there was one pattern that we didn’t know about, there are probably more, and these patterns can lead to exploits in crypto. It would be a big shock. So we really want to make sure that doesn’t happen. We’ve been convinced of things like the Riemann hypothesis over time. Some of it is experimental evidence, and some is that the few times we’ve been able to make theoretical results, they’ve always aligned. It is possible that the consensus is wrong and we’ve all just missed something very basic. There have been paradigm shifts in the past in scientific history. But we don’t really have a way of measuring this, partly because we don’t have enough data on how math or science develops. We have one timeline of history, and we have maybe 100 stories of turning points in history. If we had access to a million alien civilizations, each with a different development of history and science in different orders, then maybe we’d actually have a decent shot at understanding how we measure what progress is and what is a good strategy. We could maybe start formalizing it and actually having a framework. Maybe what we need to do is start creating lots of mini-universes or simulations of AI solving very basic problems in arithmetic or whatever, but coming up with their own strategies for doing these things and having these little laboratories to test. There are people who investigate what’s the smallest neural network that can do 10-digit multiplication and things like that. I think we could learn a lot just from evolving small AIs on simple problems. You have to learn about new fields not only very rapidly, but deeply enough to contribute to the frontier. So in some sense, you’re also one of the world’s greatest autodidacts. What is your process of learning about a new subfield in math? What does that look like?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.