Terence Tao: evaluation

Transcript context

…You made this distinction between artificial cleverness and artificial intelligence. I would like to better understand those concepts. What is an example of intelligence that is not just cleverness? Intelligence is famously hard to define. It’s one of these things that you know when you see it. But when I talk to someone and we’re trying to collaboratively solve a math problem together, there’s this conversation where neither of us knows how to solve the problem initially. One of us has some idea and it looks promising, so then we have some sort of prototype strategy. We test it, and it doesn’t work, but then we modify it. There’s adaptivity and continual improvement of the idea over time. Eventually, we’ve systematically mapped out what doesn’t work and what does work, and we can see a path forward, but it’s evolving with our discussion. This isn’t quite what the AIs do. The AIs can mimic this a little bit. To go back to this analogy of these jumping robots, they can jump and fail, and jump and fail. But what they can’t do is jump a little bit, reach some handhold, stay there, pull other people up, and then try to jump from there. There isn’t this cumulative process which is built up interactively. It seems to be a lot more trial and error and just repetition: brute force. It scales, and it can work amazingly well in certain contexts. But this idea of building up cumulatively from partial progress is what’s still not quite there yet. Interesting. You’re saying if Gemini 3 or Claude 4.5, whatever, solves a problem, it is not the case that its own understanding of math has progressed.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.