Adam Brown: belief

Transcript context

…What is required to ace a test? Obviously, they probably have read about all the generality textbooks, but I assume to ace a test, you need something beyond that. Is there some way you'd characterize physics? Physics problems, compared to math problems, tend to have two components. One is to sort of take this word question and turn it, using your physics knowledge, into a math question and then solve the math question. That tends to be the typical structure of these problems. So you need to be able to do both. The bit that maybe only LLMs can do, and wouldn't be so easy for other things, is step one of that: turning it into a math problem. I think if you ask them hard research problems, you certainly can come up with problems that they can't solve. That's for sure. But it's pretty noticeable, as we have tried to develop evaluations for these models, that as recently as a couple of years ago, certainly three years ago, you could just scrape from the internet any number of problems that are standard, totally standard high school math problems that they couldn't do. And now we need to hire PhDs in whatever field, and, you know, they come up with one great problem a day or something. The difficulty, as these LLMs have got stronger, the difficulty of evaluating their performance has increased. How much do they generalize from these difficult problems to not only that domain of physics but just generally becoming a better reasoner overall? If they just see a super hard GR problem, are they better at coding now?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.