Nate Silver: evaluation

Transcript context

…Like a Math Olympiad tournament. They just did gold medal performance. I said this last year. I said, “In a year, they’re going to do gold medal.” A year ago, they weren’t sure how many r’s were in the word strawberry. You don’t think on superforecasting, they can — I think it’s very different when you’re dealing with a static problem as compared to a dynamic system where the inputs are changing all the time. Currently, the large language models are very, very bad at poker. They’re not trained on poker data. I’m sure if you did train them . . . There are these things called solvers that are trained on poker data that do very well, but they cannot quite impute the general patterns just from mediocre text data or an amateurish kind of hand analysis. If you probe them on why they’re bad, they’re like, “Yes, maybe it’s tough for us when you have a complicated, evolving game theory dynamic, and you have to develop exploitative strategies very quickly.” If you have a computer solve a poker hand to get to one where there’s enough loss minimization, a very strong computer can take minutes, whereas a poker player makes those calculations implicitly in a handful of seconds, for example. I don’t know. I worry, with the Math Olympiad stuff — there’s a little bit of teaching to the test where, because you set this as the goal that a large language model should have, that therefore there’s a lot of prestige when you meet that goal, potentially — Isn’t teaching to the test what we should do, even with humans in a sense. The test is what you think is important, and that’s what you ought to teach.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.