Shawn Wang: evaluation

Transcript context

…He does. I actually found out about the guy on Reddit because he does extremely long threads about the different models he evaluates and the kind of questions that they get right or wrong. He does his evaluations in German, if I remember properly. So it's usually very interesting to see how he does it. He's super rigorous, but he does, I don't know, 15 prompts. So a rigorous vibe check. So I've read those things on the local LLM subreddit and it's a little bit excessive. I don't know if I need all that, but I'm glad somebody does it. To me, he's my automated vibe eval. I don't know who he is, but he shows up, so that's about it. So we wanted to cover specific choices around the new leaderboard. So congrats on launching it. You corrected a bunch of very fundamental data science things, like the variance between the benchmarks, as well as selecting for better benchmarks. I think obviously MMLU Pro is the top one, just because that's the top number that a lot of people report. The headline figures are, for example, it's 10 choices instead of four, and it's actually reviewed by experts instead of just not reviewed by experts. Any other sort of special notes that you would, basically, I want to do a quick tour around the ones that you picked, right? MMLU Pro, GPQA, I think these two are very well regarded. I have Eval as well. I noticed that Apple Intelligence is the only benchmark that Apple Intelligence used. Everything else was their own internal evals, but Apple Intelligence picked IFEval as their benchmark. Anyway, so do you want to comment quickly on some of the ones that you picked? Yeah. So for IFEval, I think it's a very interesting one because it's like unit tests, but for language, right? When you evaluate coding LLMs, you give them a bunch of unit tests, and you see if the functions that the LLM has written is able to make all the unit tests work. And IFEval behaves in literally the same way. They are giving prompts with very strict instruction formatting, and they are only evaluating instruction following. And I find it very interesting because it's not a metric which is ambiguous at any step. A lot of evaluations which are looking at the content are going to be using bag of words or embeddings to try to get like semantic similarity. Here you don't care. You are literally evaluating on understanding instructions. And I think it's a very smart data set. I loved it. We also added GPQA, which I've wanted to add to the leaderboard since it came out. Basically MMLU, but PhD level. Super complex questions which have been written by PhD experts and which are easy kind of to answer. If you have a PhD in the field, but not if you don't. So I think those ones are super interesting. They are only in science.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.