Sander Schulhoff: evaluation

Transcript context

…We'll link to this report where people are really curious about the history. I am actually, but let's stay focused on techniques. What are some other techniques that are towards the advanced end of the spectrum? There's certain ensembling techniques that are getting a bit more complicated. And the idea with ensembling is that you have one problem you want to solve. And so, it could be a math question. I'll come back and again and again to things like math questions because a lot of these techniques are judged based off of data sets of math or reasoning questions simply because you're going to evaluate the accuracy programmatically as opposed to something like generating interview questions, which is no less valuable, but just very difficult to evaluate success for in an automated way. So ensembling techniques will take a problem and then you'll have multiple different prompts that go and solve the exact same problem. So I'll take maybe a chain of thought prompt, let's think step by step. And so I'll give the LM a math problem. I'll give it this prompt technique with the math problem, send it off, and then a new prompt technique, send it off. And I could do this with a couple different techniques or more. And I'll get back multiple different answers and then I'll take the answer that comes back most commonly. So, it's like if I went to you and Fetty and Gerson to a bunch of different people, and I asked them all the same question. And they gave me back in slightly different responses, but I take the most common answer as my final answer. And these are a historically known set of techniques in the AI ML space. There's lots and lots and lots of ensembling techniques. It's funny, the more I get into prompting techniques, the less I remember about classical ML. But if you know random forests, these are a more classical form of ensembling techniques. So anyways, a specific example of one of these techniques is called mixture of reasoning experts, which was developed by a colleague of mine who's currently at Stanford. And the idea here is you have some question, it could be a math question, it could really be any question. And you get yourself together a set of experts. And these are basically different LLMs or LLMs prompted in different ways, or some of them might even have access to the internet or other databases. And so you might ask them, I don't know, how many trophies does Real Madrid have? And you might say to one of them, okay, you need to act as an English professor and answer this question. And then another one, you need to act as a soccer historian and answer this question. And then you might give a third one, no role but just access to the internet or something like that. And so you think, all right, like the soccer historian guy and the internet search one, say they give back 13 and the English professor is four. So you take 13 as your final response. And one of the neat things about, well, roles as we discussed before which may or may not work, is that they can activate different regions of the model's neural brain and make it perform differently and better or worse on some tasks. well, roles as we discussed before which may or may not work, is that they can activate different regions of the model's neural brain and make it perform differently and better or worse on some tasks. So if you have a bunch of different models you're asking and then you take the final result or the most common result as your final result, you can often get better performance overall.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.