Sander Schulhoff: prediction

Transcript context

…But explain it for folks that don't know what you're talk about. Sure. Role prompting is really just when you give the AI you're using some kind of role. So you might tell it, "Oh, you are a math professor," and then you give it a math problem. You're like, "Hey, help me solve my homework," or "this problem," or whatnot. And so looking in the GPT-3, early ChatGPT era, it was a popular conception that you could tell the AI that it's a math professor, and then if you give it a big data set of math problems to solve, it would actually do better. It would perform better than the same instance of that LLM that is not told that it's a math professor. So just by telling it it's a math professor, you can improve its performance. And I found this really interesting and so did a lot of other people. I also found this a little bit difficult to believe because that's not really how AI is supposed to work, but I don't know, we see all sorts of weird things from it. So I was reading a number of studies that came out and they tested out all sorts of different roles. I think they ran a thousand different roles across different jobs and industries, like, you're a chemist, you're a biologist, you're a general researcher. And what they seemed to find was that [inaudible 00:19:21] roles with more interpersonal ability, like teachers, performed better on different benchmarks. It's like, wow, that is fascinating. But if you looked at the actual results, data itself, the accuracies were 0.01 apart. So there's no statistical significance, and it's also really difficult to say which roles have better interpersonal ability. And even if it was statistically significant, it doesn't matter. It's 0.1 better, who cares?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.