Speaker unverified: belief

Transcript context

…But yeah, I agree with this thing. We should have a benchmark for it and it'll get better. Yeah, I hope so. And I guess a bunch of other things would be interesting to evaluate. I think that robustness to prompting, nobody does it because it's too expensive. But if I prompt a model with 10 variations of the exact same prompt in terms of content, I don't want to get 10 different answers, right? And it's kind of linked to calibration. It's something that should be taken for granted in LLMs, but it's actually not working that well. And if I had to take a third choice, because I'm very greedy, you asked me for one, but you're getting three. I would love to see more things about psychofancy and basically all the ways into which models can be problematic in their interactions and put people in basically thought bubbles. You don't want people to be on social network too when they talk to a chat model, right? You want the chat model to be assertively saying what is factually true or not. Some things are factually true, right? The earth is round, gravity exists. A lot of things should not be debated and models should be assertively telling users that they are wrong if they are saying that those do not exist. Awesome. This was a great kind of run through the leaderboard and a lot of the questions we already took a lot of your time. Before we wrap, maybe just one last thing. Any predictions for like leaderboard v3? Like if you go one year from now, do you think most models will have kind of top this new v2 too? Or how long do you think it's going to last before you need a new one?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.