Sander Schulhoff: evaluation

Transcript context

…Even better. ... I ended up being right. And a couple months later, one of the researchers who was involved with that thread, who had written one of these original analytical papers, sent me a new paper they had written, and was like, "Hey, we re-ran the analyses on some new data sets and you're right. There's no effect, no predictable effect of these roles." And so my thinking on this is that at some point with the GPT-3, early ChatGPT models, it might've been true that giving these roles provides a performance boost on accuracy-based tasks, but right now, it doesn't help at all. But giving a role really helps for expressive tasks, writing tasks, summarizing tasks. And so with those things where it's more about style, that's a great, great place to use roles. But my perspective is that roles do not help with any accuracy-based tasks whatsoever. This is awesome. This is exactly what I wanted to get out of this conversation. I use roles all the time. It's so planted in my head from all the people recommending it on Twitter. So for the titles example I gave you of my podcast, I always start, you're a world-class copywriter. I will stop doing that because I don't... You're saying it won't help.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.