Evidence receipt / evaluation
Published · transcript-backedAlessio Fanelli: evaluation
14 Mar 2024 Latent Space Making Transformers Sing - with Mikey Shulman of Suno
“I think before we wrap, you have written a blog post that can show about good hearts law impact in ML, which is, you know, when you measure something, then the thing that you measure is not a good metric anymore because people optimize for it.”
Source trail
Everything needed to verify it.
- Speaker
- Alessio Fanelli
- Attribution
- Verified speaker
- Claim type
- evaluation
- Recorded
- 14 Mar 2024
- Publisher
- Latent Space
Transcript context
…Awesome. I know we covered a lot of things. I think before we wrap, you have written a blog post that can show about good hearts law impact in ML, which is, you know, when you measure something, then the thing that you measure is not a good metric anymore because people optimize for it. Any thoughts on how that applies to like LLMs and benchmarks and kind of the world we're going in today? Yeah, I mean, I think it's maybe even more apropos than when I originally wrote that, because so much we see so much noise about pick your favorite benchmark. And this model does slightly better than that model. And then at the end of the day, actually, there is no real world difference between these things. And it is really difficult to define what real world means. And I think to a certain extent, it's good to have these objective benchmarks, it's good to have quantitative metrics. But at the end of the day, you need some acknowledgement that you're not going to be able to capture…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.