Evidence receipt / prediction
Published · transcript-backedJay Baxter: prediction
27 Feb 2025 Lenny's Podcast An inside look at X’s Community Notes | Keith Coleman (VP of Product) and Jay Baxter (ML Lead)
“I don't know, for people out there who typically run A-B tests on big platforms, you may already be familiar with this, but 1% is typically an awesome effect size for any algorithm change.”
Source trail
Everything needed to verify it.
- Speaker
- Jay Baxter
- Attribution
- Verified speaker
- Claim type
- prediction
- Recorded
- 27 Feb 2025
- Publisher
- Lenny's Podcast
Transcript context
…Definitely mean these days. Okay. Before we get to the origin story, is there anything else along those lines you guys think might be really important to share, that are really interesting? Sure. I guess one other thing is that although we don't actually use the fact that a post was noted in the core ranking algorithm, which we think is a nice property. There is a really big impact just organically, meaning not from the algorithm but just from user behavior, where people will like and re-share or quote posts way less when- Quote. Posts way less when notes are applied. I don't know, for people out there who typically run A-B tests on big platforms, you may already be familiar with this, but 1% is typically an awesome effect size for any algorithm change. We saw more like 30 to 40% engagement rate drops for likes and reposts in A-B tests we were ran when showing a post with or without a note, which is just crazy big. That's just an A-B test on the engagement rate, so that's not the network effect. If you capture the overall network effect of how post spread less by that person's repost, basically if you look top line with a difference in differences approach, multiple different external research groups have both found consistently that there's a 50 or 60% drop in total reposts, which is just nuts after a note is applied. It's having a really big impact on spread actually, too. That's so great to hear. It's what I would want to see and it's incredible impact. Basically, an AI image of something false would just go crazy on Twitter, and did before Community Notes came out, and now what you're saying is just adding that context, not actually... Like you're saying, the algorithm doesn't demote it. If there's something incorrect, it's just people are like, "Okay, this is false, why would I want to retweet this?" That makes sense.…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.