High Signal Podcasts Evidence ledger
Method
Browse
← Back to evidence

Evidence receipt / prediction

Published · transcript-backed

Jay Baxter: prediction

27 Feb 2025 Lenny's Podcast An inside look at X’s Community Notes | Keith Coleman (VP of Product) and Jay Baxter (ML Lead)

“I don't know, for people out there who typically run A-B tests on big platforms, you may already be familiar with this, but 1% is typically an awesome effect size for any algorithm change.”

— Jay Baxter

Source trail

Everything needed to verify it.

Speaker
Jay Baxter
Attribution
Verified speaker
Claim type
prediction
Recorded
27 Feb 2025
Publisher
Lenny's Podcast

Transcript context

…Definitely mean these days. Okay. Before we get to the origin story, is there anything else along those lines you guys think might be really important to share, that are really interesting? Sure. I guess one other thing is that although we don't actually use the fact that a post was noted in the core ranking algorithm, which we think is a nice property. There is a really big impact just organically, meaning not from the algorithm but just from user behavior, where people will like and re-share or quote posts way less when- Quote. Posts way less when notes are applied. I don't know, for people out there who typically run A-B tests on big platforms, you may already be familiar with this, but 1% is typically an awesome effect size for any algorithm change. We saw more like 30 to 40% engagement rate drops for likes and reposts in A-B tests we were ran when showing a post with or without a note, which is just crazy big. That's just an A-B test on the engagement rate, so that's not the network effect. If you capture the overall network effect of how post spread less by that person's repost, basically if you look top line with a difference in differences approach, multiple different external research groups have both found consistently that there's a 50 or 60% drop in total reposts, which is just nuts after a note is applied. It's having a really big impact on spread actually, too. That's so great to hear. It's what I would want to see and it's incredible impact. Basically, an AI image of something false would just go crazy on Twitter, and did before Community Notes came out, and now what you're saying is just adding that context, not actually... Like you're saying, the algorithm doesn't demote it. If there's something incorrect, it's just people are like, "Okay, this is false, why would I want to retweet this?" That makes sense.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.

Search evidence