High Signal Podcasts Evidence ledger
Method
Browse
← All source episodes

Lenny's Podcast / episode intelligence

An inside look at X’s Community Notes | Keith Coleman (VP of Product) and Jay Baxter (ML Lead)

27 Feb 2025 31 published claims 3 attributable people

Speakers in the public record

Claim mix

belief 18evaluation 6uncertainty 2prediction 2commitment 1preference 1recommendation 1

Evidence policy

Every row below preserves an exact excerpt. Identified speakers are linked; unresolved voices are labeled and excluded from people counts.

Claim ledger

The useful parts, with receipts.

31 published records

01 / belief

One other thing that was, I think, nice to see working then was, one criticism of Community Notes some people bring up is, well if you always need agreement from people who typically disagree, then in these super polarized settings, that conflict being probably number one, then you wouldn't see any notes.

“One other thing that was, I think, nice to see working then was, one criticism of Community Notes some people bring up is, well if you always need agreement from people who typically disagree, then in these super polarized settings, that conflict being probably number one, then you wouldn't see any notes.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

02 / belief

Whatever is going to produce the best outcome is what we should do. So I think I feel more attached to the product being helpful than to anything else, and so to whatever degree it might seem like low ego is probably more a result of wanting to actually solve the problem.

“Whatever is going to produce the best outcome is what we should do. So I think I feel more attached to the product being helpful than to anything else, and so to whatever degree it might seem like low ego is probably more a result of wanting to actually solve the problem.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

03 / belief

There's this talk for a while of just there's no more facts, nobody believes there is a single true fact anymore, everything is subjective, and I think Community Notes proves the opposite.

“There's this talk for a while of just there's no more facts, nobody believes there is a single true fact anymore, everything is subjective, and I think Community Notes proves the opposite.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

04 / belief

I think even though there was original research before Birdwatch had even started, or Community Notes had even started, from external researchers showing that crowdsourced fact-checkers, laypeople can do about as well as fact-checkers and actually the agreement rates were kind of similar between the groups.

“I think even though there was original research before Birdwatch had even started, or Community Notes had even started, from external researchers showing that crowdsourced fact-checkers, laypeople can do about as well as fact-checkers and actually the agreement rates were kind of similar between the groups.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

05 / belief

I think often before Community Notes you could have just been living in a little news filter bubble, or maybe there were fact checks out there that you should have been reading but you weren't discovering them.

“I think often before Community Notes you could have just been living in a little news filter bubble, or maybe there were fact checks out there that you should have been reading but you weren't discovering them.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

06 / belief

I think all of the same aspects of recruiting that apply to external startups and apply internally, and if you can have an exciting vision, that is key.

“I think all of the same aspects of recruiting that apply to external startups and apply internally, and if you can have an exciting vision, that is key.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

08 / belief

I want to start with just giving people a brief understanding of what is Community Notes. I think a lot of people kind of heard about it, kind of maybe see it on X.

“I want to start with just giving people a brief understanding of what is Community Notes. I think a lot of people kind of heard about it, kind of maybe see it on X.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

09 / belief

There's a community moderation aspect, so I think we also thought that this would be specifically for adding context to misleading or potentially misleading information, but what you can see is that there are some notes that go beyond that towards calling out content that they think is spammy or something.

“There's a community moderation aspect, so I think we also thought that this would be specifically for adding context to misleading or potentially misleading information, but what you can see is that there are some notes that go beyond that towards calling out content that they think is spammy or something.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

11 / belief

I think we had done a lot of good work getting momentum going, and people in the us and in the industry had tried things to deal with misleading information, but nothing was really working.

“I think we had done a lot of good work getting momentum going, and people in the us and in the industry had tried things to deal with misleading information, but nothing was really working.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

12 / belief

I think the opt in is important, and it may even be that you want to opt in at one point in your life, and maybe at another point in your life something else is better.

“I think the opt in is important, and it may even be that you want to opt in at one point in your life, and maybe at another point in your life something else is better.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

13 / belief

I think that's another great example, OKRs, I understand why they can be helpful, but they can also be not necessarily the right cadence at which to set goals.

“I think that's another great example, OKRs, I understand why they can be helpful, but they can also be not necessarily the right cadence at which to set goals.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

15 / belief

Related to that and your list, Lenny, one thing we missed that's super important is that on this project, and I think of successful projects like it in startups, is that people are self-selecting to join.

“Related to that and your list, Lenny, one thing we missed that's super important is that on this project, and I think of successful projects like it in startups, is that people are self-selecting to join.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

16 / uncertainty

Something on my mind, and I don't know if you guys want to go here, but it's something I think a lot of people think about is when Elon came in, he let go of 80% of folks.

“Something on my mind, and I don't know if you guys want to go here, but it's something I think a lot of people think about is when Elon came in, he let go of 80% of folks.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

17 / belief

Vitalik Buterin had a blog post where he talks about his explorations, making sure the algorithm really does what it says it does. And I think just the fact that a handful of people have done this, there's enough people who have done it that there's someone you'd probably trust who's verified it.

“Vitalik Buterin had a blog post where he talks about his explorations, making sure the algorithm really does what it says it does. And I think just the fact that a handful of people have done this, there's enough people who have done it that there's someone you'd probably trust who's verified it.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

18 / belief

I think there's a lot there that a lot of people think they should do when they set these teams up, but they don't actually do, and it feels like each of these is just a really key ingredient to it to actually succeeding.

“I think there's a lot there that a lot of people think they should do when they set these teams up, but they don't actually do, and it feels like each of these is just a really key ingredient to it to actually succeeding.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

19 / belief

I think you may have shared with this with me, when someone noticed you guys were testing this and they took screenshots and tweeted it, and I think Elon replied, "This is cool.

“I think you may have shared with this with me, when someone noticed you guys were testing this and they took screenshots and tweeted it, and I think Elon replied, "This is cool.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

21 / commitment

I would also say there is this reputation component to the algorithm as well. So if you consistently rate notes in a way that is counter to the bridging-based consensus, then we will stop counting your ratings.

“I would also say there is this reputation component to the algorithm as well. So if you consistently rate notes in a way that is counter to the bridging-based consensus, then we will stop counting your ratings.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

22 / prediction

I think back in 2020 before we started building anything here, whether this could work at all, I think a room of ML engineers would say, "Oh, you have to keep it closed source.

“I think back in 2020 before we started building anything here, whether this could work at all, I think a room of ML engineers would say, "Oh, you have to keep it closed source.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

23 / evaluation

I will say that I think people were sort of open-minded to it, generally speaking, and I think everyone had a sense that what was being done at the time wasn't really working that well or solving the problem, and people were open to new ideas, so that's a good foundation.

“I will say that I think people were sort of open-minded to it, generally speaking, and I think everyone had a sense that what was being done at the time wasn't really working that well or solving the problem, and people were open to new ideas, so that's a good foundation.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

25 / prediction

I don't know, for people out there who typically run A-B tests on big platforms, you may already be familiar with this, but 1% is typically an awesome effect size for any algorithm change.

“I don't know, for people out there who typically run A-B tests on big platforms, you may already be familiar with this, but 1% is typically an awesome effect size for any algorithm change.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast

26 / evaluation

The other approach is even if you had 10,000 people doing it, the real issue is that they don't work that well because they're not trusted or they don't scale or they're too slow.

“The other approach is even if you had 10,000 people doing it, the real issue is that they don't work that well because they're not trusted or they don't scale or they're too slow.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

29 / evaluation

I think the point you made about people opting into that is important, 'cause I think a lot of people hearing that would be like, "I would never want to be asked to build something like that in two weeks.

“I think the point you made about people opting into that is important, 'cause I think a lot of people hearing that would be like, "I would never want to be asked to build something like that in two weeks.”
Speaker
Lenny Rachitsky
Publisher
Lenny's Podcast

30 / preference

The classic career path is, I don't know, rewards or running a large organization or being a manager, or things like that, but I think, at the end of the day, you got to work on stuff you love, you got to be having fun, and I think people want to be having impact.

“The classic career path is, I don't know, rewards or running a large organization or being a manager, or things like that, but I think, at the end of the day, you got to work on stuff you love, you got to be having fun, and I think people want to be having impact.”
Speaker
Keith Coleman
Publisher
Lenny's Podcast

31 / recommendation

I think one other thing that's key is when you are forced to have such a small team, well, this is important anyways, but deleting code is more important than writing it a lot of the time. So I think so often maybe due to promotion incentives or just regular human tendency, engineers have a tendency to add these little incremental wins that actually add more of a long-term maintenance cost than is clear, because you just run a little one month A-B test, you see this significant win and you don't realize the maintenance burden you just added to your team for the rest of eternity until you turn the thing off.

“I think one other thing that's key is when you are forced to have such a small team, well, this is important anyways, but deleting code is more important than writing it a lot of the time. So I think so often maybe due to promotion incentives or just regular human tendency, engineers have a tendency to add these little incremental wins that actually add more of a long-term maintenance cost than is clear, because you just run a little one month A-B test, you see this significant win and you don't realize the maintenance burden you just added to your team for the rest of eternity until you turn the thing off.”
Speaker
Jay Baxter
Publisher
Lenny's Podcast
Search evidence