Speakers in the public record
Claim mix
belief 6evaluation 5recommendation 3observation 2prediction 2uncertainty 1commitment 1preference 1
Evidence policy
Every row below preserves an exact excerpt. Identified speakers are linked; unresolved voices are labeled and excluded from people counts.
Claim ledger
The useful parts, with receipts.
21 published records
“There's a fun book called Calling Bullshit, which despite the name, which is a little extreme, I think, for a title, it actually has a lot of amazing insights that I love. And it sort of embodies, in my opinion, a lot of the Twyman's law showing that things that are too extreme, your bullshit meter should go up and say, "Hey, I don't believe that." So that's my number one recommendation. There's a slightly older book that I love called Hard Facts, Dangerous Half-Truths And Total Nonsense by the Stanford professors from the Graduate School of Business. Very interesting to see many of the things that we grew up with as well understood turn out to have no justification. So a stranger book, which I love, sort of on the verge of psychology, it's called Mistakes Were Made (But Not by Me), about all the fallacies that we fall into, and the humbling results from that.”
- Publisher
- Lenny's Podcast
“So a stranger book, which I love, sort of on the verge of psychology, it's called Mistakes Were Made (But Not by Me), about all the fallacies that we fall into, and the humbling results from that.”
- Publisher
- Lenny's Podcast
“There's this holy grail of experiments that I think people are always looking for of one hour of work and it creates this massive result.”
- Publisher
- Lenny's Podcast
“I think one of the mistakes that some company makes is they launch a lot of experiments and never go back and summarize the learnings.”
- Publisher
- Lenny's Podcast
“We'll put in the notes how to find it, and some discount for people who managed to stay all the way to the end of this podcast.”
- Publisher
- Lenny's Podcast
“I think the ability to give people honest feedback and have them appreciate, and have it stay after the meeting in these notes on the document, just amazing.”
- Publisher
- Lenny's Podcast
“" We came up with a lot of follow on ideas that helped a lot. But to me, this was an example of a tiny change that was the best revenue generating idea in Bing's history, and we didn't rate it properly.”
- Publisher
- Lenny's Podcast
“Coming back to this kind of concern people have of experimentation, keeps you from innovating and taking big bets, I know you have this framework overall evaluation criterion, and I think that helps with this.”
- Publisher
- Lenny's Podcast
“In fact, if you look at one investment, one big investment that was done at the time was online experiences, and the initial data wasn't very promising. And I think today, it's a footnote.”
- Publisher
- Lenny's Podcast
“Do you do you collect a list of, "Here's things that often work when we want to move" there's some you could share. I don't know if you have a list in your head.”
- Publisher
- Lenny's Podcast
“I teach this in my class, but I think I've posted this on LinkedIn and answered some questions.”
- Publisher
- Lenny's Podcast
“If in peace time you're wrong two thirds to 80% of the time, why would you be subtly right in wartime, in Covid time? So I don't believe in the idea that because bookings went down materially, the company should suddenly not be data driven and do things differently.”
- Publisher
- Lenny's Podcast
“And the question is, what are you optimizing for? And it's a much harder question that people think because it's very easy to say we're going to optimize for money, revenue.”
- Publisher
- Lenny's Podcast
“You figure out how you can help Microsoft." And I said, "I'm going to build an experimentation platform," because nobody at Microsoft is running experiments.”
- Publisher
- Lenny's Podcast
“I think one of the things that I will say at Airbnb is the analysis was relatively weak, and so lots of data scientists were hired to be able to compensate for the fact that the platform didn't do enough.”
- Publisher
- Lenny's Podcast
“Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact.”
- Publisher
- Lenny's Podcast
“This wasn't the case when I started working. So when I started running experiments at Amazon, we were building the platform because nothing existed.”
- Publisher
- Lenny's Podcast
“We need that okay button because you want to be able to debug the reasons, and sometimes the metrics help you understand why you have a sample ratio mismatch. So we blanked out the scorecard, we had this button, and then we started to see that people pressed the button and still presented the results of experiments with sample ratio mismatch.”
- Publisher
- Lenny's Podcast
“I will say that when you say things are going well, I think we don't know the counterfactual.”
- Publisher
- Lenny's Podcast
“Hold that dinner, investigate, see, because there's a large probability that something is wrong with the result. And I will say that nine out of 10, when we call it Twyman's law, it is the case that we find some flaw in the experiment.”
- Publisher
- Lenny's Podcast
“So if I'm pitching the book here, there is no financial gain for me from having more copies sold. I think we made that decision, which was a good decision.”
- Publisher
- Lenny's Podcast