High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Ronny Kohavi

Published podcast speaker

Claims
18
Episodes
1
Shows
1
Named items
2

Books, apps, and tools

The evidenced stack.

Browse the grouped index →

book / likes

Mistakes Were Made (But Not by Me)

“So a stranger book, which I love, sort of on the verge of psychology, it's called Mistakes Were Made (But Not by Me), about all the fallacies that we fall into, and the humbling results from that.”

Lenny's Podcast · 27 Jul 2023

Evidence receipt · Source ↗

book / likes

Hard Facts, Dangerous Half-Truths And Total Nonsense

“There's a fun book called Calling Bullshit, which despite the name, which is a little extreme, I think, for a title, it actually has a lot of amazing insights that I love. And it sort of embodies, in my opinion, a lot of the Twyman's law showing that things that are too extreme, your bullshit meter should go up and say, "Hey, I don't believe that." So that's my number one recommendation. There's a slightly older book that I love called Hard Facts, Dangerous Half-Truths And Total Nonsense by the Stanford professors from the Graduate School of Business. Very interesting to see many of the things that we grew up with as well understood turn out to have no justification. So a stranger book, which I love, sort of on the verge of psychology, it's called Mistakes Were Made (But Not by Me), about all the fallacies that we fall into, and the humbling results from that.”

Lenny's Podcast · 27 Jul 2023

Evidence receipt · Source ↗

Claim ledger

What Ronny said.

18 transcript-backed records

01 / recommendation

Likes Hard Facts, Dangerous Half-Truths And Total Nonsense. Likes Mistakes Were Made (But Not by Me).

“There's a fun book called Calling Bullshit, which despite the name, which is a little extreme, I think, for a title, it actually has a lot of amazing insights that I love. And it sort of embodies, in my opinion, a lot of the Twyman's law showing that things that are too extreme, your bullshit meter should go up and say, "Hey, I don't believe that." So that's my number one recommendation. There's a slightly older book that I love called Hard Facts, Dangerous Half-Truths And Total Nonsense by the Stanford professors from the Graduate School of Business. Very interesting to see many of the things that we grew up with as well understood turn out to have no justification. So a stranger book, which I love, sort of on the verge of psychology, it's called Mistakes Were Made (But Not by Me), about all the fallacies that we fall into, and the humbling results from that.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

05 / belief

I think the ability to give people honest feedback and have them appreciate, and have it stay after the meeting in these notes on the document, just amazing.

“I think the ability to give people honest feedback and have them appreciate, and have it stay after the meeting in these notes on the document, just amazing.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

06 / evaluation

" We came up with a lot of follow on ideas that helped a lot. But to me, this was an example of a tiny change that was the best revenue generating idea in Bing's history, and we didn't rate it properly.

“" We came up with a lot of follow on ideas that helped a lot. But to me, this was an example of a tiny change that was the best revenue generating idea in Bing's history, and we didn't rate it properly.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

07 / belief

In fact, if you look at one investment, one big investment that was done at the time was online experiences, and the initial data wasn't very promising. And I think today, it's a footnote.

“In fact, if you look at one investment, one big investment that was done at the time was online experiences, and the initial data wasn't very promising. And I think today, it's a footnote.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

09 / prediction

If in peace time you're wrong two thirds to 80% of the time, why would you be subtly right in wartime, in Covid time? So I don't believe in the idea that because bookings went down materially, the company should suddenly not be data driven and do things differently.

“If in peace time you're wrong two thirds to 80% of the time, why would you be subtly right in wartime, in Covid time? So I don't believe in the idea that because bookings went down materially, the company should suddenly not be data driven and do things differently.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

10 / evaluation

And the question is, what are you optimizing for? And it's a much harder question that people think because it's very easy to say we're going to optimize for money, revenue.

“And the question is, what are you optimizing for? And it's a much harder question that people think because it's very easy to say we're going to optimize for money, revenue.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

11 / commitment

You figure out how you can help Microsoft." And I said, "I'm going to build an experimentation platform," because nobody at Microsoft is running experiments.

“You figure out how you can help Microsoft." And I said, "I'm going to build an experimentation platform," because nobody at Microsoft is running experiments.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

12 / evaluation

I think one of the things that I will say at Airbnb is the analysis was relatively weak, and so lots of data scientists were hired to be able to compensate for the fact that the platform didn't do enough.

“I think one of the things that I will say at Airbnb is the analysis was relatively weak, and so lots of data scientists were hired to be able to compensate for the fact that the platform didn't do enough.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

13 / observation

Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact.

“Because again, I've observed this sort of surprising result that even small bug fixes, even small changes can sometimes have surprising, unexpected impact.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

15 / recommendation

We need that okay button because you want to be able to debug the reasons, and sometimes the metrics help you understand why you have a sample ratio mismatch. So we blanked out the scorecard, we had this button, and then we started to see that people pressed the button and still presented the results of experiments with sample ratio mismatch.

“We need that okay button because you want to be able to debug the reasons, and sometimes the metrics help you understand why you have a sample ratio mismatch. So we blanked out the scorecard, we had this button, and then we started to see that people pressed the button and still presented the results of experiments with sample ratio mismatch.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

17 / prediction

Hold that dinner, investigate, see, because there's a large probability that something is wrong with the result. And I will say that nine out of 10, when we call it Twyman's law, it is the case that we find some flaw in the experiment.

“Hold that dinner, investigate, see, because there's a large probability that something is wrong with the result. And I will say that nine out of 10, when we call it Twyman's law, it is the case that we find some flaw in the experiment.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast

18 / evaluation

So if I'm pitching the book here, there is no financial gain for me from having more copies sold. I think we made that decision, which was a good decision.

“So if I'm pitching the book here, there is no financial gain for me from having more copies sold. I think we made that decision, which was a good decision.”
Speaker
Ronny Kohavi
Publisher
Lenny's Podcast
Search evidence