Lenny Rachitsky: uncertainty

Transcript context

…I've learned a lot of things about metrics, mostly from bad metrics. I actually think you learn a lot from picking the wrong metric. Ultimately, you want to find a short-term metric you can measure that drives a long-term output. So people always talk about, "Oh, we want to drive an improvement in retention." Retention is a terrible thing to goal on because it's almost impossible to drive in a meaningful way in the short term, and yet you want to be able to experiment and iterate quickly. So what are the things that drive retention? What are the inputs? So I think it's really important to find the right inputs, and then through experimentation test whether or not those short-term inputs are driving the long-term output that you're looking for. I think that's one thing. I think keeping things simple is another thing I've learned over the years, maybe it's data scientists, but they tend to love these composite metrics with a coefficient. "We're going to wait this input at X and this input at X+2." And then you end up with a metric that nobody really understands that doesn't actually mean anything. And you're like, "I don't know if a 0.1 increase is it a lot? Is it good? Is it bad?" So they're just hard to work with. And so I always encourage folks, just pick something simple, even if it's not perfect and your composite would be more perfect. If people understand it, if they have an intuition around it, if it's something that people can talk about across the company, it's going to be a much better metric in terms of driving real outcomes than your made up composite score that nobody understands. So I think keeping things simple is also really important. And then I think the last thing I'll say is it's important to understand how metrics across the company equate to one another. And so we spend a lot of time quantifying things in terms of a common currency. So for example, if I were to lower price by a dollar, what would I get in terms of, we'll say, volume? Well, what if I lowered delivery times by a minute? What do I get for that in terms of volume? And so now you can make trade-offs between maybe your marketing team and your logistics team because you have this common currency that everyone can talk about. And so we've done that. We've tried to quantify all of the levers of our business, price, selection, quality in common terms, so that if we have, say, a dollar to spend, we know what we get depending on where we put it, over what timeframe. I think that that helps us make decisions more quickly because we know what our options are. We know we have our inventory of things that we can do, short-term, long-term, and what we get for it. So it definitely helps us to make decisions more quickly and hopefully better decisions. These are so awesome. I definitely want to follow up on some of this. This is so good. So maybe on this last one, which we did at Airbnb also like, how does everything translate into Knight's book and booking? Every decision we make, what is the actual Knight's book impact? And so I imagine in your case, I don't know if you want to talk about these things. I imagine it's transactions, or purchases, or GMB or something like that, so I'm guessing is the final metric. I don't know. Is that something you talk about or you don't talk about that? We measure things in terms of GOV, so Gross Order Value, and also volume.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.