Lenny Rachitsky: preference

Transcript context

…Correct. And I mean on growth, we're definitely in the... We want to drive metrics on a short-term basis and we can do that obviously, but we have the luxury, and the way Tobi thinks about the world and the way we operate, to really think about these longterm effects and make sure that we're holding ourselves accountable with these longterm holdouts, and then constantly refining the input metrics that we're using and getting a lot smarter about that. But because we take that long horizon, it allows us to be better in the short term and just get a lot smarter. And a lot of counterintuitive things. And I would encourage everyone, if you can, look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment. And I'll bet you'd be surprised how many times the metric is different than what you thought it would be after a year. Because where people just make a call at a certain point in time, here's the list in it, here's the list experience. I love this, because very few people have experiences running a longterm experiment, and so this is a really interesting insight that you're sharing that, I guess how often do you find this to be true in your long-term holdouts, where things end up being very different downstream? I think there's probably two things that have been very common. And I would say in quite a few cases, you get a lift on a metric up front, a more short-term metric. Number of people who become a paying shopper, number of people who make their first sale in Shopify. And then you look a year later, and there's actually no incremental lift on GMV from that cohort. And so I think it actually trains, a lot of us in growth are looking at these short-term metrics. A lot of the time it's actually more pull-forward effect, than you fully realize or an incremental user that's just really not worth that much. So that's one. And then two, so this effect size goes away. There are cases where the experiment has flipped the other way. And then there are cases, and these are the most interesting ones, where you realize that you uncovered a pocket of merchants that are actually extremely valuable entrepreneurs who go on to be successful, that you missed in your normal short-term measurement techniques. And so all across the board we see that. But actually the most common is it actually isn't a long-term lift from a lot of things that you might think of the short-term are.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.