Lenny Rachitsky: belief

Transcript context

…Oh, average sale press, [inaudible 00:43:22] MRR or some other revenue metric. Got it. This point you made about how a lot of times experiments show positive and then they end up not being anything, I had the head of growth from Shopify on the podcast, and they do this really cool thing where they keep holdouts for years of cohorts and then it auto emails them I think a year or two later, "Hey, check this and see if these cohorts, this is still higher or not. " And 40% of the time, it turns out neutral after a positive experiment long term. Interesting. It's really funny because the last time we did something similar, we had a global holdout group actually that was held out of all experiments. The experiment platform couldn't target that group at all. So 10% of all people never saw anything ever. So that's be really, really helpful because you can always compare them against whatever the experience was for any of the same vintage of cohort. I agree with you. But the other thing is I don't really love some of that thinking process just in general. It's like, "Hey, let's say an experiment does show a temporary benefit. If an experiment shows a temporary benefit, but that benefit does not persist forever, does that mean the temporary benefit was never worth it? Or does that just mean the temporary benefit was an opportunity to reach another level you just didn't capitalize on?" I don't think there's a perfect answer, is what I'm trying to say. I don't think that the fact that a benefit doesn't last forever means that you failed. But I agree with you that not trying to understand, well, what has the net benefit been, what has the net lift been is also really important too. That's why growth is so hard. Growth is part of product is so especially hard.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.