Elena Verna: recommendation

Transcript context

…So where do you find that balance? I imagine people hearing this are like, oh, yes, this makes sense to me, and then they continue to test basically everything. What's kind of some heuristics you'd recommend for knowing, okay, just don't test that. Don't worry about it. You don't need credit for that win. First of all, I think that people should trust their intuition a little bit more. Data is good, but data is only good if you have enough of it. So if you have low volume real estate, that is going to take you eight months to reach some sort of answer. Do you really want to test it for eight months? What's the point of it? My rule of thumb, if we cannot collect the sample size in the month, we shouldn't test it, period. Because it's just then it's not fast enough. We should just go and do pre versus post. Pre versus post is pretty powerful. It's also a way to assess your impact and you can still roll back if it doesn't work. But don't think that everything needs a scientific explanation to whether it needs to be moved forward or not. Also, experimentation cannot be the way that people make decisions in the company. There's still so much about knowing your user, understanding the market for your brain to connect all of those dots and to know what needs to be an experience that your customers are going to want. That if scientific data, like a very tight determination of the probability that this is a success, it's important, absolutely do it because it might be a really big strategical pivot that you're planning to do. So it's a data point to validate that all of this extra work will be needed. It might be a very high traffic real estate that even 0.1% difference will mean millions of dollars for you. But other than that, it should be just go, go, go. Do pre versus post. By the way, I'm not saying just release and move on. Release, do seven day, 24 hour readout, seven day readout, 28 day readout, even come back to it a year later and measure some of the retention or extension data that is associated with it. But to test everything is debilitating to growth teams and app paralyzes them in its place. So kind of look at your initiatives and say, where do I need precision? And it's important and I can get it best in that versus where we should just go for it. And yeah, we will fail there too, and that's okay. And we can roll back and we can figure out how to make it better. But failure is going to happen regardless. In statistics, six is tricky. Many people take it for face value versus it's just like a directional data point to say there's likely new distribution that has maybe a different mean, and 5% of the chances. If you measure in 95% statistical significance, you might not even be there, and yet really take it for so granted. Oh, it's going to drive this much lift. So I just think that people stop in this age of data, almost rely enough on their element intuition. A lot of contrary intakes here. I love this. Elena, we've reached number 10. And I know number 10 is like a special one where it's more than one. Quick. Fire.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.