Laura Schaffer: evaluation

Transcript context

…So I'll say this, this is actually very critically important. You must have this game plan set before you run something. Failure mode that I see so many teams fall into is they'll run the experiment or whatever it is and then they'll make the data fit the hypothesis. Or sometimes they'll go without a hypothesis and just be like, "This is going to do better things for our metrics," but not a core reason as to why or what exactly are we testing here. And so this is another area we could absolutely fall into that trap. "Let's [inaudible 00:45:36] on good 80. I think it's good. That Laura person said it was cool. So I think that that's fine." That will always be a trap. So it needs to be very deliberately thought of in advance as a way of like, "Hey, here's how we're going to validate this." And always, always, always, if you're going to accept more risk of a false success or false positive, false negative, you want to then be really thinking about how you're going to harden your validation of a hypothesis. For example, let's take that when we talked about with Twilio where we are kicking people out and we're sending them to the pilling hotdog experiment, and we're sending people to that experience to hide the phone number. Now in that case, let's say that we were going to accept a lower confidence interval. I would very much want to see qualitative feedback to confirm that that hypothesis was true. I want to be looking at the qualitative data from the ones where people were thrown into the existing flow and one's put into the dogs that one of them felt more confident and more like this was really easy to get through and they felt out of a territory and things like that. And I'd be wanting to hear from the ones who were in the other one, things like, "Oh, I got stuck on that [inaudible 00:46:48]." Like, "Figure this out, but it feels like it's that amount of my depth." I would want to be looking for other things to corroborate the hard data that I'm seeing. And yes, it opens the door to whenever you open the door to more risk acceptance, you are going to have some false successes there. But all of these things together can overall make it more likely that you're shipping more things that are going to positively influence the customer. And again, I can't say it enough. It is a huge risk in and of itself to not ship as much as you possibly could in a year. That is a huge risk given that very high fail rate. So to those data scientists, and I've chatted with a few of my time, what I try to explain is that that article, that data that the 80%, that's hard data about what a detriment it can be if you don't run an enough experiments. If you just run 10 in a year on there, maybe two around impact, two of a course of an entire year if you take that approach. ta about what a detriment it can be if you don't run an enough experiments. If you just run 10 in a year on there, maybe two around impact, two of a course of an entire year if you take that approach. So data scientists can understand, "Hey, if we do this, if we run this down, we can double or triple whatever it is, the number of experiments where we can run and overall net that's going to result in more successes that will overall net us to a positive place." You can still tell a data story to the data scientist about why you're doing this. Again, this is why when you asked that question identify, as a very data-driven person. But I think some of the methods that I use can sound at the service level as more like, "Oh, I'm going by my gut. " But again, very data driven is just embracing the reality of some of the hard data that I don't think we all embrace or are even aware of sometimes about that fail rate. This is awesome. This is a big idea. Have you written about this anywhere for folks that maybe want to try this approach at their company? And if not, you should.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.