Mayur Kamat: evaluation

Transcript context

…That is certainly a hot take. So the idea here, I'm curious how you operationalize this with folks at N26. Is it just like, "I don't need to see a whole strategy for the year. Just give me, here's the plan, here's what we're going to test, here's our hypothesis?" Are you actually, what do you tell your PM team? We use this tool. I'm going to give a shout-out to Statsig because they're awesome. Vijay used to run the experimentation at Facebook and has this tool. There's several of those. But if you're running proper experiments, I just look at the Statsig dashboards, right? And I'm looking at experiments, I'm looking at what metrics they're moving, I'm looking at the P-value, I'm looking at how quickly can they get to statistical significance. And I'm like, "Oh, this is working. Let's do more of these." Right? So, now there's some areas where you can't do it, like in compliance, in legal aspects, in Europe, especially pricing. In US, you can run pricing tests. In Europe, it's a little bit different. So, those areas, you would need to have a lot more kind of deeper thinking, understanding of your cohorts. You're coming up with more structured reason for why you should do it, but you can't really test and know within a couple of days or a couple of weeks at max whether this was a good idea or not. Those, if there are either irreversible decisions or they're just extremely time-consuming to find out, then do some pre-work. We look at largely, find a lot of companies that really look at data without looking at cohorts that make completely bad decisions, right, because if you look at your dashboard as a mixture of users over 10 years, 20 years, even six months, and they all behave differently. If you look at a cohort level development of certain users, you generally end up making better decisions. But even over there, it's still lot more, there's a lot of noise between the moment you start tracking it than moment you start making decisions based on it. The world has changed in that meantime. By now, this was kind of a very foreign concept when I brought this in. I'm like, oh, the conversions down now, even though the product's done really well because Bitcoin has crashed, right? Nobody wants to go sign up for an Exchange account. So, if you just measure pre and post, you would think that you have done something wrong in the product. If you measure it as an experiment, you would know that, yeah, between the variant and control, it's still doing great, even though overall conversion is down. So largely, the more you kind of, one of the first thing kind of doing when I take on a role, the company already doesn't have an experimentation culture. That's largely why they hire me in the first place, right? So, for now, the first thing is to say, how can we bring it in? First, the kind of right culture, the right incentives and the right tools. And then, once it's set up, it gets a lot of fun. Get a lot of fun for the PMs because you have democratized performance for the product managers. And the second thing, which I tell my PMs now, which is truly kind of empowering if you think about it, the challenge with being a product manager is everybody thinks they can do their job, right? You go to... The CFO might have an idea, the head of kind of accounting has an idea. if you think about it, the challenge with being a product manager is everybody thinks they can do their job, right? You go to... The CFO might have an idea, the head of kind of accounting has an idea. Anybody who uses the product thinks they have ideas, right? So, at some point in time, you're like, what is my discipline? What is my science? Nobody goes to the accounting guy and says, "Hey, I have a great idea for how to cook our books." Nobody does that because there's a science behind it. There's a science for financial forecasting. Even in technology, a lot of the times people just don't go and say, "Hey, just dump this thing and let's use this code that the AI code generator has used." Right? There's a little bit of science there. Whereas in product, you largely find that it's a combination of data and ideas and stuff, and anybody thinks they can... The moment you build experimentation, you'll now make it scientific, right? Now, somebody comes up with an idea, say, that's a bad idea. Here, this is why it's a bad idea, because we have done this experiment six times and it has failed across this user groups at this exact level of impact created. So, it kind of gives the PMs the kind of, hey, I'm not just a general purpose technician, I'm a specialist now. And it's extremely empowering once we can, it takes a long time to move the team in that direction. But once you get it there, the PMs just, it's a natural kind of dopamine hit every time you run an experiment and see more metrics.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.