Tal Raviv: evaluation

Transcript context

…I love that analogy. This episode is brought to you by Eppo. Eppo is a next generation A-B testing and feature management platform built by alums of Airbnb and Snowflake for modern growth teams. Companies like Twitch, Miro, ClickUp and DraftKings rely on Eppo to power their experiments. Experimentation is increasingly essential for driving growth, and for understanding the performance of new features. And Eppo helps you increase experimentation velocity while unlocking rigorous, deep analysis in a way that no other commercial tool does. When I was at Airbnb, one of the things that I loved most was our experimentation platform where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Eppo does all that and more, with advanced statistical methods that can help you shave weeks off experiment time and accessible UI for diving deeper into performance and out-of-the-box reporting that help you avoid annoying prolonged analytic cycles. Eppo also makes it easy for you to share experiment insights with your team, sparking new ideas for the A-B testing flywheel. Eppo powers experimentation across every use case, including product growth, machine learning, monetization, and email marketing. Check out Eppo at getEppo.com/lenny and 10X your experiment velocity. That's getEppo. com/lenny. Let's move on to another, let's say hot take that you have. You have this kind of phrase that there's a big difference between book smart decision making and street smart decision making. What's that about? Yeah. This one comes out of a lot of mistakes I've made myself and seen around me as well. So I'm so guilty of this. Book smart decision making as a PM is all the stuff we talk about all the time. That's data, design, technology strategy, frameworks, all that stuff. And it's really important and it's why we're strong at it. Street smart decision making is taking all that and then seeing something from somebody else's point of view. This goes beyond empathy. I'll tell you what I mean. It's like giving the customer's perception just as much weight as you would to logic. A really big example of this, and I won't name the company. I was at this company that changed the structure of the pricing and the change was actually not meant to make more money. It was meant to unblock a payments roadmap and enable all these feature requests that were stuck behind this fundamental change. And in preparing for this, the company did a lot of analyses and just made sure that this was actually really good for customers and that this would only, the number- This would only ... The numbers and the predictions and the models are like, "This is a positive thing." I know all the people that were involved were the most empathetic, did this because they really cared about this customer, genuinely. I've never seen people, at the executive level, at the product level, at the data level, these are the people who really genuinely wanted the best for this customer and they rolled out the change, and in reality, bottom line, it was a big revolt on the internet. It was a big deal and it was rolled back. And the bottom line why there was such a gap between everything ... And actually, by the way, all the analyses proved that to be correct. All the models, all the predictions, everything played out the numbers as predicted. The problem was the perception, the narrative behind it, what it looked like when you've logged into the product and you saw only the negative, but you didn't see all the positive because the positive was what happened 30 days later and you saw the negative immediately. A lot of things like that.I would've made exactly the same mistake. Not a criticism. I would've probably made more mistakes, but that's what opened my mind to, oh my God, you have to really think more than just logical, more than just utilitarian. Another example of this is there's a company I worked at where all the features that were on a higher plan were invisible to the lower plans just because everything was built really fast and there was time to make them visible so people could upgrade. So we got that ready and we did this and we're about to release it, and we realized tomorrow morning a bunch of customers are going to log into the product and there's all these features that they've been asking us for because they didn't know that we have them, because they were locked and only on higher plan. customers are going to log into the product and there's all these features that they've been asking us for because they didn't know that we have them, because they were locked and only on higher plan. Suddenly, from their point of view, the company built everything they'd asked for and all of it requires paying more money. How are they going to feel? And so, we didn't roll it out that way. We rolled it out in smaller pieces and in different ways because we realized that's going to feel really if you think about it. But, practically speaking, or theoretically speaking, we didn't upsell anything. These were all things we already had built. Everything's above board. That's the perception of what was going to happen.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.