Lenny Rachitsky: evaluation

Transcript context

…you made it better, but people's habits are broken or if it's not better. And one thing is, for example, to look at new user cohorts that don't have that behavior versus all user cohorts and so forth. So we went through all of this with the teams. Before we did it, I said, "This is going to be painful." There's probably going to be a lot of tweets because chances that we get it exactly right are very low. So for that reason, it hasn't been very hard on the team. It is hard ... You want to respond to people, but the right way to do it is to listen, understand, try new hypothesis to really figure out what's going on. So I think I've done it maybe three or four times now. Three maybe. One unsuccessful, two successfully. So kind of knew what I was getting into. Gustav Söderström So it's almost like you punish yourself, very painful, but also the most exciting things. And I think any product person knows that the easiest, the most straightforward thing to do is to iterate around where you are. There's no risk. You're not going to get fired, no user is going to get angry. But everyone also knows that eventually if you don't adapt new technologies, new paradigms, et cetera, you're going to get replaced. You have to find this balance of trying new things. And when you work in software, you have this tool of A/B testing and being scientific about it. When you build hardware, it's worse. If you're wrong, you're wrong. You can't update. I love this story. I so appreciate you sharing it. I imagine also with a big launch like this, you can't actually A/B test it ahead of time because of the press season. They're like, "Oh my God, look what Spotify's doing." And so you're limited there. Imagine, right? You couldn't really test this ahead of time. Gustav Söderström The hardest thing about this is if you're trying something completely new, the MVP needs to be very big so you can build a new IU, but if you didn't do algorithms for single item feed, you can't tell if it was the right idea but poor machine learning, right? UI poor machine learning. Or you have to build a lot and that gets quite expensive. That's actually ... The biggest why it's painful is not really the feedback from the outside. It is the cost you have to take on the inside. You incur a lot of costs as you're really hoping you're right. Gustav Söderström And in our cases, the changes on the homepage aren't that hard for us to do. The important thing is that the underlying hypothesis of, can we help you break out of your taste bubble actually works and then you update the acquisition funnels into that experience. But I think the problem is that you need to get so many things in place to be able to say, "You might get a false negative," just because you didn't do it well or not. That's the biggest challenge, I think, with these big rewrites where everyone has to update everything before you can know if you're right or wrong.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.