Speaker unverified: commitment

Transcript context

…fic playlist that you know you want to get to or at least a specific use case. So you come in with a high intent, you actually knew what you wanted, and maybe only 10% of the time as a true discovery, like I don't know what I want. So if you think about, that is 90% recall and 10% discovery. When we tested the design ... So the subfeeds were working and not working, but when we tested some of them on Home, we switched it from 90/10 to 10/90. So 10% recall, 90% discovery. And while people want discovery, they probably don't want 90% discovery, instead of 90% recall. So if you then look at the comments on Twitter, what they're saying is like, "Hey, I can't find my playlists anymore. Where are these things?" Gustav Söderström They're not really complaining about the discovery, they're complaining about the things they don't get anymore. And we can see this in the quant data as well. And you can see traffic shifting from home into search and into library, which is a clear sign people are trying to find the things they can't find anymore. And you can even see people then trying to use these discovery tools which are optimized for quickly understanding new things to do the recall. Where's that workout playlist I know I want? And it's actually very bad UI for recall, it's like a slot machine, right? Very unpredictable if you ever get to that workout playlist. It was optimized for finding new things, not for recall of existing things. When you do recall, you want the dense UI with many items on screen because you know what it is you're looking for. So you don't need a lot of real estate when you're doing discovery of new things. You want a lot of pixels and you probably want sound because you don't know what it is. Gustav Söderström So what we learn about our UI, and I think there's maybe a little bit of product jealousy here, you always look at other experiences. And if you look around, it could be forgiven for thinking that most other products, if you look at something like YouTube, for example, their homepage is exactly that. It's a huge single-item discovery feed with only new items. And people don't seem to tweet angrily about how angry they are at you to say they love YouTube and it's a big product. And I think what we discovered was that we actually did something really well on our homepage, which was supporting you being inside a multiple sessions at the same time. So you could be in the middle of two podcasts and an audiobook and also them actually I just want to get to that workout playlist. I don't remember the name of it, but I know it's workout. Gustav Söderström We actually did that part really well. I would venture just say much better than the other experiences where you literally have to go to some tab and into library and start browsing to get back to where you were. And so maybe it's path dependent. Because we have done recall pretty well, people got, I think, reasonably upset when they couldn't do the recall anymore. And we didn't want lose that because it was one of the things we did well and underestimated. And my takeaway is actually we do it better than other experiences. So we certainly want to keep that. re. And we didn't want lose that because it was one of the things we did well and underestimated. And my takeaway is actually we do it better than other experiences. So we certainly want to keep that. So what we did was now we're just updating their hypothesis to achieve the same goal, which is these things are working and when people want to discover, they use them and they seem to work, they can also get better. Gustav Söderström You're on this hill-climbing journey from a machine learning point of view, but the question is, how do you make sure that whenever people feel that they are in that I'm trapped in my taste bubble, they understand that these things are there and they're easy to use? So now we have a version of Home that we are also testing, obviously, where these things are very available but voluntary and you can still do all of the recall. And so from my point of view, this is the reason we A/B test because you want to be scientific about it and you want to learn as much as possible about your own product and your users. And now I'm sharing a lot of the learnings. Maybe we should keep them to ourselves, but my hunch is that it's going to make it a much better product. Gustav Söderström But what I told my teams when we went into this, because I've done this a few times, agree to signing, I think there are two fundamentally different types of product development. One is designing a new feature. It is hard, but it's voluntary for people to use. So you do the AI DJ. Some people love it, that's fine. If you don't like it, it didn't make it worse for you. But when you redesign, it is much more tricky because it's not voluntary to participate in the redesign. So there's a cost even for people who don't like it. Then you have a very tricky problem here, which is there are going to be two types of feedback. One is you did something and it was right, but people are upset because you changed stuff. The other is you did something and it wasn't right, and people are also upset but for good reasons. Gustav Söderström And so how do you separate these two? Because I think I explained this to ... When we talk through this with my teams, I think the analogy to think about is you have your desktop, your physical desktop, you have your computer in one place, you have your pencil over here, you have your notebook over there, and I come in and I just rearrange all of it. And you have spent, in our case, maybe 12 years with that setup. It doesn't matter if I have a lot of quantitative data that my new setup is better, you're going to get upset because you are effective in this old setup. And it's hard to tell those apart. The most classic use case is the Facebook newsfeed, which people are very upset about when it became a single newsfeed. But it turned out to solve a lot of user problems that you didn't have to run around all of Facebook collecting events yourself. Gustav Söderström So there are some ways of understanding if you made it better, but people's habits are broken or if it's not better. And one thing is, for example, to look at new user cohorts that don't have that behavior versus all user cohorts and so forth. you made it better, but people's habits are broken or if it's not better. And one thing is, for example, to look at new user cohorts that don't have that behavior versus all user cohorts and so forth. So we went through all of this with the teams. Before we did it, I said, "This is going to be painful." There's probably going to be a lot of tweets because chances that we get it exactly right are very low. So for that reason, it hasn't been very hard on the team. It is hard ... You want to respond to people, but the right way to do it is to listen, understand, try new hypothesis to really figure out what's going on. So I think I've done it maybe three or four times now. Three maybe. One unsuccessful, two successfully. So kind of knew what I was getting into. Gustav Söderström So it's almost like you punish yourself, very painful, but also the most exciting things. And I think any product person knows that the easiest, the most straightforward thing to do is to iterate around where you are. There's no risk. You're not going to get fired, no user is going to get angry. But everyone also knows that eventually if you don't adapt new technologies, new paradigms, et cetera, you're going to get replaced. You have to find this balance of trying new things. And when you work in software, you have this tool of A/B testing and being scientific about it. When you build hardware, it's worse. If you're wrong, you're wrong. You can't update.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.