Speaker unverified: evaluation

Transcript context

…Gustav Söderström Yeah, you're being very kind. There was a lot of negative feedback on Twitter on some of that. So let me actually dig into some detail because I think for product people listening to this, this is an interesting lesson that I think few companies talk about because you want talk about everything that went exactly as you thought they would and you don't want to talk about the things that didn't go exactly as you thought they would. Gustav Söderström So I'll go through what we are trying to achieve and what we learned. So Spotify is mainly a background application, and for a long time, we've been considered very good at background music and podcast recommendation. When the phone is in your pocket and you're listening to an EDM playlist or pop playlist or something, we're really good at inserting another EDM track there or another pop track there or something like that in the background. Gustav Söderström What we hear from users again and again, though, is that they say that they get trapped in a taste bubble. So I love my Spotify, I love this, but I'm a little bit bored with EDM now and Spotify's not suggesting something completely new. And if you think about that problem, it may sound similar to the recommendation problem, it's just another recommendation problem, but it's actually fundamentally different because when you're recommending another EDM track inside the EDM playlist, you have a lot of signal from that user that they like EDM. But if you're going to recommend a completely new genre, by definition, you have no idea. Because if you had an idea, it wasn't new to them. So you can't know anything. So back to hit rate, your hit rate is going to be incredibly low when you suggest something completely new to the user. Gustav Söderström So this problem of helping people get out of the taste bubble isn't that easy as it sounds. And we can't really take some genre that maybe isn't typical. So I'm a big fan of Reggaeton, for example. It's not typically ... It's not that common in Sweden. And if you would look the rest of my profile, it's EDM heaviness, you probably wouldn't have guessed it. And Spotify wouldn't have guessed it. So if I'm listening to my favorite EDM playlist in the background or maybe my metal playlist, metal is very big in Sweden, it's really hard for us to just insert a Reggaeton track in the middle of that. Most people are going to think Spotify's broken. Yeah. Gustav Söderström What the hell are they thinking? So that doesn't really work. So in order to help people break out of their taste bubbles, you need something different. You need something where your hit ratio can be very low and you need people to expect it to be very low. Gustav Söderström So when we recommend things in the background, our hit ratio needs to be at least nine out of 10, maybe one dud is okay, but if you get five duds, you're going to think we broke your playlist and your session. We need something where one out of 10 is a success. needs to be at least nine out of 10, maybe one dud is okay, but if you get five duds, you're going to think we broke your playlist and your session. We need something where one out of 10 is a success. If you find one gem out of 10 tries, you're very happy. So you need a completely different paradigm. And you also need to be able to go through many candidates quickly because the hit rate is so low. You can't take three minutes per item. It's like, "Okay, I didn't like this," and it's still like two minutes left before the next one comes on. You need to quickly say, "No, no, no." So the obvious candidates for this are these feed-type experience, where you can go through lots of content, you're expecting the hit ratio to be much lower. And if you don't like it, the cost is very low, you just swipe. Gustav Söderström And then this is the reason why people have been ... When they want to break out of their taste bubbles or when they come into Spotify and listen to something completely new, it is usually because they found it on one of these services, like a TikTok or YouTube or something, where they get exposed to lots of new content. So people were asking us for these tools and so that's what we wanted to solve for. And so we built a bunch of features, feed-like structures, where you can go through either a new genre with many tracks or a podcast channel with genre with many episodes or even full playlists. And we implemented those and we put them in something called subfeeds. So in the current experience, and this is roll out worldwide, if you click the podcast subfeed, you get a feed of podcast episodes. Click the music subfeeds, you get a feed of playlists where you can go through many playlists. And if you don't understand the name, you can quickly hear what they sound like and check out a few tracks and understand if this is for you. And if you go through the search and browse page, you can find completely new genres that you can quickly go through. Gustav Söderström And so those are working as we intended. People go to them when they want to find new music. They browse through them and they save new songs. So they're working as we intended. The thing that didn't work as we intended was when users asked us for this again and again, we took the sum of these things and we put it on Home because people ask so much about discovery and we can see clearly how correlated discoveries with retention on Spotify and so forth. But what we misjudged or failed or rather learned about our own homepage is that the way it works right now, and this is what you can see in the Twitter comment, if you remove the angry voices and try to see what they're saying, they're saying the following, which is actually quite clear in the quantitative data as well, that if you look at what people do on Spotify's homepage, the current one, it is almost 90% what we call recall. Gustav Söderström So it is either getting to a session that you're already in or a specific playlist that you know you want to get to or at least a specific use case. So you come in with a high intent, you actually knew what you wanted, and maybe only 10% of the time as a true discovery, fic playlist that you know you want to get to or at least a specific use case. So you come in with a high intent, you actually knew what you wanted, and maybe only 10% of the time as a true discovery, like I don't know what I want. So if you think about, that is 90% recall and 10% discovery. When we tested the design ... So the subfeeds were working and not working, but when we tested some of them on Home, we switched it from 90/10 to 10/90. So 10% recall, 90% discovery. And while people want discovery, they probably don't want 90% discovery, instead of 90% recall. So if you then look at the comments on Twitter, what they're saying is like, "Hey, I can't find my playlists anymore. Where are these things?" Gustav Söderström They're not really complaining about the discovery, they're complaining about the things they don't get anymore. And we can see this in the quant data as well. And you can see traffic shifting from home into search and into library, which is a clear sign people are trying to find the things they can't find anymore. And you can even see people then trying to use these discovery tools which are optimized for quickly understanding new things to do the recall. Where's that workout playlist I know I want? And it's actually very bad UI for recall, it's like a slot machine, right? Very unpredictable if you ever get to that workout playlist. It was optimized for finding new things, not for recall of existing things. When you do recall, you want the dense UI with many items on screen because you know what it is you're looking for. So you don't need a lot of real estate when you're doing discovery of new things. You want a lot of pixels and you probably want sound because you don't know what it is. Gustav Söderström So what we learn about our UI, and I think there's maybe a little bit of product jealousy here, you always look at other experiences. And if you look around, it could be forgiven for thinking that most other products, if you look at something like YouTube, for example, their homepage is exactly that. It's a huge single-item discovery feed with only new items. And people don't seem to tweet angrily about how angry they are at you to say they love YouTube and it's a big product. And I think what we discovered was that we actually did something really well on our homepage, which was supporting you being inside a multiple sessions at the same time. So you could be in the middle of two podcasts and an audiobook and also them actually I just want to get to that workout playlist. I don't remember the name of it, but I know it's workout. Gustav Söderström We actually did that part really well. I would venture just say much better than the other experiences where you literally have to go to some tab and into library and start browsing to get back to where you were. And so maybe it's path dependent. Because we have done recall pretty well, people got, I think, reasonably upset when they couldn't do the recall anymore. And we didn't want lose that because it was one of the things we did well and underestimated. And my takeaway is actually we do it better than other experiences. So we certainly want to keep that.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.