Lenny Rachitsky: evaluation

Transcript context

…in five hits, four out of five things are done, then you need a UI that probably at least shows five things at the same time on screen. So you have a one in five of something being relevant on screen. Gustav Söderström So you need to understand the performance of your machine learning to design for it. It needs to be fault tolerant and often you need an escape hatch for the user. So you make a prediction. But if you were wrong, it needs to be super easy for the user say, "No, you're wrong, I want to go to my library or to this or to that." So we have that principle of having fault-tolerant user interface and a user interface that corresponds to the current performance of your algorithms. And I think that is going to be true for generative machine learning as well. I think a very clear example actually is Mid Journey. If you think about the early Mid Journey user interface inside the Discord channel, actually generating an image was very, very slow. Gustav Söderström It took a long time to generate high-quality image and they could have built the silver button thing where you put in a prompt, you wait for minutes, you get an image, and I think one out of four times, it's going to be bad. So you would've been disappointed three out of four times and it's a minute each, so like four minutes later, you'd be, "This is a shitty product." What they did was they generated four simultaneous low-res images very quickly and you could say, "So apparently, their performance was probably one in four, that's why they showed four and not six." And so one in four was usually pretty good. You click that one and either continue to iterate or scale it up. So that's also an example of, I think, people understanding where the performance of generative AI was when they built the UI. So that's something that I would be inspired by. Gustav Söderström And for the AI DJ specifically, another principle is to try to avoid this urge of just wanting to show off the technology and then have this voice that talk and talk and talk and talk. You have to remember that people came there for the music. So the principle for the AI DJ coming from the team, by the way, this was a bottoms-up product actually, it required a lot of support. We actually acquired big companies and so forth to be able to build it. But the idea had been built by teams bottom up. So the principle there was literally to do as little as possible and get out of the way. And I think that was really helpful. It's not telling you what the weather is and what happened in the news and going on and on and on about this band. It is trying to get you to the music and I think that's why it's working because it is working very well for us. I love this distinction between recommendation and generation. And this begs the question of, there's this trend that I imagine you're seeing of people autogenerating music using artists catalog. There's this Drake and The Weeknd thing that came out a week or two ago. Where do you think this ends up going and how do you think artists adjust to this world where music can just be autogenerated? This play button is all of it is generated versus just like the DJ in between the songs. Gustav Söderström First, big caveat, this is just super early. No one knows anything about how this is going to play out or the legal landscape and so forth, but I think it's going to have a lot of impact. And I think if we talk about two things, one is what it could do for music, the other is the right situation, and if rights-holders are getting compensated and so forth. So we talk about the first thing in isolation. I think an interesting example is right about when I grew up, Avicii came along. And it's interesting to think about because Avicii was not really considered by the existing music industry as a real artist because he couldn't really play an instrument and he couldn't sing, and he was just sitting with this computer in this DAW, digital audio workstation. And so it wasn't really considered real music. And I think now all of us consider it very real music and that he had tremendous real musical talent. Gustav Söderström So I think right now, we're probably in the face where people say this isn't real music and it's somehow fake. I think the way to think about these diffusion models if and when they get good enough at generating music is probably the same like an instrument. It's just a much more powerful instrument and we'll probably see a new type of creator that wasn't proficient at any instrument and they couldn't assemble a full orchestra and do the thing that they had in their head and they can now generate very new things. I also think, by the way, that there is this distinction between AI music and real music that doesn't exist. For sure, very talented real musicians are using AI to get better and to help create new ideas. So that distinction doesn't really exist. It's all going to be AI. The question is what percentage, which makes the problem harder because you can't talk about if it should exist or not. Gustav Söderström You have to talk about what percentage should exist and who gets to use it or not. But I think the way to think about it is probably as an instrument and that could help create a huge amount of art. And I think this is not news to you who probably use these things a lot, but I think if you don't use these generative models, there is the perception that you tell it to create a hit and you will get that. That's not how it works. Actually, what these models do is because they've been listening to a lot of music, they are very good at doing something that sounds very similar to what already exists. Actually being original is very hard. And from one point of view, as it now gets easier to create more generic music, it will actually be more difficult than ever to be truly unique. Gustav Söderström So I still think there would be tremendous skill in creating something truly unique. And my hope would be that what happened with the DAW and that technology jump was you got a whole new genre like EDM that you couldn't really produce it with an orchestra or live.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.