Shaun Clowes: evaluation

Transcript context

…You as a product leader is at the Venn diagram center of the sweet spot of where this podcast has been going recently, which is product and growth and how AI helps you with all these things. And so to follow a thread there with synthesizing and understanding what people are saying, user research and surveys and all these things, have you found any tools that you and your team have found really useful to help you do this more efficiently versus traditionally just manually going through all the stuff and finding patterns? Yeah, so firstly, stepping back a little bit just into the motherhood and apple pie portion of qualitative research or whatever, I find that most people don't even understand or don't start with a rigorous foundation in what they're going to need to do to get the answers that they want. So for example, your listeners have probably heard about the Nielsen number before, but basically the idea is that once you interview between 7 and 14 people, you stop learning new things. Less than 7, you don't learn enough, more than 14, you start learning anything new. And so if you interviewed two people, you probably don't have enough data. If you interviewed 22, you probably had too much, so they don't even right size their efforts. So that's a problem. So they don't start that way. Then they go into these conversations asking leading questions, which really are designed to get the customer to say what they already want to be true, which is so they haven't done enough research or they've done too much and then they've blown up all of the results before they've even heard anything. If you don't right size your research and you don't set this up to learn, then you're going to lose. No amounts of applying LLMs or any type of kind of structured reasoning is going to help you. Because you just basically you're reading back what you want to hear or some weird summarized version of what you want to hear. But stepping back from all of that, what I like to do specifically getting to LLMs is I think that we live in just the most amazing time for product managers right now in terms of being able to analyze vast quantities of information and see the common threads. And so let me give you few examples of that. One might be you can do a bunch of customer interviews, you can put a bunch of customer interviews into ChatGPT and you can say, "Hey, ChatGPT, this is my strategy. Tell me where my strategy does not fit what these customers talked about." It's all about the not, not where it does, where it does not. People spend far too much time looking for what they're hoping to see, not for what they're not looking to see. So you can literally ask ChatGPT to help you find where the customer is probing at the edges of what you're trying to do, where it's wrong, where what you're saying is not what they believe. And you can ask it questions like that. You can ask it what your customers are saying would better fit what your competitors are saying. So you can basically say, you can copy and paste one of your competitor's positioning documents into ChatGPT and say, "Is this a better fit for what they have said than my thing?" Which is you can summarize your own strategy, you can take your competitors but public documents and you can ask it to summarize what their strategy probably is. And it's actually supposedly good at that because mostly your public documents are actually a summary or at least they're derivative of what your strategy is. ummarize what their strategy probably is. And it's actually supposedly good at that because mostly your public documents are actually a summary or at least they're derivative of what your strategy is. So it will give you crazy insights into what other people's, literally their product strategy at times creepy like, "Oh, they will probably do this, they will probably do that. It's more likely they would do this than they would do that." And so normally that type of insight was hard one, it took a lot of sweat work. You basically get to read a lot of stuff. You kind of had to use your brain as this big summarization machine and eventually you knew what you felt about all the things you had read, but you couldn't summarize why. LLMs let you get to that really, really, really quickly in a very structured way, but only if you push at the edges, provoke the answers you don't want to hear, provoke the problems, try and prove to yourself that you are wrong, I think is the easiest way to start trying to use some of these tools.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.