Laura Schaffer: preference

Transcript context

…This episode is brought to you by Writer. How much hype have you been hearing about generative AI? So much. But how do you take it from a shiny toy to an actual business tool that helps you do your actual job? Writer is an enterprise grade generative AI platform built specifically for the needs of businesses and already widely deployed at world-class brands like Uber, Spotify, HubSpot, and UiPath. With writer, you can break through content bottlenecks across your organization from marketing webpages to sales e-mails in product messages to creating high quality on-brand content at scale. And unlike other AI applications, writers' training happens securely on your data and your style and brand guidelines that you provide specific to your organization. The result is that you get consistent content in your brand voice at scale. Get AI that your people will love for a limited time. Listeners to Lenny's podcast and get 20% off if they go to writer.com/lenny, that's writer.com/lenny. Where do you find the best ideas come from for driving meaningful lift? Is it gut instinct type and experience bucket or is it data telling you like, "Hey, or here's a huge opportunity," in your experience? I'm a very data-driven person. I self-describe and think of myself that way. In large part because of that, I feel you have to be constantly checking yourself and data is a really great way to do that. But I definitely think that I would be described as someone who's going more by their gut when looking at date end results just because of the way that I approach it, which is I'm very comfortable and very common in using qualitative responses and things like that and supplement to quantitative data to make a decision and that puts less of a burden on the quantitative to really make an assessment of whether something was working or not. One of the things I see, I think sometimes goes against what other folks do, although I'm seeing things shift a little, is that 95% confidence rate. My background in college, I was in a lab running experiments or really publishing two journal and stuff and we had to have that 95% confidence rate, had to because the things that were coming out of the lab and being published were influencing things like how we do education and how we understand how bias works and when it shows up and therefore how we can combat it. Things were wrong. And sending a bunch of bologna, that can cause some significantly bad things like false positive, false negatives in that context can be very dangerous, for lack of a better word. And you think of other pharmaceuticals, the 95% confidence rate belongs in some companies and some industries because the risk of failure on the impact of a false success is very high. But those of us converting users and trying to upsell folks, we are very fortunate to not have that level of burden on us and we can take advantage of that. And so there are definitely times where I will advocate for and I will push for and I will myself use lower confidence intervals and 95%, especially if that doubles amount of experiments that you can run in a year. End of the day, these are all methods that we use to try to validate the hypotheses that we have. And if you're doing a 95% confidence in a role, you're still accepting a 5%, some amount of false success, do that a little bit more, challenge you to do that a little bit more. And then run way more experience. If you look at the net of what your team is doing over the course of year, what you're doing over the course of a year, you will be positive. Wow, that is a big idea. Idea of releasing the P-value confidence interval for experimentation and data teams. Everyone would be excited about this. Probably maybe not some data scientists on teams. Do you do that? How do you act? Is that how you operate on your teams? Just like we don't need 95% competence?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.