Jason Cohen: prediction

Transcript context

…Okay. I'm going to now take us to Contrarian Corner. The question here is, what's something that you believe that most other people don't? A/B testing doesn't work very well and it doesn't work on most things. It won't work on strategy, or vision, or insights, nothing actually important to the success of the company. You don't A/B test whether Uber's a good idea. And then, even when you do A/B test the details, where I agree, sometimes that can work. What happens is people will try things like, "Oh, I'll just try this verb, and that verb, and this ad, and that ad." And then like, "Oh, the seventh or eighth one, I got a positive result. That must be good." And what happens is you keep doing that. You pick the best one, and then you go on, and you find another one, and pick the best one. And then a year later you look back and you should be like 50 or a hundred percent better because you've stacked these things, and you look back, and like nothing's different. The conversion rates are the same as they've always been. You say, "What the hell happened? I thought I picked the winner." And the answer is in a combination of the tools not being statistically accurate, which they're not. And the fact that you will get false positives even if the tool is statistically accurate means that most of them are false positives. Even if the tool's 95% accurate, when the thing you're looking for is rare, which it is in the case of A/B testing, the false positives happen more often than the actual thing happens. And so most of the results you get are false positives anyway. So as a result, this isn't true of all A/B testing, but for what most people do when they just do the mundane A/B testing, you can't A/B test the important things and the details are mostly false positives, so it's an enormous waste of time unless you're incredibly sophisticated. I know there's special groups that actually are very sophisticated. Fine. If you're not doing that, it's like the poker table. If you don't know who the patsy is, it's you. If you don't have all of this information and knowledge about A/B testing, then you're the patsy. Bam. All right.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.