Stewart Butterfield: evaluation

Transcript context

…Classic. It's really, I think the essential thing to understand about this is it's not because people are evil, and it's not because they're stupid. And it's to me, very related to everything is complex. And if maybe this is my butterfly's law, I haven't thought about this way before. But I tweeted this a very, very long time ago like if you... Everything is simple if you have no idea what you're talking about. So the other side of that is like if something seems simple, probably you don't understand it. And there's obvious exceptions to that. But for anything that involves a large organization or a lot of human beings, if the problem seems simple, you don't get it. So every budget process, no head of engineering know, head of sales, no CFO, no GC, who's ever going to come back and say, "Oh, I've actually think next year we can just hire fewer people. Or we're going to keep it flat or we're going to shrink through attrition because we don't need any more people to do what we're doing." Not because they're evil, not because they're stupid, but it's almost overpowering impulse inside the organization that often leads to disastrous results. And so there's an... I'll give one example from Slack's history, and I have tried in the past to disguise this example so that no one feels bad about it but I... Unfortunately, the specifics are so important to the example that it's not disguised and so I'll just reiterate that the people involved aren't stupid or evil. And one example that's from the outside. So the example inside of Slack was we introduced threads, which was the ability to reply to a message inside of a channel. And let's say you, Lenny, post a message. I, Stewart reply to it. You will automatically get a notification. And now Sarah later on replies to the same message. Both you and I, as people who have push in that thread will receive a notification that there's been more activity, and so on. So like every single time anyone replies to it. So when the feature first was released or like when we did the final product review before it was released, the input box was pre-populated with at the person before you in the thread. And I was using the feature and I would put the insertion point there, select all delete, and then start writing my message. And even if I wanted to add someone specifically, I almost never wanted to start my sentence with at that, because it just made it hard to reference what they were saying before. So I said, "Get rid of this because, A, I think most people won't use it. Or if they did want to add someone, they're not going to want to do it at the beginning of the sentence. And by the way, you're teaching them to use the product wrong. Because it's important that everyone understand that every previous poster in this thread will automatically receive a notification unless they've figured it." So okay, we release it. Six months goes by and suddenly the at thing comes back. erstand that every previous poster in this thread will automatically receive a notification unless they've figured it." So okay, we release it. Six months goes by and suddenly the at thing comes back. And so I messaged someone around the team and I said, "Hey, there's been a regression. This is super weird. I don't know what happened. But the at thing came back." And they said, "Oh no, this is on purpose. We did a bunch of research." And so I was like, "What?" And I went through this and it was, if I recall correctly, it wasn't even P-95 certainty on this analysis. But it was something like when we do this, threads are 2.17 messages long, versus 2.14 messages long on average for when we don't do it. And so first of all, why is a longer thread better? Like maybe a shorter thread is better? It can be fewer messages that people have to go back and forth. Also, that's such a tiny difference. Also, again, I don't remember the actual statistical analysis, so I'm not going to claim that it was incorrect. But I'm pretty sure this was outside the bounds of certainty that they can have. But the real thing was, oh my God, so you guys put flags into the product, you A-B tested it. You did the instrumentation. You created tables in the database or whatever we're using to record all of that. You wrote queries to pull that. You created charts based on that data. You had meetings to discuss it. And just kind unpacking all of the things that would've had to happen for this to come back. And it's like thousands of person hours at a minimum, because any feature change at that scale of organization, it's involving like a dozen people. Engineering, QA, analytics teams, project managers, user research and stuff like that. The problem with that, so I think it was a bad idea, right? But the problem with that was the difference that you could possibly achieve between having this feature and not having this feature is like this much whatever units you want. The cost of doing the analysis was this much. So it's guaranteed to be a loser. Like there's just, there's no world in which anyone could imagine putting the at previous respondent in the thread at the beginning of the message could possibly make that much of a difference to the quality of Slack, and how much utility it provides for people and all of that. But you know that to put the feature flags in, to ship new versions of the product, to put the instrumentation in. To have it all the API calls to record every action that people take to do all the analytics, to create the dashboard. To put paste a screenshot of that into a Google Slides presentation. To send the invitations to the meeting, to reschedule the meeting because someone couldn't make it. To have everyone sit down and look at the thing. Like guaranteed loser. And I know that Fareed told you to ask me about this hyper realistic work-like activities. And so here's my grand theory. Hyper realistic work-like activities goes along with this other concept called known valuable work to do. And when I say known, I mean both you know what it is and you know that it's valuable.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.