Nathan Labenz: recommendation

Transcript context

…I think as most things are, it's multifaceted. And in the report, we theorize the cycle at play here. I do think it starts with bot sitting. I do think it starts with all of this manual work that is often a reflection of all of the AI that's being deployed in our organization and the pressure to adopt. Organizations are under massive pressure to adopt this technology and implement it. So you start to get more pressure to adopt. You have more. need to bot sit the technology, that is exhausting. It's exhausting because it takes up a lot of time, but it's also exhausting because you're not rewarded for it. And there's really not very much incentive in many organizations to bot sit well. And what we start to see is good enough. In the research, they sometimes call it satisficing. Once you see an AI output that is good enough, well, that's often permission to ship it. your case, you're an expert, you're skilled at your craft, right? You're recognizing that good enough isn't good enough. And so that's a big, I think the bot sitting is a precursor to the bot ********. It's the exhaustion. And employees hit a breaking point. They can no longer bot sit and they start to bot **** more and more. I think a big part of this is the lack of context. The lack of context in the AI tools, the fact that so many AI tools don't speak to one another. In so many organizations, employees want to use different models and different tools. And that's why I'm excited about Glean as a platform as well, because there's no situation in my mind that I can envision where we have a single model a single tool environment. Employees want that choice. We're seeing models leapfrog each other left and center. Employees want different models, different tools for different use cases. It becomes very complicated if you don't have that contextual layer to connect them and make sense of them. And not just in terms of how are the dots connected, but also in terms of recency. Knowing that this report is published this week, this month versus a report that was published 2 months ago, two years ago, that should be given different treatment. The authoritativeness of the content as well, very hard to discern in an enterprise context. So I think context is the big feeder as well from a technical standpoint. And then you have all these perverse incentives in organizations. The token maxing, the rewarding, the clicks of the tool, that is a big contributing factor as well. Yeah, I used to tell people, When I did any sort of AI advisory consulting, I would say you could do a lot worse than as a leader just watching your token consumption, but definitely don't tell the team that's how you're gonna be measuring them. So it's really super easy to cheat on that. It's amazing that's actually, that's honestly been probably years ago that I was saying that, and it's funny to see that people are still shooting themselves in the foot that way. I guess in terms of understanding my fellow human. I struggle a little bit with the idea that this bot sitting work is so onerous. My attitude, which I don't expect everybody to share, but I just will give it to you for compare and contrast, is like, I used to have to do stuff and now I get to have AI largely in many cases do this stuff for me. I still have to make a contribution. But I definitely get a lot more done a lot faster. And I also get to learn a lot more about AI and what it can do, which I find to be always a very interesting question unto itself. And just a lot more because I'm able to take on so many more different things. I'm able to learn much more broadly and satisfy my curiosity in all kinds of ways that I never could before AI. So if you combine that with a And I would say mine is even more. I was at this AI event recursive last weekend and the question of people in the audience was, if your team had to replace you plus AI as it exists today, how many of you unaided by AI would they have to hire to get the same output? The median answer in that room was like basically two. In other words, people thought that they were twice as productive thanks to AI as they would be if they were unaided. And that's pretty much where I put myself as well. So even though, leaving that aside, okay, so people are reporting 13 hours gained. There's another stat in the report that says, let me make sure I get it exactly right. People are spending 6.4 hours, basically half of that time savings on this bot sitting activity. reviewing outputs, connecting things, different products that don't connect. And I think anybody listening to this show certainly had that experience of, okay, I got a clawed prompt, I got a clawed report or plan or whatever, but now I got to put that into copy and paste. So the phrase you just use is the human becomes the integration layer. And I felt that, and it is certainly tedious, although it's honestly also just a part of general computer work, right? We've all got kind of Slack and then this other task tracker and then there's e-mail. And so it's all a little bit disjointed anyway. All that to say, I don't really get it. Like, why is it so bad to be responsible for babysitting the bots or bot sitting? What is it that's really bothering people so much or alienating them so much about that? So it's the taking away from the meaningful work. And in the report, we look at three categories of interaction with AI. One is the bot sitting. One is the using the technology. So using the technology to do real work, you prompt it and It gives you the answer or it asks you a follow-up question in a way that you're moving work forward, iterating with the technology, as opposed to you're asking it a prompt. It doesn't have the context, so you're re-prompting it. We're seeing about 36% of all AI sessions fail, meaning a worker goes to use the technology and it's not successful. They either have to start completely from scratch or do significant rework. Imagine if it was right on the first time. Imagine if you could put those 6.4 or hours into either using the technology to drive work forward or the third category, which is learning or building agents, the time savings would be significantly higher. And so I think, again, there's a small component of bot sitting that I think is healthy. And for people who are curious, this is less of a problem. And I love to believe in human curiosity. I think the reality for many, and I think Nathan, you and I are probably an outlier here, the reality is employees are too to be curious right now. They're too overwhelmed with work to spend those one, two hours tinkering with the technology and prompting 4 different. tools and picking the right answer. And that's the problem. The fact that it's not meaningful work and it's not meaningful learning with the technology. We also look at along the different dimensions of bot sitting, what is most exhausting. In the report, we call it the exhaustion multiplier. And what we see is the highest exhaustion multiplier is associated with feeding AI context, right? Because that is in the best case, something your AI should know. know where the documents are, which documents are authoritative, and you should not be supplying that as a human in most cases. The other one is the debugging. The debugging. see an output, it's wrong, but because of the nature of LLMs, you're not quite sure why it's broken, what's wrong. You try to tweak one thing because the nature of the technology is probabilistic, it's not deterministic, you're not really sure what you tweaked that worked and didn't. That is the biggest contributor to this exhaustion multiplier.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.