Nathan Labenz: evaluation

Transcript context

…So Newsbench was an evaluation that we built to look at how AI systems, particularly the leading chatbots, respond to questions about the news. And what we did was we looked at three things. We said, after talking to experts, we defined good as accuracy, getting the facts right, neutrality, so is it leaning in one direction or the other, and source quality, right, when they have to ground themselves on external sources, which of course you have to do quite a bit with with news, are those sources reliable? We looked at GPT, Claude, Grock, and Gemini, and I would say the findings certainly wasn't all bad, and there were some positive signals. We actually saw generally models improved as we tracked them version over version. So Opus 4.6 to 4.8, was a pretty significant improvement in bias, which also is consistent with Anthropic's reporting. But Fable was actually a big regression, interestingly, which I think is just an interesting point, which shows that more power doesn't, like when you're dealing with these sort of more subjective nuanced things, more power doesn't always mean better. We looked at, you know, per model, about 2,500 responses each. About 1/3 of them had a factual error. in that what could be a wrong number, a wrong date, a misattributed quote, a misstated policy, which I think was, I think we knew that was an issue, but I think that number was quite a bit higher than even we expected going into it. I'm sourcing in about 15%, like one in seven responses sourced foreign state media, like RT from Russia, So China Daily. And what's really interesting is oftentimes this wasn't even in questions about their home country. So we saw Russia, like RT and China Daily sourcing questions about US domestic politics. And so that was, I think, a particularly interesting finding that we saw across the board with all the models. From measuring AI to betting the whole company on it, our last guest, Eric Vaughn, CEO of Ignite Tech. Back in 2023, he called generative AI an existential threat, put his entire org on AI one day a week, and when most of his people pushed back, he replaced them, rebuilding around what he calls AI DNA. We started with one of his recent acquisitions, a company called Chorus. Chorus is a large acquisition. I mean, we don't release numbers, but it was a nine-digit revenue company. I don't think we could have done it without the AI DNA forward approach that we were able to take. So just for instance, there were hundreds of employees in eight different countries, and we needed to very quickly understand who they were, what they did, what they knew. how they were thinking about everything about the business. And we quickly developed an AI interviewer that asked questions before there was a human one-on-one. Before we would have set up human one-on-ones and people would have gone through 30-minute drills of hundreds of people over the next several weeks without any data. Now we all arrive with a dossier, you know, the interview process that just shortcut that and let us get familiar and categorize and understand who knew what. And that was one example. Another example was being able to communicate to the customers in a way that was timely. In our AI creation world, we wrote a new software product that's available on the market. It's a product called Eloquence AI. And Eloquence AI is an e-mail AI persona. It answers every e-mail it gets in 5 minutes or less, always in perfect grammar in 160 languages, always empathetic, doesn't miss anything, and knows when it needs to escalate to a human, and it just adds them on CC. With eloquence, we always were able to respond timely, and especially when it came to HR, empathetically and with detail, right? So that's just two examples. of how we were able to really fuel that acquisition. And the company was losing money when we bought it. The company is now profitable. We have completely transformed the operation of the company. And in one year's time, we've released 2 brand new versions of Corus' software that are fully AI enabled. And in one case, it was the rewrite of 15 years of code that we discarded and rewrote from the ground up. or one of the two products that they had. We did that in a year in the middle of all of that transition. That's not a story I could have told five years ago. It's just not possible. It just wouldn't have happened.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.