Speaker unverified: belief

Transcript context

…Yeah. First, I think shout out to Ciara, which is this very good startup, which was founded by Brad Taylor and Clay Barber. And Ciara is a startup doing conversational AI. So what they do is they they build agents for businesses. Like suppose you have a business and you have a customer service. We want to automate that part. And then it becomes very interesting because it's very different from coding a web agent or whatever people are doing, because it's more about how can you do simple things reliably? It's not about, you know, can you sample a hundred times and you find one good mass proof or kill solution. It's more about you chat with a hundred different users on very simple things. Can you be robust to solve like 99% of the time, right? And then we find there's no really good benchmark around this. So that's one thing. I guess another thing is obviously this kind of customer service kind of domain. Previously, there are some benchmarks, but they all have their limitations. And I think you want the task to be kind of hard and you want user simulation to be real. We don't have that until LLM. So data sets from 10 years ago, like either just have trajectories conversating with humans or they have very fake kind of simulators. I think right now it's a good opportunity to, if you really just care about this task of customer service, then it's a good opportunity because now you have LLMs to simulate humans. But I think a more general motivation is we don't have enough agent benchmarks that target this kind of robustness, reliability kind of standpoint. It's more about, you know, code or web. So this is a very good addition to the landscape. If you have a model that can simulate the persona, like the user the right way, shouldn't the model also be able to accomplish the task, right? If he has the knowledge of like what the person will want, then it means...…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.