Kevin Weil: evaluation

Transcript context

…And so in your case, because this is really interesting, is that you're using different levels of Chat GPT, like a 1 0 3 and stuff that's earlier because it's cheaper. There'll be parts of our internal stack. I'll give you an example. Customer support, with 400 plus million weekly active users, we get a lot of inbound tickets. I don't know how many customer support folks we have, but it's not very many, 30, 40, I'm not sure, way smaller than you would have at any comparable company, and it's because we've automated a lot of our flows. We've got most questions using our internal resources, knowledge base, guidelines for how we answer questions, what kind of personality, et cetera. You can teach the model those things and then have it do a lot of its answers automatically, or where it doesn't have the full confidence to answer a particular question, it can still suggest an answer, request a human to look at it and then that human's answer actually is its own sort of fine tuning data for the model. You're telling it the right answer in a particular case. We're using... At various places. Some of these places, you want a little bit more reasoning, is not super latency sensitive, so you want a little more reasoning, and we'll use one of our O series models. In other places, you want a quick check on something and so you're fine to use four oh mini, which is super fast and super cheap. In general, it's like specific models for specific purposes and then you ensemble them together to solve problems. By the way, again, not unlike how we as humans solve problems, a company is arguably an ensemble of models that have all been fine tuned based on what we studied in college and what we have learned over the course of our careers. We've all been fine tuned to have different sets of skills and you group them together in different configurations and the output of the ensemble is much better than the output of any one individual. Kevin, you're blowing my mind. That sounds exactly correct. And also, different people, you pay them less, they cost less to talk to, some people take a long time to answer, some people hallucinating. This is...…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.