Speaker unverified: evaluation

Transcript context

…All right. Yeah. Let me answer those separately. I'll start with structured outputs. So for the most part, when I'm doing prompting tasks and rolling my own, I don't build a framework. I just use the API and build code around it. And my reasons for that, it's often quicker for my task. There's a lot of invisible prompts at work and a lot of these frameworks, I hate that. So like you'll have this function summarizes input, but if you look behind the scenes, it's using some special summarization instruction. And if you don't have visibility on that, you can get confused by the outputs and also for research papers, you need to be able to say, oh, this is how I did that task. And if you don't know that, then you're going to be misleading other researchers. It's not reproducible. It's a whole mess. But when it comes to structured output prompting, I'm actually really excited about that OpenAI release. I have a project right now that I hope to use it on. Funnily enough, when the same day that came out, another, or a paper came out that said, when you force the model to structure its outputs, the performance, the accuracy, creativity is lessened. And that was really interesting. That wasn't something I would have thought about at all. And I guess it remains to be seen how the OpenAI structured output functionality affects that because maybe they've trained their models in a certain way where it's just not a problem. So that's, those are my opinions there. And then on the eval side, this is also very important. I saw last year, I saw this demo of a medical chatbot, which was deployed at like to real patients and it was categorizing patient need. So patients would message the doctor and say, Hey, like this is what's happening to me right now. Like, can you give me any advice? A doctor only have a limited amount of time. So this model would automatically score the need as like, they really need help right now or no, this can wait till later. And the way that they were doing the measurement was prompting the model to evaluate it and then taking like the logits values output according to like which token has a higher probability basically. And they were also doing, I think a sort of one through five scoring where they're prompting saying or maybe it was zero to one, like output a score from zero to one, one being the worst, zero being not so bad about how bad this message is. And these methods are super problematic because there is an incredible amount of instability in them in the sense that models are biased towards outputting certain numbers. And you generally shouldn't say things like output your result as a number on a scale of one through 10 because the model doesn't have a good frame of reference for what those numbers mean. So a better way of doing this is say, Oh, output on a scale of one through five, where one means completely fine, two means possible room for emergency, three means significant room for emergency, et cetera. tter way of doing this is say, Oh, output on a scale of one through five, where one means completely fine, two means possible room for emergency, three means significant room for emergency, et cetera. So you really want to assign, make sure you assign meaning to the numbers. And there's other approaches like taking the probability of an output sequence and using that to actually evaluate the, I guess these are the log props, actually evaluate the probability. That has also been shown to be problematic. There's a couple of papers that directly analyze the technique and show it doesn't work in a lot of cases. So when you're doing these sort of evals, especially in sensitive domains like medical, you need to be robust in evaluation of your own evaluation system. Endorse all that. And I think getting things into structured output and doing those scoring is a very core part of AI engineering that we don't talk about enough. But so I wanted to make sure that we give you space to talk about it.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.