Chip Huyen: evaluation

Transcript context

…This is a good segue to, you talked about supervised learning versus unsupervised learning. I love, we're getting into this, by the way. This is super interesting. So you talk about labeled data. Basically, supervised learning is AI learning on data that somebody has already labeled and told it, here's correct versus incorrect. For example, this is spam versus not spam. This is a good short story. This is not a good short story. We've had the CEOs of a lot of these companies that do this for labs, Mercor and Scale, Handshake, there's Micro, there's a few others. So is that essentially what these companies are doing for labs, giving them labeled data, high-quality data to train on? It is in a way, but I think it's more like a product of big equations. So there are a lot more different components than that. So that's why I was talking about reinforcement learning. I'm not sure if your CEO [inaudible 00:16:09] interview bring up that term. So the idea is that once you [inaudible 00:16:14]... So let's say you have a model, give the model a prompt and it produce an output. You want to buy, once you reinforce, encourage the model to produce an output that is better. So now it comes to how do we know that the answer is good or bad? So usually, people relies on signals. So one way to get a first one good or bad is human feedback. They happen to be have two responses. You can, okay, this one one's better than the other. And we do that is because as humans, we tend to, it's very hard to give a concrete score, but it's easier to do comparisons. If you ask me, okay, give this song a score, I'm not a musician and don't know how hard it is. It's like yeah, I don't know what, out of 10 I going to remove six. And if you ask me again a month from now and I completely forgotten, okay, maybe now seven, only four, I don't know. But then if you ask me, okay, here are two songs and which one would you prefer to play for the birthday party? I was like, "Okay, I can prefer this song." So comparisons a lot easier. So [inaudible 00:17:18] have a human, you have human feedback and then you use this human feedback to treat a reward model to tell which and then the reward model help you like, okay, it's a model that produce this response. It's [inaudible 00:17:30] can score, is this good or bad? And you try to bias toward producing better model, the better responses. Another ways you can, instead of using a human, so you can use AI because the response and say good or bad, right? Or in fact the thing is that people are very big on nowadays, verifiable rewards, which it's natural. So basically, they give it a math problem and then math solutions is a model app a solution. Okay, it's expected response should be 42 and if it doesn't provide 42, then it's wrong. Now it's not a good response. So yes, a lot of time, people using this human laborer, human laborers should produce, how to say, expert questions and I say expected answers and in the ways that [inaudible 00:18:16] systems that verifiable so that the models can be trained on. Yeah. Okay, I'm really glad you went there. This is essentially RLHF reinforcement learning with human feedback, which is exactly what I wanted to also talk about, right?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.