Kyle Kranen: evaluation

Transcript context

…Yeah. And that doesn’t mean like, you know, people aren’t taking a shot at this, like tinker from thinking machines, you know? Yeah. RL as a service. Yeah, totally. It’s, it also gets even harder when you try to do big model training, right? We’re not the best at training Moes, uh, when they’re pre-trained. Like we saw this with LAMA three, right? They’re trained in such a sparse way that meta knows there’s gonna be a bunch of inference done on these, right? They’ll open source it, but it’s very trained for what meta infrastructure wants, right? They wanna, they wanna inference it a lot. Now the question to basically think about is, okay, say you wanna serve a chat application, a coding copilot, right? You’re doing a layer of rl, you’re serving a model for X amount of people. Is it a chat model, a coding model? Dynamo, you know, back to that, it’s [00:33:00] like, yeah, sorry. So you we, we sort of like jumped off of, you know, jumped, uh, on that topic. Everyone has like, their own, own journey. And I, I like to think of it as defined by like, what is the model you need? What is the accuracy you need? Actually I talked to NA about this earlier. There’s three axes you care about. What is the quality that you’re able to produce? So like, are you accurate enough or can you complete the task with enough, performance, high enough performance. Yeah, yeah. Uh, there’s cost. Can you serve the model or serve your workflow? Because it’s not just the model anymore, it’s the workflow. It’s the multi turn with an agent cheaply enough. And then can you serve it fast enough? And we’re seeing all three of these, like, play out, like we saw, we saw new models from OpenAI that you know, are faster. You have like these new fast versions of models. You can change the amount of thinking to change the amount of quality, right? Produce more tokens, but at a higher cost in a, in a higher latency. And really like when you start this journey of like trying to figure out how you wanna host a model, you, you, you think about three things. What is the model I need to serve? How many times do I need to call it? What is the input sequence link was the, what does the workflow look like on top of it? What is the SLA, what is the latency SLA that I need to achieve? Because there’s usually some, this is usually like a constant, you, you know, the SLA that you need to hit and then like you try and find the lowest cost version that hits all of these constraints. Usually, you know, you, you start with those things and you say you, you kind of do like a bit of experimentation across some common configurations. You change the tensor parallel size, which is a form of parallelism I take, it goes even deeper first. Gotta think what model.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.