Speaker unverified: belief

Transcript context

…That makes sense. One of the architecture questions that I forgot to mention in there was, so just the architecture choice of like a very big, you know, 400B dense model, I actually honestly thought that maybe 175 or like, you know, was kind of the peak, you know, whatever can fit on like an H100. So basically I think the common question that people have is like, why no MoE? In a way that Mistral and the others have gone and, you know, it seems like the trend has been MOEs and you guys have bucked the trend there. I heard that question a lot, different aspects there. Why notMoEin the future? The other thing is, I think a dense model is just one specific variation of the model for an hyperparameter for anMoEwith basically one expert. So it's just an hyperparameter we haven't optimized a lot yet, but we have some stuff ongoing and that's an hyperparameter we'll explore in the future. Let's make sure we run through everything on post-training. You also had a recent tweet about RLHF versus imitation learning explained in one tweet. So we'll put this in the show notes, but it's basically like two charts about a doctor opinions. On one side, there's like whether or not the suggestion is good from like a content perspective and the chatbots rank really highly and the physicians are kind of like, you know, a bell curve as you might imagine. But then the empathetic voting, most physicians are rated not empathetic or slightly empathetic versus all the model responses are rated very empathetic and empathetic at worst. You know, most people might look at it and not really get much from it, but obviously it resonated with you. Can you run people through like some of the choices you make in post-training to like optimize for one of the two and getting the best responses?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.