Dylan Patel: evaluation

Transcript context

…We’ll talk about different benchmarks and so on but some has just a vibe. That in itself is a, let’s say quote “fire” tweet. If I’m trying to produce something where people are like, “Oh, shit.” Okay, so that’s a chance probably return to it more. How were they able to achieve such low cost on the training and the inference? Maybe you could talk to the training first. There’s two main techniques that they implemented that are probably the majority of their efficiency, and then there’s a lot of implementation details that maybe we’ll gloss over or get into later that contribute to it. But those two main things are, one is they went to a mixture of experts model, which we’ll define in a second. And then the other thing is that they invented this new technique called MLA, latent attention. Both of these are big deals. Mixture of experts is something that’s been in the literature for a handful of years. And OpenAI with GPT-4 was the first one to productize a mixture of experts model. And what this means is when you look at the common models around that most people have been able to interact with that are open, think Llama. Llama is a dense model i.e. every single parameter or neuron is activated as you’re going through the model for every single token you generate. Now, with a mixture of experts model, you don’t do that. How does the human actually work? Is like, oh, well my visual cortex is active when I’m thinking about vision tasks and other things. My amygdala is when I’m scared. These different aspects of your brain are focused on different things. A mixture of experts, models attempts to approximate this to some extent. It’s nowhere close to what a brain architecture is, but different portions of the model activate. You’ll have a set number of experts in the model and a set number that are activated each time. And this dramatically reduces both your training and inference costs because now if you think about the parameter count as the total embedding space for all of this knowledge that you’re compressing down during training, one, you’re embedding this data in instead of having to activate every single parameter, every single time you’re training or running inference, now you can just activate on a subset and the model will learn which expert to route to for different tasks. And so this is a humongous innovation in terms of, hey, I can continue to grow the total embedding space of parameters. And so DeepSeek’s model is 600 something billion parameters, relative to Llama 405B, it’s 405 billion parameters, relative to Llama 70B, it’s 70 billion parameters. This model technically has more embedding space for information to compress all of the world’s knowledge that’s on the internet down. But at the same time, it is only activating around 37 billion of the parameters, so only 37 billion of these parameters actually need to be computed every single time you’re training data or inferencing data out of it. Versus again, the Llama model, 70 billion parameters must be activated or 405 billion parameters must be activated, so you’ve dramatically reduced your compute cost when you’re doing training and inference with this mixture of experts architecture. Should we break down where it actually applies and go into the transformer? Is that useful?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.