Yi Tay: evaluation

Transcript context

…the Okay, maybe like more old school transformers. Maybe we want to just talk about the Decision on encoder decoder versus decoder only. So I, okay, I won't be able to comment about like exactly our setup, but like, I think encoder decoder are kind of very misunderstood from thing, right? So there's encoder decoder, non causal decoder, which is a prefix LLM, and then there's a decoder only model, right? Technically, a causal decoder and a non causal decoder are very similar in the sense that it's just a bidirectional mask, right? And then a prefix LLM decoder has only The only difference is that Encoder Decoder splits the inputs and targets into different non shared transformer stacks. And then, like, there's encoder bottleneck in the end, right? So, technically, people, like, kind of always associate, like, Encoder I like BERT, or like something like, like, you know, people get confused about these things, right? But I think in the UL2 paper, we really, like, kind of explored this, and also, like, maybe some of the big science papers that also talk about this, right, is that prefix LLM and causal decoders are very similar, that's a must. At the end of the day, they're all autoregressive transformers. That's actually, like, the only big benefit of encoder decoders, it has this thing called, like, I mean, what I like to call, like, intrinsic sparsity. So basically, an encoder decoder with, like, n params is, like, basically, if it's, like, It has the cost of like an N over 2 decoder model. So it is a bit like a sparse model because you actually spend the same amount of flops. It's just that you have two sets of parameters, like, for encoder and decoder, right? So it's actually flop matched with a decoder model of, like, half the parameters. So like a, like UL220B is actually about A 10 B decoder only model. Right. So you get free sparsity from that. It's, it's something that, okay. The, the, the, the OG T five paper talks about this. You, you can look at it. There's this complex detail. I, I did, I didn't like, when doing the UR two paper, I kind of like was mind blown by like, like, wow, I could decode so much more not bounded by The causal mask anymore. A lot of the efficient transformers, like a lot of the sparse transformers, like, I mean, the old, early days, that's like, , Linformer and like, whatever, things like this, they cannot maintain the causal mask, and that's why you cannot train a proper language model with this, right? If you separate out your very long context into an encoder, this encoder has no loss. Right, you could just do like aggressive pooling, you could do some crazy sparse attention that has like, final transformer or something like that, right? And then you could make that smaller than the decoder, you could make that faster than the decoder, that are just some of the advantages of like, why, , splitting into encoder and decoder could be beneficial to, like, just using a decoder only model. At the end of the day, the decoder in Encode decoder is a language model. It's still a regular autoregressive language model. and decoder could be beneficial to, like, just using a decoder only model. At the end of the day, the decoder in Encode decoder is a language model. It's still a regular autoregressive language model. So that's actually, I mean, it's not that much different from, like, a retrieval augmented language model. This is news to me. I don't know if you've ever expressed this, but…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.