Shawn Wang: recommendation

Transcript context

…But please don't come on the podcast to Noam Shazir. Sorry, we didn't mean. No, no, no. Because like, I don't really understand like what the use case is for, apart from like the therapy, role play, homework assistant type of stuff that is the norm. But anyway, one of the most interesting things, so he detailed five tricks. One thing that people talk a lot about is native int8 training. I got it wrong in our Thomas podcast. I said fp8 is int8. And I think that is something that is an easy win. We should basically, when we're getting to the point where we're over-training models 100 times past Chinchilla ratio to optimize for inference, the next thing is actually like, hey, let's stop using so much memory when training because we're going to quantize it anyway for inference. So let's pre-quantize it in training. So that makes a lot of sense. The other thing as well is this concept of global, local, hybrid architecture, which I think is basically going to be the norm, right? So he has this formula of one to five ratio of global attention to local attention. And he says that that works for the long form conversations that character has. Okay, that's great. And like simultaneously, we have independence research from other companies about similar hybrid ratios being the best for their research. So Nvidia came out with a Mamba transformer hybrid research thing. And in their estimation, you only need 7% transformers. Everything else can be state-space models. Jamba also had something like between like six to like 30 to one. And basically every form of hybrid architecture seems to be working at the research stage. So I think like if we scale this, it makes complete sense that you just need a mix of architectures It could well be that the transformer block, instead of transformers being all you need, transformers are the global attention thing. And then the local attention thing can be the state-space models, can be the RWKVs, can be another transformer, but just limited by its lighting window. And I think like we're slowly discovering like the fundamental building blocks of AI. One is transformers, one is something that's local, whatever that is. And then, you know, who knows what else is next? I mean, the other stuff is adapters but we can talk about that. But yeah, headline is that Noam, maybe he's too confident, but I mean, I believe him. Noam thinks that he can do inference at 13x cheaper than the Fireworks together, right? So like there is a lot of room left to improve inference. I mean, it does make sense, right? Because like otherwise, I don't know. Yeah, exactly. I was like, they will be losing a ton of money.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.