Alessio Fanelli: recommendation

Transcript context

…I think that the thing that we missed is the effective training of the large language models. So what do I mean by that? We've shoved all of the internet, basically all of the tokens we could into them. Obviously, OpenAI has done quite a bit of work probably to get rid of all of the toxic tokens and things like that, but it's still, it has been pretty brute force in the sense of how much data we fit. We were like, the more data, the larger, the better, and it's true, but the moment where you try to put it into an application. You're like, I don't need that thing that does math, physics, computer science, to like, tell me what color this car is. And we saw these very brutally on computer vision, like the model distillation. We started with ResNet 150s and then we, there were other models other than ResNets, but like the surprising fact over my time doing AI. Andresen is that ResNet 50 kept coming back, they would jump to VisionNet, Vision Transformers, and then they were like, oh, Vision Transformers, they don't train very well, they need tons of data, so annoying. So they would go back to ResNet 50, or like, they would try to use this other model, and then they would be like, oh, well, ResNet 50 worked out. Anyway, but that was for very constrained use cases, right? Maybe there is something interesting there for the end side of things, because maybe that means that we'll just keep going back to the model that worked. Yeah, keep going. I think that makes a lot of sense and we're still maybe in the, everybody wants something else that is not transformers, you know, uh, but maybe the, the lesson is to not, to not move away too much. Yeah, I mean, I haven't been doing super hardcore coding like I did three years ago to be in the field, but my impression when I would read the papers, I would ask like researchers at Google DeepMind and ask them, like, why did we choose this function? This function feels so arbitrary. It is because at the end of the day, it was computationally efficient, like multi head attention, the paper was like, Ooh, it trains well parallelly, as opposed to LSTMs. Right? And then that computational efficiency and ability that we had to shove more data was like the big. Big thing, uh, there, obviously there are major breakthroughs that happen. I don't want to invalidate that, but that was to me, like one of the things that got highlighted on that journey.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.