Nathan Labenz: evaluation

Transcript context

…we can discuss licensing. Once you're hitting 10 million of those revenue, let's discuss licensing. It can be multi year deal that's extremely predictable for you, so you can manage the cost etcetera. And to me it's obviously why the big guys don't want to do it because they just like this model is like way, way less lucrative economically than to creating a toll road. But our claim is that toll road isn't going to be vile to alternative because if you're offering a different business model, that's more of the win win more and more people are going to switch there. I think like GLM recently is a great example of that, right? Like once you start edging the capabilities of closed models, then a lot of people suddenly start to think about costs. We asked what people actually do with an open world model that a closed API cannot offer. Yeah, just like in terms of parameter counts, like from like my understanding that the moment that the recent closed stuff that you saw and you're going to see is that the order of magnitude of like couple of hundreds of billions of parameters, right. I didn't hear about like a world model that hits like trillion parameters just yet. Maybe I don't think like the open source is going to be that behind. We're planning to release like Moe that's going to be also around like 100 to 200 billion parameters. So the gap, I think it's going to be on the scale of LLMS right, where you have, I don't know like 2-3 quarters behind, but I think in the world modestly we're coming back to adaptations. The range there is kind of wider than with LLMS. So give it just like a bunch of examples, right? Of I think like the first things that are come to mind are like indeed like VFX and animation on a specific IP. If you have like a specific franchise, but you have like a lot of data around it, that's a kind of a bunch of seasons, then fine tuning and like focusing all the capacity of the model on this specific IP is extremely beneficial. That works very well, almost to the point where for certain use cases, think about the keyframe animation, right? Like the animators still want to do the keyframes. That's the creative part. They actually don't want to outsource at all. But like so far in the PNL of animation, the in between was like this crazy expensive part. I think like the models of the, I don't know, 10/20/30 billion parameters that are fine-tuned for a specific task are like good enough. And then it's actually a matter of costs. Like another example, let's talk about, I don't know, like a lot of marketing and advertisement use cases, right? ed for a specific task are like good enough. And then it's actually a matter of costs. Like another example, let's talk about, I don't know, like a lot of marketing and advertisement use cases, right? Or for example, creating like UGC where you basically need like an avatar's models, right? That also like really doesn't require like a trillion parameters at some point. There again, it's all about efficiency. If you want to have your personal teacher, some kind of avatar, etcetera, you want, you don't want to pay sedan's 4K prices in order to do that. And you're going to require, I don't know, like hours a day of that. So I think like around a lot of use cases. Once you start like doing to fine tuning to specific domains, costs are becoming like very important because again, you're like clear of the bar of quality. And then once you do that, it's all about cost. There are like some more unusual cases of fine tuning that I saw. You know, like one example, think about like a field of computational photography, right? Where for example, you're taking, I don't know, like a data from sensors and trying to implement algorithm like denoising, right? Like you want to take videos and low light conditions, but then to create a clean video or for example, you want to create videos with higher dynamic range, right? Because like you, the sensor dynamic range is limited and then you're losing either some details in highlights or in shadows or for example, let's say you're taking a stream from a camera and want to simulate how it looks like with a different focal length. So there are like a ton of these use cases that you don't associate with generative models but actually kind of run like that. And some people address the problem exactly like that, right? They're taking like an existing data and it could be, for example, the footage that was taken from 2 cameras that are like really close but with different kind of focal lamps and then do the adaptation. And the adaptation is done on top of the model. So again, that's like a very big, like an unusual adaptation of the model. And like surprisingly typically you don't need like a crazy enough crazy amount of data for that. The more maybe like even more kind of surprising stuff that I saw is people who are adapting the models for doing all kinds of simulations that in the past required like a really expensive solvers, right. So think about like computational fluid dynamics, right, where you're trying to understand how like the water or like the smoke or something is moving. As we surprisingly saw that people are dumping these models to that they actually like solve the equations with precise solvers is like takes a ton of time and then use use it as an input to the model. And then the model can do a simulation like kind of fairly quickly. So again, like kind of circulating back to the question of fine tuning, I kind of feel that again, the range is higher than LLMS. Sometimes you don't need like a ton of data to do the adaptations and I think it's like stresses the point that why this model should be open, right? Like you do have a lot of different pockets of physical data. ou don't need like a ton of data to do the adaptations and I think it's like stresses the point that why this model should be open, right? Like you do have a lot of different pockets of physical data. You want to make sure that the model like really excels at that. Then a first for the show a question from Q the AI Co host Prakash build running on GPT 5.5 listening live and speaking for itself Zev one area that often gets less airtime is failure modes in the creative pipeline. Where do you see the biggest gap between what your tools can reliably deliver today and what creators assume they'll get? And how are you testing against that? OK, great question. Like the gap between what we basically promised and what we deliver. So listen guys, I'm I directly clear still some gaps. I think like the major one is physics, right? Like we try to capture with the model that I don't know, even if it's like 200 to 500 billion parameters, the entire physics of the universe of at least a pirate universe. As we know we are not there yet, but we're closing the gaps pretty quickly. I think I would say probably the most creators are still going to point at the fact that the simulation isn't as correct as as controllable as they want it to be, right. If you're talking to really creative people, they typically want to control like every nuance of the appearance and that requires to somehow decompose the models like a bunch of knobs, right? Like that you have like in the classical sort where you can say, yeah, here I want to have two more light in here. I want like the splash to be bigger. So achieving this control ability. I so besides, like the physics is also like one of the, I would say open things, right? Like we, we're getting cool things, not necessarily the things that creators wants. Exactly. And it, it is a pain point and Zeb's own pick for the most underrated variable edge compute. It's kind of funny, right, when we're having all these benchmarks and we're listening about like Erdos problems like being solved, etcetera. But guys, 99% of the use cases of lens are not around Erdos problems, right? And we're like spending a lot of electricity around it. So I think, but there are going to be like this orchestrators that are going to understand what you actually need and they're going to try to address it on the edge device and if not, then kind of go to the bigger model of the data centre. I think. Yeah, that's like one of the things that is being underpriced at the moment. How much of the compute will be able to move to edge devices and once people are going to start having like these local routers that will understand the complexity of the problem and then do these decisions for you. I think there's going to be a moment of reckoning for, you know, anthropic and open eye that at the moment doing these decisions for you, but not in your favour, right? So what is Q actually from Monday's show precaution, how we built a real time AI Co host and the diarization hack that makes it work? Tell me about it. What's the tech under the hood?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.