Ramin Hasani: observation

Transcript context

…Yeah, but let's zoom in maybe a little bit more on that neuron. And I guess I'm wondering a little bit too, what is the fundamental limit or what are the first bottlenecks that you hit when you try to scale the original liquid approach? With the sigmoid type functions that we have today, those are, I think, certainly carefully selected to be easily run on GPUs. I'm guessing that these sort of multiple degrees of freedom inside a single neuron maybe present challenges in terms of executing that process on available hardware. Like we can get these amazing results from 10s of neurons. So the obvious bitter pilled question would be, what if we take that exact paradigm and go to millions, billions? But what, you must be hitting some bottlenecks along the way. What are they? The main challenge is turning sequential computation into parallel computation. So when you have degree, like when you start like going from a single neuron dynamics to multiple neural dynamics, weight parameters of your system, instead of being scalars or vectors, they become matrices and tensors. So now you're talking about matrix multiplication. You want to turn the scalar computations into tensor computations, right? You want to be able to, this is how you parallelize kind of, let's say, sequential computers. The problem is that not all the time, like if you have non-linear relationships in an equation, you cannot trivially create a one-to-one map between a vectorized computation and a tensorized computation. So those non-linear kind of non-linear attributes of liquid neural networks and any other non-linear recurrent neural network, I mean, recurrence itself adds like some degrees of complexity into the math of the whole equation. I mean, let alone if the recurrence has a non-linearity on it, becomes like a lot more complex to disentangle vectors or write them with typical linear algebra kind of methods that we know into tensors. And you have to be able to turn these computations into tensors to be able to compute in parallel, like you compute them once at a time, you see, so that's kind of the mathematical it's not related only to any mathematical operation that you want to tensorize. If it is nonlinear, you're gonna have the troubles that we're talking about. That's why the state-space models that actually came out, these are all around linear dynamics. We're talking about linear dynamical systems, right? And why those linear dynamical systems are important is just pure fact that we do not have proper ways to scale nonlinear systems. You can apply non-linearity to a linear dynamical system. For example, let's say like a gated, or let's say you can add a sigmoidal function after you perform the dynamical system in a linear way. You compute all the matrices and everything in parallel, but then you can do a point-wise kind of application of a non-linear operation on top of the entire tensor. That's what you can do. But what if the relationship between the parameters themselves are governed governed by some non-linearity. So you cannot really use typical linear algebra. You gotta always approximate that non-linear system into a linear system, then you would be able to actually parallelize the systems, right? Does that make sense? So that's the fundamental bottleneck. So how much does the closed form solution address that? And how far has the... With available computing resources, how far has the original liquid network paradigm been able to scale so far to present?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.