High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Dan Fu

Published podcast speaker

Claims
6
Episodes
1
Shows
1
Named items
0

Claim ledger

What Dan said.

6 transcript-backed records

01 / evaluation

It's a CUDA library. And one of the things that, that we found frustrating is every time that we built one of these new architectures, and I'm sure you had the exact same experience, we'd have to go and spend two months in CUDA land, like writing these, these new efficient things.

“It's a CUDA library. And one of the things that, that we found frustrating is every time that we built one of these new architectures, and I'm sure you had the exact same experience, we'd have to go and spend two months in CUDA land, like writing these, these new efficient things.”
Speaker
Dan Fu
Publisher
Latent Space

02 / evaluation

There's this NVIDIA and MIT put out this new diffusion model called SANA recently that one of their key key observations is that you can take a standard diffusion transformer diffusion model, replace the layers with linear [00:19:00] attention, and then that lets you scale to much larger much larger images, much, much Much larger sequences more efficiently. And and one thing that I don't think anybody would have called when a few years ago is that one of those gated SSM, gated states based models ended up on the cover of Science because a great group of folks went and trained some DNA models.

“There's this NVIDIA and MIT put out this new diffusion model called SANA recently that one of their key key observations is that you can take a standard diffusion transformer diffusion model, replace the layers with linear [00:19:00] attention, and then that lets you scale to much larger much larger images, much, much Much larger sequences more efficiently. And and one thing that I don't think anybody would have called when a few years ago is that one of those gated SSM, gated states based models ended up on the cover of Science because a great group of folks went and trained some DNA models.”
Speaker
Dan Fu
Publisher
Latent Space

03 / evaluation

Along those same lines, so afterwards we started putting out some work on specialized kernels, so just [00:12:00] like we have flash attention for transformers, we also have works like flash fft conf, and if you look at these lines of work oftentimes when, whenever you see a new architecture, you see a new primitive one of the, one of the table stakes now is, do you have an efficient kernel so that you can actually get wall clock speed up?

“Along those same lines, so afterwards we started putting out some work on specialized kernels, so just [00:12:00] like we have flash attention for transformers, we also have works like flash fft conf, and if you look at these lines of work oftentimes when, whenever you see a new architecture, you see a new primitive one of the, one of the table stakes now is, do you have an efficient kernel so that you can actually get wall clock speed up?”
Speaker
Dan Fu
Publisher
Latent Space

04 / prediction

rom here, if we take out this non linearity in the middle of the attention operation, and then if you compute the keys and the values operation first, you actually never hit this quadratic bottleneck.

“rom here, if we take out this non linearity in the middle of the attention operation, and then if you compute the keys and the values operation first, you actually never hit this quadratic bottleneck.”
Speaker
Dan Fu
Publisher
Latent Space

05 / commitment

I know I just gave a whole talk about it, but you know, what, what's the point of doing research if you can't, you know, play both sides. But I think one of the, so I think for both of us, the reason that we first got into this was just from the first principled questions of there's this quadratic thing.

“I know I just gave a whole talk about it, but you know, what, what's the point of doing research if you can't, you know, play both sides. But I think one of the, so I think for both of us, the reason that we first got into this was just from the first principled questions of there's this quadratic thing.”
Speaker
Dan Fu
Publisher
Latent Space

06 / commitment

Yeah, I think that if I were to turn back the clock to 2022, that's probably one of the things I would have done differently, which would have been actually get some long context benchmarks out at the same time as we started pushing context length on all these models.

“Yeah, I think that if I were to turn back the clock to 2022, that's probably one of the things I would have done differently, which would have been actually get some long context benchmarks out at the same time as we started pushing context length on all these models.”
Speaker
Dan Fu
Publisher
Latent Space
Search evidence