Public evidence record
Published podcast speaker
Claim ledger
16 transcript-backed records
01 / recommendation
“You might have synchronization issues. And that's why I'm saying practically that you should take the minimum number of GPUs that you have because those are the easier cases to debug.”
02 / observation
“The problem is that in practice, full utilization, that's the key word, right? Because in practice, there are a lot of cases where like you're spending time waiting on data movement from like the GPU to CPU.”
03 / belief
“I think in my area of research, there are two things that I think people should really care about.”
04 / belief
“I would say flash attention. Guys, just talk to Tree. And flash attention is just sort of a really great set of kernels that I thought would take a while to get to us.”
05 / belief
“I would say the Atom optimizer is good at what it does. It's sort of a broad question.”
06 / belief
“Next up would be something like selective recomputation. So in selective recomputation, which Megatron has a good paper on, and I believe the figure that we have in our blog post is from, in that case, you sort of do a weighted decision for each activation.”
07 / belief
“I would say that a lot of major GPT-based models use this scheme. A lot of them now are sort of going with just a pure zero scheme.”
08 / belief
“Other than that, I think there was like a doc from Microsoft on like their Turing NLG.”
09 / uncertainty
“I don't know if I have a smell test myself, to be honest, like maybe I'll look at like what sort of flops, what you would expect on like an A100.”
10 / evaluation
“And we did a lot of experiments there. It's a challenging system to scale to because your interconnect across nodes is kind of slow in comparison to within a node, which I think we'll get to later.”
11 / uncertainty
“I think there's a lot of questions that are unanswered for fine-tuning. For example, we know scaling laws for training.”
12 / observation
“So once the hardware moved to V100, the software moved to like mixed precision and APEX and AMP and such. And one counterintuitive part of mixed precision is that you actually require more memory when you're trained because you need an FP16 copy of the weights and an FP32 copy of the weights.”
13 / recommendation
“I always feel like you should go with the minimum number of GPUs because the more number of GPUs that you have, the more likely it is for things to break.”
14 / evaluation
“I would say most things run there, not particularly efficiently, but you're going to have weird bugs that no one has encountered before.”
15 / recommendation
“I would say I do look at ways to improve upon it using things like second order optimizers.”
16 / preference
“I would say even if you had infinite VRAM, you would still want a quantized model, just a bigger model that's quantized is what I would say. And that's because like I was mentioning there at the end, how like deep learning is very stochastic and a lot, you could have all the precision in the world, but ultimately it's meaningless when you still depend so much like on what the input is.”