Public evidence record
Published podcast speaker
Claim ledger
6 transcript-backed records
01 / belief
“I think in my area of research, there are two things that I think people should really care about.”
02 / belief
“I would say flash attention. Guys, just talk to Tree. And flash attention is just sort of a really great set of kernels that I thought would take a while to get to us.”
03 / belief
“I would say the Atom optimizer is good at what it does. It's sort of a broad question.”
04 / belief
“Next up would be something like selective recomputation. So in selective recomputation, which Megatron has a good paper on, and I believe the figure that we have in our blog post is from, in that case, you sort of do a weighted decision for each activation.”
05 / belief
“I would say that a lot of major GPT-based models use this scheme. A lot of them now are sort of going with just a pure zero scheme.”
06 / belief
“Other than that, I think there was like a doc from Microsoft on like their Turing NLG.”