Evidence receipt / belief
Published · transcript-backedQuentin Anthony: belief
16 Aug 2023 Latent Space The Mathematics of Training LLMs — with Quentin Anthony of Eleuther AI
“I would say the Atom optimizer is good at what it does. It's sort of a broad question.”
Source trail
Everything needed to verify it.
- Speaker
- Quentin Anthony
- Attribution
- Verified speaker
- Claim type
- belief
- Recorded
- 16 Aug 2023
- Publisher
- Latent Space
Transcript context
…We'll try to get that RWKV episode out. It seems like there's increasing mentions of it and they are doing pretty important work as far as scaling these models are concerned. Okay. So we discussed inference type quantization and memory requirements. And then you also had a section on training with a lot of stuff I think mentioned. I think we probably want to spend the most of our time on optimizer states and the Atom optimizer. Yeah. What are your takes on it and what should people keep in mind when they deal with these optimizers? Okay. I would say the Atom optimizer is good at what it does. It's sort of a broad question. So let me think. You have the copy of the weights and then you have your momentum and your variance that you store.…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.