Evidence receipt / recommendation
Published · transcript-backedQuentin Anthony: recommendation
16 Aug 2023 Latent Space The Mathematics of Training LLMs — with Quentin Anthony of Eleuther AI
“You might have synchronization issues. And that's why I'm saying practically that you should take the minimum number of GPUs that you have because those are the easier cases to debug.”
Source trail
Everything needed to verify it.
- Speaker
- Quentin Anthony
- Attribution
- Verified speaker
- Claim type
- recommendation
- Recorded
- 16 Aug 2023
- Publisher
- Latent Space
Transcript context
…time. So you kind of have to tune it. In both cases, getting back to like the practical case, you should be able to get linear speed up by adding more GPUs. The problem is that there are hardware failures. You tend to have problems with like maybe loss will overflow if you have too many GPUs or maybe one GPU will hang. You might have software issues. You might have synchronization issues. And that's why I'm saying practically that you should take the minimum number of GPUs that you have because those are the easier cases to debug. That make sense? Yeah.…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.