Evidence receipt / uncertainty
Published · transcript-backedShawn Wang: uncertainty
20 Jun 2023 Latent Space Commoditizing the Petaflop — with George Hotz of the tiny corp
“There's been a whole round trip with positional embeddings. I don't know if you've seen this discussion.”
Source trail
Everything needed to verify it.
- Speaker
- Shawn Wang
- Attribution
- Verified speaker
- Claim type
- uncertainty
- Recorded
- 20 Jun 2023
- Publisher
- Latent Space
Transcript context
…Yeah, well, FlashAttention, yeah. And FlashAttention is the same compute. FlashAttention is an interesting fact where it's actually the identical compute. It's just a more efficient way to do the compute. But I'm even talking about like, look at the new embeddings people are using, right? They used to use these like boring old embeddings. Now, like, Lama uses that complex one, and now there's like Alibi. I'm not up-to-date on all the latest stuff, but those tricks give you so much. There's been a whole round trip with positional embeddings. I don't know if you've seen this discussion. I haven't followed exactly. I mean, you quickly run into the obvious problem with positional embeddings, which is you have to invalidate your KV cache if you run off the context. So that's why I think these new ones,…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.