Shawn Wang: belief

Transcript context

…Yeah, yeah, yeah. So we were sitting around. Like, we've, you know, for a long period of time, we had 500 A100s, because we made a commitment and they were constantly being delayed, I think because of H100 supply, demand, whatever reasons. And it was also very hard to get a lot of compute in one place, right? And then we were locked in, and we had to wait for the compute to come, right? So, I think, It was very painful because even when the compute came, it was mostly broken most of the time. And it was broken to a very bad extent that, you know, before I left Google I was like, even in the early stage I was very optimistic about You Okay, this compute translates to this amount of flops, this is the model, right? But I never expected the reliability to be so poor that it just threw off all the calculations and then we had to, work ten times harder just to make the thing go smoothly. So, it was a bearable pain. I think the pain was bearable, but it was just way, way more than expected. I think you addressed this in your post, but the temptation would have been just to run everything on TPUs. Which is the stack that you already know very well. That that works very well. Oh, no, no. So, so, so TPUs outside Google and t inside Google are probably very different things, I think. Oh, how come? Okay. First thing is like infrastructure. Like, there was, there wasn't like a lot of good code bases like outside Google that was like still, right. And, and the code base that I was most familiar with was like T five X. It was a jack space. It would have been like, by, by the time we wanted to consider it, it was really like. Debrigaded for nine months, right? And then, TPUs I mean, we weren't sure about I mean, the availability of TPUs was not great, great.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.