Reiner Pope: belief

Transcript context

…And when you say the best chips for LLMs... I can think of multiple ways to measure best. It could be the best performance per watt. It could be the lowest latency, capable of handling the largest models. What is best? In general, there are two metrics which LLM workloads care about, which is throughput, which is really just an economics thing. I buy a chip for $30,000, and then can I do 10,000 tokens a second or 100,000 tokens per second of throughput? That determines the dollars per token. Throughput and then latency, how fast does a thing respond? As I see the market, the economics seems to be most important. Ultimately, the quality of the AI you can train and serve is constrained by, "I have only a $10 billion budget, and I want to train and serve the best model I can on that budget." If I can have more tokens per dollar, then I can get a better quality out. The product we aim to build is far ahead on throughput, but then, actually, the surprising thing is we're competitive with the best on latency as well. I think that is a unique thing in offering both in the same place. Is this for… Obviously in AI, there's training the models and then running the models' inference. Is this most interesting for inference, or is there any training angle? Incidentally, is it useful for trading, but you are trying to win inference, is that how you think about it?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.