Shawn Wang: belief

Transcript context

…I think the biggest change in this market is latency is actually not that important anymore. We lived in the past 10 years in a world where 10, 15, 20 milliseconds made a big difference. I think today people will be happy to trade 50 milliseconds to get higher quality output from a model. But still, all the tracing is all like, how long did it take? What's the thing? Instead of saying, is this quality good for this output? Like, should you use another model? We're just kind of taking what we did with cloud and putting it in LLMs instead of saying what actually matters when it comes to LLMs, what you should actually monitor. Like, I don't really care what my P99 is if the model is crap, right? Also, I don't own most of the models. So, it's like, this is the GPT-4 API performance. It's like, okay. Am I going into a moment? It's like, I can't do anything about it. So, I think that's maybe why the value is not there. Like, am I supposed to pay 100K a year? Like, I pay to Datadog or whatever to have you tell me that GPT-4 is slow? It's like, you know, and just not, I don't know. I agree, it's challenging there. Okay, so the last piece I'll mention is briefly, ML Ops is still real. I think LLM Ops or whatever you call this, AI Engineer Ops, the Ops layer on top of the LLM layer might follow the same evolution path as the ML Ops layer. And so, the most impressive thing I've seen from the ML Ops layer is from Apple. When they announced Apple Intelligence, they also announced Teleria, which is their internal ML Ops tool, where you can profile the performance of each layer of a transformer. And you can A-B test like 100 different variations of different quantizations and stuff and pick the best performance. And I could see a straight line from there to like, okay, I want this, but for my AI Engineering Ops, like, I want this level of clarity on like what I do. And there's a lot of internal engineering within these big companies who take their ML training very seriously. And I see that also happening for AI Engineering as well. And let's briefly talk about RAG and context caching maybe, unless you have other like LLM OS stuff that you're excited about. LLM OS stuff I'm excited about. No, I think that's really a lot of it. It's like move beyond being observability or like help for like making the prompt call and like actually being an LLM OS, you know? I think today it's mostly like LLM Rails, you know? Like there's no OS, but I think like actually helping people build things. That's why, you know, if you look at XLA-A2B, it's like, that's the OS, you know? Those are kind of like the OS primitives that you need around it.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.