Alessio Fanelli: belief

Transcript context

…Okay, so to round out the Ops world. So, it's a three-circle Venn diagram, right? It's frameworks. It's gateways. So, the only job of a gateway is to just be one endpoint that proxies all the other endpoints, right? And it normalizes the APIs, mostly to OpenAI's API just because most people started OpenAI. And then, lastly, it's monitoring and tracing, right? So, logging those things, understanding the latency, like P99 or whatever, and the number of steps that you take. So, LangSmith is obviously very early on to this stuff. But so is LangFuse. So is... Oh, my God. There's so many. I'm sure Datadog has some. Weights and Biases has some. It's very hard for me to choose between all those things. So, I, as a small team developer, want one tool that does all these things. And my discovery has been that there's so much specialization here. Everyone is like, oh, yeah, we do this, but we don't do that. For the other stuff, we recommend these two other friends of ours. And I'm like, why am I integrating four tools when I just need one? They're all the same thing. That is my current frustration. The obvious frustration solution is I build my own, right? Which is... We have 14 standards, now we have 15. So, it's just a very messy place to be in. I wish there was a better solution to recommend to people because right now I cannot clearly recommend things. Yeah. I think the biggest change in this market is latency is actually not that important anymore. We lived in the past 10 years in a world where 10, 15, 20 milliseconds made a big difference. I think today people will be happy to trade 50 milliseconds to get higher quality output from a model. But still, all the tracing is all like, how long did it take? What's the thing? Instead of saying, is this quality good for this output? Like, should you use another model? We're just kind of taking what we did with cloud and putting it in LLMs instead of saying what actually matters when it comes to LLMs, what you should actually monitor. Like, I don't really care what my P99 is if the model is crap, right? Also, I don't own most of the models. So, it's like, this is the GPT-4 API performance. It's like, okay. Am I going into a moment? It's like, I can't do anything about it. So, I think that's maybe why the value is not there. Like, am I supposed to pay 100K a year? Like, I pay to Datadog or whatever to have you tell me that GPT-4 is slow? It's like, you know, and just not, I don't know. I agree, it's challenging there. Okay, so the last piece I'll mention is briefly, ML Ops is still real. I think LLM Ops or whatever you call this, AI Engineer Ops, the Ops layer on top of the LLM layer might follow the same evolution path as the ML Ops layer. And so, the most impressive thing I've seen from the ML Ops layer is from Apple. When they announced Apple Intelligence, they also announced Teleria, which is their internal ML Ops tool, where you can profile the performance of each layer of a transformer. And you can A-B test like 100 different variations of different quantizations and stuff and pick the best performance. And I could see a straight line from there to like, okay, I want this, but for my AI Engineering Ops, like, I want this level of clarity on like what I do. And there's a lot of internal engineering within these big companies who take their ML training very seriously. And I see that also happening for AI Engineering as well. And let's briefly talk about RAG and context caching maybe, unless you have other like LLM OS stuff that you're excited about.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.