Raza Habib: evaluation

Transcript context

…And so this is where one of the hard questions is, right? Like, is there a graduation risk as people get very serious about logging? You brought up Datadog earlier and for sure Datadog is looking at your market as much as you're looking at theirs. So how do you, how do you think about that of, like, ultimately... At scale becomes a commodity, right, the logging. So I think that the that actually this is really different to that. So the more people use it, we find actually the stickier it becomes. It's almost the opposite, that as they get to scale. So, so you're right that the millionth feedback data point is worth a lot less than the thousandth feedback data point. But what continues to be really valuable is this infrastructure around the workflow. We see a process of prompt management, engineering, fixing things. So we see, you know, you have what happens over time is people put more and more evaluations onto Humanloop. They've got more people on their team, the product manager and also three linguists and someone else who are opening up, the data that's being logged through Humanloop back into that interactive environment they're rerunning things or plugging in other data sources. And so over time, actually the raw logs, I agree with you kind of come commoditized. But the tooling that's needed to be able to. So, of collect the data, but make it useful and do something with it to improve your model, that's the bit that becomes more valuable, right? Once you have something working at scale, then improving it by a few percentage points is like very, very impactful. So a lot of our customers early on. And Like, Oh, we can just dump our logs to like an S3 bucket or we can plug it, you know, and then like, why do we need a special purpose tool? And most of them come back to us later, because what they find is, oh, okay, I've logged something, but it's really difficult for me to like match up the log to like what model generated it and then quickly run that and try something else. Or I've like logged something and that log involves a retrieval. And I would like to know like what went wrong with the retrieval or where, which document the retrieval came from and I didn't log that information correctly, et cetera, et cetera. And like the complexity of like setting this up well is quite high. So you can either spend a lot of time at that point, at that stage, two things happen. Either people roll their own solution and early on we saw a lot of people build their own solutions. Or they come and use something like us. And I think increasingly, because we've been working on this for now more than a year the difference between something you would build yourself and, and sort of a bot solution is now quite enormous. Yeah. And so I just wouldn't recommend it. And the, I guess the difference on the Datadog point, or like other analytics tools, you mentioned Amplitude or Datadog, they're much more about passive monitoring. And I think one of the amazing things about AI is the interventions you can take can be very quick and very powerful. And so coupling very closely the ability to update a retrieval system or change a prompt to the analytics data and allowing you to run those experiments, I think is very powerful. Fantastic answer. It's almost like we we prepped for this.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.