Raza Habib: belief

Transcript context

…working on it. Put in quite a lot of work. So it says here human loop application platform includes a playground, monitoring, deployment, A B testing, prop manager, evaluation, data store, and fine tuning And based on our chat earlier, it seems like evaluation is kind of the more beta one. That's in sort of like a private beta That's correct. Yeah. So we have evaluation in private. There's, there's always been some aspect of evaluation. It was actually the first problem that we were solving for customers. But evaluation in human loop early on, early on was driven entirely by end user feedback. So so if you're building an LLM app, there's probably three different places where evaluation matters a lot. There's the evaluation that you need when you're Iterating in design, and you haven't got something in production yet. Mm-hmm. , but you just need feedback on as you're making changes, are you making things better? You're iterating on prompts, you're iterating on the context, trying out different models, like how do you know that the changes are actually improving things. Then once you're in production, there's sort of a form of evaluation you need for monitoring. Is this, you know, it seemed to work when I was in development, but now I'm putting a whole bunch of different customer inputs through it. Is it still performing the way that I expect it? And then the last one is something like equivalent to integration tests or something like this. Every time you make a change, how do you know you're not making it worse on the things that are already there? And so I think we always had a really good version of the monitoring user feedback version. But what we were missing was support for offline evaluation and being able to do evaluation during development or regression testing. And we're going to be launching something for that very soon. Yeah, this isn't slightly unintuitive to me because I just I would typically just assume they're all three are the same evals…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.