Raza Habib: belief

Transcript context

…excellent. So what is HumanLoop today? Yeah, so at its core, we help engineers to measure and optimize LLM applications. So, in particular, helping them do prompt engineering, management, and then evaluation. So evaluation is particularly difficult for large language models because they tend to be used for much more subjective applications than traditional machine learning. Definitely than traditional software, right? If you're coming from a pure software, non ML background, then the first thing you have to learn when you start working with LLMs is this stuff is stochastic which I think, you know, most people are not used to. So just playing with software that every time you run it, it's different and you can't just write unit tests. Yeah. It's the first kind of painful lesson. But then it turns out that a big piece of these applications ends up in prompts, and these are natural language instructions, but they're having similar impact to what code has. So they need to be treated with the importance of code. And so iterating on that, managing it, versioning it, evaluating it, that, those are the problems that HumanLoop helps engineers with today. And in particular, we tend to be focused on companies that are at a certain scale, because one of the challenges that, one, they tend to care more about evaluation. I think if you're a two person startup, you sort of build something quick MVP it into production. But larger companies need to have some confidence of the product experience before they launch something. And also what we've found is that there's a lot more collaboration between engineers and non engineers, between product managers and domain experts who are involved in the design, the prompt engineering, the evaluation. But are maybe not the engineering part, and they have to work together nicely, so giving them the right tools has been a really important part as well. Yeah. Something I've often talked about with other startups in this space is who's the buyer. Yeah. Because you talked about collaboration between the engineer and the PM, or... whoever. And it's not clear sometimes. Do you have a clear answer? It…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.