Shawn Wang: belief

Transcript context

…you get. Yeah, so you're validating in similar ways, but if you're doing a really subjective task, then I think the only real ground truth is what your customers say is the right answer, right? If you're building copilot, like do the customers accept the code suggestions or not? Yes. That is the thing that ultimately matters. And you can only have proxies for that in development. And so that was why those two things end up being different. Yeah. And in terms of the quality of feedback so we did an episode with Amplitude, which, you know, is an analytics platform dedicated for collecting this kind of behavioral feedback. And you mentioned Copilot. There was a very famous post about reverse engineering co pilot that shows you the degree of feedback. I think typically when people implement these things, they implement it as a sort of thumbs up, thumbs down sort of binary feedback until you find that nobody uses those. Nobody does those feedback. Like I barely use the up down on chat GPT. Yeah. So this was something we learned really early on in building human loop. And you know, we, the feedback aspects of human loop were very Customer driven. People were, the people who are getting amongst our early users, the people who are getting traction and who had built something that was working well, had jerry rigged some version of feedback, collection and improvement themselves. And they were pushing for something better here. And they all. We're collecting usually three types of feedback, and HumanLoop supports all three of these. So you have the thumbs up, thumbs down type feedback that you just described. You don't get much of it. It's useful when you get it, but you don't get that much. And then the other form of feedback, so we call that votes. And then you have actions, and these are like the implicit signals of user feedback. So I can give a concrete example here. There's a company I really like called Sudowrite. And Sudowrite, founded by James Yu, and they're building editor experience for fiction writers that helps them. So as they're writing their stories or novels, there's a sidebar and you can highlight text and you can say, like, help me come up with a different way of saying this or in a more evocative way, you know, there's many different features built in. And they had built in early on. You know, analytics around does the user accept a suggestion? Do they refresh and regenerate multiple times? How much do they edit the suggestion before including it in their story? Do they then share that? And all of those implicit signals correlate really well with the quality of the model or the prompt. And they were like running experiments all the time to make these better. And you could just see it in their traction figures. As they figured out the right prompt, the right features, the things that people were actually including, the product became much more loved by their users.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.