Ankur Goyal: recommendation

Transcript context

…and compare it. The main thing I was impressed by was that you can run all these things in parallel so quickly. Yeah, exactly. So I had built spreadsheet plus plus a few times. And there were a couple nuggets that I realized early on. One is that it's very important to have a history of the evals that you've run and make it easy to share them and publish in Slack channel, stuff like that, because that becomes a reference point for you to have discussions among a team. So at Impira, when we were first ironing out our layout LM usage, we would publish screenshots of the evals in a Slack channel and go back to those screenshots and riff on ideas from a week ago that maybe we abandoned. And having the history is just really important for collaboration. And then the other thing is that writing for loops is quite hard. Like, writing the right for loop that parallelizes things is durable, someone doesn't screw up the next time they write it, you know, all this other stuff. It sounds really simple, but it's actually not. And we sort of pioneered this syntax where instead of writing a for loop to do an eval, you just create something called eval, and you give it an argument which has some data, then you give it a task function, which is some function that takes some input and returns some output. Presumably it calls an LLM, nowadays it might be an agent, you know, it does whatever you want, and then one or more scoring functions. And then Braintrust basically takes that specification of an eval and then runs it as efficiently and seamlessly as possible. And there's a number of benefits to that. The first is that we can make things really fast, and I think speed is a superpower. Early on we did stuff like cache things really well, parallelize things, async Python is really hard to use, so we made it easy to use. We made exactly the same interface in TypeScript and Python, so teams that were sort of navigating the two realities could easily move back and forth between them. And now what's become possible, because this data structure is totally declarative, an eval is actually not just a code construct, but it's actually a piece of data. So when you run an eval in Braintrust now, you can actually optionally bundle the eval and then send it. And as you saw in the demo, you can run code functions and stuff. Well, you can actually do that with the evals that you write in your code. So all the scoring functions become functions in Braintrust. The task function becomes something you can actually interactively play with and debug in the UI. So turning it into this data structure actually makes it a much more powerful thing. And by the way, you can run an eval in your code base, save it to Braintrust, and then hit it with an API and just try out a new model, for example. cture actually makes it a much more powerful thing. And by the way, you can run an eval in your code base, save it to Braintrust, and then hit it with an API and just try out a new model, for example. That's more recent stuff nowadays, but early on just having the very simple declarative data structure that was just much easier to write than a for loop that you sort of had to cobble together yourself, and making it really fast, and then having a UI that just very quickly gives you the number of improvements or regressions and filter them, that was kind of the key thing that worked. I give a lot of credit to Brian from Zapier, who was our first user, and super harsh. I mean, he told me straight up, I know this is a problem, you seem smart, but I'm not convinced of the solution. And almost like Mr. Miyagi or something, I'd produce a demo and then he'd send me back and be like, eh, it's not good enough for me to show the team. And so we sort of iterated several times until he was pretty excited by the developer experience. That core developer experience was just more helpful enough and comforting enough for people that were new to evals that they were willing to try it out. And then we were just very aggressive about iterating with them. So people said, you know, I ran this eval, I'd like to be able to rerun the prompt. So we made that possible. Or I ran this eval, it's really hard for me to group by model and actually see which model did better and why. I ran these evals, one thing is slower than the other. How do I correlate that with token counts? That's actually really hard to do. It's annoying because you're often doing LLM as a judge and generating tokens by doing that too. And so you need to instrument the code to distinguish the tokens that are used for scoring from the tokens that are used for actually computing the thing. Now we're way out of the realm of what you can do with clod and sheets, right? In our case at least, once we got some very sophisticated early adopters of AI using the product, it was a no-brainer to just keep making the product better and better and better. I could just see that from the first week that people were using the product,…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.