Shawn Wang: belief

Transcript context

…cture actually makes it a much more powerful thing. And by the way, you can run an eval in your code base, save it to Braintrust, and then hit it with an API and just try out a new model, for example. That's more recent stuff nowadays, but early on just having the very simple declarative data structure that was just much easier to write than a for loop that you sort of had to cobble together yourself, and making it really fast, and then having a UI that just very quickly gives you the number of improvements or regressions and filter them, that was kind of the key thing that worked. I give a lot of credit to Brian from Zapier, who was our first user, and super harsh. I mean, he told me straight up, I know this is a problem, you seem smart, but I'm not convinced of the solution. And almost like Mr. Miyagi or something, I'd produce a demo and then he'd send me back and be like, eh, it's not good enough for me to show the team. And so we sort of iterated several times until he was pretty excited by the developer experience. That core developer experience was just more helpful enough and comforting enough for people that were new to evals that they were willing to try it out. And then we were just very aggressive about iterating with them. So people said, you know, I ran this eval, I'd like to be able to rerun the prompt. So we made that possible. Or I ran this eval, it's really hard for me to group by model and actually see which model did better and why. I ran these evals, one thing is slower than the other. How do I correlate that with token counts? That's actually really hard to do. It's annoying because you're often doing LLM as a judge and generating tokens by doing that too. And so you need to instrument the code to distinguish the tokens that are used for scoring from the tokens that are used for actually computing the thing. Now we're way out of the realm of what you can do with clod and sheets, right? In our case at least, once we got some very sophisticated early adopters of AI using the product, it was a no-brainer to just keep making the product better and better and better. I could just see that from the first week that people were using the product, that there was just a ton of depth here. There is a ton of depth. Sometimes it's not even just that the ideas are not worth anything. It's almost just the persistence and execution that I think you do very well. So whatever, kudos. We're about to zoom out a little bit to industry observations, but I want to spend time on Braintrust. Any other area of Braintrust or part of the Braintrust story that you think people should appreciate or which is personally insightful to you that you want to discuss it? There's probably two things I would point to. The first thing, actually there's one silly thing and then two maybe less silly things. So when we started, there were a bunch of things that people thought were stupid about Braintrust. One of them was this hybrid on-prem model that we have. And it's funny because Databricks has a really famous hybrid on-prem model and the CEO and others sort of have a mixed perspective on it. And sometimes you talk to Databricks people and they're like, this is the worst thing ever. But I think Databricks is doing pretty well and it's hard to know how successful they would have been without doing that. But because of that and Snowflake was doing really well at the time, everyone thought this hybrid thing was stupid. But I was talking to customers and Zapier was our first user and then Coda and Airtable quickly followed. And there was just no chance they would be able to use the product unless the data stayed in their cloud. Maybe they could a year from when we started or whatever, but I wanted to work with them now. And so it never felt like a question to me. I remember there's so many VCs…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.