Evidence receipt / belief
Published · transcript-backedAnkur Goyal: belief
11 Oct 2024 Latent Space Production AI Engineering starts with Evals — with Ankur Goyal of Braintrust
“At Impira, while we were having an existential revelation, if you will, we realized that the debates we were having about what model and this and that were really hard to actually prove anything with.”
Source trail
Everything needed to verify it.
- Speaker
- Ankur Goyal
- Attribution
- Verified speaker
- Claim type
- belief
- Recorded
- 11 Oct 2024
- Publisher
- Latent Space
Transcript context
…Yeah, I think everyone's keen on that. Dev mode was, I think, the first segue into that. So we're going to go into Braintrust now, about 20-something minutes into the podcast. So what was your idea for Braintrust? Tell the full origin story. At Impira, while we were having an existential revelation, if you will, we realized that the debates we were having about what model and this and that were really hard to actually prove anything with. So we argued for like two or three months and then prototyped an eval system on top of Snowflake and some scripts, and then shipped the new model like two weeks later. And it wasn't perfect. There were a bunch of things that were less good than what we had before, but in aggregate, it was just way better. And that was a holy shit moment for me. I kind of realized there's this, sometimes in engineering organizations or maybe organizations more generally, there are what feel like irrational bottlenecks. It's like, why are we doing this? Why are we talking about this? Whatever. This was one of those obvious irrational bottlenecks. Can you articulate the bottleneck again? Was it simply…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.