High Signal Podcasts Evidence ledger
Method
Browse
← Back to evidence

Evidence receipt / belief

Published · transcript-backed

Ankur Goyal: belief

11 Oct 2024 Latent Space Production AI Engineering starts with Evals — with Ankur Goyal of Braintrust

“I was saying, I think while in the short term there may be opportunities to scale agents by doing silly things, it feels super clear to me that in the long term, anything you might do to work around that limitation of an LLM will be pushed into the LLM.”

— Ankur Goyal

Source trail

Everything needed to verify it.

Speaker
Ankur Goyal
Attribution
Verified speaker
Claim type
belief
Recorded
11 Oct 2024
Publisher
Latent Space

Transcript context

…I don't know anyone who's doing that. Guy Podjarny Voyager is in the paper. My term for this, if you want to use the term, you can use mine, is core versus LLM core. This is a direct parallel from systems engineering, where you have functional core imperative shell. This is a term that people use. You want your core system to be very well defined and imperative outside to be easy to work with. The AI engineering equivalent is that you want the core of your system to not be this Shoggoth, where you just chuck it into a very complex agent. You want to sprinkle LLMs into a database. Because we know how to scale systems, we don't know how to scale agents that are quite hard to be reliable. I was saying, I think while in the short term there may be opportunities to scale agents by doing silly things, it feels super clear to me that in the long term, anything you might do to work around that limitation of an LLM will be pushed into the LLM. If you build your system in a way that assumes LLMs will get better at reasoning and get better at sort of agentic tasks in the LLM itself, then I think you will build a more durable system. What is one thing you would build if you're not working on…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.

Search evidence