Thomas Ahle: evaluation

Transcript context

…thrive. Okay. So you're saying at the moment there are these commercial, you know, verifiers and simulators and they cost a ridiculous amount of money. So is is it something like $10,000 per seat or something? Yeah. Per for 1 CPU kernel. Right. Dollars. Yeah. Like, say you wanna scale this up in a data center with 1000000 agents running like you're gonna like I mean, computer is already expensive but not that expensive. What is that $10,000,000,000 or something, right? Just for those licenses. And I think actually that's 1 of the reasons also why AI still is not as popular in the hardware space because they haven't been able to train like the models are not as trained to this kind of workloads because they just, it's not feasible. Like you don't have all of the open source code out there to start the training, but you also don't have the tools that they need to learn to use. You don't have the, yeah, all of the, like the tasks, like you can't just, all of the reinforcement learning on top of it. I mean I mean they they clearly are doing some of that. Like we can also see from model generation to model generation that they are getting better, but it's just it's kind of day and night between like Python or JavaScript. We've been sort of developing these EDA tools in house using AI, having, you know, think I probably have the world records of the longest running agents having like some 20 GPT agents running for around 6 months now, still making progress. So as I understand correctly, so similarly to Anthropic, Carlini had this blog post out and they had a functional specification of a C compiler and they had 40,000 agents, and they reproduced the C compiler. You did a similar thing for an E-forty 2. By the way, right? Well, that's an interesting thing as well. Because this is what I want to get to, right? It's tantalizing. That that there there are there are some domains that are so well evolved that we have, you know, that might be very complex, but but we have a functional specification, and and it's reasonably coherent. So so the idea is we we get a shitload of agents, and we reproduce the function of this software based on these tests, we do it recursively, agentically, and and and so on. Now, my my contention with this is I think that it's not about where you end up. It's not about the functions and the tests passing. It's about how you got there and how structured it is. Yeah. So there is this tendency with agentic coding to build a spaghetti monster, which seems to work. But because if you think about it in in this project you're talking about, I think you said there's, 500 or more, 500,000 lines of code. And in 5 years' time, there's all of this code that probably, you know, you folks haven't read most of it. What you know, that that that sounds like a problem.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.