Robert Lange: evaluation

Transcript context

…Yeah. It's it's so interesting because, you know, like a a symbolic AI person would say, oh, I don't like connectionism because it doesn't under you know, the only semantics in connectionism is this notion of similarity. It doesn't really understand things. So so they would say, well, just just start with a a an entity relationship graph and then just kind of build up using, you know, composition and first principles. That that that doesn't work. Right? So we're using neural networks because they're incredibly flexible and they understand a lot of things about the world, but they don't have the kind of constraints that we want. So what we do is we use these tricks. So Jeremy, we evolved program descriptions. On your program selection, you had a semantic novelty detection, you know, using like a Embedding based similarity. You had like a kind of self similarity metrics and you know, based on the cosines. And indeed, you've got this meta scratch pad. So what we're seeing is this fascinating spectrum of possibilities where still using neural networks, you can imbue semantics in using all of these different tricks, but they all come with trade offs. Yeah. For sure. Like, I think it's it's kind of interesting. We we've had a long period of computer science where algorithms were sort of designed by humans. Right? Then we had sort of this Android Kapathy software 2 paradigm where, like, we trained neural networks that then performed a certain function. And now we're sort of at this point where we're using LLMs to design algorithms or solutions more generally. Right? And I think, actually, like, even though, like, large frontier language models are extreme, like, let's say, black boxes or it's very hard to get a full mechanistic understanding of them, the outputs can be. Right? The programs, the instructions, and so on. Right? So I think it opens up a very sort of new paradigm of doing research or basically doing anything. Right? If you if you think about it. But I think we're we're just sort of at the starting point of figuring out the the right user interface for that. So the other innovation in the paper was using UCB, which is upper confidence bound. It comes from the multi arm bandit literature, which is this problem where you can pull these these levers and at the beginning, you don't know which levers to pull. And and over time, you kind of reduce your uncertainty and you can kind of pull the ones that work. But there's this exploration exploitation dilemma. And you've implemented that for figuring out which LLM. So it could be Gemini. It could be like, you know, Grokfur or something to figure out which 1 to use.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.