Robert Lange: prediction

Transcript context

…Beautiful. The only thing we didn't talk about was we spoke about the circle packing problem, but you also applied it to a few other things. Can you tell us about that? So 1 thing we did was we sort of used a framework called ADAS, automatic design of agentic system, where basically instead of manually writing an agent scaffold, you use an LLM to write agent scaffolds for a specific task. Right? So what we did is we looked at mathematics tasks. So Amy and we used Cinca to evolve basically an agent. Right? So using an agent to evolve an agent. And we found that there we could dramatically improve sort of the performance of very cheap models like GPT 4.1 nano, but the agent scaffold was also able to either, like, generalize to other language models or to different years of of AIMI. Right? That was 1 application. 1 important other application that we did was to ALE bench. ALE bench is basically work done by other folks at Sicana, including Yuki, who's also part of the paper, which is considering heuristic programming contests, sort of previously done and executed by Adcoder, which is like this famous Japanese competitive programming organization. And we sort of showed that Shinka can also work very well as a coscientist. So basically, we we took initial solutions obtained by an ALE agent that was previously designed, and then we optimized on top of these initial solutions with Schenka and showed that on 1 of these sort of programming tasks, if the combination of this agent and Schenka would have competed in the challenge, it would have ranked second place, basically. So I think there's some evidence that Schenka can work as a coscientist and not only for LLM agents, but potentially even for humans like we discussed before. And then finally, the final application that we looked at was designing sort of mixture of expert load balancing loss functions. So at Zekana, we've done some previous work called DiscoPop. I think we discussed this during the last podcast we did where we are using LLMs to design objective functions, and back then, we did it for preference optimization and post training. And here, did it for load balancing of mixtures of experts. Also there, we found that within, I think, like, even only 20 sort of generations, we were able to sort of explore, let's say, not only a single objective function, but sort of, let's say, a convex hull where there are different trade offs between sort of performance and load balancing and so on. So I think this is another application of Schenka where it's not only basically about sort of finding the best solution, but essentially illuminating a program space where there are always potential trade offs between, like, let's say, for example, runtime and the quality of the circle packing. Right? And having a system that can explore all of these is important as well. I'm very excited to see you apply this to the ARC challenge.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.