Thomas von Tschammer: evaluation

Transcript context

…Have we seen any surprises come out of that process? Of course, the legendary move 37, it seems like increasingly with the Mythos class Fable models, we're starting to see Move 37s might be strong, but they're definitely stepping outside. I had a really interesting experience today where I, or yesterday, where I ran a skill that's a very familiar workflow. And Fable stepped outside of the workflow and proactively asked me a bunch of questions, actually presented a webpage to me to collect information. Not instructed to do that, never been part of the process before. Opus never did anything like that, but it took it upon itself. to say, I've got all these inputs, but I think I could use some more inputs to really do a great job. And here's what I need from you, the human, to really knock this out of the park. That was like a little mini Move 37 in that it was, I've probably done this workflow 50 times over the last few months and nothing like that had ever happened. What's the sort of most Move 37-like thing that we're seeing in engineering? Yeah. We have very similar analogies, and I think to be fair, it's one of the favorite part of the job field. That's where it becomes very exciting. We have engineers that are using these workflows, right? So using AI to explore and asking the AI to explore these hundreds, thousands of configurations overnight. And when they come the morning after, they're looking at the results, and then they're getting back to us and telling us, hey, very impressive. The AI model came up with a design. that I would have never thought would be good. If you had shown me in this design like this, I would have said, hey, scrap this, is not going to work. But actually, those designs are better than anything we could come up with. And now I need to get back to the dashboard to understand why it isn't so much better, right? So I need to rethink my intuition because I didn't think it could be that good as a design, right? So you learn as well from these models. Again, because they explore this much richer space, they go out of bounds. They go beyond their intuition, similar to what you were saying, I think. So that's why it becomes super interesting because then something really clicks with the engineers, they become very excited because they understand that they can also learn from the model. I didn't get there. How can I learn from it? Because explore new physics or new phenomenal that I was not aware of when I was only working with intuition, essentially. So is it possible without even trying to reverse engineer the design themselves? learning from the AI's advances and its occasional leapfrogs over us is definitely a really exciting, thrilling, slightly scary part of this new future. Could you give us a little bit more intuition for like how, like just how radical these moments are? It may be a little bit hard for somebody not in the domain to really grok it, but I'd love to try, to get a little bit better sense on kind of, are they really good optimizations or are they really like stepping out and exploring different regions of the design space that people, because I think what made Move 37 qualitatively so compelling was like, no human would have made that move. Like initially, I think the live stream commentators like thought it was a blunder, right? When we see these surprises in engineering, like how big of a surprise are they? Are we seeing like, oh, that's kind of interesting. I wonder if that could work. Or is it like, that looks kind of crazy, but in defiance of all my intuitions, it actually does work. Just try to help me calibrate on how big those surprises are.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.