Cristopher Moore: prediction

Transcript context

…Yes. It feels though that we do something slightly more sophisticated. I completely agree with what you said about this reification, simplification, abstraction. There there's so much more information which we're leaving out in these processes. But we can design the Linux operating system. And and it feels that even though these transformers can learn structure, the types of algorithms that that they can perform are limited in in very, very, you know, problematic ways. I follow these debates about how good these things are at coding. I, you know, I follow Jonathan Blow on Twitter who's you know, I I enjoy his I I love his work on game design. He's very opinionated. I I don't really have an informed opinion about this because the coding that I do tends to be relatively small scale. I don't build large modular things that with many interacting parts. I build, like, some code to run some physical system on my laptop. So, yeah. I mean, I from the outside, I mean, I see that there's this debate about is it really just copying GitHub, and, you know, how much is it really kind of understanding the code the way a human coder does. I guess I also know that people are talking about or are are doing taking an LLM and giving it a module that it can use the way we use specialized modules. When I have a certain kind of mathematics problem, I fire up Mathematica. Right? If I want to know how some function behaves, I graph it and look at it. And I think once LLMs are given these various playgrounds and given the ability to fire them up to do literally doodle and look at it in a 2 dimensional way, the way we can with our eyes, as opposed to treating everything as 1 dimensional strings of text, I expect that we'll see much more multimodal abilities. And I know that this is already happening. It's just so interesting that a layer of in the, you know, the self attention layer, it's it's this first order genocide pooling where you just kind of, you know, take all of the possible pairs of the tokens and you stack it n times, stick an MLP on the end. And like, just the sanity test there just, you know, seems to me. How could it possibly learn a deeply factorized structured representation of problems?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.