Speaker unverified: belief

Transcript context

…h, and actually having seen oh, okay, oh, it's actually solving these problems. It's really really a fascinating thing to see. Uh you know, one of these problems actually is from a domain that I I I I did my PhD in and yeah, like seeing the model kind of come up with these ideas which I would be, you know, quite proud to come up with like in a in a week or two. Uh seeing it come up with them in like an hour or so was that was very uh yeah, it's a very weird feeling, right? Like yeah, I I think like in the past when I felt like that was like when watching our Dota bot like play just like very interesting Dota games infinitely, right? It feels like just there's some sort of magic happening because like you know, interesting things should not be like indefinite. Yeah, and so seeing that happen for math, right? For something that I believe like you know, is actually like quite representative of of of of our our our you know, precursor to a lot of the work that we're doing and a lot of the work that like really matters in the world. Um yeah, definite definitely really increase my feeling of urgency. One thing that's fascinating too is the idea that you're you're training these models and it's like, you know, you probably you throw these problems in and it's like nobody knows whether, you know, how good will they be at solving them and and I think just like it must just be fascinating to see uh something that you know so well and are in in a space that you spent so much time in and and realizing, "Hey, probably the previous generation of models wouldn't have been able to do that." and you wouldn't even thought necessarily that this was like the the benchmark to do, but it's like just generally showing the the general purpose capabilities and and improvements of the models. I mean, it it is at a stage where I'm like, you know, we needed to like seek out experts in the in the particular domains to be to be able to tell us whether these particular proofs are correct or not, but, you know, it's still much easier to like tell whether you've you've actually made progress than, you know, less than for something like uh even coding. I like because sure, the competitive programming you can evaluate, but most programming is not competitive programming and it's, you know, it's about like already abstractions are they're handling all the cases and, you know, Yeah. I guess like, you know, I feel like there was this maybe common criticism a year ago and I don't know if it's as strong now that like, "Okay, these models are like pattern matchers, but like you really want AI for science, like we're not going to get new ideas or like, you know, entirely novel things out of out of pattern matching." Feels like we continue to like chip away at that narrative. Are we getting closer to kind of fundamentally disproving that? I believe so, yeah. I mean, I think kind of on scale we're starting to see like minor advancements, right? Like not huge things, right? Like a small idea here or there. I mean, maybe maybe some like bigger papers in collaboration with with scientists, right? Like uh but, you know, was AlphaZero a pattern matcher? AlphaGo a pattern matcher? a small idea here or there. I mean, maybe maybe some like bigger papers in collaboration with with scientists, right? Like uh but, you know, was AlphaZero a pattern matcher? AlphaGo a pattern matcher? You know, our our data about a pattern matcher, like they did kind of come up with new strategies for the respective games. Yeah. >> Um it's funny that there's counterexamples to it all the way back to, you know, 2016, 2017. Right, right? And and you know, I think you can to well, I guess you going to always to floss in that, which I think is interesting like AlphaGo can be beaten with some strategy. Our data bots could have been been beaten with some with some strategy. I think I think there will be a lot of deficiencies for a while of of like these models, right? But But I think also like they they are able to discover new things because they have a lot of these capabilities. And like the way, you know, Yeah, I mean it's you know, it's taken a couple years to like get go from like this like very tiny game environments to like this much more um draw scientific research that requires kind of going through um you know, like a decent approximation of like all human knowledge in the meantime. And you know, learning all the human languages and so forth. But but I But I think the basic principle is is is very similar. Yeah. You know, it's funny. I think like when you guys had these first proof results, um if I remember right, the organizers said, you know, they were commenting these AI solutions and they were like, "This feels like, you know, 19th century mathematics of like brute force, you know, computation-heavy approaches rather than these like elegant modern techniques." Um which I'm not sure is a a feature or bug of of, you know, obviously the the way these models work. But like, you know, hearing that, I mean, does that like does that concern you, excite you? >> It doesn't concern me. I mean, I I mean, I think it's expected that like I mean, I I'm sure I I I thought for at least one of the problems I actually actually always produce pretty good pretty nice results. Quite a bit shorter than like the intended one. You know, but I think in general you would expect like, yeah, these models kind of you know, they can produce so much more reasoning in a short time than like a person can, right? Just like in terms of the raw number of like tokens or thoughts. I I don't expect that to be like kind of a long-term feature. It feels like there's so much momentum behind AI for science right now. And you mentioned obviously like, you know, at some point you do have to connect these models to the physical world. And you guys released some cool stuff with Ginkgo and like some of these other things you've been experimenting with. I'm sure you've thought a lot about like AI for a bunch of different areas of science. You know, as you've kind of dug into some of this stuff, have you developed any intuition first? You think about like 3 years from now, the spaces where of science where you're like, oh, that there's going to be crazy progress there versus the ones that might prove like a little more resistant to immediate change? You know, attempting the spaces where of science where you're like, oh, that there's going to be crazy progress there versus the ones that might prove like a little more resistant to immediate change? You know, attempting honestly would be that like, oh, you know, it's really about like, um, you know, do you uh, you know, what what are the things that it requires some some, you know, manual work like where where where the models are not not quite quite plug and the ecosystem are you know, like the the the different laboratories will also kind of evolve pretty quickly to adapt to like these new technologies. Within those STEM fields, obviously, you know, I feel like there's a question of is it like an LLM with access to the physical world or you've obviously had companies that are we've been started specifically around these domains, right? Like an Isomorphic in biology or Periodic in in material sciences or Physical Intelligence in robotics. What's your kind of gut instinct on the extent to which it makes sense to pursue some of these things like independent with different model architectures versus like all within the context of one place? Yeah, I think it's kind of similar to, you know, my answer with like the UI for, you know, for Codex which like I I I would build around the capabilities of the technology and not around its limitations so much. Um, so, you know, you definitely like if you have something that can suddenly design like a huge amount of like interesting like chemical or biological experiments, like, yeah, I mean, it makes sense to uh, you know, build labs that enable that. You know, I think if we if if we did get to a place where like the model is like very capable of designing high quality experiments, it would also make sense to like have it work with humans in a loop, right? Like we shouldn't think of it as like, oh, it's either you kind of automate it fully and you have it as like fun thing using some tools on the side. Like we will get to a world where like it's just very natural to be collaborating with, um, you know, AI scientists that are that are working hard on a problem. >> Yeah. It's interesting. It's almost like a different vision. It's like one world where this works is like, hey, you just train a model, you know, to basically run these end-to-end tasks and like be the automated like, you know, uh, biologist or, you know, chemist or whatever it is. And there's another one which is like, well, you're building really tools to, you know, both propose, run, kind of work in tandem with a bunch of human researchers. >> you know, I wouldn't necessarily characterize it this. I mean, you know, of course there are tools in some sense, but I think like, you know, we're going to a point where they're driving a lot of the like design and and ideation for the whole process. Yeah, with with like an LM architecture, but just like, you know, being able to figure out the right way the right kinds of experiments to run and and then actually design it. >> And yeah, when it comes to like different architectures and, you know, I mean, you know, for for sure like, you know, like natural language reasoning, like the kind of the kind of…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.