Joscha Bach: belief

Transcript context

…Yeah. Filtering early on, not later. Yes. It’s a slightly different paradigm and it leads to much faster convergence. We only need to look the tiny fraction of the data to become coherent. Of course, we do not have the same richness as our train models. We will not incorporate the entirety of text in the internet and be able to refer to it and have all this knowledge available and being able to confabulate over it. Instead, we have a much, much smaller part of it that is more deliberately built. To me, it would be fascinating to think about how to build such systems. It’s not obvious that they would necessarily be more efficient than us on a digital substrate, but I suspect that they might, so I suspect that the actual AGI that is going to be more interesting is going to use slightly different algorithmic paradigms or sometimes massively different algorithmic paradigms than the current generation of transformer-based learning system. Do you think it might be using just a bunch of language models like this? Do you think the current transformer-based large language models will take us to AGI?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.