Public evidence record
Published podcast speaker
Books, apps, and tools
paper / likes
“I really like Rich Sutton's paper that he wrote about the Bitter Lesson and the Bitter Lesson effectively is this nice one-page paper but the essence of it is you can try lots of approaches, but the two techniques that are incredibly effective are learning and search.”
service / likes
“One of the things I like about Google is our ambition has always been sort of something that would require pretty advanced AI. Because I think organizing the world's information and making it universally accessible and useful, actually there is a really broad mandate in there.”
Claim ledger
6 transcript-backed records
01 / prediction
“We’re trying to push the frontier of 1 million or 2 million context, which is good because I think there are a lot of use cases where.”
02 / prediction
“" Or even below that, some people are quantizing models to two bits or one bit, and I think that's a trend that definitely –”
03 / prediction
“Even though people are saying, "Oh no, we're almost out of textual data," I don't really believe that because I think we can get a lot more capable models out of the text data that does exist.”
04 / prediction
“I thought, naive me, that 32 processors would be able to train really awesome neural nets. But it turned out we needed about a million times more compute before they really started to work for real problems, but then starting in the late 2008, 2009, 2010 timeframe, we started to have enough compute, thanks to Moore's law, to actually make neural nets work for real things.”
05 / prediction
“Compute is the rough, highest-level view of these capable models because if one of the techniques for improving their quality is scaling up the amount of inference compute you use, then all of a sudden what's currently like one request to generate some tokens now becomes 50 or 100 or 1000 times as computationally intensive, even though it's producing the same amount of output.”
06 / prediction
“Some of those benefits may be improved quality, some may be less concretely measurable, like this ability to have lots of parallel development of different modules. But that's still a pretty exciting improvement because I think that would enable us to make faster progress on improving the model's capabilities for lots of different distinct areas.”