David Rosenthal: uncertainty

Transcript context

…Which was dramatically better and a big, crazy cultural thing, because you’ve got these researchers parachuting in, again led by Jeff Dean, saying, I’m pretty sure our neural networks can do this way better than the classic methods that we’ve been using for the last 10 years. What if we take the next several months and do a proof of concept? They end up throwing away the entire old code base, and just completely wholesale switching to this neural network. There’s actually this great New York Times magazine story that ran in 2016 about it. I remember reading the whole thing with my jaw on the floor, like wow, neural networks are a big effing deal. And this was the year before the Transformer paper would come out. Before the Transformer paper, yes. They do the rewrite of Google Translate, make it based on recurrent neural networks, which were state-of-the-art at that point in time, and it’s a big improvement. But as teams within Google Brain and Google Translate keep working on it, there are some limitations. In particular a big problem was that they “forgot” things too quickly. I don’t know if it’s exactly the right analogy, but you might say in today’s Transformer-world–speak, you might say that their context window was pretty short. As these language models progressed through text, they needed to remember everything they had read so that when they need to change a word later or come up with the next word, they could have a whole memory of the body of text to do that.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.