David Rosenthal: prediction

Transcript context

…Let’s do it. The ultimate point of PG’s post that Brian sent to us is that Yahoo wasn’t an engineering-centric culture and Google was, and that to win at search, you needed to have really excellent engineering and technology. This is sort of what I’ve been referring to all along. I want to make an argument here in What Would Have Happened Otherwise that maybe not the Overture specifically but had Yahoo not gone down this path with trying to build Panama and compete directly with Google, we would not have seen any or at least they would look very different, any of the great startups that emerged out of the web 2.0 era right after this and then even into today with Airbnb and Uber, because the most important thing to me that comes out of Yahoo’s attempts to compete with Google is Hadoop. And for our listeners who aren’t super familiar either with what Hadoop is or the history of it, when I was referring to the really hard math and engineering problems that Google had to solve their both the organic search results and manage the data to deliver the best search results for what people are looking for but also serve the best ads, that requires orchestrating just this huge, huge amount of data. So Google invented two things that allowed that. One was the Google File System. The second was this computation paradigm called MapReduce that allowed Google to basically operate on this huge amount of data that they were storing in the Google File System. So Yahoo, when they tried to compete with Google, they basically fund this open-source project called Hadoop which was trying to recreate. Google had published papers about how they were doing this but they were trying to recreate these technologies that really had founded sort of the “big data” world and paradigm. And Doug Cutting was a researcher who was building Hadoop and Yahoo funds him through this period in the mid 2000’s. It’s Hadoop that really then enables Facebook to do all of the data work that they do to create the newsfeed, that enables an Uber, when you order an Uber for it to get – you know, you get matched to a driver who’s two minutes away that enables when you search on Airbnb for you to find the best property. This technology that gets open-sourced was all of Google’s core technology innovation, and I think without this battle wouldn’t have just emerged for free for the community to innovate on. Yeah. That’s a great point. This is also kind of not tech theme but a company theme that Google has is they often will write a research paper about a new method or paradigm they’ve discovered and then there’s these open-source projects that come around it. And those things are lagging what Google has developed internally by about three years. So for anyone who’s sort of worked at Google and worked outside of Google, I’ve heard over and over again, “Oh man, yeah, I wish we had this tool that we had at Google. It was like this but on steroids.” It’s interesting to hear about the announcement of TensorFlow was after they had a very similar machine-learning system at Google for about three years before announcing –…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.