Sergey Levine: evaluation

Transcript context

…In terms of robotics progress, why won't it be like self-driving cars. It's been more than 10 years since Google launched its… Wasn't it in 2009 that they launched the self-driving car initiative? I remember when I was a teenager, watching demos where we would go buy a Taco Bell and drive back. Only now do we have them actually deployed. Even then they may make mistakes, etc. Maybe it'll be many more years before most of the cars are self-driving. You're saying five years to this quite robust thing, but actually will it just feel like 20 years? Once we get the cool demo in five years, then it'll be another 10 years before we have the Waymo and the Tesla FSD working. That's a really good question. One of the big things that is different now than it was in 2009 has to do with the technology for machine learning systems that understand the world around them. Principally for autonomous driving, this is perception. For robots, it can mean a few other things as well. Perception certainly was not in a good place in 2009. The trouble with perception is that it's one of those things where you can nail a really good demo with a somewhat engineered system, but hit a brick wall when you try to generalize it. Now at this point in 2025, we have much better technology for generalizable and robust perception systems and, more generally, generalizable and robust systems for understanding the world around us. When you say that the system is scalable, in machine learning scalable really means generalizable. That gives us a much better starting point today. That's not an argument about robotics being easier than autonomous driving. It's just an argument for 2025 being a better year than 2009. But there's also other things about robotics that are a bit different than driving. In some ways, robotic manipulation is a much, much harder problem. But in other ways, it's a problem space where it's easier to get rolling, to start that flywheel with a more limited scope. To give you an example, if you're learning how to drive, you would probably be pretty crazy to learn how to drive on your own without somebody helping you. You would not trust your teenage child to learn to drive just on their own, just drop them in the car and say, "Go for it." That's also a 16-year-old who's had a significant amount of time to learn about the world. You would never even dream of putting a five-year-old in a car and telling him to get started. But if you want somebody to clean the dishes, dishes can break too. But you would probably be okay with a child trying to do the dishes without somebody constantly sitting next to them with a brake, so to speak. For a lot of tasks that we want to do with robotic manipulation, there's potential to make mistakes and correct those mistakes. When you make a mistake and correct it, well first you've achieved the task because you've corrected, but you've also gained knowledge that allows you to avoid that mistake in the future. With driving, because of the dynamics of how it's set up, it's very hard to make a mistake, correct it and then learn from it because the mistakes themselves have significant ramifications. Not all manipulation tasks are that. There are truly some very safety-critical stuff. This is where the next thing comes in, which is common sense. Common sense, meaning the ability to make inferences about what might happen that are reasonable guesses, but that do not require you to experience that mistake and learn from it in advance. That's tremendously important. That's something that we basically had no idea how to do about five years ago. sses, but that do not require you to experience that mistake and learn from it in advance. That's tremendously important. That's something that we basically had no idea how to do about five years ago. But now we can use LLMs and VLMs and ask them questions and they will make reasonable guesses. They will not give you expert behavior, but you can say, "Hey, there's a sign that says slippery floor. What's going to happen when I walk up over that?" It's pretty obvious, right? No autonomous car in 2009 would have been able to answer that question. Common sense plus the ability to make mistakes and correct those mistakes, that's sounding an awful lot what a person does when they're trying to learn something. All of that doesn't make robotic manipulation easy necessarily, but it allows us to get started with a smaller scope and then grow from there.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.