Tim Sweeney: belief

Transcript context

…Yeah, that’s right. Humans are, by far, the hardest part of computer graphics because millions of years of evolution have given us dedicated brain systems to detect patterns in faces and infer emotions and intent because cavemen had to, when they see a stranger determine whether they were likely friendly or they might be trying to kill them. Humans, we people in the world have extraordinarily detailed expectations of a face, and we can notice imperfections, especially perfections arising from computer graphics limitations. It becomes by far the hardest problem. The metahumans effort is part of a decades-long initiative that Vladimir Mostilovic, the most talented digital humans visionary in the world, has been working on for generations and generations of games, serving individual clients around the game industry for a while. Then joining Epic as part of the three-lateral team and leading now a worldwide effort to build all of the technologies required to make digital humans realistic. One part is capturing humans. We’ve got really advanced dedicated hardware that puts a human in a capture sphere with dozens of cameras in them, taking high-resolution, high-frame-rate video of them as they go through a range of motions. Then capturing the human face is complicated because the nuanced detail of our faces and how all of the muscles and sinews and fat work together to give us different expressions. It’s not only about the shape of a person’s face, but it’s also about the entire range of motion that they might go through. Capturing one human requires a few hours of capture work in a dedicated environment like that. Then thousands of hours of processing work to capture a precise and real-time replicatable version of that human in the environment, and so one of the things that … … that human in the environment. One of the things that’s done is just capturing an actor or actress in the real world and then using them in a video game. But the much more interesting thing going on is capturing thousands of humans to form a dataset whose goal is to encompass the entire range of faces in all of humanity, so going around every culture, every continent, every age and every face variety and capturing representative people so the entire range of faces is represented. Then being able to combine and merge those together to enable recreating an arbitrary face that the system’s never seen before. One of the ideas is capture giant amounts of this high-precision data and then you use it to reconstruct a face at a consumer level, like maybe take an iPhone photo of somebody’s face and then capture a very accurate depiction of that, not by synthesizing it then and there on that device, but by combining all the known details of human faces to accurately capture the most accurate representation of that. That’s the data problem. at, not by synthesizing it then and there on that device, but by combining all the known details of human faces to accurately capture the most accurate representation of that. That’s the data problem. There’s a lot of other problems with computer graphics. There’s technology for rendering hair, which is really hard because you can’t render every, again, we know the laws of physics. It would be easy to just render every hair. It would just be a billion times too slow. You need approximations that capture the net effect of hair on rendering and on pixels without calculating every single interaction of every light with every strand of hair. That’s one part of it. There’s detailed features for different parts of faces. There’s subsurface scattering because we think of humans as opaque, but really our skin is light travels through it. It’s not completely opaque. The way in which light travels through skin has a huge impact on our appearance. This is why there’s no way you can paint a mannequin to look realistic for a human. It’s just a solid surface and we’ll never have the sort of detail you see. We should actually just linger on that. That kind of blew my mind thinking through that. I think I heard that the oiliness of the skin creates very specific nuanced, complex reflections, and then some light is absorbed and travels through the skin and that creates, would it be fair to say micro shadows or something? It creates textures that are humanized, able to perceive and it creates the thing that we consider human, whatever that is. You have to compute both that, the reflection, how light interacts with the oiliness of the skin and how it is also absorbed in, and all of that while considering all the muscles involved in making the nuanced expression, just the subtle squinting of the eyes or the subtle formation of a smile. It’s a stupid, annoying subtlety of human faces that you have to capture, the difference between a real smile and a fake smile. Man, I love human faces. I love humans in general. But the way to show the beginning of a formation of a smile that actually reveals a deep sadness, all of that. When I watch a human face, I can read that. I can see that. Again, this is the engineering and the artist. You have to have the tools that in real time can render something like that and that’s incredibly difficult. But anyway, sorry. There’s a lot of this kind of complexity in even just the lighting of a face.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.