For Tim Sweeney, advancing Epic means racing into AR and VR

What does it feel like to receive an award honoring a lifetime of achievement...before you're 50?

"I feel like maybe I'm an old fogey and should be shopping for a cane!" Epic chief Tim Sweeney tells Gamasutra, with a laugh.

This week he's being presented with the Lifetime Achievement award during the Game Developers Choice Awards at GDC, and he says it sounds like the "kind of thing you get when you're about to die."

"But I guess the game business, it's not an old enough industry to have octogenarians yet. So it's a real honor," Sweeney says. "It's cool to see that....you know, the story of Epic is just a bunch of smart, hardworking people starting a company together and learning as we went. And just making some cool things. There weren't investment bankers driving it, or people in suits. It was all completely organic."

Epic is probably best known among game developers these days for its Unreal Engine business, and with good reason: the company is at GDC this week to talk up what's new with Unreal Engine 4, and to showcase a new way the engine is being used to overlay computer graphics on camera footage of the real world in real-time.

But with a constant focus on what's coming next, it can be easy to forget that the Epic devs know now has gone through several significant changes in its 20+ year journey from shareware distributor to game engine heavyweight.

"It's been a really interesting process. I started the company, built our first couple of games, then I stepped back for a while," Sweeney recalls. "I thought I'd never be a competitive programmer again, but got back into it to ship Unreal. I had to build the engine for it, which sort of set the stage for everything we're doing now. I've constantly kept in touch with programming, doing at least research and projects and...."

At this point Epic CTO Kim Libreri, who is seated nearby, jumps in. "On one of his windows earlier was some code!"

"Yeah," admits Sweeney. "I feel it's really important to keep up to date with all the latest trends in the industry. It's funny, I was never really a serious gamer -- I just played enough of these different games to understand what they did, reverse-engineer their techniques, and learn from them. It's been a great process, and we've brought in a vast number of people who are better than me in all their individual areas."

This is something Sweeney has talked about in the past -- his efforts to pull back from the game development work at Epic by hiring people who are more effective designers or programmers or producers. Looking back, he says this strategy has mostly been effective, but that there are still some things that he feels only he can do.

Slaying sacred cows

"You find there really are some things that only a founder of a company can do. Sacred cows that need to be addressed," says Sweeney. "You know, what is Epic really about and what is it we do. How do we make the tradeoffs between running a business that works and bringing in the revenue to pay the bills, while still being incredibly respectful of our customers' rights and their privacy and everything else. And being an industry advocate when it's necessary. It's cool."

Pressed for an example, Sweeney points to Epic's (relatively) recent decision to cut UnrealScript out of its engine and make Unreal Engine 4 strictly a C++ affair.

"One of the key moments in Unreal Engine 4's development was, we had a series of debates about UnrealScript -- the scripting language I'd built that we'd carried through three generations. And what we needed to do to make it competitive in the future," says Sweeney. "And we kept going through bigger and bigger feature lists of what we needed to do to upgrade it, and who could possibly do the work, and it was getting really, really unwieldy. And there was this massive meeting to try and sort it out, and try to cut things and decide what to keep, and plan and...there was this point where I looked at that and said 'you know, everything you're proposing to add to UnrealScript is already in C++. Why don't we just kill UnrealScript and move to pure C++? You know, maximum performance and maximum debuggability. It gives us all these advantages.'"

"And everybody's like 'oh, yeah. I guess...that could happen,'" Sweeney continued. "There are some things that people just don't want to think about, because it's too radical a change. Or opening the engine up to open-source! Giving out all our source code. We decided that we wanted to do that in 2004! And it took us literally ten years to strip out all the closed-source components we used, and remove all the legacy components that prevented that from happening. And over the course of that ten years we did all that, and in 2014 we unveiled it and people were like 'oh wow, that's really smart. When did you decide to do that?' And I was like....oh my god, you have no idea."

Now, Epic is restructuring its games division to focus more on nimble, "live" design (think: multiple betas with lots of community feedback) even as it continues to try and push the Unreal Engine into arenas beyond game design. Both are potentially of interest to UE4 devs, because the former describes how Epic's philosophy of game design is evolving (which in turn influences the evolution of UE) while the latter is a force for improving the engine's capabilities.

The next big step for Unreal Engine: Digital creations composited into real-world footage in real time

As an example, Epic this week brought a tech demonstration of UE4's real-time rendering capabilities that it produced in conjunction with creative firm The Mill and car company Chevrolet. It's a short film called "The Human Race" which showcases a car-like AR tracking vehicle (dubbed the "Mill Blackbird") being filmed driving in real time on a mountain road, then overlaid with Unreal Engine 4 renderings of (in turn) a Chevy Camaro and a Chevy concept car in real time to produce a mixed-reality car commercial.

At first blush this short film might not be as immediately enticing to game developers as the VR editor Epic debuted during GDC last year, but if you pick apart how the tech demo works it suggests some intriguing potential for future augmented-reality game design. 

While computer-animated simulacrums of cars, scenes, and characters have been common in film for decades, they're usually done in post-production. In Epic's tech demo, however, a machine running UE4 was fed live video feeds and positional data from the Blackbird's Arraiy tracking system; the engine was then used to render the virtual cars and integrate them directly into the shot in real-time, creating what Sweeney calls "final pixels" as the video is being produced.

"It's just become possible, this capability of rendering final-quality pixels in real-time, with a photo-realistic level of quality," says Sweeney. "Offline photo-real rendering has always been possible; it's been done for more than 20 years now, in movies. And a lot of people have been using game engines for prototyping and preproduction in films and all these other uses where they rough something out in-engine, then recreate it in professional tools outside the engine. But now we're seeing it finally come together with Unreal producing final-quality photo-realistic pixels in real-time." 

He and Libreri went on to note they only recently learned that Lucasfilm experimented with using Unreal Engine 4 to produce real-time graphics effects in its 2016 Star Wars film Rogue One, and actually kept a few shots in the film -- specifically, in a sequence where the leads are moving through an Imperial base in disguise alongside robot companion K2SO.

"So final pixels rendered in the Star Wars movie, using the Unreal Engine, in real time. And it went into the movie! These are pixels you see in the movie! It's unbelievable," says Sweeney. "These real-world usage cases have pushed us much harder than before. Because as game developers, if we can't render a particular type of object well, we just put something else there instead, right. We can cheat. But these other industries cannot cheat. We have to solve all of the hard problems."

Footage of the Blackbird tracking vehicle, studded with sensors and trackers

Sweeney and Libreri say that this work is in turn beneficial to game devs using UE4 because it opens up what they can do with the engine, potentially making it easier to build games that can capture video footage (from say, a smartphone camera) and overlay it with realistic computer-generated graphics in real time. 

"The actual component pieces of this demo, in terms of this technology, we'll be releasing between now and next GDC, so anybody will be able to do stuff like this with the off-the-shelf version of the engine," says Libreri. "Of course we have to clean up after ourselves a little bit; these demos are pretty hectic to put together. Some features will be available sooner, but we'll get it all there by the end of the year."

Rendered Chevy models overlaid on the Blackbird tracking vehicle in real time

And of course if we're talking about using UE4 to create augmented-reality games, we have to talk about the phenomenon that was Pokemon Go. Sweeney acknowledges it was "an awesome first game," but says its developers sidestepped a lot of problems by using stylized graphics. It's a canny design choice, but one that Sweeney thinks is ultimately the wrong one.

"The worst problem with AR is making virtual objects fit into real-world scenes with proper lighting, shadows, camera effects, and all these other things," he says. "That's going to be an absolutely essential ingredient in these AR experiences, especially as you get towards the high end. And when you get to the point where there are competing AR games and some have cartoony graphics overlaid on your view of the real world and some have photoreal graphics that look like a complete, unified scene, those games with the beautiful graphics are going to win."

A live on-stage demo of the Blackbird during Epic's GDC 2017 presentation

Okay, but real talk -- how realistic is it for contemporary game devs to expect to be able to do photo-realistic graphics in a mobile AR game, in real time? Libreri hedges, but Sweeney stands firm: it's possible.

"To a limited fashion," says Libreri. "You're still limited to what the GPUs can do on the mobile devices."

"Photo-real is possible on mobile devices now, you can do it," responds Sweeney, who has a habit of highlighting Korean mobile games like Blade II as an example of high-quality UE4-powered mobile graphics.

"Yeah, but image-based lighting of that caliber with 20 million polygon cars? You're not going to be doing that for a while," says Libreri. "But there are hybrid models of these, as well. [...] I think we're going to see all sorts of hybrid graphics happening over the next few years as people sort of explore the potential."

Despite fresh focus on AR, Epic expects high-end VR to continue pulling the market forward for years to come

But while Sweeney is passionate about mobile AR and VR, he's confident that high-end PC VR headsets are going to be driving innovation in the market -- on both the design and technology fronts -- for years to come. 

"Almost all of the user time, and all of the user money, is being spent on PC VR and PSVR," Sweeney says. "So I think that's the area where developers should focus for now. Just because it's an early adopter market, people have high expectations of visual fidelity, and patience for imperfection. And it's better to reach a highly focused audience with these early apps than to reach everybody, and have general lack of pickup. Most people who have used mobile VR devices have used them a few times and never come back, and never spent a penny in them."

Sweeney estimates there are roughly 2 million PSVR or PC VR headsets in the market right now, and he says that's fine -- it's an acceptable market for Epic's current VR game projects, and one he sees as being more comparable to the dawn of personal computing than the arrival of smartphones.

"If you look back to the first year of the personal computer, Apple sold a grand total of 23,000 Apple IIs," says Sweeney, noting that VR is basically creating its own new market rather than usurping an old one the way smartphones did. "And that was the start of it! Look at it now."

As VR tech advances and becomes more commonplace, Sweeney says Epic staffers are becoming more comfortable using it in their day-to-day work. 

"We've been adopting the VR toolset more and more; you find VR headsets all over our offices," he says. Different people are using it in different amounts: Sweeney says Unreal Engine 4's VR editor is getting a bunch of use when it comes time to edit or tweak scenes, but that "it's not yet a tool that people are using from scratch to build their environments."

A live demonstration of the Unreal Engine 4 VR editor at GDC this week

This is due to a handful of limitations, some of which are personal hurdles and some of which are technological ones that Sweeney believes will be eliminated as VR tech improves.

"One thing is that the resolutions of the screen are very low, especially on an angular pixel density metric. So, text tends to be hard to read unless it's really big," admits Sweeney. "So a lot of the professional-level tools like the Unreal Editor are not yet at a level of perfection that you would want for a lot of your large-scale CAD use. One more generation of hardware, and we'll have 4K divided by two eyes and suddenly it will become much more widely adopted and we'll spend even more of our time in it."

"I would project that in six years or so, almost all of our designers will spend almost all of their time immersed in the VR editing environment," Sweeney continues. "That will rely on the evolution of maybe two more generations of headsets, improving the pixel quality and the tracking. Two more generations of controllers getting even more fine and precise motion controls, and even more tweaks to the software. But I feel like over the next decade, every piece of existing CAD software in the market will be completely reenvisioned around VR, and suddenly we'll have a much more approachable 3D content industry where anyone can go in and start building stuff."

Of course, in some ways this vision of the future makes game development work a lot less approachable -- at least until we figure out how to get the attention of someone who's immersed in a VR headset without making them jump. But even as he forecasts a future where game devs work with their heads in a box, Sweeney speaks passionately about the value of being outdoors and exploring the natural world.

He recently picked up a bit of press attention for paying $15 million to earmark 7,000 acres of North Carolina as a "conservation easement", effectively preventing it from being developed or interfered with. But Sweeney has a long history of land conservation, something he is only recently beginning to talk about openly in the press.

"It was just personally important to me," Sweeney says, by way of explanation. "I grew up in this funny town outside of Washington D.C. called Potomac, Maryland. It was just a backwater farm area when I grew up there. And over the course of my lifetime I saw all of my old stomping grounds, you know, these beautiful forests and gold mines I used to explore, they were completely paved over to build mansions for congressmen and their lobbyists as Washington expanded. That sucked."

This is part of the reason Epic moved to North Carolina in 1998 -- to get away from the crowding and rising cost of living in that area. "We thought 'oh, phew, we got away from it,'" recalls Sweeney. "But you know, then you started to see the same thing happening. So I really felt that, when I had the financial ability to do so, that I really wanted to help set some land aside that could be saved from future development. And remain in its natural state. I love being out in the woods and hiking. So, I do that when I have the chance."

"I think that everybody who has the opportunity, when we are that fortunate with our lives, should participate in something they really believe in, and understand, and know about personally," Sweeney continues. "For me, that was land conservation. There are lots of great causes in the world that people can get involved in; that's just one of them. But you know, I think if you have the ability to do something then you should do it."