Ray Bradbury once said, "You've got to jump off the cliff all the time and build your wings on the way down." In a way, that's what the team working on Microsoft Flight Simulator (2020) did, starting an ambitious project that was literally ahead of its time.

The Microsoft Flight Simulator series has been a vehicle for technological innovation since its introduction as Microsoft's first packaged product in 1982. That predates Office and even Windows.

The big selling point for the latest Microsoft Flight Simulator, released on PC in August last year and headed to Xbox Series X and S later this month, is the way the game maps out an impressively accurate and detailed planet Earth using cloud computing techniques with Microsoft Azure.

Here's basically how it works: via Azure, the game draws 2.5 petabytes of world data from Bing and combines that with vector data from OpenStreetMap, then uses machine learning that identifies and processes buildings, trees, grass, clouds, etc. to render them in believable 3D.

Through partnerships with companies like Meteoblue which follows weather conditions, and FlightAware which relays air traffic, Flight Simulator pulls data to reflect what's happening in the world in real-time. Combine that with satellite data that reflects real-time ground conditions, and on-device processing that renders seasons, time of day, aircraft, and player interaction, you get an incredibly convincing flight experience.

When the tech chases the game

It's a complex process with many moving parts, and what's more fascinating about this technical feat is that Microsoft and development lead Asobo Studio in France were relying on cloud infrastructure that did not fully exist when work on the game began in 2016.

"When we started, I would say, it might have not been possible," said Jorg Neumann, head of Flight Simulator at Microsoft. "The infrastructure to get it to people at low latency was actually foremost on my mind."

Neumann said at the start of development, there were three main challenges: storing, processing, and streaming the world of Flight Simulator. Storage wasn't so much of an issue--there's plenty of room in the cloud--but the efficiency of processing and streaming would depend on future expansion of Azure datacenters.

"There was a point in time where we said, 'Maybe [the game's availability] is just regional' and we have to wait for the infrastructure to actually catch up," said Neumann. "But it did [catch up] during the course of the development."

As for the pre-processing driving the machine learning that renders buildings, trees, and removes images of clouds in order to make way for real-time weather, that was also a cloud-dependent challenge. "The thing that probably was the least expected was the amount of work we're doing to actually process the data," said Neumann, "because basically what Bing has is [the] photogrammetry cities, like a few hundred, and they're super cool, but beyond that, literally everything is just a height field with nothing on it."

That means from way up high, buildings would look ok, but getting in closer--not so much. The solution would be cloud-based procedural construction technology that didn't really come along until 2018 or 2019, according to Neumann. Keep in mind, Flight Simulator launched in summer of 2020.

In short, this procedural tech uses data from Bing and crams it through an algorithm that takes 2D imagery and erects 3D landscape features such as buildings with stunning accuracy. There were various challenges in automating the creation of Earth's digital twin--some unexpected.

"When you have a [building's] picture in 2D, and you want to make it 3D, you actually have to figure out how the roof is structured," he said. The solution involved hiring someone with a literal PhD in rooftop reconstruction who helped with the rooftop algorithm.

And then there are the trees. The development team's tree detector was complicated to program because it's difficult to distinguish a tree from a bush or from grass. "When you look at a satellite picture, fundamentally, it's all just green," Neumann said. "So you need to train up the algorithms to actually really understand...It gets better all the time. It literally gets smarter all the time. Even since we launched, I would argue tree detection is a lot better."

Neumann said he believes it's just a matter of time to where machine learning will be able to take this imagery from the cloud and recreate trees "down to the leaf--the perfect digital twin." For him, the technology going into Flight Simulator goes far beyond trees and rooftops.

Beyond the horizon

Flight Simulator is impressive in itself, but this kind of technology isn't just limited to flight games. Neumann has been at Microsoft for two decades, but before that he was at Origin Systems, known for its aspirations of making vast believable worlds such as Ultima.

Such tech can open the door for studios, including smaller indies, to make expansive 3D worlds, he said, and lower the cost of making games. "Back at Origin, and then I was at Digital Anvil, 50 percent of our budget went to art," he said. "A lot of it was environment art, and environment [art] oftentimes is repetitive." On Flight Simulator, Neumann said for the first two years of the project, artists only worked on planes, and there was no environment artist on the entire project.

That doesn't mean artists would become obsolete, however. To Neumann, offloading more mundane environmental art duties would free up artists to focus on, say, a unique handmade castle in a fantasy game.

He also has larger aspirations when it comes to using Azure to accomplish the "digital twinning" of Earth. Neumann, who used to work on Zoo Tycoon, said while there are some animals in Flight Simulator, he would love to see a digital Earth with accurate animal populations.

"There's something called the Icarus Initiative that actually tracks all this [animal population] stuff," he said. "The data exists for completely different scientific purposes. But that means that the moment big data exists, that's the opportunity. Big data can now be put into the cloud and downstream." Animal migration patterns, herd behavior, and more could be accurately implemented.

So it seems this kind of large-scale planet-building is only limited by the kind of data that's available. Neumann enthusiastically elaborated on other possible cloud-centric possibilities in games, such as using Azure Cognitive Services for speech synthesis ("not just synthesis, you can also start translating," he said). Think of in-game voice chat that accurately turns your English into Japanese for intercontinental communication, or a world populated by NPCs that apply machine learning for more convincing interactions.

Flight Simulator has expanded Neumann's vision for the cloud. "It is literally a world computer," he said. "And I think it's growing at a very, very fast rate. It's sort of like surfing the crest of the wave of what's possible."