What is XR, and how is it radically transforming industries?

Extended reality (XR) has been seeping into modern life for decades. From virtual reality (VR) arcade games in the early ’90s to the millions of augmented reality (AR) lenses available on Snap AR, the evolution of XR is gaining steam. According to the research group Statista, the global XR market size, forecast at $31 billion in 2021, is expected to swell to $300 billion by 2024.

The growing enthusiasm for XR is no doubt fueled, in part, by Mark Zuckerberg’s recent announcement of Facebook’s name change to Meta and unveiling the company’s plans for the metaverse, a 3D evolution of the Internet where people can meet in shared virtual places to work, exercise, perform, and socialize.

Applications for XR, already well-established in entertainment and gaming, are quickly expanding to new industries. Car companies are using XR for design creation, where designers can create 3D concepts of new vehicle lines and then collaboratively review the designs. Health-care organizations such as the Mayo Clinic are applying the technology to create interactive preoperative plans, assist with heart surgeries, and train surgical teams. Construction companies are outfitting installation crews with AR glasses to improve the speed and quality of framing and wiring installations. And production studios are using XR to create fantastic, three-dimensionally lifelike worlds like those found in Disney’s The Mandalorian.

“We’ve already gone beyond two-dimensional technology and into three-dimensional technology,” says Nick Roseth, chief explorer at XPLR Design. “Whether we’re talking about visualization or three-dimensional audio, like what Apple’s doing in that space, we’re creating much more realistic and immersive experiences that go beyond just looking at our phone or a tablet or a computer.”

Here’s a look at what XR is, how it’s being used across industries today, and what the future could hold for the evolving technology.

Extended reality (XR) is the umbrella term for a spectrum of immersive and interactive technologies. It encompasses augmented reality (AR), mixed reality (MR), and virtual reality (VR). You can access XR through mobile devices, VR headsets and glasses, and other technologies.

Virtual reality

VR lets you immerse yourself in a digital world, experiencing the true depth and visual richness of digital objects. With traditional human-computer interfaces, you interact indirectly through a 2D view into a 3D world. But with VR, you are immersed in the world of that object and can experience it as if it were real.

Augmented reality

AR lets you bring virtual objects to the real world, in an interactive and spatially consistent way with real objects. Unlike virtual reality, where you enter the world of the object, augmented reality brings the object to you, in your world.

Mixed reality

MR describes the continuum between the virtual and the physical worlds. On one end of the spectrum are VR experiences; on the other end are AR experiences. Adding real-world elements, such as pass-through video, to a VR experience will push that experience toward physical reality. In contrast, adding virtual objects that replace or disguise the real world in an AR experience will push that experience toward digital reality. Imagine a hologram that can be placed in the physical world as though it were present.

In November 2021, leaders from enterprise companies such as Boeing, Mayo Clinic, McKinstry, and Bristol Myers Squibb gathered at the Augmented World Expo (AWE) in Santa Clara, CA, to discuss current and projected uses for XR. A term heard repeatedly at the conference was transformation. This new kind of transformation implies a more radical departure from the era of Rolodexes, drafting tables, and paper—a wholesale organizational shift where 3D immersive technologies that blend virtual and physical worlds are changing the way teams collaborate, design and develop products, and serve their customers.

Using XR in the architecture, engineering, and construction industry

The architecture, engineering, and construction (AEC) industry is a leader in adopting XR: It’s being used to plan building projects, review and refine mockups, improve coordination and quality control on jobsites, and manage buildings across their lifecycles.

1. Project planning

By overlaying cutaway sections of 3D intelligent models onto real-world sites, firms such as Gilbane Building Company, McKinstry Construction, and Gensler are using XR to test speculative scenarios before making costly decisions. Just as a prospective buyer at IKEA can use an AR app to place digitally rendered 3D furniture in her living room to check if it’s correctly sized and matches the decor, firms can use third-party apps to determine if design plans are congruent with actual site conditions.

“If it’s an empty parking lot, you can actually project the 2D plan in the physical space of that empty lot or visualize the 3D model using an augmented reality app,” says Viveka Devadas, technical account specialist at Autodesk. “That’s a lifesaver because you can identify potential clashes in the early planning and conceptual phases.”

Bringing designs from 3D modeling programs such as Autodesk Revit into VR at early stages can give architects a more visceral feel for a space before expensive or irrevocable decisions are made. “In architecture, you don’t get the feeling of the actual space you’re building,” says Hans Kellner, senior manager and principal engineer for Autodesk’s Emerging XR Technology division. “XR has really helped with that. Now, an architect can go into an immersive environment in VR and realize, ‘This volume is a lot larger than I expected,’ or, ‘It actually feels cold because it’s too large.’”

2. Design collaboration and review

Design collaboration no longer needs to be tethered to a physical office or local server. Assuming they have access to an XR-capable device, collaborators from nearly anywhere in the world, represented as avatars, can co-locate inside the same 3D model to conduct design reviews.

“This technology saves so much time and energy,” Devadas says. “Traditionally, we put everything on the wall, and all of us were stuck in a conference room making mockups, then going back to the computer and making changes. Now, you can make design decisions on the spot, in virtual space, and collaborate with multidisciplinary teams.”

Viewing a 3D virtual space from disparate locations is not the only advantage of XR. The configuration of a building’s skeleton—its framing, plumbing, air valves, and electrical conduits—can benefit from design-review packages such as Autodesk Navisworks that let teams navigate inside 3D models in real time and flag potential system clashes or measurement discrepancies.

3. Jobsite coordination

Companies are using simulated VR environments during preconstruction to improve the coordination of people, equipment, and materials on jobsites. Can a crane access building supplies and move them to the correct floor without interference? What’s the best route for a supply vehicle to navigate a busy city center and approach a site? These questions are best answered before materials and equipment are procured and put in place, Kellner says, and VR provides a cost-effective means to do so.

“Here in the Twin Cities, Cargill is one of our biggest companies,” Roseth says. “During COVID, they still had to do their work but couldn’t do it with a bunch of people on the jobsite. So they were finding every [Microsoft] HoloLens they could to continue remote work and stream that information back to people at their homes. Whereas before you’d say, ‘All right, let’s 10 of us meet at this place, and we’ll do an inspection or a walkthrough or some design planning.’”

4. Real-time installation guidance

At later stages of a build, AR helps installation crews see 1:1 building information modeling (BIM) data where and when it’s needed, says Dace Campbell, director of product management in construction at McKinstry. When McKinstry partnered with Emerald Development Group and Bouten Construction Company on a shared university health sciences building in Spokane, WA, crews used HoloLens headsets loaded with cloud-based Spectar software to install pipe hangers. Rather than consulting 2D plans at ground level, workers consulted holographic visual overlays of hanger bed connection points at the ceiling level.

Because installers had the designs directly in their fields of vision and didn’t have to repeatedly go up and down on scissor lifts, they worked faster, saving an estimated 3.1 minutes per hanger, or roughly 40% of the labor time. Projected across a portfolio of similar projects, the application could save McKinstry $1.4 million per year.

“XR aligns, I would argue, to a lean construction philosophy—if it is defined as providing the right information and the right materials to the right people in the right place at the right time,” Campbell says.

5. Safety and competency training

One important application for XR in AEC is training employees, first responders, and building occupants. “There are technical folks and nontechnical folks on a design team,” Devadas says. “The people who actually go to a construction site might be nontechnical and have no idea where a beam is supposed to go. Then, you also have folks like firefighters, who focus on preventing safety hazards in buildings. For these people, training in VR is very important because it prepares them for events before they actually happen and helps avoid hazardous situations, which is lifesaving.”

Applying information from a 2D schematic drawing or an instruction manual to a real-world situation requires an extra layer of spatial processing that can be cognitively taxing and increase the risk of mistakes. In a high-risk situation, such as evacuating an airplane or managing the reactor of a nuclear facility, there is little room for error.

“If you think about dumping a training manual of 200 pages on somebody’s desk that they’re unlikely to read in the first place or spending a day with them going through immersive experiences and showing them what they should do under certain conditions, the benefits are obvious,” Roseth says.

6. On-site inspections

AR can give inspectors and facility managers a kind of X-ray vision into a building’s inner mechanics. Is there sufficient clearance to access equipment panels? Is the building in compliance with codes and Americans With Disabilities Act regulations? Are fire protection systems properly installed? Have the right number of parts been put in the right places? Although quality-control checks can be performed using more traditional markups, AR provides a more discerning lens.

“Augmented reality is what I would call the ultimate stud finder,” Campbell says. “If there was work put in place that was covered up—and there’s lots of that in construction—the facility operator needs to know it’s there. They need to rely on the BIM data, whether that was originally put in BIM or described later, using reality capture technology. Either way, augmented reality gives people that kind of a superpower to see what’s behind the walls.”

7. Physical asset management

Building owners and facility managers are increasingly turning to digital twins to evaluate performance data about a building’s HVAC and MEP (mechanical, electrical, and plumbing) systems and assess maintenance needs and environmental impacts. Employing embedded sensors, artificial intelligence, machine learning, and Internet of Things technologies, digital twins provide information about an asset across its lifecycle, from design and construction to operation and maintenance—even its future use. In short, they are virtual replicas of physical structures, loaded with actionable data.

Historically, this data has appeared on dashboards, but that paradigm is shifting. Inside the AEC industry, consensus is growing that presenting information in XR, where it can be seen in context, can help teams grasp information more easily.

“Imagine that you have a fully virtual representation of the building,” says Nic Fonta, director and general manager of XR at Autodesk. “For each of the managed sensors, you have the information in the 3D space presented in context. You could have trends and insights; you could do simulations—all this is better understood, consumed, and used to inform future designs when you have the information in an immersive space.”

8. Client bidding and sales

Most people are familiar with virtual home tours offered by real-estate tech companies like Zillow. Similar to the street view in Google Maps, these tours offer potential buyers 360-degree walkthroughs of properties. By 2025, the VR market in the real-estate sector is projected to be $2.6 billion, according to a Goldman Sachs report (PDF, p. 7).

In many ways, the use case in AEC is analogous: Both build trust in the veracity of an unseen building. In the future, VR is likely to become even more interactive, helping clients understand subtle features of a space, such as its lighting patterns and material textures. “The goal is to provide the end client a good sense of what they’re buying into and what will be built in the future,” Fonta says.

Using XR in product design, manufacturing, and automotive

With a rich history of using XR in interior and exterior car design, the manufacturing sector is poised to expand adoption of the technology. Research from PwC projects the use of XR for services and product development to lead to a $360 billion increase in GDP by 2030.

Lukas Fäth, senior product manager for Autodesk’s VRED and automotive divisions, saw VR take hold in the industry about 10 years ago. He says the use of small-scale clay models and full-size automotive prototypes for automotive concept designs has, for some companies, given way to digital prototyping, which can be performed over multiple iterations at a comparatively low cost.

XR emerged as the next logical progression. From early-stage exterior styling reviews to more detailed considerations of light reflections, human-computer interactions, and interior functionality, XR has revolutionized the way cars and other manufactured products are built.

1. Team collaboration across the globe

Today, large auto manufacturers are using MR headsets supported by VRED to conduct global design reviews in immersive, photorealistic environments. At the European Design Center of Kia Motors, designers use Varjo VR headsets to transition fluidly between real and virtual cars, updating digital prototypes in real time. Streamed to teams across continents, the resolution of these visual displays, down to the glint of the metallic paint, is remarkably lifelike, Fäth says.

2. Design review and digital prototyping

The argument for designing in XR, rather than from a series of physical prototypes, is rooted chiefly in economics. Creating a synthetic clay model can cost up to $900,000, so replicating it digitally “is a huge value for manufacturers,” Fäth says. Inside VR, designers, engineers, and test participants can conduct driving simulations; examine a car’s ergonomics; and assess how a car’s dashboard, steering wheel, mirror array, and digital interface can best support driver’s comfort and safety.

3. Factory and assembly line planning and design

A prime example of using VR for factory design relates to spatial clearances for assembly line workers and maintenance crews. “You can send the actual repair person into VR and say, ‘Here are some of the repairs that you will have to do. Go in and try,’” Kellner says.

Something relatively simple, such as removing a cover of an air compressor, can be complicated by surrounding machinery and access routes. VR can flag these issues prior to construction, avoiding project delays or costly rework.

4. Product assembly and digital factories

Efficiencies can be found further downstream as well. Boeing’s use of the HoloLens and BARK (Boeing Augmented Reality Kit) as part of its digital manufacturing program exemplifies how AR can shave time and costs from later production stages. In a session at AWE, Laura Bogusch, the general manager at a Boeing airplane manufacturing site in Salt Lake City, says that mechanics used the system along with QR code guides to install 150 miles of cabling per plane. After reporting first-pass quality compliance of 88% and a 20% reduction in time, the company will use the system to wire its new Air Force One fleet.

Further afield, aerospace companies such as Airbus and Boeing plan to create digital factories in the metaverse. In these shared virtual spaces, 3D digital twins of aircrafts and manufacturing systems are supported by data reservoirs that, according to a report from Reuters, “stitch together every piece of information about the aircraft … from airline requirements to millions of parts to thousands of pages of certification documents.” By running sophisticated performance tests, if these systems evolve as executives and engineers hope, they could significantly improve production speed and quality.

5. Training for facility managers, machinists, and maintenance workers

A study by the Manufacturing Institute found that 78% of surveyed companies were “very or somewhat” concerned about the exodus of the sector’s aging workforce; as of 2017, nearly a quarter of these employees were 55 or older. Facility managers may become particularly hard to find, as 40% of them are projected to retire in the next eight years. This is one reason why the use case for XR as a training tool has emerged in manufacturing.

“I wouldn’t call facility operators a dying breed,” Campbell says. “But there are fewer of them by the day; they’re retiring out of the industry, and we’re bringing in new ones. So there’s a model to say, ‘Let’s make a remote expert out of that guy who’s been turning wrenches for 50 years.’ And younger, greener staff can go out to the jobsites or the functioning buildings while senior employees advise them about how to change a filter, what sequence to turn the valves, and so on.”

“There’s the whole assembly aspect, as well: training people to assemble the car and making sure the movements they perform are ergonomic so they won’t be harmed by doing them 50 times a day,” Fäth says.

6. Co-design with customers

Beyond workflow efficiency, digital models aim to improve the likeliness of customer satisfaction. Although many car companies start with a physical model, Fäth says, they will then replicate that model with a digital twin in MR that has a translucent layer of interactive content on the car’s surface and in its interior.

At early design phases, customers participate in Q&A sessions to help refine key features. “Eye gaze, where you’re looking, is extremely interesting at the early design phase,” Fäth says. Using heat maps to plot where customers are looking, and for how long, can provide useful insights about the user experience and how to maximize customer satisfaction during the design phase.

7. Marketing and sales

VR can also be used in manufacturing and product design to woo prospective clients. That’s not simply because it’s “a shiny new technology,” Campbell says. “It’s really about giving the client confidence that you understand the scope of work, and you’re in good alignment—like, this is what we’re going to do and how we’re going to do it.”

Many high-end automotive brands already offer sophisticated digital experiences at their dealerships. “You’re sitting in a virtual car, and they can change all the configurations,” Fonta says, referring to color and material options. “You have the flexibility to choose the final car you would actually buy—not just say, ‘I like the orange one more than the black one,’ then see, on the final delivery, there’s too much orange in the car.”

Using XR in media and entertainment

XR is seeing some of its strongest momentum in media and entertainment. From recently released VR games such as Cities: VR for Oculus (Meta) Quest 2 to Snapchat and Instagram Story AR filters that add digital layers (such as light effects or a new head of hair) to your photos, the technology is becoming familiar to many.

Whether it’s adding photorealistic scenery to video games, engaging film audiences, or creating virtual exhibitions like those found at the National Gallery in London, the convergence of virtual and real worlds is happening now. Gaming, a market valued at $11.5 billion in 2019, is projected to grow by 30% each year from now through 2027. And AR in retail is on the rise. According to a research report by Deloitte and Snap (the parent company of Snapchat), by 2025, 75% of the population—and almost all smartphone users—will be using AR frequently (PDF, p. 4). Brands such as Nike and L’Oreal have leaned into the technology, with AR experiences letting prospective customers try on sneakers or experiment with makeup using the front-facing camera on their phone.

1. Virtual production

Hilmar Koch, director of Autodesk’s Industry Futures group in media and entertainment, says one of the newest use cases for XR is a kind of TV and film staging called virtual production. Rather than placing actors in front of green screens, resulting in the counterfeit look of, say, a Seinfeld episode when Jerry and Elaine are riding in a taxi, actors can be placed before holographic LED volumes depicting imagined settings. “This is how Lucasfilm is doing their work, nowadays, on The Mandalorian,” Koch says.