Mixed Reality Solutions

Mixed Reality (MR) can be considered an enhanced version of AR which gives better immersion. It provides the ability to visualize virtual/digital elements along with the physical spaces.The integrated hybrid environment thus created is immersive, interactive, intelligent and insightful.

If you can imagine yourself in your living room where you have virtual objects that dynamically interact with you and your physical environment in a purposeful way, you would be experiencing mixed reality. For instance, while you are inside MR space, the virtual 3D modelled ball you throw, hits on your physical wall, bounces back, falls with gravity and rolls along the surface of your floor. Such interactive experiences require the use of a headset such as Microsoft HoloLens.

In summary, this mixed reality definition considers the visualization of both real world and digital objects creating a unique interactive experience. It is sometimes called a hybrid reality as it allows you to remain connected and move about in the real world and derive an experience that is augmented and enriched by digital elements.

Mixed Reality technology and devices utilize a variety of sensors that scan the physical environment and also the location of the user simultaneously. The scanned data thus obtained from these sensors is processed using complex algorithms that accurately develop a 3D map of the environment and also the current location of the user/device.

The complex processing system also recognizes the positions travelled previously by the agent and the environment it is in. This enables more accurate navigation and also the ability to interact with the mapped environment with interfaces and interactions designed specifically help to revolutionize the reality of the user experience. The expert use cases for this technology have resulted in a plethora of applications of the mixed reality technology.

SLAM or Simultaneous Localization and Mapping has several applications in the real world for a long time such as autonomous robots, self-driving cars, drones, etc. One of the exciting areas where it’s also used recently is in Mixed Reality. SLAM technology is the backbone of mixed reality applications as it delivers the ability to integrate virtual objects within the physical space and enables interaction from the perspective of the user.

SLAM is a computational optimization tool that maps an unknown environment and at the same time localizes the user with respect to that environment, particularly for situations without any prior reference point. The technology uses complex algorithms to provide the desired information for developing mixed reality solutions to meet industry needs.

The most commonly known devices to deliver Mixed reality applications and experiences are Microsoft HoloLens and MagicLeap One powered by Spatial computing (SLAM) technologies. In addition to these two, there is also NReal’s Light MR goggles. There are other headsets which are either under development or yet to be released commercially.

A mixed reality device employs various sensors that track the environment and the user/device simultaneously in real time. For example, Microsoft HoloLens has various special sensors each with a specific function such as

Multiple optical sensors (BW Cameras), with two on each side for peripheral sensing – Spatial awareness using Visual SLAM

“Holographic Processing Unit” to render holograms over the See-Through glasses into the environment

Down facing IR Depth camera (Kinect Sensor) to pick up hand movements and gesture interactions

RGB camera for capturing HD video of the surrounding environment

Array of specialized speakers, microphones, ambient light sensors

It is not absolutely necessary to have internet connectivity to use a mixed reality device and run mixed reality applications. It may be essential to set-up the device, but beyond that it can function independently without it to perform mapping and interactions. This is also dependent on the application or use case deployed such as Digital Twin which requires live streaming of IIoT data or while running applications from the Cloud where connectivity becomes necessary.

The question of mixed reality vs augmented reality can be dealt with a few details on both technologies. Augmented reality is the mere overlay of digitally created content on the existing environment where there are no interactions between the physical and digital/virtual objects.

On the other hand, mixed reality enables the interaction of digital content in the environment using SLAM technology. The experience thus obtained is significantly more enhanced and interactive than what would be obtained using AR. There is no immersion experienced in AR while the level and nature of immersion with MR is much more. With the recent advancements in technology such as Google’s ArCore and Apple’s ARKit, the line between AR applications and MR applications is getting increasingly blurred. The fully immersive experience however is obtained in a virtual reality device.

While the basic underlying computational technology such as SLAM and sensor implementations are pretty much the same, it could be argued that both are essentially providing a similar experience. These technologies have evolved tremendously since the time these terminologies were first created. As development advanced and competition for the adoption of these technologies increased, they were also described differently by different companies. Microsoft presently calls it Mixed Reality and Magic Leap calls it as Augmented Reality.

Mixed Reality and Virtual Reality are fundamentally different technologies and provide unique experiences to the user. Therefore, the mixed reality vs virtual reality debate may not be really required.

Virtual Reality requires the use of a head mounted device that completely covers the user’s vision and completely disconnects them from the real world. The user is teleported to a digitally created virtual environment where they can navigate and perform activities using gestures and hand controllers. Since the head mounted device blocks your ability to see your actual surroundings, the use of virtual reality devices should be confined to the safety of a room. Unrestricted movement could result in harm to the user as he/she is unable to see the actual environment.

With Mixed Reality applications, the user is still able to view the real world and the experience is augmented with digitally overlaid content that can actually interact with the user’s environment. The user is able remain connected and move about in a safe manner with digital experiences that are integrated to deliver a unique experience. Mixed Reality is also immersive in a different way and requires the use of a different kind of head mounted device that allows the user to view both the real and digital environment seamlessly. The applications for these technologies are therefore unique and have different use cases developed for effective utilization.

See-Through Mixed Reality uses transparent optics to view the real world with digital content superimposed on the see-through visor on the headset.

In contrast, the Pass-Through Mixed reality has cameras that provide a live video feed of the environment, rendered on the LED display with digital content superimposed. The field of view with the pass-through MR system is broader than the See-Through MR device. For example, Oculus Quest-2 VR headset utilizes its BW tracking camera sensors for providing a see-through MR experience.

The Windows Mixed Reality (WMR) Headset is essentially a Virtual Reality headset while the HoloLens is a Mixed Reality (MR) headset. The former completely encases your eyes and cuts off visual access to the surrounding real world. While HoloLens enables continuous viewing of the outside world and the digitally augmented interactive experience, the WMR headset does not do so.

Mixed Reality(MR) applications are found in many industries today and most of them can obtain maximum advantage. That depends on the proper identification of use cases and developing solutions for those.

The manufacturing and medical industries could be the biggest beneficiary of MR. The applications in this industry are in areas such as design, operations, maintenance, training, safety, reliability etc. The areas mentioned above are common to many industries such as oil & gas, petrochemicals, mining, nuclear, utilities, automotive, pharma, healthcare, real estate, electronics and many more. MR delivers greater remote collaboration, increased productivity, better decisions, improved efficiency, cost savings and customer satisfaction.

Mixed Reality technology is used for many different applications by enterprises today. Here are some mixed reality examples. For instance, design teams can visualize design elements in the desired environment or conditions and remotely collaborate on design reviews in real time and dramatically reduce the time and cost incurred to complete the overall design process. This brings new products faster to the market.

Mixed Reality applications also enables overlay of critical information that supports the execution of critical tasks by relaying real-time guidance or decision factors from experts outside the workspace. This remote expert support advantage lends to greater success rates and more wholesome outcomes for the enterprise. The remote support application is used in industries in remote locations, hospitals, critical facilities and even academic institutions.

Medical professionals can train on physical examination and surgery with virtual models using MR headsets without the logistical challenges associated with cadavers. This type of training can be provided anywhere and anytime and opens the doors to unlimited possibilities.

In the automotive industry, mixed reality solutions enable customers to visualize a car with the ergonomics, interiors, preferred colors and accessories and helping the buyer decide on customization desired even before the card is swiped. Therefore, the usage of MR is endless and depends on how enterprises wish to deploy it.

Microsoft Mesh is an innovative technology that helps blending virtual and real worlds to enable better collaboration across multiple geographies and deliver effective teamwork. Humans tend to produce their best work in face-to-face interactions and by working in teams.

With work being outsourced globally and the advent of the COVID-19 pandemic, remote work and collaboration has become the new normal. Microsoft Mesh enables the presence of virtual avatars and eventually holographic representations of individuals themselves to be “holoported” into the virtual remote collaborative workspaces leading to better outcomes. HoloLensmixed reality applications utilizing Microsoft Meshcan enable participation in the MR space for the desired multiuser remote collaboration.

As per Microsoft, the HoloLens 2 Industrial edition is considered Intrinsically Safe as it is suitable for use in Class I, Division 2-Groups A, B, C, and D hazardous location environments since these devices do not generate enough energy that could ignite a hazardous mixture.This enables mixed reality applications to be used in areas where some flammable materials are expected.

Users may be interested to know that AR devices such as Realwear HMT-1Z1 are certified as Intrinsically Safe (IS) which are truly safe for use in hazardous environments and known for its ruggedness. Such areas are defined by the Class/Division or Zone system and IS devices are identified appropriately.

Virtual Reality is completely immersive as it fully blocks the user from the outside world with the highest Field-of-View (around 110 degrees of FOV), while mixed reality overlays interactive virtual elements in the physical environment that appears less immersive (55 degrees FOV). Mixed reality could provide unnecessary distraction as it uses the surrounding environment and the level of immersive experience that it does provide gets affected.

The visual quality of the synthetic VR environment does not come close to the natural environment seen through MR goggles, but it can be made highly realistic as required by the user. The visual quality of VR would be more consistent than the graphics rendered in MR under variable ambient conditions.

Outdoor lighting varies depending on the sun and cloud cover which impact the MR experience in the following ways.

Using Mixed Reality glasses in a bright outdoors will degrade the visual quality of the graphics rendered since the bright ambience washes it out.

Also, under such bright conditions the B&W tracking cameras used for Visual SLAM would get saturated and nothing much can be seen. Conversely, if there’s less light, these cameras would not be able pickup much details from the environment causing instability in the mapping and tracking functions.

Therefore, ambient lighting level of 500-1000 lux would be considered optimal for mixed reality devices.

The biggest benefit of Mixed Reality applications is the ability to integrate or merge the digital and real worlds. With such integration, the visualization of new realities, data and information can be accomplished that deliver improved performance, productivity and returns on investment.

For example, engineers can collaborate on design reviews of virtual prototypes and models in the shop environment or customers fit-out and decorate their new home by trying various options before finalizing purchasing decisions. Mixed Reality solutions are deployed in manufacturing, product design and development, training, marketing, maintenance, reliability, safety, health care, customer service and much more.

Some of the practical challenges affecting the implementation of mixed reality are organizational rather than technological. However, these are not huge challenges and companies can overcome implementation challenges quite easily.

One of the first steps towards successful MR implementation is having an internal champion or lead, typically a senior executive, who understands the benefits of this technology in the context of the organization’s vision and long-term goals. Having such a change agent helps explain the purpose and uses of this technology to leadership and stakeholders for buying into the concept and obtaining necessary budgetary support for project implementation.

Companies should also ensure that budgetary cycles are aligned with project cycles and necessary capital budget allocation happens. They also need to remember to progress steadily and not rush for results. The adoption and maturing of technologies take time and sustained effort from all internal stakeholders can deliver better outcomes with mixed reality technologies.

The implementation of 5G mobile networks would have a positive impact with the implementation of mixed reality solutions. The 5G network not only provides higher connectivity speeds, but also delivers with lower latency which is essential for a seamless MR experience.

The higher 5G bandwidth enables faster processing and transmission of the graphics workload necessary for higher quality and frame rates while the lower latency neutralizes lags in tracking head and hand gestures/movements.

Such advantages enable a wider adoption of mixed reality solutions as they need not be constrained by processing limitations and could take advantage of the cloud to host applications and data. As the 5G trials are completed across the world, developers and solution providers need to accelerate efforts to address use cases and develop appropriate solutions to companies worldwide.

For a detailed comparison of 4G Vs 5G please click here (FAQ No. 10)

Mixed Reality applications are definitely going to transform the way we work, play and live in the 21st century. The tremendous advances in these technologies in the past few years is proof of its potential to be transformative. The COVID-19 pandemic has accelerated the use of mixed reality as remote working, collaboration and assistance has been a necessity to ensure business continuity. Such advances are only expected to be adopted more widely as accessibility increases and cost decreases for all industrial and commercial applications. Mixed reality will find greater applications in education, research, industry, safety and of course, entertainment.