Challenges in HoloLens Application Development

Microsoft HoloLens is a holographic computer built into head-mounted mixed-reality smart glasses. It is powered by Windows 10, which supports the Universal Windows Platform (UWP) application architecture. This means that all UWP applications can be installed on HoloLens if the application satisfies the minimum required configuration for the HoloLens device family. While HoloLens specific features can be factored in at the design stage when building new UWP applications, there are practical challenges when upgrading existing UWP applications to leverage HoloLens capabilities. This paper discusses the lessons we learned and various challenges we faced when developing new UWP apps and updating existing UWP applications to make them work on the HoloLens. Furthermore, we present UWP application design considerations to be taken into account early in the development lifecycle to support the HoloLens device family. Finally, we provide  plausible solutions and suggestions to make the upgrade process less complex.

Adding support for HoloLens devices to an app can bring several benefits. First, Microsoft HoloLens uses 3D holograms and spatial mapping to provide an immersive user experience. Most application developers target both mobile and desktop devices for their applications, which use their 2D frame to render the app’s content. If the app’s objects are presented using a 3D model, for example machine CAD models and building architecture designs, users can more clearly understand the design intent. Additionally, by adding HoloLens to the set of supported devices, developers can make their applications available to additional groups of users. 

Hololens features

In a nutshell, HoloLens can create hologram objects which are made of light and sound. It projects 3D holograms which are intangible and can be placed in the real world. The holographic shell represents a 'Mixed-Reality World' which is a combination of real-world objects and the holograms created by the system. HoloLens can create 3D holograms which can be placed alongside with real world objects, and the user can interact with them. Holograms can be attached to horizontal or vertical planes, such as room walls or floors. For example, movies can be played on the wall, internet browsers mounted on the walls, or Skype calls follow you as you walk around the space. Interactive object models can be created using Holograms to demonstrate proofs of concept. HoloLens recognizes the room model with its geo-coordinates which helps us to create location-aware applications. Applications running on it can leverage Spatial Mapping and Spatial Sound capabilities. The following is the list of the key features of HoloLens:

• HoloLens uses the holographic shell, an interaction model, and perception APIs in the app model.
• It scans the surroundings, such as room, and creates a 3D object model for the room.
• It uses the room model to place the holograms in the environment around you.
• It allows to pin and unpin holograms to real world objects and locations, and sizing and scaling them in real time.
• Holograms can respond to human interactions such as gaze, gestures, and voice commands.
• Holograms created by HoloLens can be shared with other HoloLens users.
• HoloLens allows us to capture the mixed world frames and save it as image or video using Mixed Reality Capture (MRC) capability.
• Holograms can be displayed either in 2D or 3D frames.
• HoloLens uses the real world as the canvas to display holograms.
• Based on the dimensional nature of the application, HoloLens renders them in the mixed world as 2D or 3D holograms.
• HoloLens delivers an immersive and interactive user experience with clear hologram projection, spatial mapping, spatial sound.

Even though HoloLens is packed with many useful features, it has a few shortcomings, including:

• The viewport of the HoloLens appears within a small rectangular box in the center of the display. We have to remember where we have placed the holograms in the room.
• Even though it is possible to deploy existing UWP 2D applications on HoloLens, the application’s boundary is very much limited within the 2D window. This will not provide a good user experience when handling user input operations like gestures.
• Considering the cost of the device and its adaptability, updating small scale applications for HoloLens will not be profitable at the moment.
• Manually identifying unsupported APIs and features used in the existing application and implementing alternative functionality is very difficult.

Updating existing UWP apps

The new Universal Windows Platform introduced the classification of devices into Device Families. A device family shares a set of APIs from the core features of the OS. Microsoft UWP provides a common app platform and core API layer for every device that run on Windows 10 OS. Windows Runtime (WinRT) APIs are common to all device families, but not all the APIs are necessarily supported for all device families. In addition to WinRT, APIs applications can call APIs specific to the device family the app is running on.

All existing UWP applications, mostly targeting mobile phones and the desktop, are expected to work on HoloLens, too. Since these mobile and desktop applications are two dimensional in nature, they are rendered on the 2D surface of the HoloLens. Even though applications created using Unity appears to be 3 dimensional in nature, they are also rendered in the 2D canvas. Existing UWP applications can be made installable on HoloLens by declaring the TargetDeviceFamily with Name="Windows.Holographic" in the Dependencies section of Package.appxmanifest file.

Challenges in updating apps

HoloLens does not support some of the Universal APIs and features that have reached the same level of support on desktop or mobile. Features like Notifications, File Explorer, Messaging, tiles, etc., are not fully functional in HoloLens. Microsoft has released the list of unsupported APIs on their website. They also mentioned that the list will be updated as and when the APIs are supported in future SDK releases. We can classify the challenges in updating an existing app into two wider groups based on the nature of their impact on development and implementation.

Implementing Universal APIs

Most of the existing UWP applications are developed targeting mobile and desktop families and may be using device family-specific features. In one of our existing UWP projects, Push Notifications is implemented for the Mobile device family. Since this feature is not supported fully in HoloLens, we had to use Windows.Networking API as a workaround solution.

HoloLens Specific Challenges

Holograms, Spatial Mapping, and Spatial Sounds are specific to the HoloLens device family. When updating 3D applications created by Unity, all the 3D game objects are rendered as holograms with the real world as a canvas. Running the existing 3D application as it is in HoloLens will deliver a poor user experience. Adding spatial mapping and spatial sounds makes the application much more realistic. To add spatial characters, the existing game objects require a good amount of effort. Most of the existing game objects need to be modified to add spatial mapping and sound to individual objects.  In addition to gaze input, which closely simulates mouse cursor, gestures and voice commands should be added to the scene. If we have used custom gestures in our existing application like circle gesture, swipe left to delete, swipe right to confirm, etc., we will find it very difficult to make it work on HoloLens. HoloLens is packed with Interaction Manager, which can be extended to create new gestures. We found it difficult to identify the hand position in the 3D frame. We are exploring the HandInput class available in Microsoft’s Galaxy Explorer project to identify hand position. Alternatively, this can be achieved using third-party computer vision libraries like OpenCV.

Issue Analysis

Issues related to running UWP applications on HoloLens are widely discussed in forums, blogs, and articles. We have experienced difficulties and issues while running existing UWP applications on the HoloLens device and emulator. Since this is an initial stage, we can list such issues and problems under four high-level categories:

1. Deployment Issues
2. Debugging Issues
3. Environment Issues
4. Development Issues

Deployment Issues

Deployment to emulator will be easy since Visual Studio knows the IP of the emulator. When targeting real devices for deployment, though, we cannot use the IP address given by Cortana since it is the external address. The internal address can be obtained from Advanced Options in device Settings. Using that IP address,  HoloLens can be paired with Visual Studio 2015. in case of incorrect IP address mapping the debug console will display the error following message :

“DEP8000: Unexpected deployment failure: OperationFailedException: The network event being waited on triggered an error.”

Deployment errors

Some deployment errors and their probable resolutions are listed below for better understanding.

1. Failure reason: Failed to start deployment. Failure text: Install failed. Please contact your software vendor. (0x80073cf9)

There are multiple causes for this error such as the application being already installed on the device, outdated Windows SDK or HoloLens Emulator, and incorrect architecture of the application. You need to check AppXDeployment-Server log for additional error information. To view the AppXDeployment-Server log,

• Goto Start menu →  Programs or All Programs → Administrative Tools → Event Viewer.
• Then expand Applications and Services Logs → Microsoft → Windows → AppXDeployment-Server → double click Microsoft-Windows-AppXDeploymentServer/Operational

2. Failure reason: Failed to start deployment. Failure text: A Prerequisite for an install could not be satisfied. (0x80073cfd)

This means that some or all of the dependencies are not available on the device. Check the minimum and target build numbers in the Package.appxmanifest file. All the prerequisites should be satisfied for successful deployment.

3. App manifest references the image 'images\Square71x71Logo.png' which does not have a candidate in main app package

UWP has set scale-200 as the default image scaling factor. But Visual Studio is still using the older specifications which creates scale-100 image by default and not scale-200 image. This error can be fixed by adding scale-200 images,by removing “.scale-100” from the file name, or by adding a “Square142x142.png” file (200% of 71x71) to the asset mapping. We will not get this error in debug mode. This error can only be reported only when we build the application in release mode.

1. DEP6957: Failed to connect to device '192.168.xyz.abc' using Universal Authentication.

This error occurs when the HoloLens device is not paired with Visual Studio correctly. Uninstalling the application from HoloLens and reinstalling it will help resolve the issue.

Some of the error messages displayed in the console are not having exact error information or description. For example, when trying to deploy an application on a HoloLens emulator or device using Device Portal Apps Manager console, if the application is already installed on the device, you will get: “Please contact your software vendor”, which doesn’t explain the issue. Instead, error messages like “Application already installed” make more sense and the developer can fix  it promptly by first removing the application before deploying.

The most common resolutions to Deployment issues are listed below:

• Changes in AppxManifest configuration
• Updating Visual Studio project properties
• Checking SDK version numbers
• Build settings for Debug and Release configurations
• Checking whether all the dependencies are installed
• Uninstalling the application from device or emulator and reinstalling it (Sideloading)
• Asset availability with required configurations (dimensions and resolution)

Debugging Issues

In this category fall issues related to enabling debug mode, attaching the debugger to the device or emulator, connecting to the device portal, remote debugging, managed, unmanaged, and mixed-mode debugging.. Any issues related to Hyper-V management and Ethernet switches are also listed here. This category also covers issues related to the NuGet Package Manager configuration and settings to enable automatic download of missing packages or dependencies.

One of the debugging issues we faced is described below.

“The program '[4764] ModelExplorer.exe' has exited with code 255 (0xff)”

We learned that this issue can be avoided by using proper debug build settings:

1. Select build debug/x86 for debugging
2. Set Application Process to Mixed (Managed and Native)

Microsoft may consider the below mentioned suggestions to avoid the debugging issues:

• Providing the exact error descriptions explaining the issue
• Better documentation on settings to enable debug mode
• Proper connectivity guides to connect to the emulator and device (over USB or Wi-Fi for remote debugging)
• Dependency Manager (NuGet) configuration changes

Environment Issues

This category includes issues related to the development, debugging, testing, and system environments. Most of the issues related to the SDK can be solved by correctly installing the required SDKs (Windows 10 SDK, HoloLens Emulator, Visual Studio 2015). If the HoloLens emulator software is not installed properly it will not be visible in the Visual Studio 2015 device list, so it will not be possible to directly run the build. We should ensure  correct versions of software builds and update patches are installed. For Example, if the Windows 10 SDK Version 10.0.143.93 is installed on Windows 10 OS version 1511, the design mode in the Visual Studio XAML Editor will display a warning message like the one shown below (see Figure 1), since this version of Windows 10 SDK requires the minimum OS version to be 1607. If the development machine is not running Windows 10 Anniversary Edition, we can use an alternative XAML editor like XamlPadX, a basic visual editor for XAML which can be downloaded from Microsoft Blog.

Figure 1. Error due to Network Adapter issue

Incorrect installation of the HoloLens emulator software will result in issues when accessing the Emulator Device portal or when deploying the application directly on the emulator from Visual Studio 2015. The emulator will show a warning as displayed below (see Figure 2) if the network adapter of the emulator is having issues.

Figure 2. Warning message for OS version minimum requirement

“Updating the SDK” and “Installing correct versions of the software” will most probably fix the issues related to software environment.

Development Issues

All the issues and problems encountered when developing or upgrading an app are listed under this category. Since Visual Studio is not displaying compile-time errors or warnings for unsupported APIs and features, the developer has to be careful to implement alternate workarounds to ensure the required functionality is not broken. For example, we have faced issues when adding custom gesture listeners to an existing 2D UWP application. The problem is that every time the user tries to initiate a swipe gesture, the gaze points outside the 2D frame, so the application is pushed into suspended mode. When this happens, the app cannot recognize any gestures even if they pass through the 2D frame. Dwight Goins, Computer Science Instructor at TEALS, Microsoft and HoloLens developer, has answered a similar question in the HoloLens Forum providing two suggestions:

1. Create a 3D app, scan the room, and then switch to a 2D View and then, you can recognize gestures, and raise double click events to handle them.
2. Another idea is to create a very large 2D window, and within the window, highlight or make a small rectangle very visible to show the UI, to recognize the clicks, and then select the command.

We followed the second suggestion since we target applications for multiple device families and adding one bigger window to the app only for HoloLens can be implemented with less effort.

Testing UWP apps on hololens

Testing HoloLens applications is similar to testing Windows applications. If you already have experience testing windows application  you will likely feel comfortable testing HoloLens apps. HoloLens applications can be tested either on the emulator or on real devices. The Perception Simulation library is used for automation testing to send fake human input to the app. We can simulate gestures and gaze input using the IPerceptionSimulationManager API. Also, simulating the room model is possible in the emulator, so we can run automation testing in the emulator.

Minimizing go-to-market time

Minimizing the go-to-market time is very important when planning for an app upgrade. We have identified some key points which will help you upgrade existing applications to support HoloLens:

• Configuration upgrade: use a tool to upgrade the configuration of existing UWP apps. Such a tool will modify existing projects by programmatically updating their manifest file to add support for the HoloLens Device Family. For example, the tool can remove <PropertyGroup> elements that are configured for the 'Debug|AnyCPU' or 'Release|AnyCPU' builds  since  AnyCPU is not supported for Windows 10 apps.
• Compatibility Testing Tool: use an automated testing tool to identify libraries and APIs required by existing applications that are not supported in HoloLens device. Developers can use this tool to check whether all the libraries used in their existing application are compatible with HoloLens. This tool can output only the list of unsupported APIs used in the existing UWP application.
• API Wrapper: Providing wrapper functions agnostic to device families to account for unsupported APIs. These API wrapper will be useful when the developer wants to provide the unsupported API functionality with a different implementation.
• Extending HoloToolkit: HoloToolkit is a collection of prebuilt scripts and components provided by Microsoft.

Effective usage of HoloToolkit will certainly reduce the development time of some functional modules and increase the productivity of the developer.

Extending HoloToolkit with additional code components and reusable dependency packages to accelerate holographic app development.

Suggestions

Based on the our learnings, a few things can be suggested to overcome the challenges in HoloLens application development. These suggestions will help you to reduce the go-to-market time and improve the application usability. Before starting application development, identify and aggregate the issues already reported by other developers from various forums. You can narrow down your search to the application specific features and functionality. List the identified issues and problems under various categories and sub-categories as per your application’s requirements. Since you cannot rely on the solutions provided by the forum contributors, you have to analyze and test the solutions provided for the issues reported in online forums and collaboration websites.

If you are planning to add support to HoloLens devices in your existing UWP application, you have to identify all the unsupported APIs and libraries used in the existing app. Prepare a detailed document explaining how to wrap every unsupported API based on device family. Try to use alternate API functions, wherever possible, which are supported in HoloLens. Some features like Voice Commands for navigation menu items and action button events would also contribute to improve the user experience of your application on HoloLens.

Conclusion

When planning for new app development, all existing challenges can be factored in. Conversely, to upgrade existing apps and add Holographic support to them, we need to carry through a thorough analysis of our application code, process flow, and APIs and features used. It can be useful to prepare a comparison chart of various capabilities supported by different device families including mobile, desktop, HoloLens, IoT, and Xbox, which will help in identifying the possibilities of adding HoloLens specific capabilities.

Existing UWP 2D applications leverage mobile phone and desktop computer software and hardware capabilities. When adding HoloLens support to an app, there is a huge opportunity to take advantage of the additional holographic capabilities. Some of the existing 2D functional and UI design patterns can be automatically updated to leverage HoloLens features. For example, it is possible to automatically create and enable voice commands for the navigation menu items when upgrading the app. This can either be added to the HoloToolkit library or a to the separate upgrade tool which can inject all the additional features to the application based on the developer’s requirement.

Microsoft has already released Project Centennial to convert existing desktop applications to the UWP space. This can be extended to add HoloLens support to the converted apps. We can develop tool to check the API compatibility which will reduce the Go-To-Market and provide fallback mechanism for the unsupported APIs. Developing wrapper APIs for unsupported APIs would give a structured way for developers to take action as per their requirements.

About the Author

Vinothkumar Arputharaj is working as a Senior Technical Lead in Technology Focus Group, part of the Technology Office in Engineering and R&D Services group of HCL Technologies Ltd. He's got a Master’s degree in Computer Applications from the Bharathiyar University, Coimbatore, India in 2009.