Writing your first holographic application

It is about time we start writing some code. I will show you the different options you have when you want to write software for the HoloLens and give you some helpful pointers for setting up an ideal development environment for these kinds of projects.

The tools we need to write software

First of all, you will need Windows 10. The version you use does not matter if you have a device; however, when you want to use the emulator, you need to have the professional edition of Windows 10. The home edition does not support Hyper-V, the virtualization software that the emulator uses to run a virtual version of the HoloLens software.

Next, you need a development environment. This is Visual Studio 2015 Update 2 at least. You can use the free community edition, the most expensive and compete enterprise edition, or any version in between--the code will work just fine no matter what version you use.

When you install Visual Studio, make sure that you have the Universal Windows Platform (UWP) tools (at least version 1.3.1) and the Windows 10 SDK (at least version (10.0.10586) installed. If you want to use the emulator, and trust me, you do want this, you have to download that as well. Again, in this case, you need to have Hyper-V on your machine and have it enabled.

Your very first application

I assume that you have your environment set up and you have Visual Studio running. Let's start with our first program. Just follow along:

1. Choose File | New | Project. In the templates, select Visual C# | Windows | Universal | Holographic | Holographic DirectX 11 App (Universal Windows):

New Project wizard in Visual Studio

2. After this, you will get a version selector. The versions displayed depend on the SDKs you have installed on your machine. For now, you can just accept the defaults, as long as you make sure that the target version is at least build 10586.

SDK version selector

3. Name the project and the solution HelloHoloWorld.

That's it! Congratulations! You have just created your very first HoloLens application. Granted that it is not very spectacular, but it does what it needs to do.

As a side note, I will use the term "app" from now on to distinguish mobile or holographic applications from-full-blown applications that run on desktop computers. The latter are usually called "applications", whereas the former, usually smaller items, are called apps.

Deploying your first app to the emulator

To make sure that everything works fine, we will first see if we can deploy the app to the HoloLens emulator. If you have decided not to install this and choose to use a physical device instead, please skip to the next part.

In the menu bar, you can choose the environment you want to run the app on. You might be familiar with this, but if you are not, this is where you specify where the app will be deployed. You have several options, such as the local machine and a remote machine. Those two are the most used in normal development, such as if you are writing a universal Windows platform application or a website, but that will not work for us. We need to deploy the app to a Holographic capable device such as the emulator. If you have installed that, it will show up here in the menu, as follows:

The deployment options

The version of the emulator might be different on your system--Microsoft continuously updates its software, so you might have a newer version. The one shown here is the latest one that was available during the writing of this book.

Now that you have selected this, you can select Run (through the green arrow, pressing F5, or through the Debug | Start Debugging menu). If you do that the emulator will start up. This might take some time. After all, it is starting up a new machine in the Hyper-V environment, loading Windows 10, and deploying your app.

However, after a couple of minutes, you will be greeted with the following view:

Our first HoloLens app in the emulator!

You will see a rotating multicolored cube on a black background.

How to use the emulator

Before we can dive into the code, it would be worthwhile to learn to use the emulator a bit. Obviously, you cannot use the gestures we have talked about before--it will not see what you are doing and you have to emulate these. Fortunately, it is not too hard to get used to the controls. You can use the mouse or the keyboard, or you can attach an Xbox controller to your machine and use that as well. We will cover the main controls; the more advanced options will be discussed when we need them in later chapters.

The following table shows the Keyboard, mouse, and Xbox controls in the emulator:

Desired gesture	Keyboard and mouse	Xbox controller
Walk forward, back, left, and right	W, S, A, and D	Left stick
Look up, down, left, and right	Click and drag the mouse; use arrow keys on the keyboard.	Right stick
Air tap	Right-click on the mouse. Press Enter on the keyboard	A button
Bloom gesture	Windows key of F2	B button
Hand movement for scrolling, resizing, and zooming	Press Alt key + right mouse button, then move the mouse.	Hold right trigger and A button, then move the right stick

In the center of your view, you will see a dot; this is the cursor. Do not try to move this dot--this will always be in that spot. Move the emulator with the walk and look options to position the dot on the menu item you want to choose, then perform the air tap with the options mentioned above. This takes some practice, so try it out for a little while; it will be worth the effort.

When you are using the emulator, you will note that Alt + Tab does not get you back. This does make sense since you are working in a separate computer and Alt + Tab does not work on HoloLens. You really have to click somewhere out of the emulator screen to redirect the input back to your own computer.

Deploying the app to the device

Using the emulator can be a great tool in your toolbox, especially when you are working in a team; having a device for each developer is usually not an option. The devices are pretty expensive, and you don't need one to develop your app. Of course, when you are working on a project, it is invaluable to deploy to the device every now and then and see how your code behaves in the real world. After all, you can only experience the true power of the HoloLens by putting it on.

So, we need to deploy the code to the device. First of all, you need to make sure that you have the Developer Mode switched on. You can do this by going to the Settings app and selecting the For Developers option in the Update part. There, you can switch the Developer mode on.

Once you have done this, you can start the deployment from Visual Studio to the device. To do this, you have two options:

• Use Wi-Fi deployment
• Use USB deployment

Since you already have set up the device to use the network, you can use the option. This one tends to be slightly slower than using the USB deployment, but it has the advantage that you can do this without having a wire attached to the device. Trust me--this has some advantages. I have done a deployment through USB only to discover I could not immediately stand up and walk around my holograms--I forgot the cable was still attached to the device.

Deploying through Wi-Fi

If you chose the Wi-Fi deployment option, you can choose Remote as your deployment target. If you do this the first time, you will get a menu where you can pick the device out of a list of available devices. The autodiscovery method does not always work--every now and then, you need to enter the IP address manually.

You can find this IP address in the Settings | Network & Internet | Advanced Options menu on the device. This is the same IP address you use to launch the device portal.

The Remote Connections dialog

Make sure that you have the Authentication Mode set to Universal, otherwise your device will not accept the connection.

You need to take an additional step to deploy--your device needs to pair with your computer. This way you can be sure that no other people connect to it without you knowing it. If you start the debugger using the Remote Machine option or try to deploy to the Remote Machine, you will get a dialog box asking you for a PIN.

Visual Studio asking for the PIN to pair

This pin can be obtained in the device by navigating to the Settings app, then to Update, and then selecting For Developers. Here, you can pair the device. If you select this, you will get a PIN globally for each machine you want to connect. Enter this pin in the dialog and you are all set to go--the device should receive your app now. If you choose Deploy in Visual Studio, the app will show up in your list on the device. You can find it by starting the Start screen with the Bloom gesture, then select the + on the right-hand side of the menu. There, you will find the list with all the installed apps, including your new HelloHoloWorld project. Air-tap this to start it. If you are done with it, you can do the Bloom again to go back to the main menu.

If you happen to forget the pin, do not worry. All you have to do is enter a wrong number three times in a row. After that, the device will ask you to reconnect once again from scratch, enabling you to think of a new pin.

If you choose Debug, the app will start automatically--you do not have to select it in the device.

Deploying through USB

Since the device has a USB port, we can connect it to the computer. Unfortunately, it does not show up in the explorer--we cannot access the filesystem, for instance. However, we can use this to deploy our app to the device. All you have to do in Visual Studio is select as target Device and it will deploy it through the USB cable. As I said before, this is slightly faster than using the Wi-Fi connection, but it gives you fewer options to walk around during debugging.

Let's have a look at the code

The code you just wrote is a C#, UWP Windows 10 app using DirectX through the SharpDX library. In later chapters, we will examine this much more closely. DirectX is the technology that allows us to write fast-running apps that use a lot of graphics, such as the holograms we create. However, DirectX is a C++ library and can officially only be used in C++ programs. Luckily, the people at SharpDX have written a wrapper around this, so we can use this in our C# applications and apps as well, this is the reason why the HoloLens SDK developers included this in the template. For more information about SharpDX, I suggest that you have a look at their site at http://sharpdx.org/.

The app uses some libraries from the Windows SDK. However, if you look closely at the References part in the solution, you will note that there are no Holographic-specific libraries included. The reason for this is that the APIs needed to run Holographic apps are standard in the Windows 10 runtime. That is right; your Windows 10 computer has all the code it needs to run Holographic apps. Unfortunately, the hardware you have will not support these APIs, so they are not available to use. If you try to deploy this app to a normal machine, you will get errors--the required capabilities are not available and the runtime will refuse to install the app.

This means that our app is a standard Windows 10 UWP app with some extra capabilities added. If you right-click on the Package.appxmanifest, you will find the following tag somewhere:

 <Dependencies> 
    <TargetDeviceFamily Name="Windows.Holographic" MinVersion="10.0.10240.0" MaxVersionTested="10.0.10586.0" /> 
  </Dependencies> 

This is what makes sure that our device accepts our app and other systems do not--the app is marked for usage in Windows. holographic-capable device. The numbers you see in MinVersion and MaxVersionTested may differ--these depend on the SDK versions that you have installed and chose when you created the project.

The structure of the project

If you have written UWP apps before, you will see that the structure of this app is quite different. This is mostly because of the code SharpDX needs to start up. After all, a Holographic app needs a different kind of user interface than a normal screen-based app. There is no notion of a screen, no place to put controls, labels, or textboxes, and no real canvas. Everything we do needs to be done in a 3D world. An exception to this is when we create 2D apps that we want to deploy to the 3D environment. Examples of this are the Edge Browser and the now familiar Settings app. Those are standard UWP apps running on HoloLens. We will see how to build this later on.

The project should look like this:

The project structure

As you can see, we have the normal Properties and References parts. Next, we have the expected Program.cs and Assets folder. That folder contains our logos, start screen, and other assets we usually see with UWP apps. Program.cs is the starting point of the app. We also can identify the HelloHoloWorld_TemporaryKey.pfx. The app needs to be signed in order to run on the device, and this is the key that does this. You cannot use this key to deploy to the store, but since we have set our device in developer mode, it will accept our app with a test key such as this one. Package.appxmanifest we have already looked at.

The other files and folder might not be that familiar.

The Common folder contains a set of helper classes that help us use SharpDX. It has some camera helpers that act as viewport to the 3D world, timers that help us with the animation of the spinning cube, and so on.

The Content folder has the code needed to draw our spinning cube. This has the shaders and renderers that SharpDX needs to draw the cube we see when we start the app. We will have a closer look at how this works in later chapters.

Inside the AppView class

The AppViewSource.cs and AppView.cs files contain the AppViewSource and AppView classes, respectively. These are the boilerplate code files that launch our scene in the app. AppViewSource is used in Program.cs, and all it does is start a new AppView. AppView is where most of the Holographic magic happens.

The following are the five different things this AppView class does:

• Housekeeping: The constructor and the Dispose() method live here. The code needs to be IDisposable since most DirectX-based code is unmanaged. This means we need to clean things up when needed.

• IFrameworkView members: IFrameworkView is the interface DirectX uses to draw its contents to. This contains the methods to load resources, run the view, and set a window that receives the graphics, and so on.

• Application Lifecycle event handlers: If you have developed Store apps before, you will recognize these; this is where we handle suspension, reactivating, and other lifecycle events.

• Window event handlers: Next to the lifecycle events, there are some other events that can occur during runtime: OnVisibilityChanged and OnWindowClosed can happen. This is where they are handled.

• Input event handler: I mentioned before that you can hook up a Bluetooth keyboard to the device. If you do that, the user can press keys on that keyboard and you need to handle those. This is where that part is done. I would recommend against using this in normal scenarios. People walking around with a HoloLens will not normally carry a keyboard with them. However, in certain use cases, this might be desirable.

In the IFrameworkView.Initialize method, the app creates a new instance of the HelloHoloWorldMain class. This class is where our custom code is placed, so basically this is our entry point. The rest can be considered as boilerplate code, stuff the app needs to do anything at all.

In the following chapters, we will adapt this class heavily, but for now I want to show you another way to create HoloLens apps that will give you much nicer results much more easily. We will start to use Unity3D.