This year for the second time around I attended Microsoft Build 2015 developer conference in San Francisco. Among the biggest highlights of the show was the demonstration of Microsoft HoloLens - the revolutionary holographic reality device coming with Windows 10. I was lucky enough to be able to experience it first hand so I would like to share it now with you.

Register, quickly!

In the first day's keynote speech it was announced, that the some of the attendees will be able to try out the HoloLens experience for themselves by registering through a dedicated website. One thing that was noted was the fact that the registration will be open right after the keynote ends and it is able to accommodate just a small portion of the large number of developers who are present on the event. So I acted fast! After the keynote ended I quickly browsed the registration web site on my phone (slowed down by the extreme traffic that all available Wi-Fi networks in the area were getting) and was offered with a choice of three different activities - the Holographic Academy, Holographic presentation and HoloLens One on One. Of the three the One on One session offered the exact activities that were demonstrated in the HoloLens video ad, so I naturally thought of it as the best choice. Sadly, when I tried to choose that activity, my phone (in early preview of Windows 10 Mobile) decided, that the Holographic Academy was the better choice and sent this as my choice. And in the end, I was glad, this one was actually the right one for me!

After an hour, just before lunch, I was lucky enough to receive an invitation in my mailbox to attend the Holographic Academy in a nearby five-star hotel. I had just about 30 minutes to get there so I packed my lunch in the bag and run directly there.

The preparation

In the hotel's reception area all the invited attendees were anxiously waiting for the signal of the HoloLens staff members that directed them to their chosen activities. The Holographic Academy participants were apparently the largest group and the waiting to be let in was appropriately the longest.

When all was ready, we were let in the elevators to a floor that was dedicated to Microsoft for the purposes of this event. Right in front of the entrance hall in a glass cube was the Microsoft HoloLens itself for display, but no touching or photography allowed! We were led to a registration area to sign NDA and had to agree with the fact that we were going to be recorded for demonstration purposes and were handed out a locker key to be able to safely store our personal belongings and all electronic devices, which were of course not allowed on the event itself.

Then we had received our badges and got a measurement of the distance between our eyes. This was for our reference later when we had to configure the initial setup of the device for the optimal experience.

Enter the dev area

After all the formalities were finished and everyone was lined up, we were divided into pairs and each pair of attendees was assigned a supervisor, which acted as our personal assistant throughout the afternoon. With our supervisor we entered the room where the whole event was about to happen. This room was a medium sized conference hall with a stage in the middle and several sections around it. Each section had a conference table surrounded by three to four tables with two computers each. Every each of us was assigned a seat and a dedicated development machine. Surprisingly, there was no HoloLens on sight yet! The atmosphere of the room was awesome. The anticipation was palpable and it was clear everyone was eager to get their hands on with HoloLens. In the video below you can see for yourself (by the way, you can catch me at the 0:06 mark).

Introduction

After everyone was in the room and the doors were closed, the one of the event's presenters entered the stage to introduce us. After a short speech about the device itself and the technology behind it, we were finally handed out the HoloLens device that was going to be "ours" for the rest of the event. Mind you, HoloLens back then was still in its preview phase. The device no longer required a large battery backpack as in the previous demonstrations and looked like a finished product, but the true holographic version of Windows 10 Holographic was not yet present on the device for example. Our supervisor helped us set up the device using a preliminary web interface that acted as a temporary substitution for the device's "so far" non-existent OS interface and allowed us to directly install application packages on it. The OS itself was just an environment where the apps run (just as the Windows IoT Preview looks).

Hello, holograms

After we finished the initial setup we could install our first sample application on the device. After the app installed, we could finally put the device on. The first great surprise - HoloLens is amazingly light. Although it is a full-fledged computer with high processing power and multitude of sensors, it is really easy to wear and when adjusted correctly it doesn't even lie on the nose and is extremely comfortable. The sample application allowed us to place a toy car on any surface around us and then using our gaze and hand gestures instruct the car to drive around the environment. The car was intelligent enough to avoid real world obstacles and was even able to perform stunts like jumping from the table to the floor or driving over my own shoe. All this was extremely realistic as if the car itself was truly a physical object aware of its surroundings and the intricacies of three dimensional reality. The feeling that something like this is being created and presented to me through the lens of a small device on my head was profoundly impressive. This is what is called a mind-blowing experience. Another thing worth noticing was the image's stability. No matter how I moved around, the toy car was perfectly stable and didn't move a bit from its "real world" position. This was maybe even more surprising. I have no idea how the team behind HoloLens was able to achieve this level of stability. Day before the Holographic academy I had a chance to try out the Oculus Rift and I have to say that the difference between those two was like night and day. Sure, you are probably waiting for me to comment on the field of view, that was described as the biggest setback of HoloLens. Yes, the FOV of the device is smaller than was demonstrated in the previous live showcases and in the advertisements and Microsoft acknowledged that this is a drawback that has to be accepted for the initial release of the device targeted for businesses and developers. I suppose it is mainly an issue of battery life and expect that the second release will expand the field of view enough to be usable for immersive gaming for example. However, I do not feel the field of view was small for normal usage. Most of the objects are visible completely without problems, only a close-up view can cause some cropping of the holograms, which I find more than acceptable.

Let's //build/!

To bring us back to reality, a bell announced that we should take off HoloLens and come back to pay attention to the stage. The presenter walked us through the agenda of the rest of the event. We were going to build our first holographic application and become the first developers outside Microsoft to try building for HoloLens. Our main tools were Visual Studio and Unity - both in pre-release internal beta versions. Unity had a special HoloLens SDK installed that contained prepared tooling for building holographic applications with things like spatial mapping, holographic camera and ambient sound. Throughout the afternoon we went through the following stages of building a simple app called Project Origami.

Stage 1: Display a hologram

The first and basic stage was just displaying the 3D object in space in front of us. We dragged an pre-composed scene of a origami paper creation in the scene in Unity and placed the Holographic camera. After preparing the scene, we exported the app as a Windows 10 app and then opened the solution in Visual Studio to allow its deployment to HoloLens itself.

Stage 2: Gaze

First interaction to add to our model was gaze. This enabled HoloLens to track our eye movement and highlight the spot we "look" at. This eye tracking was once again very precise.

Stage 3: Air tap

The second phase of the tutorial added hand gesture - so called air tap. The 3D model contained two paper "balls" levitating in the air. Gazing on one of the two balls and using the air tap gesture, the ball suddenly gained gravity influence and fell down on the floor. In this part of the demonstration however it fell right through the floor into infinity.

Stage 4: Voice

Next up we added voice control as an alternative to the hand gesture. Using our voice we could cause the paper balls to fall and also reset the scene. This made the interaction more natural and demonstrated the freedom the user of HoloLens has when choosing the form of input appropriate for the specific situation. Also worth mentioning is the fact that HoloLens does react only to you and is not confused with people around saying the same thing.

Stage 5: Spatial mapping

To make the holograms aware of the surroundings we added spatial mapping. This was just another of the prepared "plug in" elements in HoloLens SDK for Unity. We just added it and it worked - we could place our hologram on any surface around us and then the paper ball no longer fell down through the floor but correctly interacted with the object around including the table or a sofa. We were even able to see how HoloLens "sees" the world. Spatial mapping creates a virtual "wireframe" of the environment and very precisely captures the shapes of objects around us.

Stage 6: Ambient sound

To make the experience even more realistic, we added ambient sound. Firstly, the hologram itself exuded a calm, relaxing music and also the falling and rolling of the paper balls was accompanied by sound effect to make them feel real. HoloLens has great sound system built in so the user actually hears the sound as coming from a distance in the right direction. Turning your back to the hologram it sounds like it is truly behind you. Coming closer makes the sound louder. It sounds like a small detail but it makes the experience unimaginably more complete.

Stage 7: A bonus!

To make the experience complete, there was a bonus ready for us at the end of the tutorial. We added an object that was not described more closely to us. When we tried the app then, the falling paper ball caused the floor to break on the impact and revealed a fantasy world below the crack, full of living paper cranes flying around.

Overall impression

Microsoft HoloLens is truly a remarkable technology that sounds, looks and feels like future. I was really blown away by its power and potential and I am sure that anyone who will have a chance to try it will experience the same. Building apps for HoloLens is be possible in the same way as all other Windows 10 Universal apps. Normal apps will normally work on HoloLens, but if you want to go beyond the boundaries two dimensions, which I definitely recommend, you will really appreciate Unity development environment and the easy-to-use tooling that is at your disposal. Even in its preview form the SDK was extremely straightforward and capable. The first public release of the hardware is not too far away and I am looking forward to see how the device evolves and what creative uses it can have not only for users, but also for enterprise and even healthcare. Microsoft has really created something revolutionary here.