User agency - interacting with the world through controllers

The HMD is not everything, although it is certainly the most important part. Being able to see a VR world is great, but at some point you want to be able to interact with it. If the world was static, you would feel like a disembodied ghost. It is Virtual Reality when you can interact with the world.

Eventually, something like full suit haptics (physical feedback) and body tracking, along with sophisticated software, will allow us to reach out and touch the virtual world. This is something to look forward to in the future.

For now, the way that we usually interact with the world is through various hand-held controllers. Different controllers have radically different capabilities and requirements. Controllers for high-end (but still consumer available) VR setups, like the Rift and Vive, work considerably differently from mobile VR controllers. We'll discuss the higher-end systems first, then discuss mobile VR controllers.

High-end controllers for PC, Mac, and Linux

With PC VR, such as the HTC Vive or Oculus Rift, controllers give a very important ability to interact with the VR world. These controllers are tracked in 3D space so that the software knows where they are. Developers can code these to look like hands, guns, and so on. This allows you to reach out and touch the world around you--very important to make the Virtual Reality you are interacting with actually something you can interact with.

To do this, both the Oculus and Vive controllers require external tracking hardware. With the Vive, these are lighthouses or VR base stations that are placed at the corners of the VR area. (There is a diagram here, and it is available at http://bit.ly/VIVEManual). These small unobtrusive cubes send out IR tracking signals that the controllers and headsets pick up and use to accurately position them in the real 3D world. With the Rift, there are two or three sensors that also track the devices to give them a real-world position:

The base stations and tracking hardware are really important for the HMD itself as well. 

This tracking of your real world position (your actual head/hands) is what makes movement, turning your head, moving your hands/controllers seem real, because the position, orientation, and movement of the headsets and controllers is so precisely tracked in real 3D space, any head movement seems real once the software displays the VR world to the user.

In practice, this means that the PC controllers will seem right where you see them. My first experience with an HTC Vive at a tech demo was amazing--I'd put on the HTC Vive headset, and, in the virtual world, saw the controllers in front of me. I expected to sort of fumble around until the controllers were where I thought they would be. I reached out, and my fingers felt the controllers exactly where my eyes saw them--through the HMD.

I was hooked! The virtual world really was Virtual Reality! The phantom controllers I was looking at were real even though I knew I was looking at a small screen in front of my face.

How do they work?

HTC Vive

The HTC Vive uses two small square cubes, called base stations or Lighthouses, at opposite ends of the area they are covering. These send out beams of infrared light over 120 degrees; this means if they are in a corner, they can be a few inches away from the corner and still cover the walls (otherwise, you'd have to dig a hole in your wall to put the lighthouses at the right spot!)

Usually, you mount the two base stations at opposite sides of a room, up to about 16 feet or 5 meters apart, and above head height, around 2 meters or 6 feet 6 inches. It's okay if you're taller--mount them higher!

The base stations can also be mounted on microphone stands with the right adapters or with custom stands. Not everyone has a large living room, so these arrangements may help fit it in.

The Vive can also be used in a seated configuration, although the real point is what is called Room Scale.

Room Scale VR means that you can walk around in the VR world, as if you are walking around the real world. No teleporting or other tricks needed. Of course, the area needs to be clear of furniture, which is one of the issues with VR in general; not everyone has a big room they can clear.

The Vive will keep you safe by showing bounds or guards at the edges of your space, if you get too close.

Make your room bounds slightly less than the actual room, if it's a wall or other area. If it's a couch or chair, you can go right to the end of the chair.

We do this so you don't hit your arm on the walls. This is easy to do if you are standing up against the wall, but still in the virtual world and therefore can't see the wall, and you swing your arm, your hand won't go through the real wall!  

Going to the edge of a sofa is nice, as your shins will run into the sofa before your hands hit the wall. In practice it's not that much of a problem, as you'll see the guards before you get close. Pay attention to the Vive/Steam VR tutorial!

The HTC Vive works by having some Inertial Measurement Units (IMUs), which detect where the HMD as well as the controllers are located. These IMUs drift, so the base stations have an infrared beam that sweeps over the room. When the controllers, trackers, or HMD detects these beams, they recenter themselves. This re-centering is completely undetectable. The advantage of such a system is that, even if a controller goes out of sight from one of the base stations or Lighthouses, the VR system still knows where that item is and where it is pointed. 

The overall effect is precision and presence, although the main effect is stability. If you cross your hands and a controller briefly goes out of sight of the base station, the controller won't lose lock.

Try not to put your VR space in an area with a lot of windows or mirrors.

The Infrared beams can reflect off of them, leading to instability.

Oculus Rift

The Rift first came out just as a headset, without controllers. It's initial base stations are two cameras that you place on the left and right of a desk; they point at the HMD and are used to position it in the world.

Soon after, Rift added the capability of third camera; with three cameras, you can do room-scale VR. The positioning of them is slightly different than the Vive; take a look at the Rift documentation for the best positioning.

Be careful with cables. As I wrote this book, the Rift is cabled directly into the back of the PC. If you trip over the cables, you could yank them out of the PC fairly hard, leading to damage.

The Vive has a breakout box, so if you trip over a cable, you'll hopefully pull it out of the box. 
Don't trip over the cables.

The purpose of this book is not to be an analysis of why the Vive or Rift is better or worse than another; both of them work roughly the same way with the base stations/cameras helping the controllers and HMD to keep track of their position and rotations. Here is a typical setup:

In it, the Vive base stations are mounted on the walls; we have a desktop PC and a VR user viewing a 3D model as if it was real. The VR user is holding two Vive controllers; the virtual image is holding a Xbox-style game controller.

This image also shows an Oculus Rift 3 camera tracker system. They are the light-gray items sitting to the left and right of the screen, and on the credenza on the back side of the couch (right in front of us).

That's right, the cybernetic robot is the user. She doesn't need an HMD; the system pumps in the video straight to her eyes. The virtual object is the human seeming to sit at the desk. 

The preceding figure is the view that a third person in front of the couch might see of the scene.

What the lighthouses see is actually a little different, but interesting. They actually have a couple of infrared bars that sweep across the view and the controllers see these lines tracking across. When they do, the controllers (and HMD) will resync their inertially tracked positioning. This means that even if a controller is out of view of a base station, it still keeps tracking, although you don't want to hide a controller for very long. Inertially tracked systems will drift. The visual issue with drift is that your arm would seem to slowly move away from your body - which is obviously highly disconcerting. The Vive lighthouses and Rift cameras keep the drift from happening. The Vive angle of view that a lighthouse projects is around 120 degrees. This is what the rear, rightmost lighthouse would see if there were a camera with this field of view at the lighthouse:

You can see both controllers and the HMD through this Lighthouse. There is, however, an issue. Notice the red circles--the large mirror on the left is actually a gigantic TV, but it's shiny. As a result, the Lighthouse IR beams will bounce off of it and the controllers will sense two beams: one directly and one reflection.

This may cause the HMD and your point of view to jump or your controllers to move about inexplicably.

Avoid shiny objects, mirrors, and windows in your VR room.

You might need to draw drapes, or even throw sheets over TV's, glass china cabinets, and the like.

Art requires sacrifice!

From the other lighthouse, one of the controllers is blocked, but it is still tracked 100% through its internal inertial tracking and the other lighthouse.

Mobile VR

For mobile VR, there are also the Google Daydream and Samsung Gear VR controllers. Due to the simpler hardware used, which keeps the price more reasonable, these are not fully 3D tracked.

With mobile VR, due to the absence of room tracking external sensors, that the Vive and Oculus both have, the VR controllers are not so precisely tracked. In practice they will seem just as real, but will periodically drift. It's as if your hand slowly moved off to the right without your control. Mobile VR thus has a reset controller button that will move the controller to a predefined position, such as near your hip. Your hand might be held straight out, but if you hit the Home button the VR display will show your hand now at your hip.

This might take some getting used to. There are advantages to this setup; it's cheaper, requires less external hardware, and there are many more such systems out in the world. However, the PC hardware does give a better VR experience.

The other wrinkle with the mobile controllers is that there are only three Degrees of Freedom (DOF). This means they track tilt, yaw, and roll, but not position; if you move the controller flat to your left, in the game your controller hasn't moved at all. This is why you can't grab things with a mobile controller. The Vive and Rift both have 6 DOF controllers, so you can move them around and grab things.