What Virtual Reality is and how it works

We have many senses. To make us feel an alternative reality is real, we need to involve these senses to fool the brain. Most VR systems make use of two: sight and sound; touch is also used but not in a full reach-out-and-touch-someone sense (although people are working on it!). 

Tor Nørretranders compiled data about the senses and their relative bandwidth, in computer terms. This is a bit like comparing apples and motor oil, although useful to see how it applies to VR.

So, we can see if we make you see something that is virtually real, we may be able to convince the brain it is real. Yet, simply putting a vision screen in front of us is not the complete answer.

Giving someone depth perception is most of the answer.

This is a fairly complicated topic, but the main method of showing depth of objects is stereoscopic depth perception. Remember those ViewMaster toys? Here is an example of one:

You put in a disk that had a left eye and a right eye image. The images on the left and right nearly look identical, but they represent what the right and left eyes would see if you were standing at that location; each slightly different due to parallax. From the disk, here, we can see each of the left and right eye images. The lenses in the preceding View-Master focuses your eyes on the images.

Your brain would look at these two images and fuse the images into something that looked real. This uses a depth perception technique called stereoscopic depth perception. 

Yes, the View-Master was an early Virtual Reality viewing device!

Now, what is really going on here? How does Stereo work?

When you look at something, perspective and separation between your eyes will make you focus your eyes differently at something closer as opposed to something far away. In this diagram, the yellow lines show our sight lines to a near object, and the orange lines show the sight lines to a distant object. Note that the angle between the yellow lines is larger than the narrow angle of the orange lines:

A friendly robot loaned us the lower half of her eyes to make this image (that's why it shows circuit boards). Your real eye is constructed somewhat similar; I omitted the light rays and where they fall on the back of the eye for illustration's sake.

Your brain will automatically figure out if your eyes are pointed at a close or far object by the difference in angles between the yellow and the orange lines.

This is just one method our brain uses to distinguish depth. Another that is also vital to Virtual Reality is the use of parallax.

Parallax refers to the way that, not only do the left and right android eyes point differently (as would your eyes, when they are attached to your head), but that each eye sees a slightly different view of the same objects. This will work even with one eye if you move your head to the left and right, and is how people with mono-vision perceive depth (among other ways).

This is how your left eye sees the scene:

This is how the right eye sees the same object:

Parallax refers to the way that an object that is more distant will be less to the right/left than a nearby object, when viewed with the other eye, OR (an extension) when moving your head left to right. Our brain (as well as the brains of animals) will instinctively see these as closer/further.

The red cube is either next to the blue cube or the green cube, depending on what eye sees the image. Your brain will integrate this, coupled with how the cubes move if you move your eye from side to side, to also give you a sense of depth.

Don't despair if you are in the percentage of the population that do not perceive 3D movies. They strictly rely on stereoscopic depth perception and do not take parallax effects into account; they are pre-recorded.

With true VR (computer generated or light field based 360 video), if you move your head, you will see the parallax effect and the VR can seem real just like someone with stereoscopic depth perception sees.

I have mono-vision because I have a nearsighted eye and a farsighted eye, and VR works great for me. Your mileage may vary, but if you don't like 3D movies, give VR a try (then again, I really like 3D movies).

Parallax depth perception will work even if you have one eye, when you move your head right to left.

There is one additional way that your brain will use to determine depth of an object - focusing. (Actually, there are many ways other than those listed, such as blue shifting of objects in the far distance, like mountains, and other effects). Focusing on an object in the real world will make that object and other objects at roughly the same distance appear in focus, and objects both further and closer will appear blurry. Sort of like this:

Current HMD's cannot accurately show focus as an effect. You are looking at a small screen that generally has a fixed focus of about 5 feet in front of you. All objects, close and far, will appear to be focused the same as they are actually just being shown on the screen. This can cause a mild VR discomfort, called the accommodation-vergence conflict. Basically, if you focus on the far focus cube (the salmon colored one), your eyes will still focus as if the salmon cube was located where the red cube is; your eyeballs will, however, aim stereoscopically as if it was located where it should be. This effect is most pronounced with very close objects.

The accommodation-vergence conflict is most severe with close objects - so try not to have anything, such as a GUI, located too close to the user's location. You will reduce sickness this way.

This means you may need to float GUI elements out into the room instead of having them very close. This may cause overlapping UI elements. 

VR design is challenging. I'm looking forward to what you design!

Use of stereo and parallax in Virtual Reality

As far back as 1968, Ivan E. Sutherland first observed that objects with stereoscopic depth perception, and that seemed to be positioned in space when the users head moved (motion parallax), seemed real.

The system that he and Bob Sproul developed, often referred to as the The Sword of Damocles, just displayed a handful of glowing lines in the air, yet:

"Even with this relatively crude system, the three dimensional illusion was real."

-Ivan E. Sutherland, AFIPS '68 (Fall, part I) Proceedings of the December 9-11, 1968, fall joint computer conference, part I: http://bit.ly/2urAV5e

Real in this instance meant that despite a total lack of realistic rendering - just a glowing cube - people thought of it as real. This is due to the stereoscopic rendering and the parallax effect. People could turn their heads and move a little bit from side to side.

They invented the first VR Headset, or Head Mounted Display (HMD).

The person widely regarded as creating the term Virtual Reality, Jaron Lanier, said:

“It's a very interesting kind of reality. It's absolutely as shared as the physical world. Some people say that, well, the physical world isn't all that real. It's a consensus world. But the thing is, however real the physical world is – which we never can really know – the virtual world is exactly as real, and achieves the same status. But at the same time it also has this infinity of possibility that you don't have in the physical world: in the physical world, you can't suddenly turn this building into a tulip; it's just impossible. But in the virtual world you can …. [Virtual Reality] gives us this sense of being able to be who we are without limitation; for our imagination to become objective and shared with other people.”

- Jaron Lanier, SIGGRAPH Panel 1989, Virtual Environments and Interactivity: Windows to the Future. http://bit.ly/2uIl0ib

A researcher named Mel Slater has performed further work on this concept, coining the further terms presence and plausibility. Some people call all of this Immersion. 3D images on a screen are not as compelling as when you have an HMD on and your only sight is of the constructed 3D world. You feel a sense of presence due to the audio and visual cues, even if the rendering isn't just like the real world. Plausibility means that what you see has rules and works even if it's not exactly what you see in the real world.

The combination of being limited to seeing just what is in the HMD, along with the parallax and sterescopic views, and any audio (sounds are very important, if done well), will immerse you in the VR world. With all of these things, even if the graphics are not real, you will feel immersed, and it becomes real. For more academic detail see http://bit.ly/2vGFso0, although I'll explain more about this in this section. 

This really works.

VR does not have to look anything like reality, but it will seem as if it is real. For example, take a look at the game Quell4D:

The graphics are block like, the images look nothing like reality. Yet, when the Ancient Triple Trunked Elephant Necro Shaman come at you, you will be scared. They seem real. To you, when you play the game, they are literally deadly real, meaning that your (in-game) person will die if you don't take them seriously.

VR simulations of fire will send about 10% of the people out of the room in a panic, even though the flames don't look anything like real flames.

VR is here. We do not have to wait until the graphics get better. Many people say this about VR, but it is because they haven't tried it yet and are making assumptions about what it has to be.

Dive in, the water is fine!

So, Virtual Reality is something that will seem real, not something that necessarily looks real (but it helps if it does!).

You do not have to wait until better graphics come around. 

Why does Virtual Reality work if it doesn't look 100% real?

Our eyes are possibly our most important senses for showing us what the world consists of. If we substitute images and immerse someone with these images, they will start to seem real. When you first get into VR, your initial reaction is, "that doesn't really look real", but with a good VR setup, you will get to a point where you think "whoa, that was real" even though you know you are looking at what is basically a computer game.

A fast frame rate (speed of display) and enough resolution will trick your brain into thinking what it sees visually really does exist. This is a powerful effect that most, but not all, people will have when immersed in such images (not everyone with normal eyesight sees 3D movies either).

In fact, the sense of reality is so good that people can get sick by watching VR. This happens because your eyes may say that's real, but your other senses, like your inner ear, says we aren't jumping 10 feet in the air. If your eyes think you are bouncing up in the air and your leg muscles (the sense of proprioception) say you're on the ground, your skin says you don't feel the wind and your inner ear says you are not tilting as you fly forward, your mind will be confused at a very deep level.

When your senses disagree powerfully, your body has a defense mechanism. It thinks you've been poisoned; as a result, your body will feel nausea and possibly even get sick. Your body is concerned that your eyes don't see what the rest of your body feels, so it may try to get rid of everything in your stomach, just in case something you ate had poisoned you.

Yes, it's no fun; different people will react differently.

However, not all VR does this! Generally speaking, poorly constructed VR will give you this feeling. Academic papers have been written on this effect. This book will summarize these discussions into a few simple rules that will make your VR much more comfortable for people.

Another important aspect of Virtual Reality is that it is something that you can interact with (the reality itself). This brings up mechanical difficulties; not everyone owns 3D controllers. We cover this in the section: User Agency - interacting with the world through controllers. True VR can be interacted with even if it is something as simple as gaze detection - look at something (gaze) and things happen - movement occurs, you're teleported, an animation plays.

Other types of VR; AR, XR, SR/FR

There is another type of imagery which is sometimes called VR, which is 360 Video. There are special video players that record in all directions. Sophisticated software will stitch the different camera inputs together to make a video stream that playback software will project to be all around you. When you turn your head, you seem to change your point of view inside the filmed world. It's as if you're in the real world, looking around at whatever you want to see.

The 360 Video looks possibly better than most computer graphic generated VR, but to me it's not reality because, at best, you are a disembodied ghost. Sure, the world looks great, but you can't reach out and touch things, because it's been filmed. The 360 Video and systems like it are beyond the scope of this book. Having said this, I do think the 360 Video is certainly a valid art form and something to pursue--just not covered by this book.

Please understand - I don't mean this dismissively of 360 video, just because it's not real VR. (Pronouce that second to last word like it had air quotes around it). 360 Video can be very heart warming, intense, emotional dramas. You do get a hint of presense, and the visuals are astounding. It is an area where we should see amazing art being made as more people are familiar with it and work out the details.

I am proposing a new term for 360 Video; Filmed Reality (FR) or Surround Reality (SR). (Although no one uses actual film for these, the phrase 'filmed' still means to record something through a lens, but maybe SR is better. You choose!)

There are other types of VR. So many that some people use the phrase XR which means (Anything) Reality; mainly to mean AR and VR. What is AR?

The HMD, which consists of some small displays and sophisticated optics, will allow you to see a stereo 3D image when you put the headset on. Most VR headsets intentionally block out the rest of the world while you are in them, to further immerse you in VR. This is an important component of VR, although there is a type of VR called Augmented Reality (AR) where the VR items are projected into the real world by wearing a type of HMD that is transparent. There are a number of manufacturers, although the Microsoft Hololens may be the best well known. There is also the game Pokemon Go, which is a type of AR. People hold up their cell phone, which shows images layered on top of reality. This is not a headset, but is still AR. Reality has been augmented with the Pokemon world.

VR systems can also be windows on world systems, although this is not commonly called VR today. In other words, a real, persistent 3D world that you sit at your keyboard and view through your screen. During the last wave of VR, years ago, this was referred to as VR, although today it's commonplace enough that people do not call it VR. You may have heard of World of Warcraft.

This is a type of Virtual Reality; although it's not (usually) in 3D, it is a persistent world in an alternate reality. It is also a full 3D world that you can see by looking at your screen; the screen transports you to a virtual reality so it's similar to a Windows on World system (although not head tracked). 

Watching movies can be viewed as a valid form of VR; you are transported to another world, and, for a short time, feel as if you are immersed in the story. Television is a type of VR.

The first use of the term VR in fact referred to the theater. While many people today would say that's not VR, they spend much of their lives watching other realities and not paying attention to the people sitting next to them. How is this not Virtual Reality? You are pretty immersed with Dancing with the Stars, but do you know any of them? They are virtually real.

Still, this is not what most people think. This book will use the modern (2014+) interpretation of Virtual Reality as being something viewed through a VR headset or HMD of some type. Today, the term VR usually implies a headset or HMD, and is fairly often coupled with some form of hand controllers. Good, effective HMDs are all commercially available now to consumers. It is a great time to be interested in VR.

The nice thing about WebVR though, is that we can still see these VR worlds through the browser without an HMD; this is great for testing and for people without the hardware.

WebVR is very inclusive.