What are the essential components of an immersive experience?

The point of explaining and defining immersion and immersive experiences in the preceding section was intentional. When we talk about VR or any other variation of XR, we are talking about different types of experiences that engage the user in distinct ways. It is important to understand that as fact rather than opinion because some technical aspects and elements make the experiences what they are. They can be clearly defined and formulaic. With most VR experiences, the user will have an experience with an HMD such as an Oculus Quest. With augmented reality (AR), the user will most likely have an experience through their smartphone. The list can go on.

The true impact as a creator and developer is taking the core elements of a formulaic experience and infusing abstract and creative elements into it so that people have a memorable experience they want to share with friends and colleagues or even promote to the world. At face value, all VR experiences are a variation of putting on a plastic headset and responding to stimuli that are not real, but the experiences people reflect on afterward with VR are a lot more formative and expressive. They will describe what they did, what they saw, and how the VR experience made them feel. The following are some examples of general (non-specific) experiences that can be enhanced using VR:

• Entertainment through 360-degree videos: You can immerse yourself in the video as if you are there. 360 videos provide a passive VR experience that allows you to see in all directions rather than at just a monitor screen in front of you. Think about someone base jumping with a 360-camera attached to them. In VR, you can tag along with them as they go on an epic adventure.
• Games: Instead of sitting on the couch with a controller, you can be the player in the game dodging all the obstacles and scoring all the points. Such games include the following:
  • Story/role-playing games: You take on the persona of a character in a story and evolve as the story progresses.
  • FPS games: First-person shooters allow you to go to battle with others in a game of survival. You can navigate environments to evade gunfire and take out opposing players.
  • Foraging/exploration games: Games where you can traverse vast worlds, climb high peaks, and scavenge for resources. These games normally focus on puzzles and creating lighthearted experiences.
  • Sports games: Instead of going to the field or the court in real life, you can play your favorite sport in VR.
  • Artistic games: These games are abstract because they are all about using interactions to elicit a certain effect. This can mean shooting paintballs at a 100-foot canvas to make a painting or fishing in a field on another planet.
  • Survival games: Much like first-person shooters, you are immersed in an environment where the goal is to think outside the box to increase your chance of survival.
• Social/virtual meetings: You can meet up with friends, watch movies, and go to meetings in virtual environments with friends across the globe. Geographic locations won’t hinder you from connecting with others.
• Medicine: You can improve patient outcomes in a variety of areas, from therapy to training. With VR, you can create simulation modules for healthcare professionals to improve their training and provide immersive learning experiences for patients to better understand their health.
• Education: In the classroom, VR can give students the ability to explore learning in a more exploratory way. This allows them to retain information better. Students can go on museum and gallery tours to places across the globe and interact with content beyond a textbook.
• Military training: You can make training more accessible and cheaper with simulations of the tasks at hand. The military uses VR to simulate combat environments.
• Utility/productivity: VR can be used to extend your office beyond its physical location. Instead of using a computer monitor, your headset can create countless virtual monitors so that you can multitask and work in a variety of locations. Want to work on a project on the beach with a 200-foot monitor screen? You can do that in VR.
• Real estate: You can tour homes from your living room using digital replicas of the places you intend to learn more about. Digital twins allow for deep exploration of real-world locations without you having to physically be there.
• Engineering: You can design and prototype before you move to manufacturing. This saves countless work hours and allows for rapid revisions and iterations, thus saving money.
• Exercise: Instead of going to a workout class, you can bring the workout class to you. In VR, you can gamify your workouts with others and/or in fantastical ways using digital enhancements.
• Content creation: You can ditch the keyboard and mouse to sculpt and paint content for projects in VR. If you like the kinetic experience of sculpting but still want to work digitally, you can put on a headset and do what you do best in the way that feels the most natural.

Skills required to build immersive experiences

Some technical and nontechnical skills are valuable in the XR industry and for developing VR experiences that will have an impact. In many cases, if you have developed skills and worked on projects in other industries, you can integrate those skills into making engaging immersive experiences. Understand that you do not exist within a vacuum. You have skills and ideas that can push the culture of XR forward in new and exciting ways. I can speak from personal experience. I was a Division 1 college football player who did software development for Windows Mobile in the early 2000s. When Windows Mobile went defunct, I shifted to comic illustration and visual storytelling. My creative endeavors evolved from newspaper comics to webcomics, to animation on YouTube, to live-action visual effects, and ultimately to XR creation. I did all that while playing football, retiring, and getting into medical school. Over this 10+ years’ creative and professional journey, I developed skills in a variety of areas that further informed my workflow and ideas to create and pursue.

The reason your skills are so valuable is because XR is just a medium. It is a manifestation of the ideas you think of and write on paper. Those ideas can become books, animated shows, live-action movies, training modules, mobile apps, and so on. You just so happen to want to create VR experiences. In many ways, there are projects and ideas only you can produce to a specific end because you ultimately infuse your skills and experiences into the work you do. Whether it is naming conventions or artistic style, the things you create will have a touch of you in them. If 10 developers and creators get the same prompt, which is a brief description of a project idea that a client or developer hopes to create, you will get 10 different projects. Some will be better than others based on the utilization of tools and execution of the prompt. Here are the skills that will prove most useful to you when you are trying to build immersive experiences:

• Project management: Immersive experiences can be large in scale and require experience managing multiple elements. If you don’t know how to navigate both people and a variety of content sources, you can easily become overwhelmed. Although this is often lost on developers and creators, project management skills are crucial to completing projects. To build a portfolio and further your career, you must be able to complete projects.
• Creative direction: XR has yet to reach its peak market value. As a result, grand ideas exist that are yet to be manifested and translated into experiences. The value of a creator and developer is not only measured by the technical skills you offer but also by the vision you present to explore the technology and push it to new heights with your ideas. Having experience in a variety of content creation workflows and pipelines can help you explore the possibilities of XR. Creative directors develop and manage projects from ideas to finished products. They typically have experience in a variety of areas, such as marketing, illustration, business, product development, and more. These skills serve as valuable assets on the creative journey.
• Software development: Having technical skills is very valuable in developing XR experiences. Although Unity makes building experiences easier, to get the most out of the medium and the platform, you need to know how to open the hood and unlock certain features. Even if you aren’t a seasoned coder, being familiar with code structures and functions can lead to major growth and innovations in the space.

Technological components of building immersive experiences

Every immersive experience has the same core elements. The difference among all experiences is the degree to which each of the core elements is incorporated. Whether you are doing a simulation or playing a game, you will need animation, a user interface, lighting, and audio. Creating experiences is more about navigating the required elements to fit the scope of the project rather than redefining what it means to build an experience. Innovation is taking what already exists and improving upon it with ideas that show the true potential of the tools and the medium. The following comprise some of the core components of building immersive experiences:

• Game engine: Most VR experiences are built on game engines because of their ability to render objects in real time instead of pre-rendering the objects, such as animation and compositing software. In this book, we will be using the Unity game engine, the most versatile engine used to build XR applications and games. Another popular engine is Unreal Engine.
• Rendering: I mentioned that game engines use real-time rendering to provide a platform for building interactive experiences. Within Unity, there are render pipelines that determine the way objects are rendered within the experience. The difference in rendering pipelines is usually device-dependent. There are render pipelines for lower-end devices such as smartphones and high-definition pipelines for higher-end devices. Since VR headsets have hardware specifications, knowing which one to use for your project early in the development process is crucial to providing the best experience for your users.
• 3D content: You can’t have a virtual experience without 3D content (2D content on some occasions). In a virtual experience, you interact with the 3D objects in the world you are experiencing. This content can comprise characters, buildings, environments, and even digital twins. Content can make or break the experiences you build because the user chooses the VR experiences they want based on the content in the experience.
• Shaders, materials, and textures: Often associated with 3D content, shaders, materials, and textures provide an element of variation to the 3D world that can elicit various emotions and responses. They provide the color and character to the polygons and pixels of the digital world. If 3D models are the architecture and foundation for the world, then materials and shaders are the paint and decorations. When the materials, shaders, and textures are used correctly, they become recognizable and familiar to the user.
• Levels design and architecture: Design is crucial to experiences because you need to give users environments to anchor their experiences to. Without a map to navigate or cities to traverse, they have no direction.
• Audio: Sound is crucial to having an immersive experience. Hearing the sound of your feet on the pavement as you walk or increasing the volume of music as you walk closer to the source creates something subtle yet impactful. If you can create and integrate sounds, you can harness an important component of an immersive experience.
• UX/UI: User interfaces are the blueprints of interactions within the experiences. If you design a world but people don’t know how to explore the world, then you need to find ways to design elements that feel natural and intuitive.
• Animation: Animation brings life to a static world. In a real-time game engine, VR makes animation an integral part of the interactive experiences. It is often the element people respond to the most because animation alters the world as time progresses. If we were playing a sports game, the location of the animated character or object can determine how the player would respond. In a still world, a player has no incentive to engage in the experience.
• Lighting: Nothing is fun if you can’t see. Lighting 3D environments is a necessary skill because it gives you full control of every element that can improve visibility and influence mood.
• Performance: Not every experience will work on every device. Being able to develop something vast but also performant can maximize your reach and leave a lasting impression on users.
• Software development: Even though we interact with the content we see in a virtual world, those interactions are dependent on elements that you code and integrate using C#. Knowing how to utilize code so that it works for your project can unlock unlimited possibilities.

Now that we understand what immersive experiences are and the components that make up such experiences, let’s dive deeper into XR.