The Infinite Retina

Introduction

Spatial Computing comprises all software and hardware technologies that enable humans, virtual beings, or robots to move through real or virtual worlds, and includes Artificial Intelligence, Computer Vision, Augmented Reality (AR), VR, Sensor Technology, and Automated Vehicles.

Seven industry verticals will see transformational change due to Spatial Computing: Transportation; Technology, Media, and Telecommunications (TMT); Manufacturing; Retail; Healthcare; Finance; and Education.

These changes are what is driving strategy at many tech companies and the spending of billions of dollars of R&D investment. Already, products such as Microsoft’s HoloLens AR headset have seen adoption in places from surgery rooms to military battlefields. Devices like this show this new computing paradigm, albeit in a package that’s currently a little too bulky and expensive for more than a few of the hardiest early adopters. However, these early devices are what got us to be most excited by a future that will be here soon.

Our first experiences with HoloLens, and other devices like it, showed us such a fantastic world that we can predict that when this new world arrives in full, it will be far more important to human beings than the iPhone was.

We were shown virtual giant monsters crawling on skyscrapers by Metaio years ago near its Munich headquarters. As we stood in the snow, aiming a webcam tethered to a laptop at the building next door, the real building came alive thanks to radical new technology. What we saw at Metaio had a similar effect on Apple’s CEO, Tim Cook. Soon, Apple had acquired Metaio and started down a path of developing Augmented Reality and including new sensors in its products, and the capabilities of this are just starting to be explored. Today’s phones have cameras, processors, small 3D sensors, and connectivity far better than that early prototype had, and tomorrow’s phones and soon, the glasses we wear will make today’s phones seem similarly quaint.

In Israel, we saw new autonomous drones flying over the headquarters of Airobotics. These drones were designed to have no human hands touching them. Robots even changed memory cards and batteries. New Spatial Computing technology enabled both to “see” each other and sense the world around us in new ways. The drones were designed to fly along oil pipelines looking for problems, and others could fly around facilities that needed to be watched. Flying along fences and around parking lots, their Artificial Intelligence could identify things that would present security or other risks. These drones fly day or night and never complain or call in sick.

Focusing on just the technology, though, would have us miss what really is going to happen to the world because of these technologies. Our cities and countryside will reconfigure due to automation in transportation and supply chains as robot tech drives our cars, trucks, and robots rolling down sidewalks delivering products. We’ll spend more time in virtual worlds and metaverses. More of our interfaces, whether they are the knobs on our watches, cars, doors, and other devices, will increasingly be virtualized. In fact, many things that used to be physical may be virtualized, including stores and educational lessons from chemistry experiments to dissection labs.

Computing will be everywhere, always listening, always ready to talk back, and once we start wearing Spatial Computing glasses, visual computing will always be ready to show us visualizations of everything from your new designs to human patterns in stores, on streets, and in factories and offices. Some call this “invisible computing” or “ambient computing,” but to us, these systems that use your eyes, voice, hands, and even your body as a “controller” are part of Spatial Computing.

At the same time, all this new computing is joined by radically fast new wireless technology in the form of 5G. The promises of 5G are threefold. First, we’ll have more than a gigabit of data if we have the highest bitrates, and even the lowest rates promise to give us more bandwidth than current LTE phones. Second, wireless will soon add almost no latency, which means that as soon as you do something, like throw a football in a virtual game, it’ll happen, even if you have many players viewing the football in real time. Third, 5G supports many more devices per tower, which means you will be able to live stream even a Taylor Swift concert with tens of thousands of fans filling a stadium.

When you combine 5G with all the new things under the Spatial Computing umbrella, you get a big bang. All of a sudden, cars can work together, where one of the cars will be sending detailed 3D imaging of streets instantly to other cars behind it. New kinds of virtual games will be possible in the streets, where hundreds of people can play virtual football games in parks and other places. Crazy new virtual shopping malls appear where virtual celebrities show you around, and you can imagine new products directly in a virtual scan of your home and in other places as well.

A range of new capabilities will appear over the next few years in devices you wear over your eyes. There will be very light ones that are optimized for utility, showing you navigation, notifications, reminding you where you have left things, or nagging you to do some exercise or meditation to keep on top of your physical and mental health. There also will be heavier devices that will be more optimized for everything from detailed design or architecture work to entertainment and video game work. We can even imagine owning several different kinds of Spatial Computing devices, along with some smart contact lenses, that will let us go out on a date night without looking like we have any computing devices on at all.

We currently have VR devices that cost a few hundred dollars that are great for games and a few other things, like corporate training. On the more expensive side of the scale, we have devices that can be used by car designers, or even as flight simulators to train airline pilots. The expensive ones, though, will soon look as out of date as one of the first cell phones does today. By 2028, the computing inside will shrink to a fraction of the size of today’s devices and the screens inside will be much sharper and capable of presenting virtual and augmented worlds to us that far exceed what we can experience today.

It is this next wave of devices that will usher in the paradigm shift in computing and in human living that we are discussing here. Already these changes are benefiting many enterprises, raising productivity. Inside many warehouses, hundreds of thousands of robots scurry about, moving products from trucks to packages. These new warehouses have evolved over the past decade and enable retailers to keep up with floods of new online orders that, back in 2000, were only dreamt of in futuristic books like this one.

The productivity gains will spread to many jobs. At Cleveland Clinic, surgeons are already using similar technology that shows them digital views from ultrasound, CAT scans, and other sensors. Like the warehouse worker who sees a blue line on the floor telling her how to find the product she’s looking for, in this case, when a surgeon navigates to the right place to cut out a cancerous tumor, it lights up like a missile guidance system and tells the surgeon they are in the right place.

Other systems help workers “phone a friend” with new remote assistance features. This can help companies that have expensive machinery, or other work forces, including surgeons, architects, and engineers, save money. At some plants, the savings will be substantial. It took us 30 minutes to simply walk across the Boeing floor where it builds airliners. Asking someone for advice virtually might save someone an hour of walking just to come over and see your problem in a plant like that.

New devices let these remote helpers see what you are dealing with, and they can often show you visually what to do. Imagine trying to remove an engine while holding a phone or tablet in your hand. These systems, because they use wearable glasses, can let workers use both of their hands while talking and showing the remote assistant what is happening. The savings in downtime can be extreme. Imagine a problem that is causing a shutdown in a line at Ford. Every minute it is down, Ford loses about $50,000.

Even for salespeople and managers, the cost savings add up. A flight from San Francisco to Los Angeles usually costs about $700, including airfare, hotel, a decent meal, and an Uber/taxi ride or two. Increasingly, these meetings will be replaced by ones held in Virtual Reality. Two headsets are less than $800. In just a couple of virtual meetings, the headsets pay for themselves. Social VR from Facebook, Spatial, Microsoft, and others, are rapidly improving to make these virtual meetings almost as good as physical ones. The time and cost saved will add up to big numbers, and workers will be happier. Workforces will be less likely to pick up viruses from travelers too, which will also add up to big savings for corporations and reduced risks.

More lives could be saved, too. Mercedes-Benz had an Augmented Reality system built to show first responders how to cut apart a wrecked car. The app showed where fuel and electrical lines were so that firefighters working to free an accident victim wouldn’t start a fire or electrocute themselves. This isn’t the only example we have of this technology helping first responders, either through better training or by giving them various assistance on scene. One such system helps police gather evidence and then be able to recreate crime scenes for juries that they can virtually walk around.

Here, we’ve given you a taste of just how much the world is about to get reconfigured because of Spatial Computing technologies. Let’s dig into what this book will cover.

What this book covers

Part I: Why Spatial Computing and Why Now?, The Significance of Spatial Computing

Chapter 1, Prime Directive, Mobile phones soon will give way to headsets and glasses that bring computing to every surface. What is driving all of this new technology? We have a need for complex technologies to keep us around on this planet longer and in a more satisfied and productive state. What will drive us to build or buy new headsets, sensors, and vehicles, along with the connected systems controlled by Artificial Intelligence? Augmentation is coming, and that can mean a lot of different things, which we will explore.

Chapter 2, Four Paradigms and Six Technologies, we will look at the previous three foundations of personal computing and include the new Spatial Computing paradigm. The six technologies discussed are those that enable Spatial Computing to work: Optics and Displays, Wireless and Communications, Control Mechanisms (Voice and Hands), Sensors and Mapping, Compute Architectures (new kinds of Cloud Computing, for instance), and Artificial Intelligence (Decision Systems).

Chapter 3, The Impact of Generative AI on Spatial Computing, we discuss how Generative AI is revolutionizing Spatial Computing by enhancing immersive experiences, enabling hyper-realistic simulations, and personalizing interactions across industries. Generative AI plays a pivotal role in training and education by creating highly detailed virtual environments for aviation, medical, and military applications. AI-driven simulations now incorporate emotional and psychological realism, making training more effective and preparing professionals for real-world scenarios.

Part II: The Seven Visions, A Road Map to Spatial Computing Disruption in Seven Industries

Chapter 4, Vision One – Transportation Automates, where you will discover that soon you will tell your glasses, “Hey, I need a ride,” and you’ll see your ride arrive. Sounds like Uber or Lyft, right? Look closer, there isn’t a driver inside. Now think of the cost and the other advantages of that. Economists see that such a system could be a fraction of the cost, and could do many other things as well: “Hey car, can you go pick up my laundry and then dinner for our family?” The problem with such a world is that it is probable, many tell us, that we’ll see much more traffic near cities as we use transportation to do new things, like pick up our laundry. This is why Elon Musk came up with the Boring Company to build tunnels under cities. We show some other solutions pioneers have come up with, including special roads for these vehicles and new kinds of flying vehicles that will whisk commuters into city centers, passing above all that new traffic.

Chapter 5, Vision Two – Virtual Worlds Appear, provides details on Technology, Media, and Telecommunications, another of our seven industry verticals to be disrupted. We start out by detailing the different kinds of devices that are available to bring a spectrum of Spatial Computing capabilities to your face, from Virtual and Augmented Reality headsets to lightweight smart information glasses, and even contact lenses with displays so small that it will be very hard to tell that your friend is wearing one.

Chapter 6, Vision Three – Augmented Manufacturing, you’ll learn about how Spatial Computing is changing how factories are even designed. Increasingly, these factory floors are using robots. The robots are different than they used to be, too. The older ones used to be kept in cages designed to keep humans away. Those can still be found welding, or like in Ford’s Detroit factory, putting windshields into trucks. Newer robots work outside cages and sometimes, can even touch humans. These types of robots are called “cobots” because they cohabit with humans and can greatly assist workers.

Chapter 7, Vision Four – Robot Consumers, we detail changes that make retail stores, even traditional ones, more efficient and better for both consumers and sales for producers, and useful new Augmented Reality technologies that make shopping at home much easier.

Chapter 8, Vision Five – Virtual Healthcare, we discuss how the virtual healthcare system guides the surgeon to the right spot to cut out a patient’s cancerous tumor.

Chapter 9, Vision Six – Virtual Trading and Banking, we cover the future uses for Spatial Computing in the financial industry.

Chapter 10, Vision Seven – Real-Time Learning, we talk with educators and others who are using technology aggressively to make learning more virtual. It isn’t only for kids, either. Soon, because of automation, we’ll need to retrain millions of adults around the world, and schools and universities are responding with new curricula, new learning programs for Virtual and Augmented Reality, and new support systems to enable even truck drivers to change careers. Speaking of careers, already at companies like Caterpillar, it is using Augmented Reality glasses to train workers to fix their expensive tractors in real time. Many new VR-based training systems are being developed, from simulators to help police learn how to deal with terrorist situations to ones that show quarterbacks how to perform better, to training at Walmart that shows retail workers how to manage stores better. Verizon even trained its retail store workers on what to do if they are being robbed using VR-based training. What if, though, the system could do even more, we asked, and predict what we might do next and assist us with that?

Part III: The Spatial Business, Productivity, Efficiency, and Making Profits Are About to Undergo Radical Changes

Chapter 11, The Always Predicted World, we show how that data will be used in each of our seven disruptable industries to serve users in radically new ways.

Chapter 12, The Business of Generative AI, we discuss how Generative AI and Spatial Computing are transforming businesses by enhancing creativity, efficiency, and customer engagement. Companies use these technologies to develop innovative products, streamline operations, and gain a competitive edge.

Chapter 13, The Impact of the Apple Vision Pro, we explore the Vision Pro’s technological features, applications across industries, content development, market impact, challenges, and its role in the future of Spatial Computing. visionOS, the core of this device, introduces a three-dimensional interface that adapts to user behavior, making digital interactions feel more natural and efficient.

Chapter 14, How Human?, we provide a philosophical framework put forth by L.A. Paul of Yale University in her book, Transformative Experience, that explains why human beings tend to have cognitive issues with radical new technologies. We then discuss recent issues regarding privacy, security, identity, and ownership, and how they relate to Spatial Computing. Finally, we take up how Spatial Computing technologies can be utilized to bring about human social good.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email [email protected] and mention the book’s title in the subject of your message. If you have questions about any aspect of this book, please email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you reported this to us. Please visit http://www.packtpub.com/submit-errata, click Submit Errata, and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com/.

Share your thoughts

Once you’ve read The Infinite Retina, Second Edition, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.