Introduction
Given that we are just starting our journey together to explore the topic of this book—the building, managing, and ongoing success of high-performing advanced analytics teams—let's ensure that we are on the same page. Let's take a moment to set the stage, to synchronize our understanding of what we will be examining and discussing in this book.
My primary motivations in writing this book are to:
- Serve fledgling and experienced front line managers in the field of advanced analytics by helping them avoid mistakes of the past and to assist them in taking the appropriate paths to building sustainable and synergistic teams that can and will engage productively in the exciting process of building these new levels of machine intelligence, and
- Help senior managers and executives understand the investments needed, the timelines required, the problems that can be addressed, and the value to be derived from including talented teams and appropriate managers and management in their organizations.
Throughout this book, we will be talking about teams involved in building advanced analytics and artificial intelligence systems. These are systems that learn and improve over time. We are not talking about static business intelligence applications, dashboards, or reports that outline, visualize, or describe the past. No matter how often a dashboard is refreshed, even a real-time interactive dashboard is still a representation of the past.
Throughout our dialog and this book, the terms advanced analytics and artificial intelligence will be used interchangeably. For clarity, advanced analytics is a broader term encompassing the practice and use of statistics, machine learning, simulation, and optimization. Artificial intelligence refers to machine learning and other analytical approaches that learn from data over time.
In this book, our discussion of advanced analytics systems will encompass analytical applications, individual models, ensembles of models, systems, platforms, and cloud, on-premises, and hybrid environments.
Our discussion will also outline analytical applications, models, and environments that are built on and utilize the following analytical techniques: descriptive statistics, Bayesian approaches, mathematical principles and theories, artificial intelligence (AI), machine learning (ML), simulation, and optimization. Our discussion of advanced analytics will be as broad as possible.
Don't worry—this is a non-technical treatment of the topics. We will not delve into the finer points of ML or any of the subject areas listed here. If you are interested in the technical details of any of these fields, there are numerous consultants, experts, pundits, academic papers, presentations, conferences, symposia, books, and classes that you can engage with to enrich and deepen your technical knowledge. There are too many organizations and events to list or refer you to, but for a solid overview, you might want to start your line of inquiry by using online learning platforms like Coursera, Udacity, or Udemy. Given that those platforms are aggregators from some of the premier universities from across the United States and around the world, you will find much of what you seek from a technical perspective on those platforms.
A few words about what we will not be discussing or what advanced analytics and AI is not. We will not be exploring the topic of sentient machines or artificial general intelligence. These areas of development and topics are interesting to me, and many people, but they are more in the realm of science fiction at this point. We will be focused on the topics of building an analytics function in your organization and how to build and manage a high-performance team. Advanced analytics applications are focused on describing, predicting, prescribing, simulating, or optimizing the immediate present or the future. Artificial intelligence systems are dynamic; they are like a rocket in that they are always off course, but continually course correcting. The teams that build these systems know this.
They know that they are building living and live systems. They know that they must consider a staggeringly wide range of scenarios. They know that the systems that they build can dramatically improve business operations and, in the end, the results of those operations.
Let's start to examine in detail the factors and forces that are affecting advanced analytics in the general market, looking at jobs created, technological evolution, level of success achieved, public perception of the value of advanced analytics, government regulations, and more.
The future of jobs and AI
The impact AI will have upon employment is a widespread topic of discussion in the popular, technical, and industry press and among employers, employees, and thought leaders.
The question I am asked in almost every internal and external presentation that is attended by more than a handful of people is something like, "How many and what type of jobs will be eliminated by AI?"
The first few times I was asked this question, I brushed it off with a brief answer, assuming that it was a passing curiosity by the person asking the question and there was no real concern or emotion behind the inquiry. I was wrong in that regard.
This question is on the minds of many people and it is weighing on people as a real concern. In the past year, I have been asked the same or similar questions in presentations and discussions in Australia, United Kingdom, Germany, Switzerland, and the United States.
Rather than continuing to brush aside the question, I have started answering the question with one of the many studies that has proven, again and again, that AI and related technologies and systems are net job creators in the short and long term. Let's examine a few of those recent studies.
AI is an engine of job creation
One of the relevant studies is from the World Economic Forum's Center for the New Economy and Society, The Future of Jobs Report, 2018. The report includes research and findings that illuminate and explain the detailed job changes that are expected to be seen, country by country, on a global basis. The report suggests that the new jobs created will be significantly larger in number than those eliminated, and those new jobs will be higher paying and have a more secure future.
A 2018 report from the World Economic Forum (WEF) even suggested that, while we may displace 75 million jobs globally by 2022, we'll create a net positive of 133 million new ones. The WEF believes — with the current data in mind — that robots and algorithms will improve the productivity of existing jobs and create several new ones in the future. Perhaps future workers won't get a job — they'll create their own. No amount of angry hand waving or puerile legislation can stop this. We cannot even begin to fathom some of the otherworldly technologies and new career fields that'll one day arise. [1]
I have experienced this exact dynamic in the workplace. In one instance, people were hired to execute a rather dull and rote process in the finance department to move data from one system to another. The people hired were young, smart, eager, and willing to learn. They discovered that the organization had licensed robotic process automation (RPA) software.
The young staff members took it upon themselves to learn the software and become proficient in automating the repetitive processes, thereby eliminating the jobs that they were hired to do. Did they lose their jobs? Yes. Did the company recognize their initiative and talent? Yes. What do they do now? They automate manual processes across the company.
Now, the company has fewer openings, and no-entry level staff members to execute manual data movement, but the firm now has a number of open positions around the world for entry-level staff members to build automated data movement processes in RPA software. Previously, the data entry roles were lower paying, dead end jobs with few to no development paths or planned ways to move up in the organization. Now, the jobs are entry-level analyst roles with higher pay and a planned path to a better job and an explicit development plan.
I am aware that the previous example is not an AI case, but many organizations cannot grasp the leap to AI without taking an easier first step in an area like process automation.
As a manager or executive who is interested in and wants to drive change, you must be aware of the ability of your organization to understand, enact, fund, and assimilate change. You may want the organization to begin operating like a top tier firm in relation to advanced analytics and AI, but the organization may be run by fast followers, laggards, or even worse, luddites.
Keep in mind and look closely at the people who are the senior executives in the firm; how did they get to their positions and how long have they been in the firm? More than likely, they will be the gating factor in how quickly the organization changes and how the organization changes.
I am betting that once you take a close look at these people, you may want to recalibrate your ambitions regarding the timeline to achieve success with AI and related technologies.
Many jobs will never be changed by AI
Numerous people ask me if there are any jobs that will not be automated out of existence by AI. Rather than asking this question, I think it is more insightful and helpful to ask, "Why are there so many jobs today that have not been automated away?"
The essence of the problem can be found in Polanyi's Paradox. Michael Polanyi, a British-Hungarian philosopher, stated, given that "We can know more than we can tell, we shouldn't assume that technology can replicate the function of human knowledge itself." [2]
We humans operate on and with a substantial amount of tacit knowledge that we have a very difficult time expressing to other people. One of the core elements of automating a task or replacing a person with AI is that we need to understand and describe what the job entails at a sufficiently detailed level, in order to replicate the job with automation tools and/or with AI. Without this ability, we cannot automate the task and we certainly cannot expect AI to undertake the work. One timely example can be summed up as, just because a computer can know everything there is to know about a car, doesn't mean it can drive it.
In late 2019, Rob May posited a related idea. In reality, advanced analytics will create whole new industries, or at least subsegments of industries, where people who can afford the services will seek out human curated goods and services that have a high degree of creativity and customization. These services and goods will be sought after because they contain an element of elegance or personalization that is only possible through the involvement of human thought, expression, and craftsmanship. [3]
Regarding the jobs created in the aforementioned category, there will not be a significant number of jobs that will move the employment numbers in any one country, and there is no hard data to back up this claim, but I do believe that May is correct in his core assertion. AI will not create deeply personal experiences.
AI will predict outcomes and it will make operations more effective and efficient, but it will not deepen most, if any, experiences for people. The lesson to learn from this example is that there are a number of market segments that will be created for industrious people. With these, they will serve firms and individuals in ways that are made more valuable by being in opposition to the mass change created by AI.
These types of jobs and businesses will be small, but the prestige and expense of engaging with these firms will be extremely high. These firms and offerings will be the opposite of Amazon, Walmart, and other firms that operate high-velocity, low-margin businesses. The offerings from these companies will be deeply personal, highly connected, coveted, limited, and very expensive.
If AI will create more jobs, let's prepare for those jobs
In my view, the bottom line on employment is that AI will create and enrich jobs in a net positive manner on all accounts. In some cases, maybe not the job you have today but there will be lots more jobs available because of AI.
AI will drive change in the job market. Foundationally, the changes will be that the rote and robotic elements of jobs will be automated away, and those elements will be accomplished through software and hardware. A relevant and salient aspect of automation and the enablement of systems through advanced analytics is that humans and machines are good at different sets of tasks. Jobs will lose the robotic and mechanistic elements of work and they will gain, or become more focused on, the elements that people are good at, tasks like creative thinking, writing, presenting, and collaborating.
Gartner, the technology and analyst firm, maintains that:
AI Will Create 2.3 Million Jobs in 2020, While Eliminating 1.8 Million. The number of jobs affected by AI will vary by industry; through 2019, healthcare, the public sector, and education will see continuously growing job demand while manufacturing will be hit the hardest.
Starting in 2020, AI-related job creation will cross into positive territory, reaching two million net-new jobs in 2025. In 2021, AI augmentation will generate $2.9 trillion in business value and recover 6.2 billion hours of worker productivity. [4]
Jobs will evolve and AI will drive that evolution
My experience with job evolution is a personal one. I started out working for my father as an automotive mechanic. I moved on to working on farms and eventually, I ended up running mills, drill presses, and lathes in factories that fed the automotive and defense industries in rural Michigan. As a teen, I saw that jobs were being eliminated and families were in distress as the old manufacturing base in Michigan was contracting. It is unlikely if you are reading this book that you or your children are working in these types of jobs, but the changing dynamic of work that we are discussing in this book is the same dynamic that I was faced with when I was 18 years old.
We need to be aware that when economists conclude there is no evidence of overall job losses that can be directly attributed to any one technology or market evolution, including AI, they are talking at a global, macroeconomic level. Of course, the effects of AI and automation will differ from region to region and from country to country.
Sean Fleming, Chief Economist of the World Bank, remarked that:
This has been mostly as a result of the use of robotics and automation in the manufacturing sector, which has displaced large numbers of workers. In some cases, those former manufacturing workers have found employment in the service sector.
But there are also pockets of left-behind communities in parts of the developed world, where several generations of families are adjusting to life without work. [5]
At the macro level, there will be more jobs and better jobs for workers and employees, but let's not gloss over the fact that real change will occur, and is underway. A recent study found that a growing proportion of manufacturing jobs now require a college degree:
More than 40% of manufacturing workers have a college degree today, according to a Wall Street Journal analysis of workforce data. That's up from 22% in 1991. If growth continues at the same pace, college-educated manufacturing workers will overtake the number of workers with a high school degree or less within the next few years, the Journal found. [6]
People around the world need to be aware that change is constant, and it has been this way for generations. It is hard to look forward and determine the jobs that will be in demand in 20 to 40 years. Yes, I did mean 20 to 40 years. If you are reading this book, it is highly probable that you will not be working in 20 to 40 years, but your children and perhaps their children and the staff members that you are mentoring will be working. You want to be able to help them, to counsel them on where the future is heading, and this will be in relation to employment and a fulfilling and engaged career.
Looking decades out into the future is challenging for some and invigorating for others. I vividly remember telling my mother that I was quitting my solid and reliable job as an automotive mechanic at a local Ford dealership to attend college, to study a new field called computer science. Her response was, "You are ruining your life." It was hard for her to look out into the future.
Luckily for me, I ignored her advice and counsel. Now, my son, who recently graduated with a degree in computer science, and our daughter, who is studying data science and user experience design, joke, "Those computers, I think that there is a future in working with them." Each time we say it, we laugh and smile knowingly.
AI and data serve us, not the other way around
Another dynamic to keep in mind, at least in the United States, is that the social safety net is not in place to take care of everyone's economic needs in full. For the most part, people in the United States will have to work longer and, in many cases, they will not have the traditional or historical retirement at the end of a fixed number of working years. Finding work that is intellectually stimulating, engaging, and that will not be automated away or redesigned to eliminate human involvement will continue to present an enormous challenge for everyone.
Remember the old saying, find something that you love to do, and you will never work a day in your life. Hackneyed, but there is a kernel of truth in that statement. If you enjoy or love what you do, you will engage in it almost effortlessly, it will not tire you out, you will do it for longer, and you will have the zeal to evolve the role to fit the needs of the market today and in the future. When I think of my work and profession through this lens, I can see being involved in the market and industry for another 20 to 40 years, and still being passionate about my work and contribution.
I worked at IBM in the 1990s. My manager was Dave Carlquist. Dave was, and is, smart and driven, and possesses substantial emotional intelligence, far more than I have or will ever have. Years later, I was waiting to board a flight in Chicago's O'Hare airport. We were all standing there looking at our phones, and I looked up, and Dave was standing next to me. I smiled and poked him with my elbow.
Dave looked up and we laughed and started to catch up on the events of the intervening ~15 years. We were about to board the plane and Dave said to me, "You know, when you were going on and on about how data and analytics were going to be the lifeblood of all organizations, I thought that you were out of your mind." I smiled and said, "Not out of my mind, just early to the party."
These societal, economic, and technological factors and trends point out that the following are useful premises to keep in mind:
- Human creativity will not be taken over by AI; in fact, just the opposite will be true. Human creativity will be more valued and valuable in the future.
- Collaboration between people cannot be automated away. Again, just the opposite will be true. Technology will facilitate better collaboration, but the essence of effective collaboration will become more valued and valuable.
- Communication skills in all forms will become crucial and more valuable.
- User experience (UX) design and construction, the interface between technology and people, will become more important. The UX will become paramount in the engagement of people with systems, applications, and platforms.
- Advanced analytics and AI developers will be in high demand.
- AI will make simple, transactional interactions more efficient, but will do little to enrich sophisticated, nuanced interactions.
- The technology roadblocks that we face today, and some of the very recent past, like the efficiency of machine learning models to be effectively trained, computing capacity, natural language processing, quantum computing, and a wide array of other issues, will be solved in the near future.
Change is constant – aim as far as you can see
The farm job that I had when I was 16 doesn't exist anymore, but there are farmers and farm workers who are working each day to bring food to the market. Farming is a widely varied industry. From artisanal, organic family farms to large corporate organizations running farming operations, farming still exists, but for some, it is not the farming that their grandparents were engaged in, and for some people, that is a loss, but not an inevitable outcome.
Find what you are passionate about, look far into the future, find the intersection of the two, or the multiple relevant intersections of trends and evolving markets that you care about, and work toward them. Listen to everyone who wants to give you advice, even if you'll ignore most of it. You will find your path and you can help others find theirs.
Employment is a very personal experience. There will continue to be the need for employees, managers, executives, entrepreneurs, innovators, and solo agents. Look decades out into the future. Think about what excites you, what the essence of the value is that you bring today and how you might want to bring even more value in the coming weeks, months, years, and decades to come. Continue to learn, engage, and guide people forward, and you will have a long and exciting career.
Let's turn our attention to the future and where opportunity lies for broad sections of the population and general workforce.
The future is long – there is much work to be done
Being deeply involved in the evolution of data, analytics, and learning systems for over 30 years, I never seriously believed, and do not believe, that AI systems will outright replace most humans in the realm of work.
The idea that people will be replaced by software across a wide range of industries in a short time period—causing despair, depression, and loss of motivation and engagement across societies—will not happen. There are people across the world who are prone to fearmongering and have an interest in furthering the dialog at the extreme end of the spectrum. Software that provides process automation and enables predictions will not replace entire industries in the short term.
In my first book, Analytics: How to Win with Intelligence, I wrote:
It has taken 50 years to completely automate transactional systems. It has also taken about 50 years to build out the first layer of information management systems, and we have not completed that ecosystem yet.
Looking 10 to 15 years in the future, we foresee having very sophisticated and automated modeling, data preparation, and model management systems. During that time, we will also continue evolving the math and analytical techniques used. For such systems, we have typically approached horizontal problems first – marketing effectiveness, customer loyalty, manufacturing quality, cyber security, and so on. It will take another 20 to 30 years to perfect these systems.
Somewhere in that time window, we will start to build vertical applications for specific industries, such as automotive, healthcare, pharmaceutical, energy, security, and telecommunications. This will be a long, complex process.
Thus, in total, we foresee between 90 to 120 years' worth of work before we complete our analytics journey. Obviously, we have much to do, but thankfully the work has been both interesting and engaging. [7]
Given the difficulty we have seen in the market for self-driving cars and related technologies, I stand by my prediction that we will not see widespread deployment of AI-based platforms that will cause significant job transformation and reformulation until 2150.
If you were counting on the creation, provision, and delivery of universal basic income due to losing your job to an AI system, you will be very disappointed. Best to keep your skills sharp and keep going to work each day. For nearly every advanced analytic system that my teams and I have built, we have worked with the subject matter experts and their teams after the implementation, and the employees are happier and are focused on higher-value work. Moreover, the humans often take credit for the better decisions being made that are either completely the work of the analytical model, or substantially supported by the output of the applications and/or models. No one ever complains about being on target more often through the augmented workflow of the human/AI collaboration. And the AI system never asks for credit, so it works out well for all involved.
In my personal experience, there is a significant amount of mundane and boring work to be automated away, and a corresponding amount of critical thinking and higher-level decisions to be made every day. These higher-level decisions remain in the remit of human cognition.
Learn and leap
"The lessons of technological innovation remind us that progress always entails thinking the unthinkable and then doing things that were previously impossible," Tim O'Reilly, the founder and CEO of O'Reilly Media, says in chapter 15 of his most recent book, WTF?: What's the Future and Why It's Up to Us. That's why he's optimistic that technology will augment, not replace, jobs. But, he says, "learning will be an essential next step with each leap forward in augmentation." [8]
In their new book, Augmented Intelligence: The Business Power of Human–Machine Collaboration, authors Judith Hurwitz, Henry Morris, Candy Sidner, and Dan Kirsch define augmented intelligence in this way:
…beyond artificial intelligence, there is augmented intelligence, which can significantly transform how we can leverage knowledge, artificial intelligence (especially machine learning), and various tools that support advanced analytics. So, what is augmented intelligence? Augmented intelligence is an approach that uses tools from artificial intelligence to perform well-defined tasks, such as those that are part of decision making. But for augmented intelligence, the human works in collaboration with the machines. Humans need to evaluate the results of automated tasks, make decisions in non-routine situations, and also assess if and when the data must be changed due to changing business needs and demands. [9]
We are on an evolutionary journey in developing and deploying artificial intelligence systems. A few facts will help set the stage as to the global effort in developing systems with advanced analytics and AI:
According to Evans Data Corporation, there were 23 million software developers in 2018, this number is expected to reach 26.4 million by the end of 2019 and 27.7 million by 2023. 29% of developers worldwide were using some form of AI or ML as of 2018 and an additional 5.8 million are expected to start using AI or ML within the next 6 months. [10]
The number of people who are developing these systems may seem like an overwhelming army of people toiling away, in every company imaginable, to automate away every job possible. This is not the case. While there are a significant number of developers working with AI and the number is growing, the majority of those developers are just beginning to experiment with AI-based technologies. And while AI and advanced analytics as a general topic garners an outsized amount of coverage from the press and pundits, we are in the early stages of a long journey.
Let's recap the main points discussed up to this point. AI will have an effect on jobs—it will actually, in all likelihood, create more jobs than it eliminates, but we need to be aware that these new jobs will be different and require higher-level skills than the jobs that are replaced. AI will augment and extend existing jobs. The idea that AI will eliminate jobs is overblown; in most cases, AI will remove the mundane aspects of work and allow people to focus on elements of work that call on more creative and subjective skills. Jobs have been evolving since the creation of work. AI is just another factor in that continuing evolution. AI is different and it has the potential to drive wide-ranging changes, but it is just another factor in the evolution of work.
To bring this process of job evolution into sharper focus, let's outline a recent example. One of my teams built a forecasting application, which we will discuss in greater depth in Chapter 8, Operationalizing Analytics – How to Move from Projects to Production. In relation to the operational staff who were employed to update and run the previous version of the forecast in a spreadsheet-based system, all of them were rendered unnecessary by the new forecasting application. Did all those 30+ people lose their jobs? No, they did not. Rather than having the staff manually obtain data, clean data, and load the data into a spreadsheet, the employees were retrained to be business analysts.
They went from being spreadsheet managers to analysts. Their job composition changed from 80% data management to 75% business analytics. Their new jobs are harder to learn, but pay more, are more secure, and provide for more job advancement and mobility. Now, the challenge is for the employer to retain them, given their newly acquired analytics skills. This is a much better place for the employees to be in from a career perspective, and the employees are more valuable to the employers. Everyone wins.
Now that we have discussed the potential impacts of AI upon jobs, and how those jobs can evolve into more secure and fulfilling jobs, let's consider another area where AI could bring major changes: the global education system.
