Artificial intelligence (AI) is not exactly a new topic of discussion: we have always desired computers that are able to do more for us. In the tools we use every day, AI is present. Whether you are using your smartphone's virtual assistant or recommendations it is making to you based on the patterns it has cataloged from your activities, you are using AI. For instance, I do my grocery shopping on Sunday mornings, and my phone knows where I am going and tells me how long it will take to get there. This is becoming more pervasive in our lives every day, whether using our phone, computer, thermostat, sprinkler system, or the far too many more applications to mention. Unless you are willing to up and move "off the grid", AI is here to stay, so you had better get used to it!

Many organizations are looking to employ AI to help streamline operations and help their businesses be even better at what they are already good at. With this in mind, Microsoft created Azure Cognitive Services as a quick and easy way to implement AI for deployment in a multitude of ways. By giving customers the ability to implement cognitive services and connect to them various applications the business is using, it helps streamline the process of taking advantage of what AI can do for an organization.

As we go through the book and provide samples and solutions, keep an eye out for areas where we see the most improvement. The reason behind this is likely to be a better design choice than building a solution from scratch. The development of these products took many years and cannot be replicated in a reasonably timely fashion. Of course, there will always be unique situations that cannot be satisfied with "canned" software products, but the heart of the conversation of this book is building value for your business in both a cost-effective and a timely manner.

Now, let's get started by looking at our fictional scenario: Ocean Smart.

Ocean Smart is a fictitious seafood procuring, processing, distribution, and retailing company with locations scattered throughout the United States (US) and Canada that specializes in quality lobster, scallops, clams, crab, and various species of fish. Depending on the type of product and the medium by which customers would procure the product, they may go to their local supermarket to purchase live lobster, clams or crabs, fresh scallops or fish, and even "value-added" products that use any combination of the raw materials that can be processed into some sort of seafood pie, frozen and breaded fillets of fish, or a seafood medley to be served over rice or pasta. This encompasses the retail outlets from which a standard purchaser could buy a product to make a meal at home; however, it is just the beginning of where Ocean Smart-produced products may go. As we go deeper into the book, we will discuss how Ocean Smart is able to take advantage of Azure Cognitive Services to streamline operations; improve quality control (QC), customer support, engagement, and services; and predict what intermediate customers (distributors and wholesalers) and end customers will purchase.

The company was founded in the early 1980s when there was not a whole lot of regulation in the industry, and standards were dramatically different from company to company. This meant that the quality of seafood provided ranged widely depending on the source and how the product was handled in transit. Ocean Smart is just as much a success story in the world of logistics as it is in the other areas of expertise and quality standards achieved over time. This section will serve as an overview of company operations for how products are procured right through to how they are sold to end consumers and includes how the company strives to keep customers happy. We will see an overview of each of the areas in which Azure Cognitive Services has been able to help streamline operations and the other results Ocean Smart has realized because of these implementations.

As Ocean Smart decides it wants to use AI to improve operations, it evaluates the ability to build its own solutions by creating machine learning (ML) models and developing code. The company is able to wrap around those ML models and create development-operations (DevOps) and ML-operations (MLOps) practices to ensure standardization. It even goes as far as to compare public cloud vendors in what their offerings are regarding data science and related tools. What the company finds is that there is significant overhead—personnel, training, and much more—required to build all these types of solutions from scratch. Plus, there is a need to have in-house experts on business operations as well as data scientists who could either work with those experts directly or have at least some specific domain knowledge to be able to work independently to build the solutions. As a result of this discovery, Ocean Smart decides that it will be dramatically easier to use Azure Cognitive Services because of the service's ability to quickly and easily plug into any software that is running within the organization via the application programming interfaces (APIs) that are provided. This also helps the company to take advantage of the award-winning ML models that are already developed and deployed by Microsoft researchers, many of which have or are remarkably close to human parity, leading in many of the categories that all tools available in the space are compared against. The company wants to be able to compare how close to a human a machine can get in recognizing the same types of patterns, whether they be in vision, speech, text, or other areas of interest.

Now, let's take a look at a quick overview of some of the operations of the company and areas where Azure Cognitive Services can help streamline operations.

Dealing with paperwork using Knowledge Mining

Today, the company consists of boats for fishing, wharves for receiving products from its own and other fishermen, trucks for transportation of products, live, fresh, or frozen at sea, production facilities for processing of seafood, short-term holding facilities for keeping the product alive, and cold-storage facilities for longer-term storage of the products after processing into a consumable form. Each step of this process requires paperwork, comprising the following:

• Purchase receipt of the product off the boat
• Trucking paperwork, customs documents, and cost receipts for transportation
• Storage facility receiving and shipping documents
• Receiving documents at production facilities containing critical date code and weight information
• Paperwork required to capture information related to disassembly of the product and downstream production
• Shipping paperwork for transport to a storage facility or customer
• Sales information about the product when sold
• Quality reports of samples, protein levels, bacterial concerns, and so on

And this is just an idea of the amount of paperwork related to each transaction to ensure the safety of the consumers who will ultimately end up eating the product. Of course, there are also the regulatory requirements that need to be satisfied for the US Food and Drug Administration (FDA) and other governing bodies, who will want all traceability information about the products from source to table with accompanying paperwork.

When we see a glut of documents of any kind, but especially ones from a variety of sources including both typed and handwritten text, as well as documents containing images and other areas of analysis, it immediately becomes a great use case for a knowledge mining solution. Traditional knowledge mining or document management solutions are great for collecting documents, capturing some detail from within them, and adding tags and references to later be able to pull some details from the documents. However, when using Azure Cognitive Search with related cognitive services, the information that can be unlocked brings the solution to a whole different category of enterprise search. It really is the next-generation engine for relating documents, finding tone in the documents, and gathering significant details of everything that is contained within.

A knowledge mining solution is a large undertaking but produces tremendous value for the companies who are willing to take the time and implement the solution so that it spans the entire organization and becomes a resource for all. The following diagram shows a sample architecture containing some of the services that will be used alongside the Cognitive Search functions, which we will elaborate further on in the second part of the book:

Figure 1.1 – Knowledge mining sample architecture

Next, we are going to use Form Recognizer to process financial documents.

Using Form Recognizer to process financial documents

Another area we will focus on is process- and expense-related documents for automatic processing using the Form Recognizer cognitive service. Form Recognizer allows us to capture relevant details in these documents that we can then store in a database or elsewhere for further use—for instance, processing an invoice using an accounts payable system and feeding the appropriate data we capture into the fields required, with appropriate validation steps along the way. This also helps develop an ML model in the background that can be trained to provide more accurate model training over time.

The Form Recognizer cognitive service offers the ability to streamline how documents, forms, receipts, and other assets are processed within an organization and is a key component in how robotic process automation (RPA) systems are implemented for optimized operations. Some common uses of these technologies are listed here:

• Expense report processing
• Accounts payable/accounts receivable approvals
• Other uses where data needs to be extracted from a form and feeds a system downstream

The following diagram shows a sample architecture using Form Recognizer, some related cognitive services, and other Azure services commonly deployed together to build a full solution:

Figure 1.2 – Form Recognizer sample architecture

The next service we will discuss will help you to understand anomalous behavior among processes in your network.

Using Anomaly Detector for discovering abnormalities

The operations at Ocean Smart are complex, and overhead costs are incredibly significant, especially when it comes to the product market being among the most volatile in the whole food industry. The supply is limited to what is caught, and much more challenging to produce than a traditional farm-raised product such as cattle, pork, chicken, and other land-based animals. Ironically, the industry does not follow the traditional economic models of supply and demand. The independent fishermen are resistant to any one organization, or a small group of organizations, having control of market prices, so buying in greater volume does not necessarily mean buying prices can be reduced as a result. In a very cutthroat industry, relationships matter, but do not necessarily guarantee preference when it comes to purchasing power.

Natural disasters can wreak havoc on the entire industry and can cause significant cost fluctuations for the end consumer. Even today, you can walk into many seafood restaurants throughout the world and find seafood prices marked as Market. This volatility also causes challenges in planning what will be produced and sold at the manufacturing stage of the overall process and what can be promised to distributors. When it comes to finished goods, in whatever form (fresh, frozen, or live), however, there is much opportunity to take advantage of pricing based on supply and demand. The process by which purchase prices are settled can depend on which distributor has which product. There is a weekly, and sometimes daily even, tug of war in the industry between buyers and sellers to determine product pricing that requires significant visibility into which product is moving where and how much of each type is available. As the details have been captured over time regarding the settled prices, fishers, product type, product quality, product region, time of year, and more details, we could use the Anomaly Detector service to test if one of our parameters is somehow out of a "normal" range.

The Anomaly Detector service monitors all the data being collected and calls out areas of concern to the procurement team, giving them the ability to accept the anomaly and provide feedback to the system. Again, this will improve an ML model that has been developed over time or allow the team to go in a different direction with the transaction.

The following diagram provides a sample architecture for Anomaly Detector and other common services in Azure used alongside it for building a solution.

Figure 1.3 – Anomaly Detector sample architecture

The process pictured in the preceding diagram in a simplified version is outlined here:

1. Data is ingested from some source in Azure.
2. Some compute—in this case, Azure Databricks—is used to manipulate and prepare the data and serve it to the Anomaly Detector service for monitoring.
3. When anomalous behavior is detected, an event is triggered and sent to Azure Service Bus for further action.
4. An alert is sent to appropriate parties for follow-up action using the service of the developer's choosing.
5. The output of the data is then logged in Azure Data Lake Storage (ADLS).
6. Power BI is used to visualize the results of the logged data.

Next, we are going to discuss how human sight can be augmented using the Computer Vision service.

Using Computer Vision to detect product quality issues early

After procurement, the manufacturing (or production process, as it is commonly known in the industry) is when the raw materials are disassembled, filleted, prepared, or enhanced by adding seasonings or breading, then cooked, frozen, and packed for distribution. A key part of this process is determining the quality of the product while it is being produced. Simply put, we take the incoming weight of the product and the cost of procurement, including landing the product at a production facility, do whatever production is to be performed to add further value to the product, and understand the overall cost of goods sold as a basis for what the selling price will be. Knowing that the selling price will fluctuate depending on demand and seasonality means that we need to take considerable care when setting these prices. Of course, each of the parts has a different value, similarly to the beef industry where various parts of the animal have different values (Filet mignon is highly sought after compared to chuck steak) so, for instance, the tail of a lobster is valued significantly higher and generates more saleable product than the legs of that same lobster.

As an example, the cost of a pound of tail meat would be four to five times the cost of a pound of leg meat to a grocery store. Because of the value assigned to each of the parts, it is critical to know if the product is somehow damaged upon procurement or within the production process. A tail missing a flipper or having an obvious crack is not going to be sold at the full price but will not be thrown out either. It will either be put in a "B-grade" batch or pushed to a different location to be processed in a different way— perhaps cooking. On the other hand, if a leg were to be crushed, cracked, or similar, it would likely just be thrown away because it is not worth salvaging. This is true for many of the varieties of saleable products, and each will be handled in a way that responds to their market value.

A process for capturing quality issues at the production level will significantly help avoid downstream customer complaints and returns, meaning the product would need to be destroyed and Ocean Smart needs to provide a refund to the customer. As a result, we will employ the Custom Vision cognitive service to capture potential quality issues that get past initial QC testers. By capturing images of lobster tails or irregular scallops prior to packaging, and alerting key personnel to the concern, the downstream losses can be avoided in the future.

The Computer Vision API gives the ability to handle images in several ways. Here are just a few examples of how the service can be used:

• Classifying images based on the content and aligning with similar images
• Grabbing data from screenshots or images captured for cataloging event-related information
• Comparing captured images from cameras and comparing against "known good" images for QC purposes

The Computer Vision service has ML models already developed for image classification and other uses but we can also use Custom Vision for training and deploying custom ML models to compare against.

The following diagram shows a sample architecture of the Custom Vision service and a simplified flow of how the service processes images it receives and compares or other activities downstream:

Figure 1.4 – Computer Vision sample architecture

Next, we are going to discuss how to protect corporate internet services from inappropriate content being displayed.

Applying Content Moderator to avoid the posting of inappropriate material

Once all goods are produced, they are ready for sale to a distributor, restaurant, grocery chain, fish market, via direct sales through a website, or another outlet. Ocean Smart is concerned about having a recognizable brand to consumers, and it typically runs various social campaigns and has a modern, interactive website, plus other marketing materials that will help make the brand stand out to consumers. But this comes with the caveat that the company maintains its reputation for quality. If a review of Ocean Smart's product is published in a forum somewhere or on a social networking website, positive or negative, it is important for the sales and marketing teams to be aware. The last thing the team wants to hear is that the public thinks poorly of their product or the Chief Executive Officer (CEO) comes to them with negative publicity. The company needs to be very aware of any situations that could enhance or serve as a detriment to the Ocean Smart family of brands in the market. All these channels need to be monitored for content as well as responded to in a timely manner to avoid any escalation of a negative situation. With data that is collected from social media outlets and review websites, we can use the Text Analytics cognitive service to gauge sentiment analysis in positive, neutral, or negative form, then build in automated responses as well as information being reported to customer services for appropriate action to be taken. We can also use the Content Moderator cognitive service to ensure no inappropriate content can be published to any of the Ocean Smart digital platforms or brand websites.

The Content Moderator cognitive service has three key areas that can help to moderate content: images, text, and video. The intent behind the service is to ensure that content does not get displayed on a website or other digital outlet where one of the preceding media types could be consumed by an inappropriate audience. When it is deployed, several other services can be expected to be used as part of the architecture, as follows:

• Event Hubs will capture live data streams.
• Azure Functions will parse text and content.
• Azure web services will host the application.

The architecture shown in the following diagram gives a good overview of the flow of information when the Content Moderator service is deployed for the protection of a regional chat service online:

Figure 1.5 – Content Moderator sample architecture

Next, we're going to discuss a service that companies can deploy on their website for making personal recommendations to customers.

Using the Personalizer service for product recommendations

The end consumer of a product is not generally known by the team on the production floor except perhaps by the type of packaging that is being used. There are several ways a product can be packed, depending on if it is going to a distributor who will then sell the product to a restaurant, chain of restaurants, or another type of food producer. There are also products that are specifically packaged for optimal display in retail outlets such as grocery stores, fish markets, and specialty stores where the product is much more visible and presented in a way that will try to entice a consumer, as with most retail packaging.

Customers can order the product of their choice directly from Ocean Smart through its e-commerce website. As a customer makes purchases and patterns are determined, the website can either make recommendations for previously purchased products or suggestions for comparable items the customer may like based on their history. There is also a capability that allows for additional recommendations when the customer puts a product into their "cart" such as a marinade or additional product from the store that might pair well with the product they have already added to their cart.

The Azure Personalizer service is a useful tool for just this type of application. Whether looking to make recommendations for other projects or to show commonly paired products that are purchased together by customers, the Personalizer service helps to build those recommendations. The service will improve its accuracy over time, and more of a user profile is built about the consumer of the content. Also included with the service are tools for building a "mock" scenario that could be presented to a user of the site, leading to an improved recommendation.

The Personalizer cognitive service is a resource that helps organizations to align products or make recommendations for location placement on screen. These recommendations are based on optimal positioning backed by an ML model that is developed over time and offers the ability to improve the model with a feedback capability. Because the service is embedded within a web application, many of the services required will be a part of a traditional website deployment.

The architecture in the following diagram gives an overview of how the Personalizer service is used to add recommendations in real time based on what a customer has chosen to add to their cart or explored on the website:

Figure 1.6 – Sample Personalizer service architecture

By implementing the Personalizer service, there was a 15% increase in profit from website sales at Ocean Smart due to the paired items that were offered and new items the customer had not purchased in the past when these were suggested. This increase in profit also drove the direct sales teams to start making similar suggestions when making sales to their customers to also increase sales. We will discuss this in more detail in Chapter 12, Using Personalizer to Cater to Your Audience.

Next, we will explore some of the options and capabilities you have when applying Speech cognitive services to your organization.

Applying Speech services for call center improvements

With any quality product being sold, there needs to be a reputation for backing that product with quality customer service. In the case of Ocean Smart, if a customer has a poor experience with one of its products and they contact customer services for some sort of resolution, there are several ways they can approach this. The website has a "contact us" form where a user can leave feedback for the team. There is a need to route the feedback to the correct employee or team for handling the feedback in the appropriate manner, but most important is how communication is handled where a customer complains about the quality of the product directly. Another more likely way customers will complain is by calling customer services directly.

As has become standard for corporations, Ocean Smart has an automated attendant phone system that helps to triage and route the nature of calls coming in. An example of this could be the following:

• "If you would like to let us know that your experience with our product is not quite what you expected, press 1, or say 'issue with product'."
• "If you would like to tell us how great our product is, press 2, or say 'this product is great'."
• "If you would like to make a suggestion, press 3, 'I'd like to make a suggestion'."

Each of these responses, whether verbally or through the keypad, will then route the call to a team/person or, depending on the time of day, for example, route the call to a voicemail box that is likely to be emailed to a team/person for handling during normal business hours. Using Speech cognitive services, the system also recognizes several different languages to help customers who do not speak English fluently, further enhancing the customer service experience. There has been considerable care in the way these types of concerns can be expressed so that customers feel as though they are being heard by the company and that the company will make it right. Ocean Smart is using the following services together to build a "call center" reference architecture:

• Speech-to-text
• Language Understanding Intelligent Service (LUIS)
• Text-to-speech
• Sentiment
• Translator

By employing this group of services, Ocean Smart can greatly enhance the customer service experience and maintain brand loyalty.

The Speech services can be applied to a host of activities related to the translation of speech, speech-to-text, text-to-speech, and language recognition activities such as keyword, speaker, and intent from recorded or real-time audio. There are many existing use cases for how speech services can be used, so the example later in Part 3, The other Cognitive Services that will help your company optimize operations, of this book will cover a call center reference architecture for improving the customer service experience.

The following diagram shows a common architecture deployed when working with the Speech services for transcribing call center activities into a text-readable format. These files are stored in Blob storage for use with later activities such as sentiment analysis, translation, and other related services downstream:

Figure 1.7 – Speech services sample architecture

LUIS is used to take natural language and process it to literally put words into action. Rudimentary versions of this service involve calling an automated phone system and being prompted "Say yes or no", for example. Depending on the response, the caller is then prompted to respond to the next question as the flow of information continues within the automated system until the caller is routed to the correct person, department, answering service, and so on. The following diagram shows a reference architecture representative of a solution that can be used to deploy an intelligent chatbot with related LUIS services:

Figure 1.8 – LUIS sample architecture

Next, let's look at how we can calculate ROI

When you are considering an AI workload in your business, there are many factors you must consider as part of the process. This section of the book will give you some helpful hints as you strive to deploy solutions to do the following:

• Recognize areas where you may be able to take advantage of AI easier than in other areas by understanding the complexity of deployment of each of the solutions.
• Calculate the ROI, because we all know that we have a boss who will want to know what it is going to take to recoup what is spent on this project.
• An overview of what you can expect your TCO to be when the solution is fully deployed, and calculating cost growth as your solution grows.
• How to pull all the necessary information together to ensure you have a complete solution that will bring value to your business and make you an AI hero!.

When we begin to discuss the areas where AI and cognitive services can be of benefit to an organization, we really need to put on our business analyst (BA) hat. Or, perhaps a BA has already started the process of identifying key areas where there are major inefficiencies in your organization. Each chapter of the book will include some helpful hints on how to be sure you are optimizing the services for cost and simplicity. That said, you will need to evaluate the value of implementing a cognitive service to help streamline a process, reduce human intervention, or be able to outline the benefit your organization will receive as a result.

Building your justification

There are costs associated with the implementation of these services, and without a full analysis of what the TCO and ROI are for the implementation, it may be very challenging to justify the out-of-pocket expenses required. Try to avoid deploying technology for the sake of technology. What I mean by this is that too many technologists really want to embrace new technology because it is "cool" or "exciting" to learn and build something new.

As an alternative scenario, you may have an executive or key stakeholder from your organization who really thinks that by implementing "x" technology, the business will be greatly enhanced. This is a dangerous scenario to have to deal with when it comes to your career at your organization, and you will have to be very careful with how you proceed. In reality, there may not be much value for the use case they are pursuing, or the data available for the use case isn't clean, accurate, or good enough. In situations such as these, you will have to decide whether it is worth your reputation to push back, or even go forward with, such an implementation.

To avoid the possible fallout from a failure, it is advisable that you build your TCO and ROI models with as little conjecture as possible. Break down the use case into simple math with parameters that can help you justify your position either way. Present your findings to the stakeholder and state your position. Let that person then make a decision on whether to proceed, hopefully reducing the pressure on you and the team you are working with.

In either case, you need to get a starting point for how to build these models. Here are some thoughts on ways to get started with the data you will need for doing so using a Form Recognizer scenario to automate the accounts payable process. Be sure to work with someone in finance or accounting to understand what the average overhead is of one of these individuals to build a baseline of your averages:

• How much, on average, does an accounts payable clerk cost the company per hour?
• How much time, on average, gets spent per week on manually entering the details of that accounts payable transaction into whatever system is being used for paying vendors?
• Is there already a process for capturing forms in a digital format, such as scanning to a repository? This could add additional costs to your calculation if there is not, or if equipment is required for such a process to be implemented.
• What are the costs required to insert data into the system, and with how much confidence can it be inserted? Are you getting handwritten slips, as in the seafood industry, making it harder to be accurate, or is everything typed?
• Is there already an electronic data interchange (EDI) process in place, and the only human intervention required is validation? Find your alternate scenarios and boil costs down in the same way so that you are sure to capture the full picture.
• What is the risk or cost if payment of an incorrect amount is made or it is made to the wrong vendor? Many of these systems can be built with very high accuracy; however, they are not completely infallible, so knowing what the downstream effects are as a result may need to be factored in.
• What is the cost of the development time required to build the solution? Can you put together a sample project plan with approximate timelines to evaluate development expenses?
• What will the solution cost the organization after it is implemented? There will be recurring costs, which we will cover in each chapter, that will need to be factored in.
• Do you have clean enough data to build an ML model to run against when processing invoices for payment? Good, clean data is critical for any deployment of AI in any scenario.

This list should provide a good starting point for where you can begin to evaluate what will be required when building out your cost models. Work through the costs and the savings to build the complete model and understand how long it will take to recoup the costs that are required to the point where you start to save the company money. Is it 3 months or 3 years? That break-even point can be a critical point that helps your solution float or sink, so be sure to have a trusted advisor look at your completed analysis.

Proving out your solution

After a complete evaluation, if you still feel it makes sense to push forward with building the solution, you will also likely have to build a proof of concept (PoC) for demonstration. Microsoft provides many tutorials for getting started with the technology through their documentation and GitHub, or you can use the solution we provide as your baseline with relevant documents for demonstration.

You have built your models, checked them with another trustworthy source, built a PoC, and tested your demonstration, so now, it is showtime. Be mindful—there is certainly no exact science to building a perfect presentation for management, as each team will have its own priorities. However, the goal is to try to make sure that you are as prepared as possible to be able to start one of these projects and build in some alternatives or answer those curveball questions that come at us when we are being challenged. The steps in the preceding list should hopefully prepare you for these challenges and will hopefully help you avoid chasing after a solution that won't provide as much value to the organization as you or the stakeholders originally thought.

Of course, there is always the possibility that the budget may not be available for any investment. If we believe there is still a compelling case to be made and it is worth pursuing, this is when we should flip the conversation to "this is how much revenue we believe can be expected as a result of implementing these solutions", if this is feasible. There is certainly a softer side of this conversation, as sometimes we need to evaluate best-case scenarios where maximum value can be achieved, without having the hard data to back it up. If "selling" your solution isn't one of your core skills, enlist a teammate, friend, advisor, or sponsor that you trust and see if they are willing to join your crusade of helping your company be more efficient. With time, the technology will improve and the costs will decrease for implementation, so maybe it is best, instead, to wait for the next budget cycle, possibly? Use your best judgment on how to have the most impact on your organization.

In this chapter, we have seen an introduction to Ocean Smart, with some of the reasons it is using AI for optimizing operations, improving the customer experience, and increasing sales. We also discussed why the company chose to build solutions using cognitive services rather than building applications from scratch, and why it is enhancing the ML models over time for better accuracy. We wanted to give an overview of the types of scenarios that are common in many organizations where cognitive services can be quickly and easily implemented for almost immediate business value. Of course, not all of the solutions will be identical to what you are facing in your organization, but the intent is to give you the ability to get an understanding of how the technology works for each of the scenarios and tailor this to your own situation.

In Chapter 2, Why Azure Cognitive Services?, we will elaborate further on the Azure cognitive services we have briefly discussed in the prior section, with more detailed examples and use cases for each service that are commonly helping organizations.