Understanding OMS architecture and data flow

This section will explore important architectural concepts and considerations for the various OMS services, and provide you with an understanding of how OMS receives and processes data. A good grasp of how data flows to OMS for the various management functions will enable you to better follow the subsequent recipes.

Getting ready

OMS is a collection of cloud-based services that provide you with hybrid cloud management capabilities, and through four key solutions offerings, OMS provides you with flexible access to the management capabilities that you need. Each of the four solution offerings require specific cloud services to be enabled in Azure in order to access the underlying management capabilities that they provide.

How to do it...

To get started, you should determine which of the key solutions and underlying capabilities you need, and understand how the various OMS cloud services facilitate their respective capabilities. For instance, if you are primarily concerned with insight and analytics capabilities for log collection and searches, and for network health monitoring, you would make use of the Log Analytics service in Azure. If, however, you are interested in protection and recovery capabilities and would like to ensure the availability of your applications and data, you would make use of the Backup and Site Recovery services in Azure. When evaluating the key solutions, note the following capabilities included with each offering:

Insight and Analytics	Automation and Control	Protection and Recovery	Security and Compliance
Log collection and search	Azure Automation Desired state configuration (DSC)	Back up to Azure and restore from Azure	Security and audit capabilities with threat intelligence
Network health monitoring and application insights	Update management and automated remediation	Site recovery to Azure and the customer's secondary datacenters	Malware threat analysis
Application and server dependency mapping (Service Map)	Change tracking	Replicate and failover to Azure, and failback from Azure	Integration with Azure Security Center for in-depth Azure services security management

Table 1.1 Solutions

How it works...

To understand how OMS works, you need to know about the various services that enable the various management functions in OMS.

Log Analytics

As mentioned earlier, Log Analytics is an OMS service that enables you to monitor your environments' availability and performance. Log Analytics does this by collecting data from sources that you connect to the service. The following are some examples of such sources:

• Windows and Linux agents
• Azure VMs and resources
• System Center

For Windows and Linux operating systems, Log Analytics collects data through agents that must be installed on the host computers. These agents then collect data from the server and relay the data directly to OMS endpoints. If, however, the computer(s) are part of a System Center Operations Manager (SCOM) management group, then no additional agent is required because through SCOM-to-OMS integration, and depending on the management solution enabled in OMS, the SCOM agents will collect data from the servers they are deployed to and send it either to OMS via the SCOM management group, or they will simply send the data directly to OMS.

In addition to collecting data from Windows and Linux computers and System Center, Log Analytics can also collect data from Azure resources such as Azure Diagnostics and Azure Monitor. Azure Diagnostics data can be written directly into Log Analytics, or sent to Azure storage, where Log Analytics is then able to read the storage logs. Log Analytics can also collect data from other Azure resources using connectors, which enable data to be sent from services such as Application Insights to Log Analytics. In addition, Log Analytics provides a REST API that enables data collection from other Azure services, third-party applications, and custom management solutions that can't send data through any of the aforementioned means.

Once sources are connected to Log Analytics, data sources are then collected from the various connected sources, based on data source configurations that are delivered to agents either directly, for directly connected computers, or through SCOM management packs, for agents that report to a SCOM management group that is integrated with OMS. Some examples of data sources include Windows Event logs, custom logs, Windows and Linux performance counters, and Syslog, among others.

Once the agent receives the data source configurations, it collects the specified data, and, depending on the collected data source - directly or via SCOM - it sends the data to Log Analytics. Once the collected data gets to OMS, it is then stored as records in the OMS repository. You will then be able to make use of the log search feature in Log Analytics to query and analyze the indexed data to glean insights about your cloud and on-premises environment and consume the data in various ways (visualize, alert, automate, integrate into workflows, and so on), which we will take a look at later in this book.

The following diagram depicts the flow of data from various connected sources to OMS and to the OMS repository for storage:

Figure 1.1 Log Analytics data collection

Automation

As mentioned earlier, the Azure Automation service lies at the heart of configuration management, process automation, and other automation-related capabilities in OMS. The Automation service uses Azure technologies and Windows PowerShell to provide you with process-automation capabilities using runbooks, and configuration-management capabilities using desired state configuration (DSC) for your Windows and Linux resources that may reside on-premises, in Azure, or other cloud service.

To automate processes such as long-running and repetitive tasks, you will make use of a set of tasks called runbooks. These enable you to perform automated processes in Azure Automation. You can perform automation tasks with runbooks just like you can with PowerShell, because runbooks in Azure Automation are based on Windows PowerShell or PowerShell workflows. Azure Automation runbooks execute in Azure and can be run against any cloud resources and any other resources that you can access. To execute runbooks against your on-premises resources, you can make use of the Hybrid Runbook Worker feature, which enables you to designate one or more computers on premises as resources, on which Azure Automation can execute runbooks to manage resources on premises.

Each worker will require the Microsoft Management Agent (MMA) and will connect to both the Automation account in Azure Automation and OMS Log Analytics. Azure Automation delivers the runbooks to the workers, and all other automation processes are executed in Azure Automation. You can then monitor the behavior of the management agent using Log Analytics. There are other considerations for making the Hybrid Runbook Worker feature highly available using groups, and we'll explore these later in this book.

Azure Automation also provides you with configuration management capabilities, using Azure Automation DSC. Azure Automation DSC is based on PowerShell DSC fundamentals, and is, in fact, a cloud-based solution for PowerShell DSC, and uses a declarative PowerShell syntax to enable you to manage, deploy, enforce, and monitor configuration for your computers. Because it is cloud based, you will manage your DSC resources in Azure Automation and apply your desired configurations to any computers on premises or in the cloud. Your computers then retrieve the configurations from a DSC pull server in Azure. You can then use the reporting capabilities in Azure Automation DSC to monitor the application according to your criteria, and identify and manage drift.

The following diagram depicts the Azure Automation data flow, process automation using runbooks in Azure and Hybrid Runbook Workers on premises, and configuration management using Azure Automation DSC:

Figure 1.2 Azure Automation workflow

Azure Backup

Azure Backup is one of the services that enables the protection and recovery management functionality in OMS. It is a service based in Azure that enables you to protect and restore your data from the Microsoft cloud, and includes support for the protection and recovery of files, folders, application workloads, and Azure virtual machines. Azure Backup provides various components to meet your protection and recovery needs, and depending on your protection goals, you can use one of the following components to protect your data in a Recovery Services vault in Azure:

• Azure Backup (MARS) agent
• System Center Data Protection Manager (DPM)
• Azure Backup Server
• Azure IaaS VM Backup

Note that while all Azure Backup components enable you to protect your data using a Recovery Services vault in Azure, the Azure Backup Server also enables the storage of backup data to a locally attached disk, and the System Center DPM component enables the protection of backed-up data to a locally attached disk and on-premises tape libraries. Azure Backup also provides some support for protecting Linux computers.

When storing backups in Azure, depending on the backup component you utilize, once the data is backed up at the protection point, it is compressed and stored in an Azure-based online storage entity called a Recovery Services vault, and, based on your storage needs, you can enable high availability through locally-redundant or geographically redundant storage replication. You can monitor backup metrics and connect to the OMS Monitoring solution for Azure backup..

The following figure depicts an Azure Backup data flow, a backed-up data relay to an Azure Recovery Services vault, storage replication of protected data, the monitoring of backup statistics, and the viewing of backup reports with Power BI, as well as the monitoring of backup parameters with the OMS monitoring solution:

Figure 1.3 Azure Backup workflow

Azure Site Recovery

Azure Site Recovery (ASR) enables the recovery management capabilities for OMS. ASR is a service in Microsoft Azure that facilitates your disaster recovery and business continuity strategy by enabling you to replicate, failover, and recover your workloads in the event of a failure. With ASR, you can replicate on-premises VMWare, Hyper-V VMs and Windows and Linux physical servers to either Azure storage or to a secondary datacenter. You can also use ASR to replicate Azure VMs to another Azure region.

ASR supports the replication of VMs in the following scenarios:

• Replication and recovery to and from Azure of on-premises Hyper-V VMs on Hyper-V standalone hosts and clusters that are managed with or without System Center Virtual Machine Manager (VMM)
• Replication and recovery to and from Azure of on-premises VMWare VMs
• Replication and recovery to and from Azure of on-premises physical servers
• Replication and recovery to and from secondary datacenters of on-premises Hyper-V VMs on Hyper-V standalone hosts and clusters that are managed in VMM clouds
• Replication and recovery to and from secondary datacenters of on-premises VMWare VMs
• Replication and recovery to and from secondary datacenters of on-premises Windows and Linux physical servers
• Replication and recovery of Azure VMs from one Azure region to another

The ASR replication process varies according to the scenario you implement, and will be explored in greater detail in Chapter 4, Protecting and Recovering Data with OMS, of this book. In general, if replicating workloads to Azure from an on premises location, you will need to set up requirements for the Azure components, including an Azure account, a storage account, and an Azure network.

For VMWare VMs and Physical server replication to Azure, you will also need ASR component servers (configuration and process servers) and a master target server for failback. You will need to set up a Recovery Services vault in Azure, which is the storage entity that houses the data in Azure. In the vault, you can specify the replication target and source, set up the configuration server, add sources, define your replication policy, and perform other recovery tasks, such as test failovers and failbacks.

Similarly, for replicating Hyper-V VMs to Azure, if the hosts are configured in VMM clouds, you can register the VMM server(s) in the Recovery Services vault and install the Site Recovery Provider to orchestrate replication with Azure. If hosts are not located in VMM clouds, then you will install the Site Recovery Provider directly on the hosts.

Once the infrastructure is set up and the replication configured, protected on-premises machines will replicate an initial copy of the data, after which delta changes will be replicated. Traffic is then replicated over the secure internet connection or Azure ExpressRoute to Azure storage endpoints. For Azure VMWare VMs and Windows/Linux physical servers, this traffic can also be replicated over a site-to-site VPN connection.

For Azure VM replication, you will need to enable Azure VM replication in the Azure portal, after which resources are created automatically in the target region that you designate. Once replication is enabled, the Site Recovery Extension Mobility Service will be installed automatically on the Azure VM and will then be registered with Site Recovery, and then the VM will be configured for continuous replication. At this point, any data written to the Azure VM disks will continuously get transferred to the cache storage account in the source environment. Site Recovery will then process the data and send it to the target storage account in the target environment.

ASR also provides you with failover testing and the failover of protected resources to a target protection environment (Azure or a secondary site), as well as the ability to fail back to the source site. The following figure depicts the ASR data flow for the replicated data of protected resources from the source location to the target location in Azure or a secondary site:

Figure 1.4 Azure Site Recovery

There's more...

While some configuration is needed to enable some of the key OMS service offerings, such as the protection and recovery capabilities, as shown previously, the deployment requirements for onboarding to OMS are minimal because the underlying functionality is provided by services in Azure.

See also

The following is a useful link to information relating to OMS architecture and data flow:

• Microsoft TechNet - OMS architecture:
  https://docs.microsoft.com/en-us/azure/operations-management-suite/operations-management-suite-architecture