This chapter has been designed to provide an understanding of the underlying Virtual Machine Manager (VMM) modular architecture, which is useful to improve the implementation and troubleshooting VMM.

The first version of VMM was launched in far 2007 and was designed to manage virtual machines and to get the most efficient physical server utilizations. It has been dramatically grown from the basic tool to the one of the most advanced tool, with abilities to work even with different type of clouds.

The new VMM 2016 allows you to create and manage private clouds, retain the characteristics of public clouds by allowing tenants and delegated VMM administrators to perform functions, and abstract the underlying fabric to let them deploy the VM's applications and services. Although they have no visibility into the underlying hardware, there is a uniform resource pooling which allows you to add or remove capacity as your environment grows. Additionally, it supports the new Windows Server 2016 capabilities including software-defined storage, networks and shielded VMs (simply put, Software-Defined Datacenters (SDDC's)). VMM 2016 can manage private clouds across supported hypervisors, such as Hyper-V and VMware, which can be integrated with Azure public cloud services as well.

The main strategies and changes of VMM 2016 are as follows:

• Application focus: VMM abstracts fabric (hosts servers, storage, and networking) into a unified pool of resources. It also gives you the ability to deploy web applications and SQL Server profiles to configure customized database servers along with data-tier applications. However, virtual application deployment based on Server App-V, which was available in older versions of VMM, is no longer existing in VMM 2016. Although, if you upgrade VMM 2012 R2 to VMM 2016, your current service templates with Server App-V will continue to work with some limitations related to scale-out scenarios.
• Service deployment: One of the powerful features of VMM is its capability to deploy a service to a private cloud. These services are dependent on multiple VMs tied together (for example, web frontend servers, application servers, and backend database servers). They can be provisioned as simply as provisioning a VM, but all together.
• Dynamic optimization: This strategy will balance the workload in a cluster, while a feature called power optimization can turn off physical virtualization host servers when they are not needed. It can then turn them back on when the load increases. This process will automatically move VMs between hosts to balance the load. It also widens and replaces the VM Load Balancing feature that is available for Windows Server 2016 Failover Clusters.
• Software-Defined Datacenter: Network virtualization (software-defined networking or simply SDN) was introduced in VMM 2012 SP1 and quickly became popular due to a possibility to define and run multiple isolated networks on a single physical network fabric. It was based on NVGRE abstraction mechanism. VMM 2016 goes beyond and brings Azure's network model closer to your datacenter by introducing network controller as a central point, VXLAN for abstraction from the underlying physical network and integration with software load-balancers and gateways. In addition to SDN, Windows Server 2016 features like Storage Spaces Direct (S2D), Storage Replica, and Quality of Service (QoS) complement each other and are also supported by VMM 2016.
• Advanced Security: Modern data center requires protection for customer's sensitive data from hackers and even technical staff or other persons who can somehow access such data without your permission. To help protect against that problem, VMM supports managing and creating a new guarded fabric with a set of shielded VMs, guarded hosts and hosts with guardian services.
• Multivendor hypervisor support: If we compare the list of managed hypervisors in VMM 2012 R2 to VMM 2016, it's been cut. VMM 2016 now manages only Hyper-V and VMware, covering all of the major hypervisors on the market so far. Support for Citrix XenServer has been removed:

This is the first step. You need to do an assessment of your current environment to find out how and where the caveats are. You can use the Microsoft MAP toolkit (download it from http://www.microsoft.com/en-us/download/details.aspx?id=7826) or any other assessment tool to help you carry out a report assessment by querying the hardware, OS, application, and services. It is important to define what you can and need to address and, sometimes, what you cannot virtualize.

Currently, Microsoft supports the virtualization of all MS infrastructure technologies
(for example, SQL, Exchange, AD, Skype for Business, IIS, and File Server).

With the assessment report in hand, it is recommended that you spend a reasonable amount of time on the solution design and architecture, and you will have a solid and consistent implementation. The following figure highlights the new VMM 2016 features and others, which have been carried over from older versions, for you to take into consideration when working on your private cloud design:

In VMM, before deploying VMs and services to a private cloud, you need to set up the private cloud fabric.

There are three resources that are included in the fabric in VMM 2016:

• Servers: These contain virtualization hosts (Hyper-V and VMware servers) and groups, PXE, update servers (that is, WSUS), and other servers.
• Networking: This contains the network fabric and devices configuration (for example, gateways, virtual switches, network virtualization); it presents the wiring between resource repositories, running instances, VMs, and services.
• Storage: This contains the configuration for storage connectivity and management, simplifying storage complexities, and how storage is virtualized. For example, you can configure the SMI-S and SMP providers or a Windows 2016 SMB 3.0 file server.

If you are really serious about setting up a private cloud, you should carry out a virtualization assessment using MAP, as discussed above and work on a detailed design document covering hardware, hypervisor, fabric, and management. With this in mind, the implementation will be pretty straightforward.

System Center 2016 will help you install, configure, manage, and monitor your private
cloud from the fabric to the hypervisor and up to service deployment. It can also be integrated with public cloud services( for instance, Azure Site Recovery to protect and replicate your VMs to Azure public cloud).

System Center 2016 VMM has six components. It is important to understand the role of each component in order to have a better design and implementation.

For small deployments, test environments, or a proof of concept, you can install all of the components in one server, but as is best practice in production environments, you should consider separating the components.

Let's start by reviewing each component of VMM 2016 and understanding the role it plays:

• VMM console: This application connects to the VMM management server to allow you to manage VMM, to centrally view and manage physical and virtual resources (for example, hosts, VMs, services, the fabric, and library resources), and to carry out tasks on a daily basis, such as VM and services deployment, monitoring, and reporting.

By using the VMM console from your desktop, you will be able to manage your private cloud without needing to remotely connect it to the VMM management server.

• VMM management server: The management server is the core of VMM. It is the server on which the Virtual Machine Manager service runs to process commands and control communications with the VMM console, the database, the library server, and the hosts.

Think of VMM management server as the heart, which means that you need to design your computer resources accordingly to accommodate such an important service.

As is the best practice for medium and enterprise production environments, keep the VMM management server on a separate cluster from the production cluster, due to its crucial importance for your private cloud.

• Database: The database server runs SQL Server and contains all of the VMM data. It plays an important role when you have a clustered VMM deployment by keeping the shared data. The best practice is to also have the SQL database in a cluster or an availability group.

• VMM library: The VMM library servers are file shares, a catalog that stores resources, such as VM templates, virtual hard drive files, ISOs, scripts, and custom resources with a .cr extension, which will all be visible and indexed by VMM and then shared among application packages, tenants, and self-service users in private clouds.

The library has been enhanced to support services and the sharing of resources.
It is a store for drivers for Bare Metal deployments, SQL data-tier apps, (SQLDAC), and web deploy packages.

In a distributed environment, you can group equivalent sets of resources and make them available in different locations by using resource groups. You can also store a resource in a storage group that will allow you to reference that group in profiles and templates rather than in a specific virtual hard disk (VHD); this is especially important when you have multiple sites and VMM will automatically select the right resource from a single reference object. This essentially enables one template that can reference an object that can be obtained from multiple locations.

You can also have application profiles and SQL profiles (answer files for configuration of the application or SQL) to support the deployment of applications and databases to a VM after the base image is deployed. Application profiles can be web applications, SQL data-tier, or a general for deploying both application types and running any scripts.

• Self-service portal: The web-based self-service portal, was removed from SC 2012. In System Center 2012 SP1/R2, App Controller was being used as a replacement to the self-service portal, however, it was also finally removed in System Center 2016.

• VMM command shell: VMM is based on PowerShell. Everything you can do on GUI, you can do by using PowerShell. VMM PowerShell extensions make available the cmdlets that perform all of the functions in VMM 2016.

As you may have noticed, although VMM management is the core, each component is required in order to provide a better VMM experience. In addition to this, for a real-world deployment, you also need to consider implementing other System Center family components to complement your design. Every System Center component is designed to provide part of the private cloud solution. The Microsoft private cloud solution includes the implementation of VMM 2016 plus the following utilities:

• System Center 2016 Configuration Manager: This provides comprehensive configuration management for the Microsoft platform that can help users with the devices and applications they need to be productive while maintaining corporate compliance and control
• System Center 2016 Data Protection Manager: This provides unified data protection for the Windows and also VMware environment, delivering backup and restore scenarios from disk, tape, off-premise, and from the cloud
• System Center 2016 Endpoint Protection: This is built on the System Center Configuration Manager and provides threat detection of malware and exploits as part of a unified infrastructure for managing client security and compliance to simplify and improve endpoint protection
• System Center 2016 Operations Manager: This provides deep application diagnostics and infrastructure monitoring to ensure the predictable performance and availability of vital applications, and offers a comprehensive view of the datacenter, private cloud, and public clouds
• System Center 2016 Orchestrator: This provides the orchestration, integration, and automation of IT processes through the creation of runbooks to define and standardize best practices and improve operational efficiency
• System Center 2016 Service Manager: This provides flexible self-service experiences and standardized datacenter processes to integrate people, workflows, and knowledge across enterprise infrastructure and applications

When deploying System Center, there are some other systems and configurations you
need to consider. There are some old components that have also been described here in order to help you to understand your current infrastructure before, for instance, migration to the new VMM from older versions.

WAP is a free solution to manage resources that integrates with System Center and Windows Server to provide a customizable self-service portal for managing services such as websites, Virtual Machines, SQL or MySQL servers, and Service Bus; it also includes capabilities for automating and integrating additional custom services. For more info see http://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack/.

Service Provider Foundation (SPF) is provided with System Center Orchestrator, a component of System Center since 2012 SP1. Service Provider Foundation exposes an extensible OData web service that interacts with Virtual Machine Manager (VMM). It's main interface for communication between WAP, SCOM, and VMM.

Service Reporting, an optional component of System Center 2012 R2, enables IT (particularly hosting providers) to create detailed views, for each customer (tenant), of the virtual machine's consumption of the resources (CPU, memory, storage, and networking). For more info see http://technet.microsoft.com/en-us/library/dn251058.aspx.

Although the domain controller is not part of the System Center family and it is not a VMM component, it plays an important role in the deployment of a private cloud as VMM requires it to be installed on a domain environment.

WSUS plays an important role with reference to the private cloud as it is used to update the Hyper-V hosts, library servers, or any other role for compliance and remediation.

The App Controller provides a self-service experience through a web portal that can help you easily configure, deploy, and manage VMs and services across private, third-party hosters (that support Microsoft Hyper-V) and public clouds (Azure). For example, moving a VM from a private cloud to Azure, creating checkpoints, granting access, scaling out deployed services, and so on.

The App Controller has been used as a replacement of the VMM self-service portal since SC 2012 SP1. It was deprecated in the SC 2012 R2 time and finally removed in SC 2016. As noted above, you should plan Azure Pack deployment instead of current App Controller instance.

Azure Stack is a hybrid-cloud platform, bringing core public Azure services to your datacenter. These services are mostly dedicated to Azure PaaS and IaaS and help you out with building unified ecosystems between private and public clouds. Azure Stack is delivered as an integrated system, with software installed on the hardware built by partners like HPE and Cisco. Azure's familiar pay-as-you-go model is mainly being used in Azure Stack and you can stretch the same subscriptions out for both Azure and Azure Stack clouds. If you have unstable or restricted connection to Azure, you may choose to use Azure Stack in disconnected mode with a capacity model pricing package - a fixed fee annual subscription based on the number of physical cores. It's important to note that you can manage WAP VMs from Azure Stack using a special connector, though it's under review and not recommended for production use: https://aka.ms/wapconnectorazurestackdlc.

The following table will guide you through choosing which System Center component is necessary as per your deployment:

High availability is important when your business requires minimum or no downtime, and planning for it in advance is very important.

Based on what we learned about each component, we now need to plan the high availability (HA) for each VMM component.

Start by planning the HA for the core component, followed by every VMM component of your design. It is important to consider hardware and other System Center components, as well as the OS and software licenses.

When planning for highly available VMM management servers, you should first consider
where to place the VMM cluster. As per best practices, the recommendation is to install
the VMM cluster on a management cluster, preferably on some physical servers, if using converged network for your virtual network. However, if you plan to install highly available VMM management servers on the managed cluster, you need to take into consideration the following points:

• Only one highly available VMM management server is allowed per Failover Cluster.
• Despite the possibility to have a VMM management server installed on all cluster nodes, only one node can be active at a time.
• To perform a planned failover, use Failover Cluster Manager. The use of the VMM console is not supported.
• In a planned failover situation, ensure that there are no running tasks on the VMM management server, as it will fail during a failover operation and will not automatically restart after the failover operation.
• Any connection to a highly available VMM management server from the VMM console will be disconnected during a failover operation, reconnecting right after.
• The Failover Cluster must be running Windows Server 2016 in order to be supported.
• The highly available VMM management server must meet system requirements. For information about system requirements for VMM, see the Specifying the correct system requirements for a real-world scenario recipe in this chapter.
• In a highly available VMM management deployment, you will need a domain account to install and run the VMM management service. You are required to use distributed key management (DKM) to store the encryption keys in Active Directory.
• A dedicated and supported version of Microsoft SQL Server should be installed. For supported versions of SQL Server for the VMM database, see the Specifying the correct system requirements for a real-world scenario recipe.

The following sections are the considerations for SQL Server and the VMM library in an
HA environment.

In an enterprise deployment of VMM, it is recommended that you have a SQL Server cluster to support the HA VMM, preferably on a cluster separated from the VMM cluster. VMM 2016 supports SQL Server Always On Availability Groups. The following link will show you a good example of how to set it up: See the Configure SQL Server with AlwaysOn AGs recipe in Chapter 3, Installing VMM 2016.

As it is the best practice in an enterprise deployment, a highly available file server for hosting the VMM library shares is highly recommended as VMM does not provide a method for replicating files in the VMM library, and they need to be replicated outside of VMM.

As a suggestion, you can use the Microsoft Robocopy tool to replicate the VMM library files if you have distributed the library type.

When planning a VMM 2016 design for deployment, consider the different VMM roles, keeping in mind that VMM is part of the Microsoft private cloud solution. If you are considering a private cloud, you will need to integrate VMM with the other System
Center family components.

In VMM, you can add the hardware, guest operating system, SQL Server, and application profiles that will be used in a template to deploy virtual machines. These profiles are essentially answer files to configure the application or SQL during the setup.

You can create a private cloud by combining hosts, even from different hypervisors
(for example, Hyper-V and VMware), with networking, storage, and library resources.

To start deploying VMs and services, you first need to configure the fabric.

Create a spreadsheet with the server names and the IP settings, as seen in the following table, of every System Center component you plan to deploy. This will help you manage and integrate the solution:

The following rules need to be considered when planning a VMM 2016 deployment:

• The computer name cannot contain the character string SCVMM (for example,
  srv-scvmm-01) and cannot exceed 15 characters.
• Your VMM database must use a supported version of SQL Server to perform a VMM 2016 deployment. Express editions of Microsoft SQL Server are no longer supported for the VMM database. For more information, check the system requirements specified in the Specifying the correct system requirements for a real- world scenario recipe in this chapter.

• VMM 2016 does not support a library server on a computer that is running Windows Server 2012; it now requires Windows Server 2012 R2 as a minimum, but for consistency and standardization, I do recommend that you install it on a Windows Server 2016.
• VMM 2016 no longer supports creating and importing templates with the Server App-V packages. If you are upgrading from a previous version of VMM that has templates with such applications, you will continue to manage them with VMM, but you will not be able to upgrade the application.
• Hosts running the following versions of VMware ESXi and VMware vCenter Server are supported:
  • ESXi 5.1
  • ESXi 5.5
  • ESXi 6.0
  • vCenter 5.1
  • vCenter 5.5
  • vCenter 6.0
• Upgrading a previous version of VMM to a highly available VMM 2016 requires additional preparation. See Chapter 2, Upgrading from Previous Version of VMM, for this purpose.
• If you're planning for high availability of VMM 2016, be sure to install SQL Server on a cluster and on separate servers as it cannot physically be located on the same servers as your VMM 2016 management server. In addition, AlwaysOn availability groups can be used for the VMM database.
• The VMM management server must be a member of a domain. (This rule does not apply to the managed hosts, which can be on a workgroup.)
• The startup RAM for the VMM management server (if running on a VM with dynamic memory enabled) must be at least 2048 MB.
• VMM library does not support DFS Namespaces (DFSN) or DFS Replication (DFSR). This support is being planned.
• VMM does not support file servers configured with the case-insensitive option for Windows Services for Unix, as the network filesystem case control is set to ignore. Refer to the Windows Services for UNIX 2.0 NFS Case Control article available at http://go.microsoft.com/fwlink/p/?LinkId=102944 to learn more.
• The VMM console machine must be a member of a domain.

For a complete design solution, there are more items you need to consider.

VMM provides support for both Block level storage (Fibre Channel, iSCSI, and Serial Attached SCSI (SAS) connections) and File storage (on SMB 3.0 network shares, residing on a Windows file server or on a NAS device).

By using storage providers, VMM enables discovery, provisioning, classification, allocation, and decommissioning.

Storage classifications enable you to assign user-defined storage classifications to discovered storage pools for Quality of Service (QoS) or chargeback purposes.

In order to use this feature, you will need the SMI-S provider.

VMM 2016 can discover and communicate with SAN arrays through the Storage Management Initiative (SMI-S provider) and Storage Management Provider (SMP) provider.

If your storage is SMI-S compatible, you must install the storage provider on a separately available server (do not install on the VMM management server) and then add the provider to VMM management. Some devices come with built-in SMI-S provider and no extra are tasks required in that case. If your storage is SMP-compatible, it does not require a provider installation either.

CIM-XML is used by VMM to communicate with the underlying SMI-S providers since VMM never communicates with the SAN arrays themselves.

By using the storage provider to integrate with the storage, VMM can create LUNs (both GPT and MBR) and assign storage to hosts or clusters.

VMM 2016 also supports the SAN snapshot and clone feature, allowing you to duplicate a LUN through a SAN Copy-capable template to provide for new VMs, if you are hosting those in a Hyper-V platform. You will need to provision outside of VMM for any other VMs hosted with VMware hosts, for example.

This capability enables VMM 2016 to identify the hardware, install the operational system (OS), enable the Hyper-V or file server role, and add the machine to a target-host group with streamlined operations in an automated process.

PXE capability is required and is an integral component of the server pool. The target server will need to have a baseboard management controller (BMC) supporting one of the following management protocols:

• Data Center Management Interface (DCMI) 1.0
• Systems Management Architecture for Server Hardware (SMASH) 1.0
• Intelligent Platform Management Interface (IPMI) 1.5 or 2.0
• Custom protocols such as HPE Integrated Lights-Out (iLO) or Integrated Dell Remote Access (iDRAC)

Enterprise and hosting companies will benefit from the ability to provide new Hyper-V servers without having to install the operational system manually on each machine. By using BMC and integrating with Windows Deployment Services (WDS), VMM deploys the OS to designated hosts through the boot from the VHD(X) feature. The right BMC configuration presence is also a requirement for one of the most interesting features, called OS Rolling Upgrade, which will be discussed in detail later.

To ensure that users can perform only assigned actions on selected resources, create tenants, self-service users, delegated administrators, and read-only administrators in VMM using the VMM console, you will need to create Run As accounts to provide necessary credentials for performing operations in VMM ( example, for adding hosts).

Run As accounts are very useful additions to enterprise environments. These accounts are used to store credentials that allow you to delegate tasks to other administrators and self-service users without exposing sensitive credentials.

There are several different categories of Run As accounts:

• Host computer: This is used to provide access to Hyper-V and VMware ESXi hosts
• BMC: This is used to communicate with BMC on the host computer,
  for out-of-band management or power optimization
• Network device: This is used to connect to network load balancers
• Profile: This is to be used for service creation in the OS and application profiles as well as SQL and host profiles
• External: This is to be used for external systems such as System Center
  Operations Manager

Only administrators or delegated administrators can create and manage Run As accounts.

When designing the VMM implementation, you need to plan which ports you are going to use for communication and file transfers between VMM components. Based on the chosen ports, you will also need to configure your host and external firewalls. See the Configuring ports and protocols on the host firewall for each SCVMM component recipe in Chapter 3, Installing VMM 2016.

The recommendation is to create a big CSV volume. CSV spreads across multiple disk spindles and it will give great storage performance for VMs, as opposed to creating volumes based on the VHD purpose (for example, OS, data, and logs).

VMM 2016 supports management up to 1000 physical hosts and 25000 VMs. Therefore, the best practice is to have a separate management cluster with running VMM components to manage the production, test, and development clusters.

In addition to this, although you can virtualize the domain controllers with Windows 2016, it is not the best practice to have all the domain controllers running on the management clusters, as the cluster and System Center components highly depend on the domain controllers. If it's possible, place one or more DCs on the physical hosts or VMs in the location or fault domains different from the management cluster.

The following figure shows a two-node hyper-converged management cluster, with System Center 2016 components installed in separate VMs to manage the production cluster. All hosts are running on Windows Server 2016 with enabled Storage Spaces Direct to provide hyper-converged solutions which help to maximize the server's efficiency and reduce overall costs:

In a small environment, you can have all the VMM components located on the same server. A small business may or may not have high availability in place, as VMM 2016 is now a critical component for your private cloud deployment.

Start by selecting the VMM server's location, which could be a physical server or a
virtual machine.

You can install SQL Server on the VMM server as well, but as VMM 2016 does not support SQL Express editions, you will need to install SQL Server first and then proceed with the VMM installation.

If you are managing more than 10 hosts in the production environment, my recommendation would be to have SQL Server running on a separate machine.

It is important to understand that when deploying VMM in production environments (real-world scenarios), the business will require a reliable system that it can trust.

The following figure illustrates a real-world deployment where all VMM 2016 components are installed on the same VM and SQL is running on a separate VM.

I would recommend up to 50 hosts in a lab environment with SQL Server and all VMM components installed on a single VM. It will work well, but I would not recommend this installation in a production environment.

In a medium-scale or large-scale environment, the best practice is to split the roles across multiple servers or virtual machines. By splitting the components, you can scale out and introduce high availability to the System Center environment.

In the following design, you can see each component and what role it performs in the System Center Virtual Machine Manager environment:

When designing an enterprise private cloud infrastructure, you should take into consideration some key factors such as business requirements, company policies, applications, services, workloads, current hardware, network infrastructure, storage, security, and users.

Following is a sample of a real-world infrastructure that can support up to 3000 VMs and 64 server nodes running Windows 2016 Hyper-V.

The number of VMs you can run on an implementation like this will depend on some key factors. Do not take the following configuration as a mirror for your deployment, but as a starting point. My recommendation is to start understanding the environment, then run a capacity planner such as a MAP toolkit. It will help you gather information that you can use to design your private cloud.

I am assuming a ratio of 50 VMs per node cluster with 3 GB of RAM, configured to use Dynamic Memory (DM):

• Servers
  • 64 servers (4 clusters x 16 nodes)
  • Dual processor, 6 cores: 12 cores in total
  • 192 GB RAM
  • 2 x 146 GB local HDD (ideally SDD) in Raid 1
• Storage
  • Switch and host redundancy
  • Fiber channel or iSCSI or S2D (converged)
  • Array with capacity to support customer workloads
  • Switch with connectivity for all hosts.
• Network
  • A switch with switch redundancy and sufficient port density and connectivity to all hosts.
  • It provides support for VLAN tagging and trunking.
  • NIC Team and VLAN are recommended for better network availability, security, and performance achievement.
• Storage connectivity
  • If it uses a fiber channel: 2 (two) x 4 GB HBAs
  • If it uses ISCSI: 2 (two) x dedicated NICs (recommended 10 GbE)
  • If it uses S2D: 2 (two) x dedicated 10Gbps NICs (recommended RDMA-capable adapters)
• Network connectivity
  • If it maintains a 1 GbE connectivity: 6 dedicated 1 GbE (live migration, CSV, management, virtual machines' traffic)
  • If it maintains a 10 GbE connectivity: 3 dedicated NICs 10 GbE
    (live migration, CSV, management, virtual machines' traffic)

System Center 2012 SP1 VMM introduced multi-tenancy. This is one of the most important features for hosting companies as they only need to install a single copy of System Center VMM, and then centralize their customer management, each one running in a controlled environment in their own domain. Hosters always want to maximize their compute capacity and VLAN segment hardware so you can't maximize its capacity. Network virtualization moves the isolation up to the software stack, enabling the hoster to maximize all capacity and isolate customers via software-defined networking
VMM 2012 R2 takes advantage of Windows Server 2012 R2 features, VMM 2012 R2 delivers Site-to-Site NVGRE gateway for Hyper-V network virtualization. This capability enables you to use network virtualization to support multiple Site-to-Site tunnels and direct access through a NAT Firewall. The networking virtualization (NV) uses NVGRE protocol, allowing network load balancers to act as NV gateways. Plus, switch extensions can make use of NV policies to interpret the IP information in packets being sent and communication between, for example, Cisco switches and VMM 2012 R2.

VMM 2016 and Windows Server 2016 continue to improve Hyper-V Network-Virtualization (HNV) and helps you move to an efficient SDDC solution. VMM 2016 introduces flexible encapsulation which supports both NVGRE (HNVv1) and new VXLAN (HNVv2) to create overlay networks in which original packets from VMs with its MACs, IPs and other data (Customer Address network) are placed inside an IP packet on the underlying physical network (Provider Address network) for further transportation. VXLAN is the default in VMM 2016 and works in MAC distribution mode. It uses a new Network Controller (NC) as a central management point that communicates with Hyper-V hosts and pushes network policies down to NC host agents running on each host. In short, NC is responsible for the address mapping, and the host agents maintain the mapping database. NC also integrates with Software-Load Balancer (L3 and L4), network layer datacenter firewall and RAS gateways which are also included in Windows Server 2016. Consequently, NC is the heart of SDN in VMM 2016 and should always be considered in a cluster configuration.

There is also a new way of deploying converged networking that has been introduced in Windows Server 2016 and VMM 2016 to ease and improve SDN deployment. Switch-Embedded Teaming (SET) allows you to group up to eight identical adapters into one or more software-based virtual adapters. Prior to VMM 2016 you needed to have two different sets of adapters: one to use with traditional teaming and the one to use with RDMA because of its incompatibility with teaming and virtual switch. SET eliminates this limitation and supports RDMA convergence as well as QoS, RSS, VMQ, and both versions of HNV noted earlier. Furthermore, creating a general Hyper-V virtual switch with RDMA NICs would be also supported:

When we discussed possible architectures for management clusters, I referred to a new feature in Windows Server 2016 and VMM 2016, Storage Spaces Direct (S2D). S2D uses industry-standard servers with local storage which could be direct-attached enclosures or internal disks. S2D provides similar shared storage pools across cluster nodes by leveraging Cluster Shared Volume, Storage Spaces, Failover Clustering and SMB3 protocol for file access (SOFS). Hyper-converged and converged solutions can now be based on software-defined storage running on Windows Server 2016. So, you have a choice: to buy external enterprise SAN or to use S2D. If your goal is a software-defined datacenter, the answer to all questions is very clear - S2D and SDN implementation. The main competitor to S2D is a well-known VMware Virtual SAN (vSAN) that was first released in vSphere 5.5 and is still present in the newest vSphere 6.6. S2D, just like a vSAN, has special licensing requirements.

Furthermore, improved Storage QoS in VMM 2016 provides a way to centrally monitor and manage storage performance for virtual machines residing on S2D or another device. Storage QoS was first introduced in 2012 R2 version. You could set maximum and minimum IOPS thresholds for virtual hard disks (excluding shared virtual hard disks). It worked well on standalone Hyper-V hosts, but if you have a cluster with a lot of virtual machines or even tenants, it could be complicated to achieve the right QoS for all cluster resources. The feature automatically improves storage resource fairness between multiple virtual machines using the same file server cluster. In other words, QoS for storage will be distributed between a group of virtual machines and virtual hard disks:

Another feature available only in Windows Server 2016 Datacenter edition is Storage Replica (SR). Previously, we needed to find third-party solutions for SAN-to-SAN replication. And building stretched clusters required a huge amount of money. Windows Server 2016 and VMM 2016 can help to significantly reduce costs and enhance unification in such scenarios. SR is the main component of multi-site clusters or disaster recovery solutions supporting both asynchronous and synchronous replication between any storage devices, including Storage Spaces Direct. Also, you are not required to have identical devices on both sides. However, at the time of writing, only synchronous replication is supported in VMM fabric, and deployment is limited to PowerShell.

For more information, see the following references:

• The Planning for high availability recipe
• The Nested Virtualization in Windows Server 2016 page at https://rlevchenko.com/2015/11/26/nested-virtualization-in-windows-server-2016/
• The Configuring ports and protocols on the host firewall for each VMM component recipe in Chapter 3, Installing VMM 2016
• The Deploying hyper-converged cluster with S2D and Hyper-V recipe in Chapter 5, Configuring Fabric Resources in VMM
• Rapid Provisioning of Virtual Machines Using SAN Copy Overview: http://technet.microsoft.com/en-us/library/gg610594.aspx
• For more on Storage Management Initiative (SMI-S), refer to the following link: http://www.snia.org/ctp/conformingproviders/index.html
• For more information on SDN, visit the following link: https://docs.microsoft.com/en-us/windows-server/networking/sdn/software-defined-networking
• VMM templates for SDN deployment: https://github.com/Microsoft/SDN/tree/master/VMM/Templates

In a real-world production environment, you need to specify a system according to the design and business requirements.

When specifying the hardware for your private cloud deployment, take into consideration future growth needs. It is also important to apply the latest OS and software updates.

Use the following tables to carry out an extensive documentation of the hardware and software requirements for your deployment.

Create a document that outlines every solution component, describing the system requirements, before starting to implement.

The following table shows the supported OS and servers for SC 2016:

Following are the hardware requirements to consider when specifying your VMM environment. Although for SMB, POC or demo scenarios you can have SQL installed on the VMM management server, the recommendation is to have SQL Server installed on another server. And you also won't run SQL and Library Servers on the VMM server if you want to manage more than 150 hosts.

Following are the hardware requirements for VMM management server:

Following are the hardware requirements for VMM database server:

Following are the hardware requirements for VMM library server.

The minimum and recommended requirements for a VMM library server will be determined by the quantity and size of the files that will be stored:

Following are the hardware requirements for the VMM console:

Following are the hardware requirements for the Windows Azure Pack.

Before the WAP installation, you need to consider which type of deployment and components you really need. In an express deployment, all core components are installed on the same machine. This type of deployment is recommended for demo and POC scenarios. Distributed deployment is when WAP portals and databases are running on dedicated virtual machines (up to 8, except for VMs for optional resource providers like VM Clouds or Web Sites):

Following are the requirements for VMM management server for SC 2016:

The following table shows the requirements for the VMM console:

Following are the requirements for core WAP components:

For more information, see the following references:

• Deploying Windows Azure Pack for cloud management recipe in Chapter 9, Managing Clouds, Fabric Updates, Resources, Clusters and new Features of 2016
• How to install Windows Azure Pack: https://rlevchenko.com/2014/11/12/step-by-step-installation-of-windows-azure-pack/

• Service Provider Foundation installation guide at https://rlevchenko.com/2014/10/29/step-by-step-installation-of-service-provider-foundation-2012-r2/

• Integrating VMM with WAP page at https://rlevchenko.com/2015/05/22/windows-azure-pack-how-to-add-and-troubleshoot-vm-clouds-2/

System Center 2016 is licensed with two versions, Standard and Datacenter. As with System Center 2012 R2, the same capabilities across editions are differentiated only by virtualization rights. All System Center components are included in these two editions. The main difference between SC 2012 R2 and SC 2016 is the licensing model that has been moved from CPU-based to core-based in order to simplify licensing across multi-cloud infrastructures.

The license is required only to manage endpoints. If you have existing software-assurance (SA) subscription, you can move to the new SC 2016 at any time. SC 2-processor licenses with active SA will be exchanged for a minimum of 8 two-core pack licenses (16 cores) or the actual number of physical cores in use on the server under management.

As part of the private cloud design solution, you need to define which license you will need, based on your solution design and business requirements.

License summary for System Center 2016:

• Core-Based licensing: Licensing is based on the number of physical cores on the servers under management, consistent with the Windows Server 2016 model. You need to license all physical cores in the server being managed. Minimum of 8 cores licenses is required for each processor and minimum of 16 cores required for each server. If you have, for example, even one 4-core CPU in server, it would be required to buy eight 2-core packs to license that server. The price of eight two-core packs will be the same as 2-processor licenses for SC 2012 R2.
• Consistent licensing model across editions: Core-based licenses for server management. User-based or operating system environment (OSE)-based license for client management.
• For endpoints being managed: No additional licenses are needed for management servers or SQL Server technology used in the System Center:

This recipe will take you through the process of troubleshooting VMM and its supporting technologies for a successful VMM deployment.

Having an understanding of the core technologies that VMM depends on to work correctly is the initial step to troubleshooting VMM:

• WS Management (WinRM)
• WMI
• BITS
• DCOM
• WCF

Troubleshooting is never an easy task, but VMM 2016 provides tools and ways to help you find and remediate an issue.

Following are some techniques you can use to troubleshoot:

• Event logs

A good starting point is to look at the event logs. Look for OS- and VMM-related errors or failures. A problem with the operating system (OS) or one of its core services could result or lead to a problem in VMM.

For example, if you are running SQL Server on the same server and it did not start, VMM management service will not start either and VMM operations will fail as a direct result of this. You can easily find this by looking for errors in the system or application logs, errors that would indicate, in this example, that the service is not running (for this example, you can also check Services.msc).

• VM manager log

When looking for VMM errors, it is recommended that you to look at the VMM log as well. To do so, perform the following steps on the VMM running Windows Server 2016:

1. 1. On the Server Manager window, click on Tools
   2. Select Event Viewer, expand Applications and Services logs, then go to the Microsoft and select the VirtualMachineManager log

• VMM installation-related troubleshooting logs

VMM records information about the VMM agent installation. However, if the installation logging is not sufficient to determine the cause of failure, you can enable tracing by using the VMM MPS Reports tool and then restart the installation.

• VMM server setup logging

Installation logs are written, by default, to the C:\ProgramData\VMMLogs
hidden folder

• VMM agent installation logging

When installing an MSI package, such as installing the VMM agent manually, you can enable logging using the following syntax:

        msiexec /I MSIPackageName.msi  /L*V path\logfilename.log 

For example, using the syntax, we can come up with something like the
following command:

        msiexec /I "C:\setup\vmmAgent.msi" /L*V vmmagent.log 

The local agent installation information is logged in the C:\ProgramData\VMMLogs hidden folders.

Look for the logfile vmmAgent.msi_m-d-yyy_hh-mm-dss.log.

• Troubleshooting WinRM

To check if WinRM has remote access, check if:

• • The SID in RootSDDL maps to the VMM Servers local group on each Hyper-V host
  • The local group contains the account that VMM management service runs as a service

A good understanding of what a successful installation log contains from a POC or a pilot environment is important to identify possible issues, especially if it appears when deploying VMM on a production environment, as you can then compare both logs.

Run the following command on the Hyper-V host:

winrm id 

This should produce an output similar to the following output:

IdentifyResponse
ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
ProductVendor = Microsoft Corporation
ProductVersion = OS: 10.0.14393 SP: 0.0 Stack: 3.0
SecurityProfiles
SecurityProfileName = http://schemas.dmtf.org/wbem/wsman/1/wsman/secprofile/http/spnego-kerberos

If the result shows an error, run the following command for a quick configuration of WinRM:

winrm qc 

If prompted, answer Yes. You will receive a response like the following:

WinRM already is set up to receive requests on this machine. 
WinRM is not set up to allow remote access to this machine for management. 
The following changes must be made: 
Enable the WinRM firewall exception 
Make these changes [y/n]? 
WinRM has been updated for remote management. 
WinRM firewall exception enabled. 
WinRM can now be tested again by typing 'winrm id' as before 

Now check the listener:

winrm enum winrm/config/listener 

Run the following command on the VMM management server:

winrm id -r:http://HyperVHost.yourdomain.local:5985 -u:YOURDOMAIN\AdminUser 

The result will be similar to the following:

IdentifyResponse
ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
ProductVendor = Microsoft Corporation
ProductVersion = OS: 10.0.14393 SP: 0.0 Stack: 3.0

Otherwise you will receive the following error:

Error number: -2144108526 0x80338012 
The Client cannot connect to the destination specified in the request 

This could indicate communication issues, so check your network, host firewall, and connectivity.

Most WinRM-related events appear in the system or application event logs. The Service Control Manager often contains the error, as the WinRM service has terminated or restarted for some reason.

To avoid this scenario, conduct the following checks:

• Make sure you installed all of the prerequisites
• Check the firewall rules and make sure the ports are configured correctly
• Open the command prompt (Run as Administrator) and type the following command:

      winrm qc -q 
      winrm set winrm/config/service/auth @{CredSSP="True"} 
      winrm set winrm/config/winrs @{AllowRemoteShellAccess="True"} 
      winrm set winrm/config/winrs @{MaxMemoryPerShellMB="2048"} 

You can check if the WinRM can communicate with OS WMI providers by running the
following command:

winrm enum wmi/root/cimv2/Win32_ComputerSystem -r :http://servername:5985 [-u:YOURDOMAIN\AdminUser] 

By running the following command, you can check if the WinRM can communicate with Hyper-V WMI providers:

winrm enum wmi/root/virtualization/v2/msvm_computersystem  
  -r:http://servername:5985 [-u:YOURDOMAIN\AdminUser] 

Also, to check if the WinRM can communicate with the VMM agent WMI provider, run the following command:

winrm invoke GetVersion wmi/root/scvmm/AgentManagement -r:servername [-u:YOURDOMAIN\AdminUser] @{} 
 

Following are the troubleshooting tools available for use:

• Windows Management Instrumentation Tester (wbemtest.exe)
  • The wbemtest.exe gives you the ability to query WMI namespaces on local or remote servers.
  • Connecting to a namespace locally indicates that it is properly registered and accessible via the WMI service. By connecting to a remote server additionally, it also indicates that WMI connectivity between the two machines is working.
• WMI Service Control Utility
  • This tool configures and controls the WMI service, allowing namespace permissions to be modified.
  • To open this tool, in the command prompt type the following:

              wmimgmt.msc

• • Then perform the following steps:
    1. Right-click on WMI Control (Local)
    2. Select Properties
    3. Click on the Security tab and then select Root
    4. Click on the security button to check the permissions

BITS transfers files between machines, providing information about the operation's progress. The transfer can be asynchronous.

In VMM, BITS is used for encrypted data transfer between managed computers. Encryption is done by using a self-signed certificate generated when the Hyper-V host is added to VMM.

You can use BITSadmin to verify that BITS is working properly outside of VMM.

BITSadmin is available in Windows Server. Some useful examples of BITSadmin are described at: http://msdn.microsoft.com/en-us/library/aa362812(VS.85).aspx.

The following tools are used to collect data surrounding VMM issues:

• VMM tracing tools: VMM tracing tools provide the ability to manage, collect, and view various traces and diagnostic information in a VMM environment:
• Gathering trace information: When you face an issue and need to report it to Microsoft, you can gather the trace by performing the following steps:
  1. In the VMM server, open the command prompt with administrative rights and type the following command:

logman create trace VMMDebug -v mmddhhmm -o
$env:SystemDrive\VMMlogs\DebugTrace_$env:computername.ETL 
-cnf 01:00:00 -p Microsoft- VirtualMachineManager-Debug                           
-nb 10 250 -bs16 -max 512

• 2. Start the trace collection by executing the following command:

logman start VMMDebug 

1. 3. Next, try to reproduce the issue, and at the end stop the trace collection by executing the following command:

              logman stop VMMDebug  

1. 4. Send the ETL file located in %SystemDrive%\VMMlogs\DebugTrace_%computername%.ETL to Microsoft.
   5. Delete the debug information by executing the following command:

              logman delete VMMDebug

After gathering the trace, you can use netsh trace convert command on the traces. This command converts the ETL binary trace logs into TXT files that can be viewed in any text editor.
To convert the ETL file:

1. Run CMD or PowerShell with administrative privileges.
2. Start conversions by executing the following command:

      nets  trace convert C:\VMMlogs\yourfilename.etl

You will receive a response like the following:

       Input file: C:\VMMlogs\VMMLog_VMM01_09242130.etl
       Dump file: C:\VMMlogs\VMMLog_VMM01_09242130.txt
       Dump format: TXT
       Report file: -
       Generating dump ... done

3. Open the generated TXT-file in Notepad by using the following command:

      notepad c:\VMMlogs\yourfilename.txt

• Configuring ports and protocols on the host firewall for each SCVMM component recipe in Chapter 3, Installing VMM 2016