Troubleshooting vSphere Storage: All vSphere administrators will benefit big-time from this book because it gives you clear, practical instructions on troubleshooting a whole host of storage problems. From fundamental to advanced techniques, it's all here.

ESXi presents its storage to a VM using host-level storage virtualization techniques which essentially provide an abstraction layer between the actual physical storage, whether that is attached via a Storage Area Network (SAN), an Ethernet network or locally installed, and the virtual machines consuming the storage. This abstraction layer consists of many different components all working together to simulate that of a physical disk inside a virtual machine.

When a virtual machine is created, it will normally have at least one virtual disk assigned to it. When a virtual disk is assigned to a VM, a piece of virtual hardware called a virtual storage adapter is created in order to facilitate the communication between the VM and its underlying virtual hard disk (vmdk). The type of virtual storage adapter that is used greatly depends on the Guest Operating System setting that has been chosen for that specific VM (see the following table). This newly created SCSI adapter provides the interface between the OS and the VMkernel module on the ESXi host. The VMkernel module then locates the target file within the volume, maps the blocks from the virtual disk to the physical device, forwards the request through the Pluggable Storage Architecture, and finally queues the appropriate adapter on the ESXi host depending on the type of storage present (iSCSI NIC/Hardware Initiator, Fibre Channel Host Bus Adapters (FC HBA), NFS – NIC, or Fibre Channel over Ethernet (FCoE NIC/CNA)).

The following table outlines the various virtual SCSI adapters available:

VMware ESXi supports a couple of different filesystems to use as virtual machine storage; Virtual Machine File System (VMFS) and Network File System (NFS).

In order to begin troubleshooting vSphere Storage, we need to be aware of how vSphere identifies and names the storage devices, LUNs, and paths available to our hosts. During the process of troubleshooting of vSphere Storage, there are a lot of situations where we need to provide the identifier of a storage device or path in order to obtain more information about the issue. Due to the uniqueness of these identifiers, ESXi will often use them when logging issues to syslog.

Viewing device identifiers

We are able to view device identifiers in a couple of different places; within the vSphere Client and within the ESXi Shell. Let us have a look at each in turn.

Within the vSphere Client

We can view the device identifiers within the vSphere Client by performing the following steps:

1. Click on the Configuration tab of the host whose storage you wish to view.

2. Click on the Storage section under Hardware.

3. Switch to the Devices view and right-click on the header bar to add and remove desired columns if needed.

   

   Device identifiers from the vSphere Client

Within ESXi Shell

The following command will give us similar information as to what we see in the vSphere Client and should return similar information to that of the following screenshot:

esxcfg-scsidevs –c

Device identifiers from within the vSphere CLI

The vSphere Pluggable Storage Architecture is essentially a collection of plugins that reside inside the VMkernel layer of an ESXi host. The following figure shows a graphical representation of all the components of the PSA. The top-level plugin in the PSA is the Multipathing Plugin (MPP). The MPP defines how vSphere will manage and access storage including load balancing, path selection, and failover. The MPP itself can be provided by the storage vendor (IE EMC PowerPath) or you may also use the VMware provided Native Multipathing Plugin (NMP).

So essentially, the VMware provided NMP is in itself a MPP. The NMP is loaded by default for all storage devices, however, it can be overridden and replaced by installing a third-party MPP. Within each MPP, including the VMware NMP are two subplugins; Storage Array Type Plugin (SATP) and Path Selection Plugin (PSP). The SATP handles the details about path failover, whereas the PSP handles the details around load balancing and which physical path to use to issue an I/O request.

The VMware Pluggable Storage Architecture

Confused yet? I know the PSA is a lot to take in but it is essential to understand when you are troubleshooting storage issues. Let's have a look at each individual plugin included in the default VMware NMP in a little more detail to better understand the role it plays.

esxcli storage core plugin list

esxcli storage nmp device list

Storage Array Type Plugin roles and commands

The SATP plugin, which is a subplugin of the overall MPP, performs the following functions:

• Monitors the state of paths to the physical storage system

• Determines when a physical path is to be declared failed or down

• Handles the switching of physical paths after a path failure has occurred

• VMware provides a number of SATP plugins depending on which supported storage array is being used and also some generic active-active/active-passive SATP's for unknown storage arrays

To list the currently loaded SATP plugins along with their default PSP information, run the storage nmp namespace with esxcli.

esxcli storage nmp satp list

To change the default PSP associated with a given SATP, you can use the esxcli storage nmp satp set –b <boottime> -P <Default PSP> -s <SATP> command similar to the one shown in following screenshot:

Associate a default PSP with a SATP via esxcli

Path Selection Plugin roles and commands

The PSP, which is a subplugin of the overall MPP, provides the PSA with the following functionality:

• Responsible for choosing a path to issue an I/O request.

• Differs from SATP in terms that the PSP is a load balancing mechanism and deals with only active paths. The SATP determines which paths are active/standby/failed.

• VMware provides three default PSP plugins; Fixed, Most Recently Used, and Round Robin.

• The VMware NMP will select a default PSP based on which SATP plugin has been loaded for the storage array.

To list all of the available PSP plugins, you can use the storage nmp psp namespace of esxcli as shown:

esxcli storage nmp psp list

More information in regards to each of the default policies that VMware provides is listed in the following table:

Policy	Explanation	Use
VMW_PSP_FIXED (Fixed)	Host uses a designated preferred path if configured; otherwise it uses the first available path at boot time. The host will failover to other paths if preferred path is down and will return to the initial preferred path when connection is restored	Default policy for most active-active arrays.
VMW_PSP_MRU (Most Recently Used)	Host will select the path that was used most recently. Upon failover, the host will move to another path. When the connection is restored, it will not revert back to the initial path.	Default policy for most active-passive arrays.
VMW_PSP_RR (Round Robin)	The host will cycle IOPs through all active paths on active-passive arrays and all paths on active-active arrays.	Default for a number of active-active and active-passive arrays.

To list certain configuration of the different PSP's on certain devices, you can use the esxcli storage nmp psp <PSP Namespace> deviceconfig get –d <device identifier command similar to the one shown in the following screenshot. On the flip side, you can set certain parameters by replacing get with set.

Retrieving configuration from a devices PSP via esxcli

Although we have used the ESXi Shell to obtain all of the information mentioned previously, we should note that it is possible to retrieve and change some of the information from within the vSphere Client as well. Most of these operations are done in the Storage section of the Configuration tab of a host.

Now that we have a general understanding of how ESXi presents storage to a virtual machine and handles load balancing and failover, let's have a look at an I/O request from start to finish. The following figure shows a graphical representation of the following steps:

We should now have a basic understanding of the common storage concepts, terms, methodologies, and transports that vSphere uses. We discussed the various types of storage that vSphere supports, including VMFS and NFS along with the many transports it utilizes to access these such as Fibre Channel and iSCSI. We went over how to identify storage LUNS, devices, and paths and have a basic understanding of the Pluggable Storage Architecture and how data flows from a virtualized application right through to its underlying storage.

In the next chapter, we will look at developing a proper troubleshooting methodology as well as review some of the most commonly used tools that can help us when troubleshooting vSphere storage.