Copying Data by Using AzCopy

AzCopy is a utility that can be used for copying files to and from Azure Storage accounts through a command-line-based utility. Authentication for the tool can be conducted using either an Active Directory account or a SAS token from storage. The tool provides many other functions but is primarily designed for file copying. It can also be used for monitoring and managing various other copy jobs. Some of the other features are its delete capability, its sync functionality for replicating storage from one location to another, and even the ability to make containers and file shares. A couple of use cases for AzCopy are replicating data from one storage account to another as a means of backup and migrating from one environment to another (such as development to production). You might also need to process data from the storage account on your local machine or server; AzCopy will enable you to copy files for local processing.

The AzCopy commands are structured in the following format: azcopy [command] [source] [destination] [flags].

In the activity that follows, you will learn how to install the AzCopy tool, and then in the subsequent exercises, you will learn how to copy data using it.

Downloading and Installing

You can download AzCopy from here: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10.

In this exercise, you will copy data to your Azure Blob Storage using a SAS token:

1. Download and install the relevant AzCopy installer from the preceding link.
2. Extract the files from the archive you downloaded and place them in a location that suits you, such as C:\AzCopy.
3. Launch PowerShell, navigate to the folder using the cd command, and press Enter.
   

Figure 9.1: Changing directory in PowerShell

You now have a copy of AzCopy on your machine ready to work with.

Copying Data by Using AzCopy

In this demonstration, you will copy data using the AzCopy utility and SAS tokens. This exercise can also be conducted using Entra ID credentials. Follow these steps to complete the exercise:

1. Identify a file you would like to copy to the Azure storage account and note the path. For simplicity, place it in the same path as AzCopy.
2. Navigate to the Azure portal by opening https://portal.azure.com.
3. Select a storage account and create two containers in the storage account, one named azcopysource and the other named azcopydestination. These can be named anything you like in later implementations.
4. On the left menu for the storage account, click Shared access signature under the Security + networking context. Create a new SAS key by selecting the Container option from the Allowed resource types list.

Figure 9.2: SAS resource type

5. Set Allowed permissions to enabled (ticked). Some of the permissions you may not be familiar with are Immutable storage (data storage that imposes a restriction whereby once data is written, no alterations or deletions to the data can be made within a predefined period of time) and Permanent delete (whereby you can delete a snapshot or blob version permanently, even before the retention period’s predefined end date) options. Immutable storage is typically instituted for governance and compliance reasons, as well as data security. The Permanent delete option is a high-risk permission and should be handled with care to prevent accidental permanent deletion.
6. Your allowed permissions should match those in the following figure:

Figure 9.3: SAS allowed permissions

7. Set the time for Start to 5 minutes before your current time and click Generate SAS and connection string. Copy the SAS token – remember that you are only ever presented with this once.
8. Copy the filename for the file identified in Step 1 and insert this name in the following script. The following script will copy the file you enter in SourceFilePath to the blob container you specified with StorageAccountName and ContainerName:

   # Change all Variables Below
   $SourceFilePath = "C:\AzCopy\MyFile.txt"
   $StorageAccountName = "az104storageaccountdemo1"
   $ContainerName = "azcopydestination"
   $SASToken = "?sv=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%3D"
   # Run AzCopy Command
   ./azcopy.exe copy "$SourceFilePath" "https:  //$StorageAccountName.blob.core.windows.net/$($ContainerName)$SASToken"

9. The script can either be saved as a PowerShell script file (*.ps1) and called in PowerShell, or you can copy and paste your edited script code into PowerShell and press Enter for it to run.

Now that you have seen AzCopy in action, you will complete the same task of copying files from a source container on a storage account to a destination container in the same storage account.

Copying Data between Containers Using AzCopy

You will now perform a similar copy task to the previous section, except this time, you will be copying data from a source container in a storage account to a destination container in the same storage account. Note that this technique can also be used across storage accounts as the principle is the same. Follow these steps:

1. Navigate to the Azure portal by opening https://portal.azure.com.
2. Select a storage account and create two containers on the storage account, one named azcopysource and the other named azcopydestination. These can be any name you choose should you want to implement this again later for other environments; just remember to update these names in your copy script.
3. Upload a file to the azcopysource container.
4. In the left menu for the storage account, click Containers under the Data storage context, then click on the azcopysource container. Click Shared access tokens under the Settings context in the left menu. Create a new container-level SAS by setting Signing method to Account key, Signing key to Key 1 or Key 2, Permissions to Read and List, and Start to 5 minutes before your current time. Then, click Generate SAS token and URL.
5. Copy the SAS token – remember you are only ever presented with this once. Perform the same operation for the destination container. This time, set the Shared access tokens permissions to Read, Add, Create, Write, Delete, and List.
6. The following script will copy the files from the source container, azcopysource, to the destination container, azcopydestination. Note the extra switches (flags) used by the following script. –overwrite=ifsourcenewer performs the operation of overwriting files on the destination if the source files are new. The --recursive flag recursively copies data from the source container and subsequent folders in any filesystem you copy from, essentially copying all the files and folders it finds:

   # Change all Variables Below
   $StorageAccountName = "az104storageaccountdemo1"
   $SrcContainerName = "azcopysource"
   $DestContainerName = "azcopydestination"
   $SourceSASToken = "sp=rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%3D"
   $DestSASToken = "sp=rxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx%3D"
   # Run AzCopy Command
   ./azcopy.exe copy "https://$StorageAccountName.blob.core.windows.net/$($SrcContainerName)?$SourceSASToken" "https://$StorageAccountName.blob.core.windows.net/$($DestContainerName)?$DestSASToken" --overwrite=ifsourcenewer --recursive

7. After running the preceding script, you will notice text like this appearing, indicating that a file copy operation has been completed:

Figure 9.4: AzCopy Script Run

You have just learned how to copy data between containers using AzCopy, which brings this section to an end, where you have learned what AzCopy is, how to download it, how it works, and also how to copy data between different containers. In the next section, you will learn about storage replication and lifecycle management.

Note

You are encouraged to read up further on the topic by using the following links:

AzCopy documentation: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10

Authorizing to AzCopy using Azure AD/Microsoft Entra ID: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-authorize-azure-active-directory