Introduction

In this chapter, we will take a look at the storage layer, which is HDFS, and how it can be configured for storing data. It is important to ensure the good health of this distributed filesystem, and make sure that the data it contains is available, even in the case of failures. In this chapter, we will take a look at the replication, quota setup, and balanced distribution of data across nodes, as well as covering recipes on rack awareness and heartbeat for communication with the master.

The recipes in this chapter assume that you already have a running cluster and have completed the steps given in Chapter 1, Hadoop Architecture and Deployment.

Note

While the recipes in this chapter will give you an overview of a typical configuration, we encourage you to adapt this proposal according to your needs. The block size plays an important role in the performance and the amount of data that is worked on by a mapper. It is good practice to set up passphrase less access between nodes,...