Hbase Essentials

Chapter 1. Introducing HBase

A relational database management system (RDBMS) is the right choice for most of the online transactional processing (OLTP) applications, and it also supports most of the online analytical processing (OLAP) systems. Large OLAP systems usually run very large queries that scan a wide set of records or an entire dataset containing billions of records (terabytes or petabytes in size) and face scaling issues. To address scaling issues using RDBMS, a huge investment becomes another point of concern.

The world of Big Data

Since the last decade, the amount of data being created is more than 20 terabytes per second and this size is only increasing. Not only volume and velocity but this data is also of a different variety, that is, structured and semi structured in nature, which means that data might be coming from blog posts, tweets, social network interactions, photos, videos, continuously generated log messages about what users are doing, and so on. Hence, Big Data is a combination of transactional data and interactive data. This large set of data is further used by organizations for decision making. Storing, analyzing, and summarizing these large datasets efficiently and cost effectively have become among the biggest challenges for these organizations.

In 2003, Google published a paper on the scalable distributed filesystem titled Google File System (GFS), which uses a cluster of commodity hardware to store huge amounts of data and ensure high availability by using the replication of data between nodes. Later, Google published an additional paper on processing large, distributed datasets using MapReduce (MR).

For processing Big Data, platforms such as Hadoop, which inherits the basics from both GFS and MR, were developed and contributed to the community. A Hadoop-based platform is able to store and process continuously growing data in terabytes or petabytes.

Note

The Apache Hadoop software library is a framework that allows the distributed processing of large datasets across clusters of computers.

However, Hadoop is designed to process data in the batch mode and the ability to access data randomly and near real time is completely missing. In Hadoop, processing smaller files has a larger overhead compared to big files and thus is a bad choice for low latency queries.

Later, a database solution called NoSQL evolved with multiple flavors, such as a key-value store, document-based store, column-based store, and graph-based store. NoSQL databases are suitable for different business requirements. Not only do these different flavors address scalability and availability but also take care of highly efficient read/write with data growing infinitely or, in short, Big Data.

Note

The NoSQL database provides a fail-safe mechanism for the storage and retrieval of data that is modeled in it, somewhat different from the tabular relations used in many relational databases.

Use cases of HBase

There are a number of use cases where HBase can be a storage system. This section discusses a few of the popular use cases for HBase and the well-known companies that have adopted HBase. Let's discuss the use cases first:

Handling content: In today's world, a variety of content is available for the users for consumption. Also, the variety of application clients, such as browser, mobile, and so on, leads to an additional requirement where each client needs the same content in different formats. Users not only consume content but also generate a variety of content in a large volume with a high velocity, such as tweets, Facebook posts, images, bloging, and many more. HBase is the perfect choice as the backend of such applications, for example, many scalable content management solutions are using HBase as their backend.
Handling incremental data: In many use cases, trickled data is added to a data store for further usage, such as analytics, processing, and serving. This trickled data could be coming from an advertisement's impressions such as clickstreams and user interaction data or it can be time series data. HBase is used for storage in all such cases. For example, Open Time Series Database (OpenTSDB) uses HBase for data storage and metrics generation. The counters feature (discussed in Chapter 5, The HBase Advanced API) is used by Facebook for counting and storing the "likes" for a particular page/image/post.

Some of the companies that are using HBase in their respective use cases are as follows:

Facebook (www.facebook.com): Facebook is using HBase to power its message infrastructure. Facebook opted for HBase to scale from their old messages infrastructure which handled over 350 million users, sending over 15 billion person-to-person messages per month. HBase was selected due to the excellent scalability and performance for big workloads, along with autoload balancing and failover features and so on. Facebook also uses HBase for counting and storing the "likes" contributed by users.
Meetup (www.meetup.com): Meetup uses HBase to power a site-wide, real-time activity feed system for all of its members and groups. In its architecture, group activity is written directly to HBase and indexed per member, with the member's custom feed served directly from HBase for incoming requests.
Twitter (www.twitter.com): Twitter uses HBase to provide a distributed, read/write backup of all the transactional tables in Twitter's production backend. Later, this backup is used to run MapReduce jobs over the data. Additionally, its operations team uses HBase as a time series database for cluster-wide monitoring / performance data.
Yahoo (www.yahoo.com): Yahoo uses HBase to store document fingerprints for detecting near-duplications. With millions of rows in the HBase table, Yahoo runs a query for finding duplicated documents with real-time traffic.

Tip

The source for the preceding mentioned information is http://wiki.apache.org/hadoop/Hbase/PoweredBy.

Installing HBase

HBase is an Apache project and the current Version, 0.98.7, of HBase is available as a stable release. HBase Version 0.98.7 supersedes Version 0.94.x and 0.96.x.

Note

This book only focuses on HBase Version 0.98.7, as this version is fully supported and tested with Hadoop Versions 2.x and deprecates the use of Hadoop 1.x.

Hadoop 2.x is much faster compared to Hadoop 1.x and includes important bug fixes that will improve the overall HBase performance.

Older versions, 0.96.x, of HBase which are now extinct, supported both versions of Hadoop (1.x and 2.x). The HBase version prior to 0.96.x only supported Hadoop 1.x.

HBase is written in Java, works on top of Hadoop, and relies on ZooKeeper. A HBase cluster can be set up in either local or distributed mode. Distributed mode can further be classified into either pseudo-distributed or fully distributed mode.

Note

HBase is designed and developed to work on kernel-based operating systems; hence, the commands referred to in this book are only for a kernel-based OS, for example, CentOS. In the case of Windows, it is recommended that you have a CentOS-based virtual machine to play with HBase.

An HBase cluster requires only Oracle Java to be installed on all the machines that are part of the cluster. In case any other flavor of Java, such as OpenJDK, is installed with the operating system, it needs to be uninstalled first before installing Oracle Java. HBase and other components such as Hadoop and ZooKeeper require a minimum of Java 6 or later.

Installing Java 1.7

Perform the following steps for installing Java 1.7 or later:

Download the jdk-7u55-linux-x64.rpm kit from Oracle's website at http://www.oracle.com/technetwork/java/javase/downloads/index.html.
Make sure that the file has all the permissions before installation for the root user using the following command:
```
[root@localhost opt]#chmod +x jdk-7u55-linux-x64.rpm
```

Install RPM using the following command:

[root@localhost opt]#rpm –ivh jdk-7u55-linux-x64.rpm

Finally, add the environment variable, JAVA_HOME. The following command will write the JAVA_HOME environment variable to the /etc/profile file, which contains a system-wide environment configuration:
```
[root@localhost opt]# echo "export JAVA_HOME=/usr/java/jdk1.7.0_55" >> /etc/profile
```
Once JAVA_HOME is added to the profile, either close the command window and reopen it or run the following command. This step is required to reload the latest profile setting for the user:
```
[root@localhost opt]# source /etc/profile
```
Tip
Downloading the example code
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com . If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

The local mode

The local or standalone mode means running all HBase services in just one Java process. Setting up HBase in the local mode is the easiest way to get started with HBase and can be used to explore further or for local development. The only step required is to download the recent release of HBase and unpack the archive (.tar) in some directory such as /opt. Perform the following steps to set up HBase in the local mode:

Create the hbase directory using the following commands:

[root@localhost opt]# mkdir myhbase
[root@localhost opt]# cd myhbase

Download the hbase binaries as the archive (.tar) files and unpack it, as shown in the following command:
```
[root@localhost myhbase]# wget http://mirrors.sonic.net/apache/hbase/stable/hbase-0.98.7-hadoop2-bin.tar.gz
```
In the preceding command, http://mirrors.sonic.net/apache/hbase/ can be different for different users, which is based on the user's location. Check the suggested mirror site at http://www.apache.org/dyn/closer.cgi/hbase/ for the new URL.
Note
HBase version 0.98.7 is available for Hadoop 1 and 2 as hbase-0.98.7-hadoop1-bin.tar.gz and hbase-0.98.7-hadoop2-bin.tar.gz. It is recommended that you use Hadoop 2 only with HBase 0.98.7, and Hadoop 1 is available as a deprecated support. In the local mode, a Hadoop cluster is not required as it can use the Hadoop binaries provided in the lib directory of HBase. Other versions of HBase can also be checked out at http://www.apache.org/dyn/closer.cgi/hbase/.
Once the HBase binaries are downloaded, extract them using the following command:
```
[root@localhost myhbase]# tar xvfz hbase-0.98.7-hadoop2-bin.tar.gz
```
Add the environment variable, HBASE_HOME. The following command will write the HBASE_HOME environment variable to the /etc/profile file, which contains system-wide environment configuration:
```
[root@localhost myhbase]# echo "export HBASE_HOME=/opt/myhbase/hbase-0.98.7-hadoop2" >> /etc/profile
```
Once HBASE_HOME is added to the profile, either close the command window and reopen it or run the following command; this step is required to reload the latest profile settings for the user:
```
[root@localhost opt]# source /etc/profile
```
Edit the configuration file, conf/hbase-site.xml, and set the data directory for HBase by assigning a value to the property key named hbase.rootdir and hbase.zookeeper.property.dataDir, as follows:
```
  <property>
    <name>hbase.rootdir</name>
    <value>file:///opt/myhbase/datadirectory</value>
  </property>
 <property>
      <name>hbase.zookeeper.property.dataDir</name>
      <value>/opt/myhbase/zookeeper</value>
  </property>
```
The default base directory value for the hbase.rootdir and hbase.zookeeper.property.dataDir properties is /tmp/hbase-${user.name}, that is, /tmp/hbase-root for the "root" user which may lead to the possibility of data loss at the time of server reboot. Hence, it is always advisable to set the value for this property to avoid a data-loss scenario.

Start HBase and verify the output with the following command:

[root@localhost opt]# cd /opt/myhbase/hbase-0.98.7-hadoop2
[root@localhost hbase-0.98.7-hadoop2]# bin/start-hbase.sh

This gives the following output:

HBase also comes with a preinstalled web-based management console that can be accessed using http://localhost:60010. By default, it is deployed on HBase's Master host at port 60010. This UI provides information about various components such as region servers, tables, running tasks, logs, and so on, as shown in the following screenshot:

The HBase tables and monitored tasks are shown in the following screenshot:

The following screenshot displays information about the HBase attributes, provided by the UI:

Once the HBase setup is done correctly, the following directories are created in a local filesystem, as shown in the following screenshot:

The pseudo-distributed mode

The standalone/local mode is only useful for basic operations and is not at all suitable for real-world workloads. In the pseudo-distributed mode, all HBase services (HMaster, HRegionServer, and Zookeeper) run as separate Java processes on a single machine. This mode can be useful during the testing phase.

In the pseudo-distributed mode, HDFS setup is another prerequisite (HDFS setup also needs to be present in pseudo-distributed mode). After setting up Hadoop and downloading the HBase binary, edit the conf/hbase-site.xml configuration file. Also, set the HBase in the running mode by assigning a value to the property key named hbase.cluster.distributed, as well as the data storage pointer to the running Hadoop HDFS instance by assigning a value to the property key named hbase.rootdir:

<property>
      <name>hbase.cluster.distributed</name>
      <value>true</value>
</property>

<property>
   <name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>

Once the settings are done, we can use the following command to start HBase:

[root@localhost opt]# cd /opt/myhbase/hbase-0.96.2-hadoop2
[root@localhost hbase-0.96.2-hadoop2]# bin/start-hbase.sh

Note

Before starting HBase, make sure that the Hadoop services are running and working fine.

Once HBase is configured correctly, the jps command should show the HMaster and HRegionServer processes running along with the Hadoop processes. Use the hadoop fs command in Hadoop's bin/ directory to list the directories created in HDFS as follows:

[root@localhost opt]# hadoop fs -ls /hbase
Found 7 items
drwxr-xr-x   - hbase users          0 2014-10-20 14:28 /hbase/.tmp
drwxr-xr-x   - hbase users          0 2014-10-20 17:29 /hbase/WALs
drwxr-xr-x   - hbase users          0 2014-10-20 14:38 /hbase/corrupt
drwxr-xr-x   - hbase users          0 2014-10-20 14:28 /hbase/data
-rw-r--r--   3 hbase users         42 2014-10-20 14:21 /hbase/hbase.id
-rw-r--r--   3 hbase users          7 2014-10-20 14:21 /hbase/hbase.version
drwxr-xr-x   - hbase users          0 2014-10-20 27:29 /hbase/oldWALs

The fully distributed mode

A pseudo-distributed mode, where all the HBase services (HMaster, HRegionServer, and Zookeeper) run as separate Java processes on a single machine, is preferred for a local development environment or test environment. However, for a production environment, fully distributed mode is a must. In the fully distributed mode, an HBase cluster is set up on multiple nodes and HBase services run on these different cluster nodes. To enable fully distributed mode, add the hbase.cluster.distributed property to conf/hbase-site.xml and set it to true; also point the hbase.rootdir HBase to the HDFS node:

<property>
   <name>hbase.rootdir</name>
   <value>hdfs://<HDFSNameNodeHost>:9000/hbase</value>
</property>
<property>
   <name>hbase.cluster.distributed</name>
   <value>true<value>
</property>

Note

This book does not touch upon information on building a fully distributed HBase cluster and also does not talk about the hardware considerations, such as, server configurations, network settings, and so on; and software considerations, such as server OS setting, Hadoop settings, and so on. For this book, it is recommended that you use either the local mode or the pseudo-distributed mode.

For understanding this mode in depth, the building blocks that play a vital role in a fully distributed HBase cluster need to be understood well. The next section will give you a glimpse of what these components are.

Understanding HBase cluster components

In fully distributed and pseudo-distributed modes, a HBase cluster has many components such as HBase Master, ZooKeeper, RegionServers, HDFS DataNodes, and so on, discussed as follows:

HBase Master: HBase Master coordinates the HBase cluster and is responsible for administrative operations. It is a lightweight process that does not require too many hardware resources. A large cluster might have multiple HBase Master components to avoid cases that have a single point of failure. In this highly available cluster with multiple HBase Master components, only once HBase Master is active and the rest of HBase Master servers get in sync with the active server asynchronously. Selection of the next HBase Master in case of failover is done with the help of the ZooKeeper ensemble.
ZooKeeper: ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. Similar to HBase Master, ZooKeeper is again a lightweight process. By default, a ZooKeeper process is started and stopped by HBase but it can be managed separately as well. The HBASE_MANAGES_ZK variable in conf/hbase-env.sh with the default value true signifies that HBase is going to manage ZooKeeper. We can specify the ZooKeeper configuration in the native zoo.cfg file or its values such as client, port, and so on directly in conf/hbase-site.xml. It is advisable that you have an odd number of ZooKeeper ensembles such as one/three/five for more host failure tolerance. The following is an example of hbase-site.xml with ZooKeeper settings:
```
<property>
  <name>hbase.zookeeper.property.clientPort</name>
  <value>2222</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
  <value>ZooKeeperhost1, ZooKeeperhost2, ZooKeeperhost3<value>
</property>
<property>
  <name>hbase.zookeeper.property.dataDir</name>
  <value>/opt/zookeeper<value>
</property>
```
RegionServers: In HBase, horizontal scalability is defined with a term called region. Regions are nothing but a sorted range of rows stored continuously. In HBase architecture, a set of regions is stored on the region server. By default, the region server runs on port 60030. In an HBase cluster based on HDFS, Hadoop DataNodes and RegionServers are typically called slave nodes as they are both responsible for server data and are usually collocated in the cluster. A list of the region servers is specified in the conf/regionservers file with each region server on a separate line, and the start/stop of these region servers is controlled by the script files responsible for an HBase cluster's start/stop.
HBase data storage system: HBase is developed using pluggable architecture; hence, for the data storage layer, HBase is not tied with HDFS. Rather, it can also be plugged in with other file storage systems such as the local filesystem (primarily used in standalone mode), S3 (Amazon's Simple Storage Service), CloudStore (also known as Kosmos filesystem) or a self-developed filesystem.

Apart from the mentioned components, there are other considerations as well, such as hardware and software considerations, that are not within the scope of this book.

Note

The backup HBase Master server and the additional region servers can be started in the pseudo-distributed mode using the utility provided bin directory as follows:

[root@localhost hbase-0.96.2-hadoop2]# bin/local-master-backup.sh 2 3

The preceding command will start the two additional HBase Master backup servers on the same box. Each HMaster server uses three ports (16010, 16020, and 16030 by default) and the new backup servers will be using ports 16012/16022/16032 and 16013/16023/16033.

[root@localhost hbase-0.96.2-hadoop2]# bin/local-regionservers.sh start 2 3

The preceding command will start the two additional HBase region servers on the same box using ports 16202/16302.

Start playing

Now that we have everything installed and running, let's start playing with it and try out a few commands to get a feel of HBase. HBase comes with a command-line interface that works for both local and distributed modes. The HBase shell is developed in JRuby and can run in both interactive (recommended for simple commands) and batch modes (recommended for running shell script programs). Let's start the HBase shell in the interactive mode as follows:

[root@localhost hbase-0.98.7-hadoop2]# bin/hbase shell

The preceding command gives the following output:

Type help and click on return to see a listing of the available shell commands and their options. Remember that all the commands are case-sensitive.

The following is a list of some simple commands to get your hands dirty with HBase:

status: This verifies whether HBase is up and running, as shown in the following screenshot:
create '<table_name>', '<column_family_name>': This creates a table with one column family. We can use multiple column family names as well, as shown in the following screenshot:
list: This provides the list of tables, as shown in the following screenshot:
put '<table_name>', '<row_num>', 'column_family:key', 'value': This command is used to put data in the table in a column family manner, as shown in the following screenshot. HBase is a schema-less database and provides the flexibility to store any type of data without defining it:
get '<table_name>', '<row_num>': This command is used to read a particular row from the table, as shown in the following screenshot:
scan '<table_name >': This scans the complete table and outputs the results, as shown in the following screenshot:
delete '<table_name>', '<row_num>', 'column_family:key': This deletes the specified value, as shown in the following screenshot:
describe '<table_name>': This describes the metadata information about the table, as shown in the following screenshot:
drop '<table_name>': This command will drop the table. However, before executing this command, we should first execute disable '<tablename>', as shown in the following screenshot:
Finally, exit the shell and stop using HBase, as shown in the following screenshot:

Note

Refer to the following link for more commands: http://wiki.apache.org/hadoop/Hbase/Shell

Ian Stirk Sep 29, 2015

Hi,I have written a detailed chapter-by-chapter review of this book on www DOT i-programmer DOT info, the first and last parts of this review are given here. For my review of all chapters, search i-programmer DOT info for STIRK together with the book's title.This book aims at getting you started in programming with HBase, how does it fare?Hadoop is the most popular platform for processing big data, and HBase is the NoSQL database included with Hadoop. The book is aimed at software developers that have no previous experience of HBase, wanting a hands-on approach.Below is a chapter-by-chapter exploration of the topics covered.Chapter 1 Introducing HBaseThis chapter opens with the observation that relational database management systems (RDBMS) are unable to scale to process huge amounts of data in a timely manner. Instead, this is the realm of big data databases such as HBase.The chapter defines big data in terms of huge volumes, high velocity, and varied types of data. Currently 20 terabytes of data is created every second, and this metric is increasing. The foundations of big data processing were established by Google in their papers on the Google File System (GFS) storage, and MapReduce processing model. Google also produced details of a scalable Bigtable database which could handle millions of columns and billions of rows, HBase is based on Bigtable.The chapter continues with a look at how to install HBase together with its prerequisites. The various modes of running HBase are discussed, these are: local, pseudo-distributed and fully distributed.The various HBase cluster components are briefly described, namely:-HBase Master – coordinates HBase cluster/admin operations-Zookeeper – centralized service for distributed synchronization-Region Servers – store horizontal rows as regionsThe chapter ends with a look at the HBase shell, this provides an environment to run interactive commands or batches, it’s a useful tool for experimenting with HBase. Various commands are introduced, including: status, create table, list, get, put, scan, delete, describe, and drop.This chapter provides a useful introduction to HBase in terms of the history of big data, the need to process huge amounts of data, and an outline of the HBase components. Useful installation instructions are provided, together with a HBase shell tutorial for playing and testing with HBase.This chapter provides well written discussions which are generally easy to read, has helpful diagrams, outputs, scripts, and brief practical walkthroughs. There are useful links to other chapters. Sometimes, terms are used before they are defined (e.g. HMaster and Zookeeper). These traits apply to the whole of the book....ConclusionThe book has well-written discussions which are generally easy to read, helpful diagrams, outputs, scripts, and brief practical walkthroughs. There are useful links to other chapters.This book aims to get you started in programming with HBase, however, it deviates from this, containing as much administration detail as programming. Sometimes terms are used before they are defined (e.g. HMaster and Zookeeper), suggesting you need some knowledge of Hadoop. It would have been helpful to list where to go next to extend your HBase knowledge.This book will help you get up and running with HBase, show you how to use HBase from various clients, and give you an understanding of its internal structure, I can recommend it as a starter book.

Amazon Verified review