Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Free Learning
Arrow right icon
Elasticsearch 5.x Cookbook
Elasticsearch 5.x Cookbook

Elasticsearch 5.x Cookbook: Distributed Search and Analytics , Third Edition

eBook
AU$46.99 AU$67.99
Paperback
AU$83.99
Subscription
Free Trial
Renews at AU$24.99p/m

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Table of content icon View table of contents Preview book icon Preview Book

Elasticsearch 5.x Cookbook

Chapter 2. Downloading and Setup

In this chapter, we will cover the following recipes:

  • Downloading and installing Elasticsearch

  • Setting up networking

  • Setting up a node

  • Setting up for Linux systems

  • Setting up different node types

  • Setting up a client node

  • Setting up an ingest node

  • Installing plugins in Elasticsearch

  • Installing plugins manually

  • Removing a plugin

  • Changing logging settings

  • Setting up a node via Docker

Introduction


This chapter explains the installation process and the configuration from a single developer machine to a big cluster, giving you hints on how to improve performance and skip misconfiguration errors.

There are different options in installing Elasticsearch and setting up a working environment for development and production.

When testing out Elasticsearch for a development cluster, the tool requires almost no configuration. However, when moving to production, it is important to properly configure the cluster based on your data, use cases, and your product architecture. The setup step is very important because a bad configuration can lead to bad results, poor performances, and kill your servers.

In this chapter, the management of Elasticsearch plugins is also discussed: installing, configuring, updating, and removing.

Downloading and installing Elasticsearch


Elasticsearch has an active community and the release cycles are very fast.

Because Elasticsearch depends on many common Java libraries (Lucene, Guice, and Jackson are the most famous ones), the Elasticsearch community tries to keep them updated and fixes bugs that are discovered in them and in Elasticsearch core. The large user base is also source of new ideas and features for improving Elasticsearch use cases.

For these reasons, if it's possible, the best practice is to use the latest available release (usually the more stable one and the less bugs free).

Getting ready

A supported Elasticsearch operative system (Linux/MacOSX/Windows) with a Java JVM 1.8 (the Oracle one is the preferred http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) or above installed. A web browser is required to download the Elasticsearch binary release. At least 1GB of free disk space is required to install Elasticsearch.

How to do it...

For downloading...

Setting up networking


Correctly setting up networking is very important for your nodes and cluster.

There are a lot of different installation scenarios and networking issues: the first step for configuring the nodes to build a cluster is to correctly set the node discovery.

Getting ready

You need a working Elasticsearch installation and know your current networking configuration (that is, IP).

How to do it...

For configuring networking, we will perform the following steps:

  • Open the Elasticsearch configuration file with your favorite text editor.

  • Using standard Elasticsearch configuration config/elasticsearch.yml file, your node is configured to bind on all your machine interfaces and does discovery broadcasting events to the nodes listed in discovery.zen.ping.unicast.hosts. This means that it sends signals to the machine in unicast list and waits for a response. If a node responds to it, they can join in a cluster.

  • If another node is available in the same LAN, they join the cluster.

    Note

    Only nodes...

Setting up a node


Elasticsearch allows customizing several parameters in an installation. In this recipe, we'll see the most used ones to define where to store our data and to improve the overall performances.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change configuration files.

How to do it...

The steps required for setting up a simple node are as follows:

  • Open config/elasticsearch.yml with an editor of your choice.

  • Setup the directories that store your server data.

  • For Linux or Mac OS X type the following command:

        path.conf: /opt/data/es/conf
        path.data: /opt/data/es/data1,/opt2/data/data2
        path.work: /opt/data/work 
        path.logs: /opt/data/logs 
        path.plugins: /opt/data/plugins 
  • For Windows type the following command:

        path.conf: c:\Elasticsearch\conf 
        path.data: c:\Elasticsearch\data 
        path.work: c:\Elasticsearch\work 
  ...

Setting up for Linux systems


If you are using a Linux system, you need to manage extra setup to improve performance or to resolve production problems with many indices.

This recipe covers two common errors that happened in production:

  • Too many open files that can corrupt your indices and your data

  • Slow performance in search and indexing due to garbage collector

Note

The other possible big troubles arise when you go out of disk space. In this scenario, some files can get corrupted. To prevent your indices from corruption and possible data loss, it is best practice to monitor the storage spaces.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe in this chapter and a simple text editor to change configuration files.

How to do it...

For improving the performances on Linux systems, we will perform the following steps:

  1. First you need to change the current limit for the user that runs the Elasticsearch server. In these examples...

Setting up different node types


Elasticsearch is natively designed for the cloud, so when you need to release a production environment with a huge number of records and you need high availability and good performances, you need to aggregate more nodes in a cluster.

Elasticsearch allows defining different type of nodes to balance and improve overall performances.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change the configuration files.

How to do it...

For advance, set up a cluster. There are some parameters that must be configured to define different node types.

These parameters are in config/elasticsearch.yml file and they can be set with the following steps:

  1. Set up whether the node can be master or not:

            node.master: true 
    
  2. Set up whether a node must contain data or not:

            node.data: true 
    

How it works...

The node.master parameter defines that the node can become master...

Setting up a client node


The master nodes that we have seen previously are the most important for cluster stability. To prevent the queries and aggregations from creating instability in your cluster, client nodes can be used to provide safe communication with the cluster.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe in this chapter and a simple text editor to change configuration files.

How to do it...

For advance set up of a cluster, there are some parameters that must be configured to define different node types.

These parameters are in the config/elasticsearch.yml file and they can set up a client node with the following steps:

  1. Set up the node as a no master:

            node.master: false 
    
  2. Set up the node to not contain data:

            node.data: false 
    

How it works...

The client node is a special node that works as a proxy/pass thought for the cluster.

Its main advantages are:

  • It can easily kill or remove the cluster...

Setting up an ingestion node


The main goals of Elasticsearch are indexing, searching, and analytics, but it's often required to modify or enhance the documents before storing in Elasticsearch.

The most common scenarios in this case are:

  • Preprocessing the log string to extract meaningful data.

  • Enrich the content of some textual fields with Natural Language Processing (NLP) tools.

  • Add some transformation during ingestion such as convert IP in geolocalization or build custom fields at ingest time

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change configuration files.

How to do it...

To set up an ingest node, you need to edit the config/elasticsearch.yml file and set up the ingest property to true:

node.ingest: true

How it works...

The default configuration for Elasticsearch is to set the node as ingest node (refer to Chapter 13, Ingest, for more info on ingestion pipeline).

As the client node...

Installing plugins in Elasticsearch


One of the main features of Elasticsearch is the possibility to extend it with plugins. Plugins extend Elasticsearch features and functionalities in several ways.

In Elasticsearch 5.x, the plugins are native plugins--they are jars files that contain application code. They are used for:

  • ScriptEngine (JavaScript, Python, Scala, and Ruby)

  • Custom Analyzers, tokenizers, and scoring

  • REST entry points

  • Ingestion pipeline stages

  • Supporting new storages (Hadoop)

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a prompt/shell to execute commands in Elasticsearch install directory.

How to do it...

Elasticsearch provides a script for automatic download and for installation of plugins in bin/directory called plugin.

The steps required to install a plugin are:

  • Call the plugin and install Elasticsearch command with the plugin name reference.

  • For installing an administrative interface for Elasticsearch...

Installing plugins manually


Sometimes your plugin is not available online or standard installation fails, so you need to install your plugin manually.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a prompt/shell to execute commands in Elasticsearch install directory.

How to do it...

We assume that your plugin is named awesome and it's packed in a file called awesome.zip.

The steps required to manually install a plugin are:

  • Copy your zip file in the plugins directory in your Elasticsearch home installation

  • If the directory named plugins doesn't exist, create it

  • Unzip the content of the plugin in the plugins directory

  • Remove the zip archive to clean up unused files

How it works...

Every Elasticsearch plugin is contained in a directory (usually named as the plugin name). The plugin directory should be filled with one or more JAR files.

When Elasticsearch starts, it scans the plugins directory and loads them.

Note

If...

Removing a plugin


You have installed some plugins and now you need to remove a plugin because it's not required. Removing an Elasticsearch plugin is easy to uninstall if everything goes right, otherwise you need to manually remove it.

This recipe covers both cases.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a prompt/shell to execute commands in Elasticsearch install directory. Before removing a plugin, it is safer to stop Elasticsearch server to prevent error due to the deletion of plugin JAR.

How to do it...

The steps to remove a plugin are as follows:

  1. Stop your running node to prevent exceptions caused due to removal of a file.

  2. Using the Elasticsearch plugin manager, which comes with its script wrapper (plugin).

    On Linux and MacOSX, type the following command:

            elasticsearch-plugin remove lang-python      

    On Windows, type the following command:

            elasticsearch-plugin.bat remove lang-python 
    
  3. Restart...

Changing logging settings


Standard logging settings work very well for general usage.

Changing the log level can be useful to check for bugs or understanding malfunctions due to bad configuration or strange plugin behaviors. A verbose log can be used from Elasticsearch community to cover problems.

If you need to debug your Elasticsearch server or change how the logging works (that is, remoting send events), you need to change the log4j2.properties parameters.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change configuration files.

How to do it...

In the config directory in your Elasticsearch install directory, there is a log4j2.properties file, which controls the working settings.

The steps required for changing the logging settings are:

  1. To emit every kind of logging Elasticsearch has, you can change the current root level logging which is:

            rootLogger.level = info 
    
  2. This needs...

Setting up a node via Docker


Docker (https://www.docker.com/) has become a common way to deploy for testing or production some application server.

Docker is a container system that allows to easily deploy replicable installations of server applications. With Docker, you don't need to set up a host, configure it, download the Elasticsearch server, unzip it, or start the server--everything is done automatically by Docker.

Getting ready

You need a working Docker installation to be able to execute docker commands (https://www.docker.com/products/overview).

How to do it...

  1. If you want to start a vanilla server, just execute:

            docker pull docker.elastic.co/elasticsearch/elasticsearch:5.1.1
  2. An output similar to the following screenshot will be shown:

  3. After downloading the Elasticsearch image, we can start a develop instance via:

            docker run -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e    
            "transport.host=0.0.0.0"    
            docker.elastic.co/elasticsearch/elasticsearch:5.1...
Left arrow icon Right arrow icon

Key benefits

  • Deploy and manage simple Elasticsearch nodes as well as complex cluster topologies
  • Write native plugins to extend the functionalities of Elasticsearch 5.x to boost your business
  • Packed with clear, step-by-step recipes to walk you through the capabilities of Elasticsearch 5.x

Description

Elasticsearch is a Lucene-based distributed search server that allows users to index and search unstructured content with petabytes of data. This book is your one-stop guide to master the complete Elasticsearch ecosystem. We’ll guide you through comprehensive recipes on what’s new in Elasticsearch 5.x, showing you how to create complex queries and analytics, and perform index mapping, aggregation, and scripting. Further on, you will explore the modules of Cluster and Node monitoring and see ways to back up and restore a snapshot of an index. You will understand how to install Kibana to monitor a cluster and also to extend Kibana for plugins. Finally, you will also see how you can integrate your Java, Scala, Python, and Big Data applications such as Apache Spark and Pig with Elasticsearch, and add enhanced functionalities with custom plugins. By the end of this book, you will have an in-depth knowledge of the implementation of the Elasticsearch architecture and will be able to manage data efficiently and effectively with Elasticsearch.

Who is this book for?

If you are a developer who wants to get the most out of Elasticsearch for advanced search and analytics, this is the book for you. Some understanding of JSON is expected. If you want to extend Elasticsearch, understanding of Java and related technologies is also required.

What you will learn

  • Choose the best Elasticsearch cloud topology to deploy and power it up with external plugins
  • Develop tailored mapping to take full control of index steps
  • Build complex queries through managing indices and documents
  • Optimize search results through executing analytics aggregations
  • Monitor the performance of the cluster and nodes
  • Install Kibana to monitor cluster and extend Kibana for plugins
  • Integrate Elasticsearch in Java, Scala, Python and Big Data applications

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Feb 06, 2017
Length: 696 pages
Edition : 3rd
Language : English
ISBN-13 : 9781786466884
Vendor :
Elastic
Category :

What do you get with eBook?

Product feature icon Instant access to your Digital eBook purchase
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
OR
Modal Close icon
Payment Processing...
tick Completed

Billing Address

Product Details

Publication date : Feb 06, 2017
Length: 696 pages
Edition : 3rd
Language : English
ISBN-13 : 9781786466884
Vendor :
Elastic
Category :

Packt Subscriptions

See our plans and pricing
Modal Close icon
AU$24.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
AU$249.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just AU$5 each
Feature tick icon Exclusive print discounts
AU$349.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just AU$5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total AU$ 220.97
Mastering Elasticsearch 5.x
AU$75.99
Learning Kibana 5.0
AU$60.99
Elasticsearch 5.x Cookbook
AU$83.99
Total AU$ 220.97 Stars icon
Banner background image

Table of Contents

18 Chapters
Getting Started Chevron down icon Chevron up icon
Downloading and Setup Chevron down icon Chevron up icon
Managing Mappings Chevron down icon Chevron up icon
Basic Operations Chevron down icon Chevron up icon
Search Chevron down icon Chevron up icon
Text and Numeric Queries Chevron down icon Chevron up icon
Relationships and Geo Queries Chevron down icon Chevron up icon
Aggregations Chevron down icon Chevron up icon
Scripting Chevron down icon Chevron up icon
Managing Clusters and Nodes Chevron down icon Chevron up icon
Backup and Restore Chevron down icon Chevron up icon
User Interfaces Chevron down icon Chevron up icon
Ingest Chevron down icon Chevron up icon
Java Integration Chevron down icon Chevron up icon
Scala Integration Chevron down icon Chevron up icon
Python Integration Chevron down icon Chevron up icon
Plugin Development Chevron down icon Chevron up icon
Big Data Integration Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Full star icon Half star icon Empty star icon Empty star icon 2.5
(4 Ratings)
5 star 0%
4 star 0%
3 star 50%
2 star 50%
1 star 0%
Charles Apr 15, 2017
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
The book contains many single recipes for many things people do with elastic.For instance 1 recipe is "indexing a document" and another is "aggregation with a filter"The recipes are nothing you can't find online and some of them are worthless, like a good chunk of the book are recipes for scala and python apis which don't really belong in a book about elasticsearch. There's also a lot written on plugin development and web UIs for elastic..I would recommend it if you need an elastic reference for when google is unavailable - and there's a few good tips on in depth technical stuff but its not a great book if you want to learn advanced elasticsearch things you can't find online.
Amazon Verified review Amazon
Rajeev Goel Sep 25, 2017
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Not upto mark, Coding examples are using syntax from Elasticsearch -5.0. Advanced features are not explained well.
Amazon Verified review Amazon
jbdemonte Mar 23, 2017
Full star icon Full star icon Empty star icon Empty star icon Empty star icon 2
Je m'attendais à un vrai cookbook, j'ai rien trouvé de plus dans ce boucain que je n'avais dans les autres
Amazon Verified review Amazon
G.S May 10, 2018
Full star icon Full star icon Empty star icon Empty star icon Empty star icon 2
Not well written, apparently written in another language originally and translated to English and it shows.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

How do I buy and download an eBook? Chevron down icon Chevron up icon

Where there is an eBook version of a title available, you can buy it from the book details for that title. Add either the standalone eBook or the eBook and print book bundle to your shopping cart. Your eBook will show in your cart as a product on its own. After completing checkout and payment in the normal way, you will receive your receipt on the screen containing a link to a personalised PDF download file. This link will remain active for 30 days. You can download backup copies of the file by logging in to your account at any time.

If you already have Adobe reader installed, then clicking on the link will download and open the PDF file directly. If you don't, then save the PDF file on your machine and download the Reader to view it.

Please Note: Packt eBooks are non-returnable and non-refundable.

Packt eBook and Licensing When you buy an eBook from Packt Publishing, completing your purchase means you accept the terms of our licence agreement. Please read the full text of the agreement. In it we have tried to balance the need for the ebook to be usable for you the reader with our needs to protect the rights of us as Publishers and of our authors. In summary, the agreement says:

  • You may make copies of your eBook for your own use onto any machine
  • You may not pass copies of the eBook on to anyone else
How can I make a purchase on your website? Chevron down icon Chevron up icon

If you want to purchase a video course, eBook or Bundle (Print+eBook) please follow below steps:

  1. Register on our website using your email address and the password.
  2. Search for the title by name or ISBN using the search option.
  3. Select the title you want to purchase.
  4. Choose the format you wish to purchase the title in; if you order the Print Book, you get a free eBook copy of the same title. 
  5. Proceed with the checkout process (payment to be made using Credit Card, Debit Cart, or PayPal)
Where can I access support around an eBook? Chevron down icon Chevron up icon
  • If you experience a problem with using or installing Adobe Reader, the contact Adobe directly.
  • To view the errata for the book, see www.packtpub.com/support and view the pages for the title you have.
  • To view your account details or to download a new copy of the book go to www.packtpub.com/account
  • To contact us directly if a problem is not resolved, use www.packtpub.com/contact-us
What eBook formats do Packt support? Chevron down icon Chevron up icon

Our eBooks are currently available in a variety of formats such as PDF and ePubs. In the future, this may well change with trends and development in technology, but please note that our PDFs are not Adobe eBook Reader format, which has greater restrictions on security.

You will need to use Adobe Reader v9 or later in order to read Packt's PDF eBooks.

What are the benefits of eBooks? Chevron down icon Chevron up icon
  • You can get the information you need immediately
  • You can easily take them with you on a laptop
  • You can download them an unlimited number of times
  • You can print them out
  • They are copy-paste enabled
  • They are searchable
  • There is no password protection
  • They are lower price than print
  • They save resources and space
What is an eBook? Chevron down icon Chevron up icon

Packt eBooks are a complete electronic version of the print edition, available in PDF and ePub formats. Every piece of content down to the page numbering is the same. Because we save the costs of printing and shipping the book to you, we are able to offer eBooks at a lower cost than print editions.

When you have purchased an eBook, simply login to your account and click on the link in Your Download Area. We recommend you saving the file to your hard drive before opening it.

For optimal viewing of our eBooks, we recommend you download and install the free Adobe Reader version 9.