Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Free Learning
Arrow right icon
Elasticsearch 5.x Cookbook
Elasticsearch 5.x Cookbook

Elasticsearch 5.x Cookbook: Distributed Search and Analytics , Third Edition

eBook
€24.99 €36.99
Paperback
€45.99
Subscription
Free Trial
Renews at €18.99p/m

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing
Table of content icon View table of contents Preview book icon Preview Book

Elasticsearch 5.x Cookbook

Chapter 2. Downloading and Setup

In this chapter, we will cover the following recipes:

  • Downloading and installing Elasticsearch

  • Setting up networking

  • Setting up a node

  • Setting up for Linux systems

  • Setting up different node types

  • Setting up a client node

  • Setting up an ingest node

  • Installing plugins in Elasticsearch

  • Installing plugins manually

  • Removing a plugin

  • Changing logging settings

  • Setting up a node via Docker

Introduction


This chapter explains the installation process and the configuration from a single developer machine to a big cluster, giving you hints on how to improve performance and skip misconfiguration errors.

There are different options in installing Elasticsearch and setting up a working environment for development and production.

When testing out Elasticsearch for a development cluster, the tool requires almost no configuration. However, when moving to production, it is important to properly configure the cluster based on your data, use cases, and your product architecture. The setup step is very important because a bad configuration can lead to bad results, poor performances, and kill your servers.

In this chapter, the management of Elasticsearch plugins is also discussed: installing, configuring, updating, and removing.

Downloading and installing Elasticsearch


Elasticsearch has an active community and the release cycles are very fast.

Because Elasticsearch depends on many common Java libraries (Lucene, Guice, and Jackson are the most famous ones), the Elasticsearch community tries to keep them updated and fixes bugs that are discovered in them and in Elasticsearch core. The large user base is also source of new ideas and features for improving Elasticsearch use cases.

For these reasons, if it's possible, the best practice is to use the latest available release (usually the more stable one and the less bugs free).

Getting ready

A supported Elasticsearch operative system (Linux/MacOSX/Windows) with a Java JVM 1.8 (the Oracle one is the preferred http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) or above installed. A web browser is required to download the Elasticsearch binary release. At least 1GB of free disk space is required to install Elasticsearch.

How to do it...

For downloading...

Setting up networking


Correctly setting up networking is very important for your nodes and cluster.

There are a lot of different installation scenarios and networking issues: the first step for configuring the nodes to build a cluster is to correctly set the node discovery.

Getting ready

You need a working Elasticsearch installation and know your current networking configuration (that is, IP).

How to do it...

For configuring networking, we will perform the following steps:

  • Open the Elasticsearch configuration file with your favorite text editor.

  • Using standard Elasticsearch configuration config/elasticsearch.yml file, your node is configured to bind on all your machine interfaces and does discovery broadcasting events to the nodes listed in discovery.zen.ping.unicast.hosts. This means that it sends signals to the machine in unicast list and waits for a response. If a node responds to it, they can join in a cluster.

  • If another node is available in the same LAN, they join the cluster.

    Note

    Only nodes...

Setting up a node


Elasticsearch allows customizing several parameters in an installation. In this recipe, we'll see the most used ones to define where to store our data and to improve the overall performances.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change configuration files.

How to do it...

The steps required for setting up a simple node are as follows:

  • Open config/elasticsearch.yml with an editor of your choice.

  • Setup the directories that store your server data.

  • For Linux or Mac OS X type the following command:

        path.conf: /opt/data/es/conf
        path.data: /opt/data/es/data1,/opt2/data/data2
        path.work: /opt/data/work 
        path.logs: /opt/data/logs 
        path.plugins: /opt/data/plugins 
  • For Windows type the following command:

        path.conf: c:\Elasticsearch\conf 
        path.data: c:\Elasticsearch\data 
        path.work: c:\Elasticsearch\work 
  ...

Setting up for Linux systems


If you are using a Linux system, you need to manage extra setup to improve performance or to resolve production problems with many indices.

This recipe covers two common errors that happened in production:

  • Too many open files that can corrupt your indices and your data

  • Slow performance in search and indexing due to garbage collector

Note

The other possible big troubles arise when you go out of disk space. In this scenario, some files can get corrupted. To prevent your indices from corruption and possible data loss, it is best practice to monitor the storage spaces.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe in this chapter and a simple text editor to change configuration files.

How to do it...

For improving the performances on Linux systems, we will perform the following steps:

  1. First you need to change the current limit for the user that runs the Elasticsearch server. In these examples...

Setting up different node types


Elasticsearch is natively designed for the cloud, so when you need to release a production environment with a huge number of records and you need high availability and good performances, you need to aggregate more nodes in a cluster.

Elasticsearch allows defining different type of nodes to balance and improve overall performances.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change the configuration files.

How to do it...

For advance, set up a cluster. There are some parameters that must be configured to define different node types.

These parameters are in config/elasticsearch.yml file and they can be set with the following steps:

  1. Set up whether the node can be master or not:

            node.master: true 
    
  2. Set up whether a node must contain data or not:

            node.data: true 
    

How it works...

The node.master parameter defines that the node can become master...

Setting up a client node


The master nodes that we have seen previously are the most important for cluster stability. To prevent the queries and aggregations from creating instability in your cluster, client nodes can be used to provide safe communication with the cluster.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe in this chapter and a simple text editor to change configuration files.

How to do it...

For advance set up of a cluster, there are some parameters that must be configured to define different node types.

These parameters are in the config/elasticsearch.yml file and they can set up a client node with the following steps:

  1. Set up the node as a no master:

            node.master: false 
    
  2. Set up the node to not contain data:

            node.data: false 
    

How it works...

The client node is a special node that works as a proxy/pass thought for the cluster.

Its main advantages are:

  • It can easily kill or remove the cluster...

Setting up an ingestion node


The main goals of Elasticsearch are indexing, searching, and analytics, but it's often required to modify or enhance the documents before storing in Elasticsearch.

The most common scenarios in this case are:

  • Preprocessing the log string to extract meaningful data.

  • Enrich the content of some textual fields with Natural Language Processing (NLP) tools.

  • Add some transformation during ingestion such as convert IP in geolocalization or build custom fields at ingest time

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change configuration files.

How to do it...

To set up an ingest node, you need to edit the config/elasticsearch.yml file and set up the ingest property to true:

node.ingest: true

How it works...

The default configuration for Elasticsearch is to set the node as ingest node (refer to Chapter 13, Ingest, for more info on ingestion pipeline).

As the client node...

Installing plugins in Elasticsearch


One of the main features of Elasticsearch is the possibility to extend it with plugins. Plugins extend Elasticsearch features and functionalities in several ways.

In Elasticsearch 5.x, the plugins are native plugins--they are jars files that contain application code. They are used for:

  • ScriptEngine (JavaScript, Python, Scala, and Ruby)

  • Custom Analyzers, tokenizers, and scoring

  • REST entry points

  • Ingestion pipeline stages

  • Supporting new storages (Hadoop)

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a prompt/shell to execute commands in Elasticsearch install directory.

How to do it...

Elasticsearch provides a script for automatic download and for installation of plugins in bin/directory called plugin.

The steps required to install a plugin are:

  • Call the plugin and install Elasticsearch command with the plugin name reference.

  • For installing an administrative interface for Elasticsearch...

Installing plugins manually


Sometimes your plugin is not available online or standard installation fails, so you need to install your plugin manually.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a prompt/shell to execute commands in Elasticsearch install directory.

How to do it...

We assume that your plugin is named awesome and it's packed in a file called awesome.zip.

The steps required to manually install a plugin are:

  • Copy your zip file in the plugins directory in your Elasticsearch home installation

  • If the directory named plugins doesn't exist, create it

  • Unzip the content of the plugin in the plugins directory

  • Remove the zip archive to clean up unused files

How it works...

Every Elasticsearch plugin is contained in a directory (usually named as the plugin name). The plugin directory should be filled with one or more JAR files.

When Elasticsearch starts, it scans the plugins directory and loads them.

Note

If...

Removing a plugin


You have installed some plugins and now you need to remove a plugin because it's not required. Removing an Elasticsearch plugin is easy to uninstall if everything goes right, otherwise you need to manually remove it.

This recipe covers both cases.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a prompt/shell to execute commands in Elasticsearch install directory. Before removing a plugin, it is safer to stop Elasticsearch server to prevent error due to the deletion of plugin JAR.

How to do it...

The steps to remove a plugin are as follows:

  1. Stop your running node to prevent exceptions caused due to removal of a file.

  2. Using the Elasticsearch plugin manager, which comes with its script wrapper (plugin).

    On Linux and MacOSX, type the following command:

            elasticsearch-plugin remove lang-python      

    On Windows, type the following command:

            elasticsearch-plugin.bat remove lang-python 
    
  3. Restart...

Changing logging settings


Standard logging settings work very well for general usage.

Changing the log level can be useful to check for bugs or understanding malfunctions due to bad configuration or strange plugin behaviors. A verbose log can be used from Elasticsearch community to cover problems.

If you need to debug your Elasticsearch server or change how the logging works (that is, remoting send events), you need to change the log4j2.properties parameters.

Getting ready

You need a working Elasticsearch installation as we described in the Downloading and installing Elasticsearch recipe and a simple text editor to change configuration files.

How to do it...

In the config directory in your Elasticsearch install directory, there is a log4j2.properties file, which controls the working settings.

The steps required for changing the logging settings are:

  1. To emit every kind of logging Elasticsearch has, you can change the current root level logging which is:

            rootLogger.level = info 
    
  2. This needs...

Setting up a node via Docker


Docker (https://www.docker.com/) has become a common way to deploy for testing or production some application server.

Docker is a container system that allows to easily deploy replicable installations of server applications. With Docker, you don't need to set up a host, configure it, download the Elasticsearch server, unzip it, or start the server--everything is done automatically by Docker.

Getting ready

You need a working Docker installation to be able to execute docker commands (https://www.docker.com/products/overview).

How to do it...

  1. If you want to start a vanilla server, just execute:

            docker pull docker.elastic.co/elasticsearch/elasticsearch:5.1.1
  2. An output similar to the following screenshot will be shown:

  3. After downloading the Elasticsearch image, we can start a develop instance via:

            docker run -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e    
            "transport.host=0.0.0.0"    
            docker.elastic.co/elasticsearch/elasticsearch:5.1...
Left arrow icon Right arrow icon

Key benefits

  • Deploy and manage simple Elasticsearch nodes as well as complex cluster topologies
  • Write native plugins to extend the functionalities of Elasticsearch 5.x to boost your business
  • Packed with clear, step-by-step recipes to walk you through the capabilities of Elasticsearch 5.x

Description

Elasticsearch is a Lucene-based distributed search server that allows users to index and search unstructured content with petabytes of data. This book is your one-stop guide to master the complete Elasticsearch ecosystem. We’ll guide you through comprehensive recipes on what’s new in Elasticsearch 5.x, showing you how to create complex queries and analytics, and perform index mapping, aggregation, and scripting. Further on, you will explore the modules of Cluster and Node monitoring and see ways to back up and restore a snapshot of an index. You will understand how to install Kibana to monitor a cluster and also to extend Kibana for plugins. Finally, you will also see how you can integrate your Java, Scala, Python, and Big Data applications such as Apache Spark and Pig with Elasticsearch, and add enhanced functionalities with custom plugins. By the end of this book, you will have an in-depth knowledge of the implementation of the Elasticsearch architecture and will be able to manage data efficiently and effectively with Elasticsearch.

Who is this book for?

If you are a developer who wants to get the most out of Elasticsearch for advanced search and analytics, this is the book for you. Some understanding of JSON is expected. If you want to extend Elasticsearch, understanding of Java and related technologies is also required.

What you will learn

  • Choose the best Elasticsearch cloud topology to deploy and power it up with external plugins
  • Develop tailored mapping to take full control of index steps
  • Build complex queries through managing indices and documents
  • Optimize search results through executing analytics aggregations
  • Monitor the performance of the cluster and nodes
  • Install Kibana to monitor cluster and extend Kibana for plugins
  • Integrate Elasticsearch in Java, Scala, Python and Big Data applications

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Feb 06, 2017
Length: 696 pages
Edition : 3rd
Language : English
ISBN-13 : 9781786465580
Vendor :
Elastic
Category :
Languages :

What do you get with a Packt Subscription?

Free for first 7 days. $19.99 p/m after that. Cancel any time!
Product feature icon Unlimited ad-free access to the largest independent learning library in tech. Access this title and thousands more!
Product feature icon 50+ new titles added per month, including many first-to-market concepts and exclusive early access to books as they are being written.
Product feature icon Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.
Product feature icon Thousands of reference materials covering every tech concept you need to stay up to date.
Subscribe now
View plans & pricing

Product Details

Publication date : Feb 06, 2017
Length: 696 pages
Edition : 3rd
Language : English
ISBN-13 : 9781786465580
Vendor :
Elastic
Category :
Languages :

Packt Subscriptions

See our plans and pricing
Modal Close icon
€18.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
€189.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts
€264.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just €5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total 120.97
Elasticsearch 5.x Cookbook
€45.99
Learning Kibana 5.0
€32.99
Mastering Elasticsearch 5.x
€41.99
Total 120.97 Stars icon
Banner background image

Table of Contents

18 Chapters
Getting Started Chevron down icon Chevron up icon
Downloading and Setup Chevron down icon Chevron up icon
Managing Mappings Chevron down icon Chevron up icon
Basic Operations Chevron down icon Chevron up icon
Search Chevron down icon Chevron up icon
Text and Numeric Queries Chevron down icon Chevron up icon
Relationships and Geo Queries Chevron down icon Chevron up icon
Aggregations Chevron down icon Chevron up icon
Scripting Chevron down icon Chevron up icon
Managing Clusters and Nodes Chevron down icon Chevron up icon
Backup and Restore Chevron down icon Chevron up icon
User Interfaces Chevron down icon Chevron up icon
Ingest Chevron down icon Chevron up icon
Java Integration Chevron down icon Chevron up icon
Scala Integration Chevron down icon Chevron up icon
Python Integration Chevron down icon Chevron up icon
Plugin Development Chevron down icon Chevron up icon
Big Data Integration Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Full star icon Half star icon Empty star icon Empty star icon 2.5
(4 Ratings)
5 star 0%
4 star 0%
3 star 50%
2 star 50%
1 star 0%
Charles Apr 15, 2017
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
The book contains many single recipes for many things people do with elastic.For instance 1 recipe is "indexing a document" and another is "aggregation with a filter"The recipes are nothing you can't find online and some of them are worthless, like a good chunk of the book are recipes for scala and python apis which don't really belong in a book about elasticsearch. There's also a lot written on plugin development and web UIs for elastic..I would recommend it if you need an elastic reference for when google is unavailable - and there's a few good tips on in depth technical stuff but its not a great book if you want to learn advanced elasticsearch things you can't find online.
Amazon Verified review Amazon
Rajeev Goel Sep 25, 2017
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Not upto mark, Coding examples are using syntax from Elasticsearch -5.0. Advanced features are not explained well.
Amazon Verified review Amazon
jbdemonte Mar 23, 2017
Full star icon Full star icon Empty star icon Empty star icon Empty star icon 2
Je m'attendais à un vrai cookbook, j'ai rien trouvé de plus dans ce boucain que je n'avais dans les autres
Amazon Verified review Amazon
G.S May 10, 2018
Full star icon Full star icon Empty star icon Empty star icon Empty star icon 2
Not well written, apparently written in another language originally and translated to English and it shows.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is included in a Packt subscription? Chevron down icon Chevron up icon

A subscription provides you with full access to view all Packt and licnesed content online, this includes exclusive access to Early Access titles. Depending on the tier chosen you can also earn credits and discounts to use for owning content

How can I cancel my subscription? Chevron down icon Chevron up icon

To cancel your subscription with us simply go to the account page - found in the top right of the page or at https://subscription.packtpub.com/my-account/subscription - From here you will see the ‘cancel subscription’ button in the grey box with your subscription information in.

What are credits? Chevron down icon Chevron up icon

Credits can be earned from reading 40 section of any title within the payment cycle - a month starting from the day of subscription payment. You also earn a Credit every month if you subscribe to our annual or 18 month plans. Credits can be used to buy books DRM free, the same way that you would pay for a book. Your credits can be found in the subscription homepage - subscription.packtpub.com - clicking on ‘the my’ library dropdown and selecting ‘credits’.

What happens if an Early Access Course is cancelled? Chevron down icon Chevron up icon

Projects are rarely cancelled, but sometimes it's unavoidable. If an Early Access course is cancelled or excessively delayed, you can exchange your purchase for another course. For further details, please contact us here.

Where can I send feedback about an Early Access title? Chevron down icon Chevron up icon

If you have any feedback about the product you're reading, or Early Access in general, then please fill out a contact form here and we'll make sure the feedback gets to the right team. 

Can I download the code files for Early Access titles? Chevron down icon Chevron up icon

We try to ensure that all books in Early Access have code available to use, download, and fork on GitHub. This helps us be more agile in the development of the book, and helps keep the often changing code base of new versions and new technologies as up to date as possible. Unfortunately, however, there will be rare cases when it is not possible for us to have downloadable code samples available until publication.

When we publish the book, the code files will also be available to download from the Packt website.

How accurate is the publication date? Chevron down icon Chevron up icon

The publication date is as accurate as we can be at any point in the project. Unfortunately, delays can happen. Often those delays are out of our control, such as changes to the technology code base or delays in the tech release. We do our best to give you an accurate estimate of the publication date at any given time, and as more chapters are delivered, the more accurate the delivery date will become.

How will I know when new chapters are ready? Chevron down icon Chevron up icon

We'll let you know every time there has been an update to a course that you've bought in Early Access. You'll get an email to let you know there has been a new chapter, or a change to a previous chapter. The new chapters are automatically added to your account, so you can also check back there any time you're ready and download or read them online.

I am a Packt subscriber, do I get Early Access? Chevron down icon Chevron up icon

Yes, all Early Access content is fully available through your subscription. You will need to have a paid for or active trial subscription in order to access all titles.

How is Early Access delivered? Chevron down icon Chevron up icon

Early Access is currently only available as a PDF or through our online reader. As we make changes or add new chapters, the files in your Packt account will be updated so you can download them again or view them online immediately.

How do I buy Early Access content? Chevron down icon Chevron up icon

Early Access is a way of us getting our content to you quicker, but the method of buying the Early Access course is still the same. Just find the course you want to buy, go through the check-out steps, and you’ll get a confirmation email from us with information and a link to the relevant Early Access courses.

What is Early Access? Chevron down icon Chevron up icon

Keeping up to date with the latest technology is difficult; new versions, new frameworks, new techniques. This feature gives you a head-start to our content, as it's being created. With Early Access you'll receive each chapter as it's written, and get regular updates throughout the product's development, as well as the final course as soon as it's ready.We created Early Access as a means of giving you the information you need, as soon as it's available. As we go through the process of developing a course, 99% of it can be ready but we can't publish until that last 1% falls in to place. Early Access helps to unlock the potential of our content early, to help you start your learning when you need it most. You not only get access to every chapter as it's delivered, edited, and updated, but you'll also get the finalized, DRM-free product to download in any format you want when it's published. As a member of Packt, you'll also be eligible for our exclusive offers, including a free course every day, and discounts on new and popular titles.