Chapter 7: Using Logstash to Extract, Transform, and Load Data
In previous chapters, one of our areas of focus was looking at how data can be indexed and searched on Elasticsearch. We looked at index mappings and the importance of defining correct mappings in downstream use cases such as computing aggregations, running alerting, and using machine learning features.
In this chapter, we look at how ETL tools such as Logstash can be used to extract data from a range of source systems (such as Syslog streams, CSV files, message-streaming platforms, or Beats agents), and transform events to their desired format before loading them into Elasticsearch. Upon completion of this chapter, you will be able to use Logstash to process and ingest a variety of data sources into Elasticsearch.
In this chapter, we will specifically focus on the following:
- Understanding the internals of Logstash and the anatomy of a Logstash pipeline
- Exploring common input, filter, and output plugins...
