Chapter 9: Centralizing Logs for Analysis

Log analysis is a critical piece of any infrastructure. The log is the first place that we usually start debugging. In a dynamic environment such as the cloud, sometimes log management becomes expensive as, because of the dynamic nature of the cloud, instances can come and go at any time if placed under an autoscaling group. The other factor that we need to bear in mind is the storage cost as these instances produce large files, which will increase your storage cost.

This chapter will start by looking at how to set up the CloudWatch agent, a centralized place to store all the logs. As we are dealing with a large amount of data in the cloud environment, we need someplace where we can store it for a quicker search, and for that purpose, we will use Amazon Elasticsearch. Finally, we will need some visualization tools to view that data, and for this we will use Kibana.

In this chapter, we are going to cover the following main topics:

...