Alerts and troubleshooting

Having just discussed logs we now have to consider the highly related concept of system alerting. I have to mention that of course logging systems themselves are also a potential source of alerts. If we use automation in our logging systems, that automation will generally be expected to either send alerts directly, or add alerts to an alerting system.

Alerts are, fundamentally, a way for our monitoring systems to reach out and tell us humans that they are in trouble and it is time for us to step in and work our human-intelligence magic. While we hope that our systems will have automation and can repair many problems themselves, the reality is that for the foreseeable future nearly all companies will have to keep working in a reality where human intervention is needed on a regular basis in systems administration. Whether it is to log in and clear a full disk or stop a broken process or identify a corrupt file or even to trigger a failover to a different...