The standard Zabbix data flow is composed of several actors that send data to our Zabbix server. Of all the sources that can send data to our Zabbix server, we can identify three main data sources:

The other agents represented in the next diagram can be of two main types:

As the diagram displays the data that gets acquired from many different sources in the form of items. At the end of the diagram, you see the GUI, which practically represents the users connected and the database that is the place where all the values are stored.

In the next section, we will dive deep into the Zabbix proxies' data flow.

Zabbix proxies can operate in two different modes, active and passive. The default setup is the active proxy. In this setup, the proxy initiates all connections to the Zabbix server, the one used to retrieve configuration information on monitored objects, and the connection to send measurements back to the server. Here, you can change and tweak the frequency of these two activities by setting the following variables in the proxy configuration file: /etc/zabbix/zabbix_proxy.conf:

ConfigFrequency=3600
DataSenderFrequency=1

Values are expressed in seconds. On the Zabbix server-side, you need to carefully set the value of StartTrappers=.

This value needs to be greater than the number of all active proxies and nodes you deployed. The trapper processes, indeed, manage all the incoming information from the proxies.

On the proxy side, another parameter to consider is:

HeartbeatFrequency

This parameter sets a sort of keep alive, which after the defined number of seconds, will contact the server although it doesn't have any data to send. The proxy availability can be easily checked with the following item:

zabbix[proxy, "proxy unique name", lastaccess]

Here the proxy unique name, of course, is the identifier you assigned to the proxy during deployment. The item will return the number of seconds as the last time that the proxy was contacted, a value you can then use with the appropriate triggering functions.

In the following diagram, you can see the communication flow between the Zabbix server and the proxy:

As you can see from the diagram, the server will wait to receive requests from the proxy and nothing more.

On the other side, proxies are usually deployed to monitor secure network segments with strict outgoing traffic policies, and are usually installed on DMZs. In these kind of scenarios, normally, it is very difficult to obtain permission for the proxy to initiate the communication with the server. Unfortunately, it's not just due to policies. DMZs are isolated as much as possible from internal networks, as they need to be as secure as they can. Generally, it's often easier and more accepted from a security point of view to initiate a connection from the internal network to a DMZ. In this kind of scenario, the passive proxy is very helpful. The passive proxy is almost a mirrored image of the active proxy setup, as you can see in the following diagram:

With this configuration, the Zabbix server will contact the proxy periodically to deliver the configuration changes and to request the item values the proxy is holding.

This is the proxy configuration to enable the proxy you need to set:

ProxyMode=1

This parameter specifies the passive proxy, you don't need to do anything else. Now, on the server side, you need to set the following parameters:

The item used to check a passive proxy's availability is as follows:

zabbix[proxy, "proxy unique name", lastaccess]

This is exactly the same as the active one.

The passive proxy enables us to gather monitoring data from otherwise closed and locked down networks with a slightly increased overhead.

If you would like to keep track of all the remaining items that the proxy needs to send, you can set up the proxy to run this query against its database:

SELECT ((SELECT MAX(proxy_history.id) FROM proxy_history)-nextid) FROM ids WHERE field_name='history_lastid'

This query will return the number of items that the proxy still needs to send to the Zabbix server. Considering that you are using MySQL as a database, you need to add the following user parameter in the proxy agent configuration file:

UserParameter=zabbix.proxy.items.sync.remaining, mysql -u <your dbname here> -p '<your password here>' -e 'SELECT ((SELECT MAX(proxy_history.id) FROM proxy_history)-nextid) FROM ids WHERE field_name=history_lastid' 2>&1

Now, all you need to do is set an item on the Zabbix server side and you can see how your proxy is freeing its queue.

Zabbix, like all the other software, can be installed in two ways:

Actually, there is another way to have a Zabbix server up and running: using the virtual appliance. The Zabbix server appliance will not be considered in this book as Zabbix itself defines this virtual appliance as not ready for productive environments. This virtual appliance is not a production ready setup for many reasons:

On the other hand, the installation from packages gives us some benefits:

The source code compilation also gives us some benefits:

It's quite usual to have different versions of Linux, Unix, and Microsoft Windows on a large environment. This kind of scenario is quite diffused on a heterogeneous infrastructure, and if we use the Zabbix's agent distribution package on each Linux server, we will have different versions of the agent for sure, and different locations for the configuration files.

The more the things are standardized across our server, the easier it will become to maintain and upgrade the infrastructure. The --enable-static option gives us a way to standardize the agent across different Linux versions and release, which is a strong benefit. The agent, statically compiled, can be easily deployed everywhere and, for sure, we will have the same location (and we can use the same configuration file apart from the node name) for the agent and his/her configuration file. The only thing that might vary is the start/stop script and how to register it on the right init runlevel, but at least the deployment will be standardized.

The same kind of concept can be applied to the commercial Unix, bearing in mind to compile it on the target environment so that the same agent can be deployed on different Unix releases of the same vendor.

$ rpm -Uvh http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm
Retrieving http://repo.zabbix.com/zabbix/2.4/rhel/6/x86_64/zabbix-release-2.4-1.el6.noarch.rpm
warning: /var/tmp/rpm-tmp.dsDB6k: Header V4 DSA/SHA1 Signature, key ID 79ea5ed4: NOKEY
Preparing…         ########################################### [100%]
1:zabbix-release   ########################################### [100%]

$ yum install zabbix-server-mysql zabbix-agent zabbix-javagateway

$ chkconfig --level 345 zabbix-server on
$ chkconfig --level 345 zabbix-agent on

$ chkconfig --list | grep zabbix
zabbix-agent    0:off   1:off   2:off   3:on     4:on   5:on   6:off
zabbix-server   0:off   1:off   2:off   3:on   4:on   5:on   6:off

rpm-build yum install

mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}

$ checkinstall --nodoc --install=yes -y

******************************************************************
 Done. The new package has been saved to
 /root/rpmbuild/RPMS/i386/zabbix-2.4.0-1.i386.rpm
 You can install it in your system anytime using:
      rpm -i zabbix-2*.4.0-1.i386.rpm
******************************************************************

/etc/zabbix/zabbix_server.conf

/etc/zabbix/

/usr/lib/zabbix/externalscripts

/usr/lib/zabbix/alertscripts

The database we will use on this book, as already explained, is MySQL.

Now, considering that you have a Red Hat server, the procedure to install MySQL from the RPM repository is quite easy:

$ yum install mysql mysql-server

Now, you need to set up the MySQL service to start automatically when the system boots:

$ chkconfig --levels 235 mysqld on
$ /etc/init.d/mysqld start

To set a password for the root, you can run these two commands:

/usr/bin/mysqladmin -u root password 'new-password'
/usr/bin/mysqladmin -u root -h hostname-of-your.zabbix.db password 'new-password'

Alternatively, you can run:

/usr/bin/mysql_secure_installation

This will also help you to remove the test databases and anonymous user data that was created by default. This is strongly recommended for production servers.

Now, it's time to create the Zabbix database. For this, we can use the following commands:

$ mysql -u root -p
$ mysql> CREATE DATABASE zabbix CHARACTER SET UTF8;
Query OK, 1 row affected (0.00 sec)
$ mysql> GRANT ALL PRIVILEGES on zabbix.* to 'zabbixuser'@'localhost' IDENTIFIED BY 'zabbixpassword';
Query OK, 0 rows affected (0.00 sec)
$ mysql> FLUSH PRIVILEGES;
$ mysql> quit

Next, we need to restore the default Zabbix MySQL database files:

$ mysql -u zabbixuser -pzabbixpassword zabbix< /usr/share/doc/zabbix-server-mysql-2.4.0/create/schema.sql
$ mysql -u zabbixuser -pzabbixpassword zabbix < /usr/share/doc/zabbix-server-mysql-2.4.0/create/images.sql
$ mysql -u zabbixuser -pzabbixpassword zabbix < /usr/share/doc/zabbix-server-mysql-2.4.0/create/data.sql

Now, our database is ready. Before we begin to play with the database, it's important to do some consideration about database size and heavy tasks against it.

DELIMITER $$
CREATE PROCEDURE `partition_maintenance`(SCHEMA_NAME VARCHAR(32), TABLE_NAME VARCHAR(32), KEEP_DATA_DAYS INT, HOURLY_INTERVAL INT, CREATE_NEXT_INTERVALS INT)
BEGIN
  DECLARE OLDER_THAN_PARTITION_DATE VARCHAR(16);
  DECLARE PARTITION_NAME VARCHAR(16);
  DECLARE LESS_THAN_TIMESTAMP INT;
  DECLARE CUR_TIME INT;

        CALL partition_verify(SCHEMA_NAME, TABLE_NAME, HOURLY_INTERVAL);
        SET CUR_TIME = UNIX_TIMESTAMP(DATE_FORMAT(NOW(), '%Y-%m-%d 00:00:00'));
        IF DATE(NOW()) = '2014-04-01' THEN
          SET CUR_TIME = UNIX_TIMESTAMP(DATE_FORMAT(DATE_ADD(NOW(), INTERVAL 1 DAY), '%Y-%m-%d 00:00:00'));
        END IF;
        SET @__interval = 1;
        create_loop: LOOP
          IF @__interval > CREATE_NEXT_INTERVALS THEN
            LEAVE create_loop;
          END IF;
        SET LESS_THAN_TIMESTAMP = CUR_TIME + (HOURLY_INTERVAL * @__interval * 3600);
        SET PARTITION_NAME = FROM_UNIXTIME(CUR_TIME + HOURLY_INTERVAL * (@__interval - 1) * 3600, 'p%Y%m%d%H00');

                CALL partition_create(SCHEMA_NAME, TABLE_NAME, PARTITION_NAME, LESS_THAN_TIMESTAMP);
                SET @__interval=@__interval+1;
        END LOOP;
        SET OLDER_THAN_PARTITION_DATE=DATE_FORMAT(DATE_SUB(NOW(), INTERVAL KEEP_DATA_DAYS DAY), '%Y%m%d0000');

        CALL partition_drop(SCHEMA_NAME, TABLE_NAME, OLDER_THAN_PARTITION_DATE); 
END$$
DELIMITER ;

CALL partition_maintenance('<zabbix_db_name>', '<table_name>', <days_to_keep_data>, <hourly_interval>, <num_future_intervals_to_create>)

mysql> Alter table history_text drop primary key, add index (id), drop index history_text_2, add index history_text_2 (itemid, id);
Query OK, 0 rows affected (0.49 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> Alter table history_log drop primary key, add index (id), drop index history_log_2, add index history_log_2 (itemid, id);
Query OK, 0 rows affected (2.71 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql -h <zabbix_db_host> -u<zabbixuser> -p<zabbixpassword> zabbixdatabase -e "CALL partition_maintenance_all('zabbix');" 

Installation of the Zabbix proxy from packages is a quite simple task. Once you've added the Zabbix repository, you only need to run the following command:

$ yum install zabbix-proxy-mysql

This will install the required packages:

Installation:
 zabbix-proxy-mysql  x86_64  2.4.0-1.el6  zabbix  390 k
Installing for dependencies:
zabbix-proxy  x86_64  2.4.0-1.el6  zabbix  21 k

The Zabbix proxy installation is quite similar to the server one. Once you've installed the server, you need to install MySQL, create the database, and import the DB schema:

$ mysql -u root -p
$ mysql> CREATE DATABASE zabbix CHARACTER SET UTF8;
Query OK, 1 row affected (0.00 sec)
$ mysql> GRANT ALL PRIVILEGES on zabbix.* to 'zabbixuser'@'localhost' IDENTIFIED BY 'zabbixpassword';
Query OK, 0 rows affected (0.00 sec)
$ mysql> FLUSH PRIVILEGES;
$ mysql> quit

Next, we need to restore the default Zabbix MySQL database files:

$ mysql -u zabbixuser -pzabbixpassword zabbix < /usr/share/doc/zabbix-proxy-mysql-2.4.0/create/schema.sql

Now, we need to start the database, configure the proxy, and start the service. In this example, we have considered to use a Zabbix proxy that relies on a MySQL with InnoDB database. This proxy can be performed in two different ways:

Here, we have chosen the second option. In a large network environment where the proxy, in case of issue, needs to preserve all the metrics acquired until the server acquires the metrics, it's better to reduce, at the minimum, the risk of data loss. Also, if you consider this scenario in a large network environment, you most likely will have thousands of subnetworks connected to the Zabbix server with all the possible network devices in-between. Well, exactly, this is necessary to use a database that can prevent any data corruptions.

The WebGUI interface will be done once more using the RPMs.

To install the web interface, you need to run the following command:

$ yum install zabbix-web-mysql

Yum will take care to resolve all the dependencies. Once you're done, the process of this component is quite easy: we need to open a web browser, point at the following URL: http://your-web-server/zabbix, and follow the instructions.

On the standard Red Hat system, you simply need to change these parameters on your /etc/php.ini file:

php_value max_execution_time 300
php_value memory_limit 128M
php_value post_max_size 16M
php_value upload_max_filesize 2M
php_value max_input_time 300

Also, set your time zone on the same file (for example, php_value date.timezone Europe/Rome).

Now, it's time to start up Apache, but before this, we need to check whether we have SELinux enabled and on which mode? To check your SELinux status, you can run:

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   permissive
Mode from config file:          permissive
Policy version:                 24
Policy from config file:        targeted

Now, you need to check whether you have the httpd daemon enabled to use the network with the following command:

# getsebool httpd_can_network_connect
httpd_can_network_connect --> off

Most likely, you will have the same kind of result, then all we need to do is enable the httpd_can_network_connect option using the next command with –P to preserve the value after a reboot:

# setsebool –P httpd_can_network_connect on
# getsebool httpd_can_network_connect
httpd_can_network_connect --> on 

Now, all that we still have to do is enable the httpd daemon and start our httpd server:

# service httpd start
Starting httpd:                              [  OK  ]

Next, enable the httpd server as a service:

# chkconfig httpd on

We can check the change done with the next command:

# chkconfig --list httpd
httpd       0:off   1:off   2:on    3:on    4:on    5:on    6:off

Once you've done this, you only need to follow the wizard, and in a few clicks, you will have your web interface ready to start up.

Now, you can finally start your Zabbix server and the first entry in the /var/log/zabbix/zabbix_server.log file will look something like the following code:

37909:20140925:091128.868 Starting Zabbix Server. Zabbix 2.4.0 (revision 48953).
37909:20140925:091128.868 ****** Enabled features ******
 37909:20140925:091128.868 SNMP monitoring:           YES
 37909:20140925:091128.868 IPMI monitoring:           YES
 37909:20140925:091128.868 WEB monitoring:            YES
 37909:20140925:091128.868 VMware monitoring:         YES
 37909:20140925:091128.868 Jabber notifications:      YES
 37909:20140925:091128.868 Ez Texting notifications:  YES
 37909:20140925:091128.868 ODBC:                      YES
 37909:20140925:091128.868 SSH2 support:              YES
 37909:20140925:091128.868 IPv6 support:              YES
 37909:20140925:091128.868 ****************************** 37909:20140925:091128.868 using configuration file: /etc/zabbix/zabbix_server.conf******************************

Next, you can start to implement and acquire all the items critical for your network.

In this chapter, we covered a large number of components. We started with defining what a large environment is. We also saw how the network setup can be designed and how it can evolve within your infrastructure. We saw the heaviest task on the server side (housekeeping) and how to avoid performance degradation due to this. We discussed MySQL partitioning in-depth. We also briefly discussed the differences between active and passive proxies; you will now be able to decide how to set them up and which one to choose once you know your network topology. Also, we saw how to acquire some critical metrics to monitor the Zabbix proxy connection and the amount of items that it still needs to send us.

As you can see, we covered a lot of arguments in just one chapter; we did this because we would like to use more space in the upcoming chapters. In the next chapter, we will explore the different appliances and protocols at layer 2 and layer 3 of the ISO/OSI stack. Also, you will see how to best extrapolate meaningful monitoring data from the collected measure for the protocol layers 2 and 3.