Introduction 

I had my share of problems trying to get an easily maintainable Icinga setup up and running - mostly because the new Icinga2 documentation is absolutely horrendous. I will now describe my own, simplistic, but very effective and scaleable setup. As usual my distribution is Debian and you have to account for potential differences in your own distribution yourself. I originally heard about this concept from a fellow student. He also has a lot of other interesting stuff and you should definitely check out his website.

If you just want the code go to my GitHub.

Package Requirements 

Be careful nsca and nsca-ng are NOT compatible.

On the master server:

apt install icinga2
apt install nsca-ng-server

On the nodes:

apt install nsca-ng-client

It you may also want to install nagios-plugins on the nodes, as it contains many helpful monitoring scripts you can use as a starting point in /usr/lib/nagios.

Master Server 

NSCA-Server 

In /etc/nsca/nsca.cfg specify the named pipeline that is used to write incoming messages to icinga and the user as which to run. The command file should already be specified in /etc/icinga2/icinga2.conf by default.

command_file = "/run/icinga2/cmd/icinga2.cmd"
user = "nagios"

Then add the host(s) from which to receive status messages. You can restrict for which services a host can submit results within the { } separated by commata, but generally that is not necessary for a simple use-case, so we just write * to allow all. This does not mean that the remote node can submit a service for another host.

authorize "example" {
    password = "PASSWORD"
    host     = "example.com"
    services = {
        "*",
    }
}

Icinga 

You have to add each host in the /etc/icinga2/ host.conf. You should have a group or a variable that identifies the host as a node, in case you ever want to have non-node servers (aka non-passive-checks).

object Host "example"{
    import 
    name = "example.com"
    address = "example.com"

    # mark as node
    var.is_node = "true"

    # default check intervals
    max_check_attempts = 7
    retry_interval = 1m     

    # add this if you want mail notifications
    vars.notification["mail"] = {
        groups = [ "icingaadmins" ]
    }
}

Then we create a template (passive check) service in services.conf and the checks themselves. Obviously you don’t need a template but it makes things shorter. The services themselves can override the check-/retry intervals if necessary. Other than that the services themselves must have the same names as the checks in the monitoring.conf on the remote server.

Template:

template Service "remote_passive" {
    import "generic-service"
    check_interval = 10m
    retry_interval = 1m
    check_command = passive
}

Service:

apply Service "serivce-name" {
    import "remote_passive"
    assign where host.var.is_node
}

It should be mentioned that you do not need to use different files for the your configuration. Every configuration file is simply included by the main icinga configuration /etc/icinga2/icinga2.conf. For (mail) notifications you need a working mail setup which is outside the scope of this article. A starting point for that should be the official icinga documentation.

Node Server 

NSCA 

On the node server you first need to configure the nsca-ng-client in /etc/send_nsca.cfg:

server   = "example.com"
identity = "example"
password = "PASSWORD"

Checks/Python-script/monitoring.conf 

Then you may simply use my python script with a configuration file that looks like this:

user<TAB>service-name<TAB>command-to-execute-with-args

For example:

nobody  test-apt /usr/lib/nagios/plugins/check_apt -o

And start it like this:

/etc/monitoring/monitoring-report.py -c /PATH/TO/monitoring.conf

Finally add this line to your crontab and you are done. Happy becoming sad because you now notice how often your shit breaks.


Feel free to send me a mail to share your thoughts!