I heard about Sensu at my job and I thought I give it a try, see what it can do, how easy it can be installed and configured into a working monitoring tool.

Before we start, let me share a few things about my OS:

nano /etc/apt/sources.list
deb http://httpredir.debian.org/debian/ wheezy main contrib non-free
deb-src http://httpredir.debian.org/debian/ wheezy main contrib non-free
deb http://security.debian.org/ wheezy/updates main contrib non-free
deb-src http://security.debian.org/ wheezy/updates main contrib non-free
deb http://httpredir.debian.org/debian wheezy-updates main contrib non-free
deb-src http://httpredir.debian.org/debian wheezy-updates main contrib non-free
deb http://ftp.debian.org/debian wheezy-backports main

Everything else is commented

nano /etc/apt/sources.list.d/erlang-solutions.list
deb http://binaries.erlang-solutions.com/debian wheezy contrib

lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 7.11 (wheezy)
Release: 7.11
Codename: wheezy

python --version
Python 2.7.3

 

1) Prerequisites:

You need to install a Data Store (Redis) and a Transport – message bus for communication (Rabbit).

apt-get update
apt-get -y install redis-server
redis-cli ping

After installing Redis Server you can confirm that Redis is ready to use by running the command:
redis-cli ping
If you get a PONG response, you are OK.
Now, let’s add the config file.

mkdir -p /etc/sensu/conf.d/
nano /etc/sensu/conf.d/redis.json
{
 "redis": {
 "host": "127.0.0.1",
 "port": 6379
 }
}

Go to /usr/local/src and start downloading all components. I am installing this way because adding Repositories and installing from there got me into dependencies issues.

cd /usr/local/src
wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb
dpkg -i erlang-solutions_1.0_all.deb
apt-get update
apt-get install esl-erlang
apt-get install socat
wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.2/rabbitmq-server_3.6.2-1_all.deb
dpkg -i rabbitmq-server_3.6.2-1_all.deb
update-rc.d rabbitmq-server defaults

Create a RabbitMQ virtual host and user for Sensu. Change the password !!!. You’ll need this password later when you configure the Sensu server and the clients to be monitored.

rabbitmqctl add_vhost /sensu
rabbitmqctl add_user sensu password
rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Create the RabbitMQ file (don’t forget to use the changed password)

nano /etc/sensu/conf.d/rabbitmq.json
{
 "rabbitmq": {
 "host": "127.0.0.1",
 "port": 5672,
 "vhost": "/sensu",
 "user": "sensu",
 "password": "password"
 }
}

 

2) Install and configure Sensu master

cd /usr/local/src/
wget https://sensu.global.ssl.fastly.net/apt/pool/sensu/main/s/sensu/sensu_0.25.3-1_amd64.deb
dpkg -i sensu_0.25.3-1_amd64.deb


Side Note:

The same commands will be used to install sensu-client on a Linux server that needs to be monitored. There is no package just for sensu client. They all come together in one package (server, client, api). On your Linux server that needs to be monitored by Sensu, there is no need to have also the server and api started, so, just disable sensu-server and sensu-api services from starting at boot with:

update-rc.d sensu-server disable
update-rc.d sensu-api disable

 

I had some problems with the order in which the sensu processes were starting, so I did the following: I disabled all sensu-services from starting at boot, and then I added them in /etc/rc.local in the order I wanted:

update-rc.d sensu-server disable
update-rc.d sensu-api disable
update-rc.d sensu-client disable

nano /etc/rc.local
service sensu-server restart
service sensu-client restart
service sensu-api restart
service uchiwa restart

Now, getting back to the task at hand, we just installed the Sensu-Server and we need to start configuring it. So, create the client file which is how you declare a new client to Sensu.

nano /etc/sensu/conf.d/client.json
{
 "client": {
 "name": "Sensu-Server",
 "address": "127.0.0.1",
 "environment": "production",
 "subscriptions": [
 "production",
 "debian" ,
 "ALL"
 ],
 "socket": {
 "bind": "127.0.0.1",
 "port": 3030
 }
 }
}

Create the transport file

nano /etc/sensu/conf.d/transport.json
{
 "transport": {
 "name": "rabbitmq",
 "reconnect_on_error": true
 }
}

Create the API file

nano /etc/sensu/conf.d/api.json
{
 "api": {
 "host": "localhost",
 "bind": "0.0.0.0",
 "port": 4567
 }
}

Create the Uchiwa file. You can declare your own user, pass and port. After that, you access it from browser http://<sensu-ip-here>:3000/

nano /etc/sensu/uchiwa.json
{
 "sensu": [
 {
 "name": "Sensu",
 "host": "127.0.0.1",
 "ssl": false,
 "port": 4567,
 "path": "",
 "timeout": 100
 }
 ],
 "uchiwa": {
 "stats": 10,
 "port": 3000,
 "user": "admin",
 "pass": "adminpass",
 "refresh": 10
 }
}

Restart all services and you should be done, Sensu is installed, you have 1 client (you are monitoring the Sensu server itself)

service sensu-server restart
service sensu-client restart
service sensu-api restart

3.1) Plugins and checks – Nagios Plugins

Ok, so now you need to start adding plugins and checks for your client(s). You only have 1 client for the moment, but that will change. I have seen people asking on forums if you could use Nagios plugins with Sensu. Yes you can! Nagios plugins are available at https://www.nagios.org/downloads/nagios-plugins/

apt-get install nagios-plugins

 

They might not be the latest version, but they will do the work you need. If you really really need the latest version, just download them from the above link and install them manually, otherwise just use apt-get to install them. Installing Nagios plugins with apt-get will put them in this location: /usr/lib/nagios/plugins/. Just do a ls -l /usr/lib/nagios/plugins/ and you will see all of them. Make sure they have +x so Sensu can execute them.

One thing worth mentioning when using Nagios plugins, is that you will see the service output together with the Nagios format performance data. This only concerns you, the human looking at the output, so if you can get past that it’s OK.

To show you what I mean, here is the output for a service check verifying the disk space on root partition (/ partition):  DISK OK – free space: / 15347 MB (84% inode=90%);| /=2740MB;19025;19035;0;19055

The blue part, is the output you need to see how much space you have on /.

The red part is performance data meant for PNP4Nagios. So, if you can look at that output and just ignore the red part, you can are OK. If you are planning to have graphs for Sensu, I have another thing that needs to be mentioned: you will not be able to use the red part (PNP4Nagios style metrics) for Graphite. I only tried Graphite as a graphing tool for Sensu, and it requires a different format for the metrics.

So why use Nagios Plugins in the first place? Because they are easy to use, and you can use them in parallel with other plugins/scripts that give you the metrics Graphite is expecting.

So, let’s start creating service checks for your Sensu Server.
Create a service check checking for number of apache processes – critical if number of processes is less than 1 or more than 30.

cd /etc/sensu/conf.d/
nano check_apache_procs.json
{
 "checks": {
 "check_apache_procs": {
 "command": "/usr/lib/nagios/plugins/check_procs -C apache2 -c 1:30",
 "interval": 60,
 "subscribers": [ "debian" ],
 "standalone": false,
 "timeout": 30,
 "ttl": 300
 }
 }
}

Check your / partition for free space – warning at 30% free space, critical at 20% free space

nano check_disk.json
{
 "checks": {
 "check_disk_root": {
 "command": "/usr/lib/nagios/plugins/check_disk -w 30 -c 20 -p /",
 "interval": 60,
 "subscribers": [ "debian" ],
 "standalone": false,
 "timeout": 30,
 "ttl": 300
 }
 }
}

Check the SWAP

nano check_swap.json
{
 "checks": {
 "check_swap": {
 "command": "/usr/lib/nagios/plugins/check_swap -w 30 -c 20",
 "interval": 60,
 "subscribers": [ "debian" ],
 "standalone": false,
 "timeout": 30,
 "ttl": 300
 }
 }
}

Check CPU Load – warning if CPU Load reaches 3,3,3 and critical if 4,4,4 or more (1 min, 5 min, 15 min values)

nano check_linux_load.json
{
"checks": {
"check_linux_load": {
"command": "/usr/lib/nagios/plugins/check_load -w 3,3,3 -c 4,4,4",
"interval": 60,
"subscribers": [ "debian" ],
"standalone": false,
"timeout": 30,
"ttl": 300
}
}
}

Then, restart sensu for the changes to take effect

service sensu-server restart
service sensu-client restart

3.2) Plugins and checks – Sensu Plugins

You can choose to go with Nagios Plugins, with Sensu Plugins or with both of them.

Here is how to install Sensu plugins.
Install sensu-plugin, because it will probably be a prerequisite for other plugins

/opt/sensu/embedded/bin/gem install sensu-plugin --no-ri --no-rdoc

After that, start installing what ever plugins you want from here:

 

https://github.com/sensu-plugins

http://sensu-plugins.io/plugins/

they all install in the same way.

/opt/sensu/embedded/bin/gem install sensu-plugins-redis
/opt/sensu/embedded/bin/gem install sensu-plugins-disk-checks
/opt/sensu/embedded/bin/gem install sensu-plugins-uchiwa
/opt/sensu/embedded/bin/gem install sensu-plugins-memory-checks
/opt/sensu/embedded/bin/gem install sensu-plugins-cpu-checks

If your plugins require a greater version of ruby, you need to install it also. If your Repo has an accepted version, go ahead and install it with apt-get. I was not able to do that, so I installed RVM.

cd /usr/local/src/
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
\curl -L https://get.rvm.io | bash -s stable --rails

RVM will take a while to install, grab some popcorn :). When the installation is done, add all users (root, sensu, uchiwa, etc) that will be using this greater version of ruby to /etc/group, under rvm group. After that execute this:

source /usr/local/rvm/scripts/rvm

make sure that /etc/default/sensu says the embedded_ruby=true
Now let’s add some checks using Sensu plugins

cd /etc/sensu/conf.d
nano check_cpu.json
{
 "checks": {
 "check_cpu": {
 "command": "check-cpu.rb -w 80 -c 90",
 "interval": 60,
 "subscribers": [ "debian" ],
 "standalone": false,
 "timeout": 30,
 "ttl": 300
 }
 }
}


nano check_mem_percent.json
{
 "checks": {
 "check_mem_percent": {
 "command": "check-memory-percent.rb -w 80 -c 90",
 "interval": 60,
 "subscribers": [ "debian" ],
 "standalone": false,
 "timeout": 30,
 "ttl": 300
 }
 }
}

Time to give your server a reboot

reboot

4) Windows Clients

Time to add some Windows clients to your Sensu. Get the Sensu Windows client from here: https://sensuapp.org/docs/latest/platforms/sensu-on-microsoft-windows.html and install it.

Double-click the msi installer package to launch the installer, accept the Sensu Core MIT License and install Sensu using the default settings (e.g. install location, etc).
WARNING: changing the default installation path from C:\opt is strongly discouraged. 

Configure the Sensu client. No “default” configuration is provided with Sensu, so the Sensu Client will not start without the corresponding configuration. In some cases, the default Sensu configuration directory (i.e. C:\opt\sensu\conf.d\) is not created by the Sensu MSI installer, in which case it is necessary to create this directory manually.

Let’s create the client file in C:\opt\sensu\conf.d\client.json

{
 "client": {
 "name": "Windows-Server-Name",
 "address": "192.168.1.2",
 "environment": "production",
 "subscriptions": [
 "dev",
 "windows",
 "ALL"
 ],
 "socket": {
 "bind": "127.0.0.1",
 "port": 3030
 }
 }
}

create the client file in C:\opt\sensu\conf.d\transport.json

{
 "transport": {
 "name": "rabbitmq",
 "reconnect_on_error": true
 }
}

create the client file in C:\opt\sensu\conf.d\rabbitmq.json. Make sure you add the correct IP address of your RabbitMQ Server – in our scenario, we installed it on the same server as the Sensu-Server. Also, make sure you put the correct password, the one you configured in  Chaper 1 of this post.

{
 "rabbitmq": {
 "host": "ip-address-here",
 "port": 5672,
 "vhost": "/sensu",
 "user": "sensu",
 "password": "password"
 }
}

Now, edit the “C:\opt\sensu\bin\sensu-client.xml” and make sure it looks like this:

<!--
 Windows service definition for Sensu
-->
<service>
 <id>sensu-client</id>
 <name>Sensu Client</name>
 <description>This service runs a Sensu client</description>
 <executable>C:\opt\sensu\embedded\bin\ruby</executable>
 <arguments>C:\opt\sensu\embedded\bin\sensu-client -d C:\opt\sensu\conf.d</arguments>
</service>

On the argument line, you could add and extra argument “-l C:\opt\sensu\sensu-client.log” but keep in mind that the log file will grow over time, filling up space. It is your choice if you add it or not, I didn’t. You could just add it for debugging purposes, and when everything is working, just remove it and restart SensuClient service.

Obtain the hostname or IP address of the Windows system where the Sensu client is installed. For the purpose of this example, we will assume 10.0.1.100 is our IP address.
Open a Command Prompt and use the Windows SC utility to create the Windows service for the Sensu client:

sc \\10.0.1.100 create sensu-client start= delayed-auto binPath= c:\opt\sensu\bin\sensu-client.exe DisplayName= "Sensu Client"

NOTE: the space between the equals (=) and the values is required.

To manually start and stop the Sensu client Windows service, use the Services.msc (Start -> Run -> services.msc -> ENTER)  utility, or via the Command Prompt (cmd.exe).

sc sensu-client start
sc sensu-client stop

Add to your Windows PATH environment variable the ruby.exe folder, which in my case is C:\opt\sensu\embedded\bin. Should be the same on your Windows machine. After that, open a cmd.exe and type this:

C:\Users\john.doe>cd C:\opt\sensu\embedded\bin
C:\opt\sensu\embedded\bin>gem install sensu-plugin
C:\opt\sensu\embedded\bin>gem install sensu-plugins-windows

You can test if they are working from command line with:

C:\opt\sensu\embedded\bin>ruby "C:\opt\sensu\embedded\bin\check-windows-cpu-load.rb"

Time to add it to Sensu Client
Create “C:\opt\sensu\conf.d\check_win_process.json”

{
"checks": {
 "check_windows_process_chrome": {
 "command": "C:/opt/sensu/embedded/bin/check-windows-process.rb.bat --processname chrome",
 "interval": 60,
 "standalone": true,
 "timeout": 30,
 "ttl": 300
 }
 }
}

Create “C:\opt\sensu\conf.d\check_win_ram.json”

{
"checks": {
 "check_windows_ram_usage2": {
 "command": "C:/opt/sensu/embedded/bin/check-windows-ram.rb.bat -w 80 -c 90",
 "interval": 60,
 "standalone": true,
 "timeout": 30,
 "ttl": 300
 }
 }
}

Create “C:\opt\sensu\conf.d\check_win_service.json”

{
"checks": {
 "check_windows_service_sensuClient": {
 "command": "C:/opt/sensu/embedded/bin/check-windows-service.rb.bat -s sensu-client",
 "interval": 60,
 "standalone": true,
 "timeout": 30,
 "ttl": 300
 }
 }
}

Create “C:\opt\sensu\conf.d\check_windows_cpu5.json”

{
"checks": {
 "check_windows_cpu5": {
 "command": "C:/opt/sensu/embedded/bin/check-windows-cpu-load.rb.bat -w 70 -c 80",
 "interval": 60,
 "standalone": true,
 "timeout": 30,
 "ttl": 300
 }
 }
 }

Create “C:\opt\sensu\conf.d\check_windows_disks.json”

{
"checks": {
 "check_windows_all_disks2": {
 "command": "C:/opt/sensu/embedded/bin/check-windows-disk.rb.bat -w 86 -c 90",
 "interval": 60,
 "standalone": true,
 "timeout": 30,
 "ttl": 300
 }
 }
 }

Restart your Sensu Service from services.msc and you should see your new Windows host in Uchiwa Interface

 

5) Client-less hosts: switches, routers, printers, etc

If you want to monitor a host where you cannot install the sensu-client, you have to add a so-called JIT client (JIT = Just In Time) or proxy clients. Those are dynamically created clients, added to the client registry if a client does not already exist for a check result.

Sensu proxy clients (formerly known as “Just-in-time” or “JIT” clients) are dynamically created clients, added to the client registry if a client does not already exist for a check result.

Proxy client registration differs from keepalive-based registration because the registration event happens while processing a check result (not a keepalive message). If a check result includes asource attribute, a proxy client will be created for the source, and the check result will be stored under the newly created client. Sensu proxy clients allow Sensu to monitor external resources (e.g. on systems and/or devices where a sensu-client cannot be installed, such a network switches), using the defined check source to create a proxy clients for the external resource. Once created, proxy clients work much in the same way as any other Sensu client; e.g. they are used to store check execution history and provide context within event data.

Basically, you just add a check file, not a client file. Go to your Sensu Server and do the following:

cd /etc/sensu/conf.d/
nano printer_ping.json
{
 "checks": {
 "printer_ping": {
 "command": "/usr/lib/nagios/plugins/check_ping -H 10.12.13.14 -w 100.0,20% -c 500.0,60% -p 5",
 "interval": 60,
 "standalone": true,
 "timeout": 30,
 "ttl": 300,
 "source": "entrance_printer"
 }
 }
}

As you can see, you are not subscribing this check to anything, I just issuing a command from my Sensu Server that does something. The ‘source’ attribute, is the hostname you will see in your Uchiwa interface. Therefore, in Uchiwa, you will see a hostname called ‘entrance_printer‘ with one single check called ‘printer_ping‘.

If you have Linux hosts, where you cannot install sensu-client (or are not allowed) and you want to perform your checks via SSH, you can do it the same way.