Sending e-mail notifications with Nagios Core

Why do we need monitoring tools?

Imagine having everyday reports extracting data from a DB smoothly when suddenly an error occurs. It is not a pleasant thought for sure, however, things could get worse if you would not be aware of it. Detecting and fixing issues before they hit, has multiple benefits, for example saving the organization’s time and money, preventing system downtime and the adverse effects, and avoiding unhappy customers. When your monitoring tools are set up to alert you on issues before they impact the customers, you can fix them before they cause you and your customers any trouble. Here is where Nagios Core comes into the game.

Why Nagios Core?

Nagios Core is a popular Linux-based open-source system and network monitoring application with the ability to alert you when a problem occurs, and when it gets resolved. Nagios Core offers basic functionalities for monitoring and managing IT environments. Because of its open-source model, there are many plugins for Nagios and the tool can easily be adapted to your needs. Besides the benefit of having access to the source code for customization and bug fixes, its flexibility is likely responsible for its popularity.

Notification methods are not directly incorporated into the Nagios Core code as it just does not make much sense. The "core" of Nagios Core is not designed to be an all-in-one application. There are a lot of different ways to do notifications and there are already a lot of packages out there that can handle it.

In this article, we will discuss an architecture for sending email notifications from Nagios Core which involves:

• Nagios Core installed on Azure VM - we use it for monitor hosts and services, and raising an alert when something occurs
• Azure Functions for triggering email notifications
• SendGrid for sending emails to end-user

Why SendGrid?

SendGrid is a third-party service that supports sending emails. If you have Nagios Core installed on Azure VM, then the supported way to send emails to external domains from Azure resources is via an SMTP relay service. Customers who create Azure subscriptions after November 15th, 2017, will have technical restrictions, like blocking emails sent from VMs directly to email providers. The reason for this is that you do not have a dedicated IP address, and it is quite possible that spammers will use Azure to send spams. In that case, spam blacklists will quickly flag the IP range of Azure data centers as sources of spam. In other words, your legitimate email will stop getting through.

Since Microsoft cannot guarantee email providers will accept these inbound emails, no requests to remove the restriction can be made. Fortunately, we can use SMTP relay to overcome this problem and we chose SendGrid. The reason we chose SendGrid is that Azure customers can unlock 25,000 free emails each month. Configuring SendGrid is very easy, it provides reliable transactional email delivery, scalability, real-time analytics along with flexible API’s that make custom integration easy, and it has a binding extension for Azure Functions. Also, there is plenty of online documentation and examples.

Why Azure Functions?

So far, we have a monitoring tool that notices a problem when it occurs, and we have a SendGrid for delivering emails to interested users. Now, we need to let SendGrid know when it should send an email and for this purpose, we use Azure Functions.

Azure Functions let you run your code in a serverless environment without having to first create a virtual machine or publish a web application. The idea behind a serverless environment is to delegate the management and maintenance of servers to third parties so that developers can focus exclusively on the business requirements. One of the benefits of a serverless approach is saving the costs as you are only paying for what you use.

Azure Functions are great for processing events, therefore naturally perfect scenario to use Azure Functions is where you have modeled things in terms of events. In our case, we use events to trigger sending an email whenever a problem occurs/resolves. Thus, as a trigger, you will use an HTTP trigger.

From the side of Nagios Core, you can use plugin check http to trigger the Azure Function.

How to setup Nagios Core command to trigger Azure Function

Covering all steps is not in the scope of this post and it would result in a very long article. Apparently, sending JSON parameters from a command line is quite tricky, it looks simple but it can give you a headache. Let’s take a look at that small part.

Prerequisites:

• Installed Nagios Core - If you don’t have Nagios installed, follow this guide.
• Configured hosts, services and contacts
• Azure subscription
• Created SendGrid Account – follow this link to create SendGrid account.
  • Copy API Key, you will need it later.
• Azure Function which will be triggered by HTTP request and send email to SendGrid

We will assume you already have installed and configured Nagios and all services you want to monitor, created SendGrid account, and have made Azure Function with HTTP trigger as input and SendGrid as output.

Steps to follow:

1. Copy Azure Function URL
2. Go to VM where you installed Nagios Core
3. Make sure Nagios Core is running
4. Open command line
5. Navigate to the directory with plugins (usually can be found at this path /usr/local/nagios/libexec)
6. Test function triggering by executing check http command. Change parameters in the following command with your values, it should be something like this:

./check_http -H <azure-function-server> -S -u <copied-url>  --method=POST --post='{"to":"user@example.com", "hostName":"testServer","hostState":"Down", "hostAddress":"127.1.1","hostOutput":"example message", "notificationType":"Warning", "longDateTime":"15.04.2018"}'

1. If you see the following response, it means Azure Function is triggered correctly and you will receive an email:

1. Change commands notify-host-by-email and notify-service-by-email like this:

   define command {
   
       command_name    notify-host-by-email
       command_line    /usr/local/nagios/libexec/check_http  -H <azure-function-server> -S -u <azure-fun-copied-url> --method 'POST' --post "{\"to\":\"$CONTACTEMAIL$\", \"hostName\":\"$HOSTNAME$\",\"hostState\":\"$HOSTSTATE$\", \"hostAddress\":\"$HOSTADDRESS$\",\"hostOutput\":\"$HOSTOUTPUT$\", \"notificationType\":\"$NOTIFICATIONTYPE$\", \"longDateTime\": \"$LONGDATETIME$\"}"
   }
   
   define command {
   
       command_name    notify-service-by-email
       command_line    /usr/local/nagios/libexec/check_http  -H <azure-function-server> -S -u <azure-fun-copied-url> --method 'POST' --post "{\"to\":\"$CONTACTEMAIL$\", \"hostName\":\"$HOSTNAME$\",\"hostState\":\"$HOSTSTATE$\", \"hostAddress\":\"$HOSTADDRESS$\",\"hostOutput\":\"$HOSTOUTPUT$\", \"notificationType\":\"$NOTIFICATIONTYPE$\", \"longDateTime\": \"$LONGDATETIME$\", \"service\":\"$SERVICEDESC$\", \"serviceOutput\":\"$SERVICESTATE$\", \"serviceLondInfo\":\"$SERVICEOUTPUT$\"}"
   }
   

2. Save changes and restart Nagios Core to apply the changes:

sudo systemctl restart nagios.service

3. And that is it. Your emails are ready to be sent.

Conclusion

Monitoring tools will give you a clear picture of how your applications and infrastructure are working. With this information, you can then detect areas that need improvements. Targeting problematic points and resolving them before incidents occur, will not only improve performance but also save your organization valuable resources, especially time and money.

There are plenty of network monitoring tools and utilities out there, including commercial products and open source solutions, which make it difficult to choose the perfect solution for your own infrastructure. Nagios can be integrated with other applications; it can perform tasks handed off to external commands and third-party applications can send control commands and data to Nagios very easily.

Still, configuring Nagios and setting up necessary plugins need time and specific knowledge. Unless you have experience with Nagios Core, make sure you consider if the time spent on configuration and potential bug fix is worth not buying a commercial monitoring tool. With Azure Functions, you only need to focus on the problem and not the resources required for the solution, although you need to have at least basic knowledge in one of the supported languages.