Alerting

Decanter provides an alerting feature. It allows you to check values in harvested data (coming from the collectors) and send alerts when the data is not in the expected state.

Service

The alerting service is the core of Decanter alerting.

It’s configured in etc/org.apache.karaf.decanter.alerting.service.cfg configuration file where you define the alert rules:

rule.my="{'condition':'message:*','level':'ERROR'}"

The rule name has to start with rule. prefix (see rule.my here).

The rule definition is in JSON with the following syntax:

{
    'condition':'QUERY',
    'level':'LEVEL',
    'period':'PERIOD',
    'recoverable':true|false
}

where:

  • condition is a Apache Lucene query (https://lucene.apache.org/core/8_5_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package.description). For instance:

    • message:foo* selects all events with message containing any string starting with foo

    • message:* AND other:* selects all events with message and other containing anything

    • threadCount:[200 TO *] selects all events with threadCount greater than 200

    • counter:[20 TO 100) selects all events with counter between 20 and 100 (included)

    • foo:bar OR foo:bla selects all events with foo containing bar or bla

  • level is a string where you can set whatever you want to define the alert level. By default, it’s WARN.

  • period is optional and allows you to define a validity period for a condition. It means that the condition should match for the period duration and, if so, the alert with be thrown after the period. The period is a string like this:

    • 5SECONDS

    • 10MINUTES

    • 2HOURS

  • recoverable is a flag that define if the alert can be recover or not. By default it’s false. The main difference is the number of alert events you will have. If not recoverable, you will have an alert for each event matching the condition. If recoverable, you will have a single alert the first time an event matches the condition, and another alert (back to normal) when the alert is back (not matching the condition).

You can use any event property in the rule condition. The alert service automatically add:

  • alertUUID is a unique string generated by the alert service

  • alertTimestamp is the alert timestamp added by the alert service

  • decanter.collector.name is the name of the origintating Decanter collector

Alerters

When the value doesn’t verify the check in the checker configuration, an alert is created and sent to the alerters.

Apache Karaf Decanter provides ready to use alerters.

Log

The Decanter Log alerter logs a message for each alert.

The decanter-alerting-log feature installs the log alerter:

karaf@root()> feature:install decanter-alerting-log

This alerter doesn’t need any configuration.

Caution
 Use the Alerter Log with caution when also using the Log Collector, decanter-collector-log, as you can create a loop. Specifically a log event will generate an alert event that will generate a log event…​..
E-mail

The Decanter e-mail alerter sends an e-mail for alerts.

The decanter-alerting-email feature installs the e-mail alerter:

karaf@root()> feature:install decanter-alerting-email

This feature also installs the etc/org.apache.karaf.decanter.alerting.email.cfg configuration file where you can specify the SMTP server and e-mail addresses to use:

#
# Decanter e-mail alerter configuration
#

# From e-mail address
from=

# To e-mail addresses, you can define here a list of recipient separated with comma
# For example: to=mail1@example.org,mail2@example.org,mail3@example.org
to=

# Hostname of the SMTP server
host=smtp.gmail.com

# Port of the SMTP server
port=587

# enable SMTP auth
auth=true

# enable starttls and ssl
starttls=true
ssl=false

# Optionally, username for the SMTP server
#username=

# Optionally, password for the SMTP server
#password=

# e-mail velocity templates
#subject.template=/path/to/subjectTemplate.vm
#body.template=/path/to/bodyTemplate.vm
#body.type=text/plain

  • the from property specifies the from e-mail address (for instance dev@karaf.apache.org)

  • the to property specifies the to e-mail address (for instance dev@karaf.apache.org)

  • the host property specifies the SMTP server hostname or IP address

  • the port property specifies the SMTP server port number

  • the auth property (true or false) specifies if the SMTP server requires authentication (true) or not (false)

  • the starttls property (true or false) specifies if the SMTP server requires STARTTLS (true) or not (false)

  • the ssl property (true or false) specifies if the SMTP server requires SSL (true) or not (false)

  • the username property is optional and specifies the username to connect to the SMTP server

  • the password property is optional and specifies the password to connect to the SMTP server

  • the subject.template property allows you to provide your own Velocity (http://velocity.apache.org) template to create the subject of the message

  • the body.template property allows you to provide your own Velocity (http://velocity.apache.org) template to create and format the body of the message

  • the body.type property allows you to define the message content type, depending if you send HTML or plain text message.

Optionally, you can add:

  • cc to add email carbon copy

  • bcc to add email blind carbon copy

The email alerter is also able to use collected data properties.

For instance, subject can look like This is my ${property} where ${property} is replaced by the property value.

The email alerter is also able to use collected data properties for subject or body (including replacement). It looks for body.template.location and subject.template.location collected data properties.

For instance, a body Velocity template looks like this:

#if ($event.get("alertBackToNormal") == true)
$event.get("alertLevel") alert: $event.get("alertAttribute") was out of the pattern $event.get("alertPattern") but back to normal now
#else
$event.get("alertLevel") alert: $event.get("alertAttribute") is out of the pattern $event.get("alertPattern")
#end

Details:
#foreach ($key in $event.keySet())
 $key : $event.get($key)
#end

where $event is the map containing all event properties.

Camel

The Decanter Camel alerter sends each alert to a Camel endpoint.

It allows you to create a route which reacts to each alert. It’s a very flexible alerter as you can apply transformation, use EIPs, Camel endpoints, etc.

This alerter creates a Camel exchange. The body of the "in" message contains a Map with all alert details (including alertLevel, alertAttribute, alertPattern and all other details).

The decanter-alerting-camel feature installs the Camel alerter:

karaf@root()> feature:install decanter-alerting-camel

This feature also installs the etc/org.apache.karaf.decanter.alerting.camel.cfg configuration file:

#
# Decanter Camel alerter
#

# alert.destination.uri defines the Camel endpoint URI where
# Decanter send the alerts
alert.destination.uri=direct-vm:decanter-alert

This configuration file allows you to specify the Camel endpoint URI where to send the alert (using the alert.destination.uri property).

For instance, in this configuration, if you define:

alert.destination.uri=direct-vm:decanter-alert

You can create the following Camel route which will react to the alert:

<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">

  <camelContext xmlns="http://camel.apache.org/schema/blueprint">
    <route id="decanter-alert">
      <from uri="direct-vm:decanter-alert"/>
      ...
      ANYTHING
      ...
    </route>
  </camelContext>

</blueprint>
Using existing appenders

Actually, a Decanter alerter is a "regular" Decanter appender. The different is the events topic listening by the appender. By default, the appenders listen on decanter/collect/. To "turn" an appender as an alerter, it just has to listen on decanter/alert/.

For instance, you can create a new instance of elasticsearch appender by creating etc/org.apache.karaf.decanter.appender.elasticsearch-myalert.cfg containing:

event.topics=decanter/alert/*
...

With this configuration, you have a elasticearch alerter that will store the alerts into a elasticsearch instance.