To log checks from sensu into logstash, I use the following sensu config:
{
"handlers": {
"logstash": {
"type": "tcp",
"socket": {
"host": "logstash.example.com",
"port": 1234
}
}
}
}
Since this is json output, logstash will be able to consume this easily.
On the logstash side, I use a tcp input to receive the messages:
input {
tcp {
codec => json {}
port => 1234
tags => ["sensu"]
}
}
and I then use filters to 'tidy up' some of the message contents (mostly throwing away fields I don't want (optional):
filter {
if ("sensu" in [tags]) {
date {
match => ["[check][issued]", "UNIX" ]
}
mutate {
remove_field => [ "host", "[client][handlers]", "[check][handlers]", "[check][history]", "[client][keepalive][handler]", "[client][keepalive][refresh]", "[client][keepalive][thresholds][critical]", "[client][keepalive][thresholds][warning]", "[client][subscriptions]", "[client][address]" ]
}
}
}
This will record a log event each time a check fires... however since tcp handlers don't honour the 'occurrences', 'refresh' or 'interval' fields of a check, this could be as often as every 30 seconds!
Since I only care about logging the first and last event (i.e. when did the service break, and when did it resolve) I use the 'throttle' filter to discard messages which 'repeat' the first message:
filter {
mutate {
add_field => { "event_id" => "%{[client][name]}_%{[check][name]}_%{[check][status]}" }
}
throttle {
after_count => 1
period => 86400
key => "%{event_id}"
add_tag => "throttled"
}
}
This config adds a new field, made up of client.name, check.name and check.status, and then tags all subsequent messages with an identical 'event_id field in any 24 hour period, as 'throttled'.
I then only output those messages where this tag isn't set:
output {
if "throttled" not in [tags]
elasticsearch { ... }
}
}
To log metrics from sensu into logstash, you need to process the message string with grok:
filter {
grok => {"message", "%{DATA:metric}\\t%{DATA:value}\\t%{INT:unixtime}"}
}
Additionally, be sure to use the line codec for the input, since sensu will send multiple metrics in one message, separated by newlines.