Insider 2017-11: netdata; stats to graphite; logstash to syslog-ng;

Dear syslog-ng users,

This is the 63rd issue of syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.

NEWS

Collecting netdata metrics

netdata is a system for distributed real-time performance and health monitoring. You can use syslog-ng to collect and filter data provided by netdata and then send it to Elasticsearch for long-term storage and analysis. The aim is to send both metrics and logs to an Elasticsearch instance, and then access it via Kibana.

syslog-ng statistics to Graphite

syslog-ng makes available various types of statistics. Data is available in a couple of forms: emitted regularly from the internal() source of syslog-ng or obtained using the syslog-ng-ctl utility from the command line. Due to the format that the internal source or the “stats” option of syslog-ng-ctl uses, it is not easy to send statistical data to Graphite or anywhere else. The syslog-ng-ctl utility provides a flexible “query” option, (available in recent versions of syslog-ng) which uses an easy-to-parse output format. Using the “jo” utility by @JPMens, you can convert the output to JSON, which can then be parsed by syslog-ng and forwarded to Graphite.

Sending logs from Logstash to syslog-ng

Logstash adds a new syslog header to log messages before forwarding them to a syslog server. In the case of syslog messages, it is problematic as there will be two syslog headers in the message. Using syslog-ng for everything logging related in an Elasticsearch environment can considerably simplify your architecture. Still, there are situations, when Filebeats and Logstash are already deployed and you need some logs from Logstash in syslog-ng. Learn how you can remove the extra syslog header.

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

Leave a Reply