My previous post described how to build a very simple parser plugin for syslog-ng in Rust. I brought a more realistic example this time: a regular expression based parser plugin. It’s so real that it is decently covered with unit tests and it has even a benchmark.
From time to time, we publish blog posts about syslog-ng and we hope you find them interesting and useful. However, we would like to become better at this: we want to give you more and better content, more regularly. Also, we would like to be transparent about it, and give you the possibility to get involved.
We are glad to inform you that a few days ago the disk-based buffering functionality has appeared in syslog-ng Open Source Edition.
Disk-based buffering can be used for storing messages on the local hard disk if the central log server or the network connection to the server becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is reestablished.
This year the syslog-ng project would like to participate again in Google Summer of Code.
During the organization registration process we had to answer some questions.
We wanted to share some of our answers to help you understand our plans,
if the syslog-ng project is accepted as a mentoring organization.
Dear syslog-ng users,
This is the 47th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.
The goal of this article is summarizing the necessary steps for a simple ‘syslog-ng and hadoop’ setup. The demo at the end of this summary demonstrates how it works. This video was recorded for the Hortonworks Data Platform certification process and as a result both syslog-ng PE 5.3+ and OSE 3.7+ are Hortonworks HDP certified.
First steps towards simple and efficient parsers
2015 was the year of language bindings in syslog-ng. From now on, people can write plugins for syslog-ng not just in C, but in Java and Python as well. Java provides access to popular big data technologies, while Python makes syslog-ng incredibly extensible by system administrators. However, syslog-ng doesn’t have bindings for a language, which is as fast as C, has automatic dependency management with simplified build, development and distribution process and ensures memory safety with extensive compile time checks.
Dear syslog-ng users,
This is the 46th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.
Google Summer of Code 2015 syslog-ng
The syslog-ng application is an open source worldwide-known logging system.
With syslog-ng you can collect logs from any source, process them in near real-time and deliver them to a wide variety of destinations including e.g. HDFS, kafka, Elasticsearch.
The syslog-ng application allows you to flexibly collect, parse, classify, and correlate logs from across your infrastructure and store or route them to log analysis tools.
This was syslog-ng’s second year participating in the Google Summer of Code (GSoC) program as an independent organization. Four of our students did particularly great.
syslog-ng server monitoring with mobile application: Krishna Kannan
Krishna has created a mobile application to monitor the syslog-ng server, thereby giving the system administrators an advantage of monitoring the server from anywhere and anytime in the world. Administrators can track the status of their syslog-ng server using this mobile app and can also perform various actions, for example gathering statistics, and so on. This mobile application is available in GooglePlay Store. https://play.google.com/store/apps/details?id=com.mobile.syslogng.monitor
The future plan is to extend the application with other functions (for example: gathering metrics).
Java language binding for syslog-ng: Peter Szabo
The syslog-ng application had Java support, but it was only for destinations. Peter’s goal was to extend this Java support and make it possible to write Java plugins for every possible syslog-ng connector (filter, parser, rewrite rule, template-function, and source).
The source part is not completely finished, and it still requires some polishment. In spite of facing a lot of difficulties during the process, finally the goal was reached. After some fine-tuning, this project will be part of syslog-ng. It is available at: https://github.com/asdwsda/syslog-ng/tree/f/java-language-binding
The future plans with java sources and destinations are: JDBC source and destination.
Riak destination for syslog-ng: Parth Oberoi
Riak is an open source, distributed database. Parth has implemented the Riak destination, and also a C library for Riak. To achieve this goal, several other open source tools were required, such as Google Protocol Buffer, Erlang and protobuf-c. The result is a pull request-ready code. For details, see: https://github.com/hTrap/syslog-ng/wiki/Riak-Destination
The pull request is syslog-ng-Incubator-ready, which means that it could be easily integrated into the syslog-ng Incubator.
The main goal is to merge the Riak destination into the main repository of syslog-ng.
For the pull request of the main syslog-ng repository, see : https://github.com/balabit/syslog-ng/pull/630
The integration is currently in progress, if there are any unresolvable issues, there is a fallback option to syslog-ng Incubator.
Graphical configuration tool for syslog-ng: Andras Mamenyak
Andras has worked on a standalone graphical tool for creating syslog-ng configuration files. syslog-ng is very adaptable: it offers numerous different drivers with several options. Fortunately, it has an elegant and easy to understand configuration syntax and a comprehensive Administrator Guide that describes all the options. Still, especially for someone new to syslog-ng, Andras’ project created a very helpful tool, a configuration software with a graphical user interface that supports creating a base configuration. This project is available at: https://github.com/mamenyaka/syslog-ng-config-qt
- syslog-ng universe
We had a lot of promising candidates for our GSoC projects but in the end we had to select only one student per project and reject others. We want to keep in touch with these students and any other developer candidates by offering mentoring support and helping them become open source developers.
- As a side effect, the syslog-ng documentation source is now publicly available on GitHub.
(András Mamenyák has requested us to make the documentation source publicly available so that he could contribute to improving it as well. It is available at: https://github.com/balabit/syslog-ng-ose-guides )
Dear syslog-ng users,
This is the 45th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.
ESK: Elasticsearch + syslog-ng + Kibana
An official Elasticsearch destination was a long awaited feature of syslog-ng. The Elasticsearch syslog-ng Kibana (ESK) stack provides high performance log collection, message parsing and filtering on the syslog-ng side and storage and analysis on the Elastic side. Read more about the benefits at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/ and https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-and-simple/
The power of SCL
SCL, the syslog-ng configuration library is a lesser known feature of syslog-ng. It can provide reusable configuration blocks, which help to format messages to a specific format, hide complex configurations. This blog shows how to create a simple SCL through the example of a Loggly (logging as a service provider) destination.
Kafka and syslog-ng
Kafka is a high-throughput distributed messaging system. It was originally developed by LinkedIn as the backbone of a website activity tracking infrastructure and is now used by Netflix, Spotify, Twitter and many other companies for message queuing, log aggregation and stream processing. Read more about how syslog-ng can collect messages from multiple sources and process as well as filter them before forwarding them to Kafka: https://czanik.blogs.balabit.com/2015/11/kafka-and-syslog-ng/
syslog-ng on Windows
Most Linux admins only consider Windows as a client machine and use syslog-ng Agent for Windows if they need to collect logs from Windows. The following use cases demonstrate why you would want to use syslog-ng Premium Edition server on Windows instead of the syslog-ng Agent for Windows.
syslog-ng @ conferences
This autumn, syslog-ng was present at many conferences. I gave presentations about syslog-ng at the annual FSF Hungary and Virtualization & DevOps day conferences (in Hungarian), and participated S– USECon, LISA and Crunch Big Data Conference as an exhibitor. Most questions I received were related to the Elasticsearch destination and also many people were interested in commercial support for syslog-ng.
In January I’ll present “syslog-ng: from raw data to Big Data” at the Southern California Linux Expo
See you there!
Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.