Writing into HDFS with syslog-ng video and setup instructions

The goal of this article is summarizing the necessary steps for a simple ‘syslog-ng and hadoop’ setup. The demo at the end of this summary demonstrates how it works. This video was recorded for the Hortonworks Data Platform certification process and as a result both syslog-ng PE 5.3+ and OSE 3.7+ are Hortonworks HDP certified.

Continue reading

Syslog-ng and Rust

First steps towards simple and efficient parsers

2015 was the year of language bindings in syslog-ng. From now on, people can write plugins for syslog-ng not just in C, but in Java and Python as well. Java provides access to popular big data technologies, while Python makes syslog-ng incredibly extensible by system administrators. However, syslog-ng doesn’t have bindings for a language, which is as fast as C, has automatic dependency management with simplified build, development and distribution process and ensures memory safety with extensive compile time checks.

Continue reading

GSoC-2015 summary

Google Summer of Code 2015 syslog-ng

The syslog-ng application is an open source worldwide-known logging system.

With syslog-ng you can collect logs from any source, process them in near real-time and deliver them to a wide variety of destinations including e.g. HDFS, kafka, Elasticsearch.

The syslog-ng application allows you to flexibly collect, parse, classify, and correlate logs from across your infrastructure and store or route them to log analysis tools.

This was syslog-ng’s second year participating in the Google Summer of Code (GSoC) program as an independent organization. Four of our students did particularly great.

syslog-ng server monitoring with mobile application: Krishna Kannan

Krishna has created a mobile application to monitor the syslog-ng server, thereby giving the system administrators an advantage of monitoring the server from anywhere and anytime in the world. Administrators can track the status of their syslog-ng server using this mobile app and can also perform various actions, for example gathering statistics, and so on. This mobile application is available in GooglePlay Store. https://play.google.com/store/apps/details?id=com.mobile.syslogng.monitor

https://pzolee.blogs.balabit.com/2015/09/monitor-syslog-ng-with-android-app/

The future plan is to extend the application with other functions (for example: gathering metrics).

Java language binding for syslog-ng: Peter Szabo

The syslog-ng application had Java support, but it was only for destinations. Peter’s goal was to extend this Java support and make it possible to write Java plugins for every possible syslog-ng connector (filter, parser, rewrite rule, template-function, and source).

The source part is not completely finished, and it still requires some polishment. In spite of facing a lot of difficulties during the process, finally the goal was reached. After some fine-tuning, this project will be part of syslog-ng. It is available at: https://github.com/asdwsda/syslog-ng/tree/f/java-language-binding

The future plans with java sources and destinations are: JDBC source and destination.

Riak destination for syslog-ng: Parth Oberoi

Riak is an open source, distributed database. Parth has implemented the Riak destination, and also a C library for Riak. To achieve this goal, several other open source tools were required, such as Google Protocol Buffer, Erlang and protobuf-c. The result is a pull request-ready code. For details, see: https://github.com/hTrap/syslog-ng/wiki/Riak-Destination

The pull request is syslog-ng-Incubator-ready, which means that it could be easily integrated into the syslog-ng Incubator.

The main goal is to merge the Riak destination into the main repository of syslog-ng.

For the pull request of the main syslog-ng repository, see : https://github.com/balabit/syslog-ng/pull/630

The integration is currently in progress, if there are any unresolvable issues, there is a fallback option to syslog-ng Incubator.

Graphical configuration tool for syslog-ng: Andras Mamenyak

Andras has worked on a standalone graphical tool for creating syslog-ng configuration files. syslog-ng is very adaptable: it offers numerous different drivers with several options. Fortunately, it has an elegant and easy to understand configuration syntax and a comprehensive Administrator Guide that describes all the options. Still, especially for someone new to syslog-ng, Andras’ project created a very helpful tool, a configuration software with a graphical user interface that supports creating a base configuration. This project is available at: https://github.com/mamenyaka/syslog-ng-config-qt

Additional results

  • syslog-ng universe

https://syslog-ng.org/universe/

 

We had a lot of promising candidates for our GSoC projects but in the end we had to select only one student per project and reject others. We want to keep in touch with these students and any other developer candidates by offering mentoring support and helping them become open source developers.

 

  • As a side effect, the syslog-ng documentation source is now publicly available on GitHub.

(András Mamenyák has requested us to make the documentation source publicly available so that he could contribute to improving it as well. It is available at: https://github.com/balabit/syslog-ng-ose-guides )

insider 2015-12: Elasticsearch & Kibana; SCL; Kafka; syslog-ng on Windows; conferences

Dear syslog-ng users,

This is the 45th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.

NEWS

ESK: Elasticsearch + syslog-ng + Kibana

An official Elasticsearch destination was a long awaited feature of syslog-ng. The Elasticsearch syslog-ng Kibana (ESK) stack provides high performance log collection, message parsing and filtering on the syslog-ng side and storage and analysis on the Elastic side. Read more about the benefits at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/ and https://czanik.blogs.balabit.com/2015/12/elasticsearch-and-syslog-ng-fast-and-simple/

The power of SCL

SCL, the syslog-ng configuration library is a lesser known feature of syslog-ng. It can provide reusable configuration blocks, which help to format messages to a specific format, hide complex configurations. This blog shows how to create a simple SCL through the example of a Loggly (logging as a service provider) destination.

Kafka and syslog-ng

Kafka is a high-throughput distributed messaging system. It was originally developed by LinkedIn as the backbone of a website activity tracking infrastructure and is now used by Netflix, Spotify, Twitter and many other companies for message queuing, log aggregation and stream processing. Read more about how syslog-ng can collect messages from multiple sources and process as well as filter them before forwarding them to Kafka: https://czanik.blogs.balabit.com/2015/11/kafka-and-syslog-ng/

syslog-ng on Windows

Most Linux admins only consider Windows as a client machine and use syslog-ng Agent for Windows if they need to collect logs from Windows. The following use cases demonstrate why you would want to use syslog-ng Premium Edition server on Windows instead of the syslog-ng Agent for Windows.

syslog-ng @ conferences

This autumn, syslog-ng was present at many conferences. I gave presentations about syslog-ng at the annual FSF Hungary and Virtualization & DevOps day conferences (in Hungarian), and participated S– USECon, LISA and Crunch Big Data Conference as an exhibitor. Most questions I received were related to the Elasticsearch destination and also many people were interested in commercial support for syslog-ng.
In January I’ll present “syslog-ng: from raw data to Big Data” at the Southern California Linux Expo

See you there!

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

Community met GSoC 2015 participants via IRC

Channel statistics

#syslog-ng@freenode
Event started at 2015.06.04 17:00 CET
42 nick’s on channel (+1 joined later)
14 active participants (at least 1 comment)

Chat around 5 GSoC topics

htrap – My project is to create a Riak Destination for Syslog-ng:
https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Riak-destination-(htrap)

Krishna_ – My project is – Syslog-ng server monitoring with mobile app:
https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-syslog-ng-server-monitoring-with-mobile-app-(Krishna41)

battila7 – My project is: syslog-ng as a command line tool:
https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-Syslog-ng-as-a-command-line-tool-%28battila7%29

mamenyaka – I am working on the Qt syslog-ng configuration editor:
https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-syslog-ng-configuration-editor-using-Qt-(mamenyaka)

asdwsda – and my project is to extend java language binding for syslog-ng:
https://github.com/balabit/syslog-ng/wiki/GSoC2015-Proposal:-syslog-ng-for-Java-(szpeter)

Short summary

Community members detailed their use case expectations, asked different questions and started brainstorming.

  • SHRREAK would be the next level of SHREK (Syslog-ng Riemann Elasticsearch Kibana) stack.
    A stands for Android for monitoring syslog-ng servers. Stat
    istics will be available also
    2nd R stands for Riak which would store events instead of ES (ES would be used only for indexing)
  • A feature proposal arrived – Qt config editor should be able to parse existing files
  • CLI example:

    throw-logs | syslog-ng-cli –parser db –dbparser /var/lib/pdb.xml –source stdin –destination stdout’

  • htrap blogs his experiences about GSoC: http://thetechtrap.com It turned out that cor edevelopers of syslog would welcome articles on https://syslog-ng.org
  • squid generated JSON parsing was discussed with syslog-ng meanwhile