insider 2014-05: Web based user interfaces for syslog-ng; GsoC; perl & python; protecting logs

Dear syslog-ng users,

This is the 34th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

Web based user interfaces for syslog-ng

One of the most popular BalaBit blogs is about syslog-ng web based graphical user interfaces (web GUIs). It’s already three years old, and many things have changed. At that time, only a single Logging as a Service solution was available, now a new one pops up regularly. Also, there were a lot less logging-related GUIs, so some not strictly syslog-ng related solutions were included as well. You can read an updated version of the blog, focusing on syslog-ng based solutions.

Protecting log data against targeted attacks

BalaBit has been saying that SIEM and other analytic tools are only as good as the underlying data. Attackers are also aware of this, and often target log management and SIEMs to hide their presence. Read this blog post for some logging best practices and how syslog-ng can help to secure your logging infrastructure.

Four GsoC students are working on syslog-ng

Thanks to Google, there are four students working on extending syslog-ng with new features during the summer. These are features, which were often requested on the mailing list or at different conferences:

  • integration with configuration management systems
  • ZMQ transport, both source and destination
  • AMQP source driver
  • TLS support for the mongodb destination

Read more about GSoC.

Python and Perl support in incubator

It is still only available in git, as it needs some more polish, but the syslog-ng incubator gained Perl and Python support during the last month. Both the perl and python destinations use the value-pairs framework to get data transferred from syslog-ng to the script, and thus, work differently from the Lua destination. With value-pairs, one can select what parts of the message will be transferred to the script. The script will need to have a queue function (settable with the queue-func() option), which will receive a hash-map of values. Additionally, one can set an init and a deinit function too, to run whenever the driver starts or shuts down.

Check it out and let us know your experiences!

NEW RELEASES

  • Check git if you are impatient 🙂

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-03: GSoC; Incubator; Conferences

Dear syslog-ng users,

This is the 32th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

Google Summer of Code (GsoC)

This year the syslog-ng project was accepted to GSoC as a mentoring organization. We vould like to thank to the Debian and openS– USE projects for vouching for us and openS– USE to let us participate previous years under the openS– USE umbrella. We don’t know yet the number of students we can mentor, but we already have a nice number of project ideas and students who expressed interest.

It’s not yet too late to browse the ideas page and write a project proposal to the syslog-ng mailing list. Contact information is available at this page.

syslog-ng incubator updates

The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository. Version 0.2.1 was released recently with many bugfixes, advanced LUA related features, support for FreeBSD and a python script based ElasticSearch destination.

The syslog-ng incubator is available in many ways. First of all, the source is available in git.

There are also packages for openS– USE and Fedora with riemann support available in a subpackage. To use it, visit one of the below web pages and add the necessary repositories, which also have syslog-ng 3.5 and the riemann C client available.

A FreeBSD port is also work in progress, once ready, it will be committed to sysutils/syslog-ng-incubator. Riemann support is not included.

syslog-ng at conferences

The next confirmed events are:

  • Open Source Data Center Conference, 8-10 April, Berlin, Germany: http://www.netways.de/osdc/, Giving a talk titled “Monitoring with syslog-ng, Riemann and Kibana” There will be two developers present, so if anyone attends, and has Incubator, Lua, Riemann, etc. related questions, we’ll be happy to answer.
  • Linux Open Admin Days, 5-6 April, Antwerp, Belgium: http://loadays.org/, Giving a talk titled “Babel Fish for DevOps: syslog-ng”. Those, who visited FOSDEM will be familiar with the basis of the presentation, but this time with a lot more details and configuration examples thanks to the more available time.
  • Infosecurity Europe, 29 April – 1 May, London, United Kingdom: http://www.infosec.co.uk/, Giving a talk titled “Finding method in the madness: the challenges of the automatic classification of log messages”. The talk will be given by Balazs Scheidler (Bazsi) who you will also be able to find at the BalaBit booth at the event.
  • LinuxTag, 8-10 May, Berlin, Germany: http://linuxtag.org, Giving a talk titled “Finding method in the madness: the challenges of the automatic classification of log messages”. BalaBit will also be sponsoring the event so you’ll be able to find us and get T-shirts and talk with our engineers. The talk will be a revised version of the talk given a week before at Infosecurity Europe.

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-02: FIPS; GSoC; Incubator; Conferences

Dear syslog-ng users,

This is the 31th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

FIPS

Certain departments of the US government and those conducting business on certain regulated markets are required to use tools utilizing cryptography certified according to the FIPS 140-2 standard to handle sensitive data. As logs often contain sensitive information, this regulation can apply to the log management system, too.

Starting with the 5.0.4 release, we are offering a version of syslog-ng Premium Edition to customers that is built and shipped with the OpenSSL FIPS Object Module and as thus usable in environments where such validation is required. (For more information about the OpenSSL FIPS Object Module, see http://openssl.com/fips/.) This is a separate version and the original version is available that uses the standard OpenSSL library. It is currently available on major Linux platforms and has the same functionality as the non-compliant version apart from a few limitations listed in the documentation.
Please visit https://www.balabit.com/lp/fips if for more information.

Google Summer of Code (GsoC)

Based on the success of last years syslog-ng Gsoc participation (in co-operation with the openS– USE project), this year we plan to participate in GsoC again with a number of syslog-ng related projects. The current list of ideas range from new sources and destination through under the hood improvements to integration with configuration management systems.

syslog-ng incubator updates

The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository.

Since last month there are many smaller changes and brand new features. The master branch goes now hand in hand with syslog-ng’s master branch, which is currently 3.6 pre-alpha code. If you are working on syslog-ng 3.5, use the 3.5/master branch of syslog-ng-incubator.

On the feature side the Lua destination was enhanced, a monitoring source was added, together with a Graphite output. Among the Lua examples one can also find a script to push logs into ElasticSearch.

To try the new features, check out https://github.com/balabit/syslog-ng-incubator from git. You can read more about the Lua destination at https://talien.blogs.balabit.com/2014/02/lua-the-undiscovered-country/

syslog-ng at conferences

In the past few weeks we visited three conferences. While there we talked about syslog-ng to many people, some during organized presentations, and many more between presentations and social events of the conferences.

A look back at FOSDEM and Config Management Camp: https://czanik.blogs.balabit.com/2014/02/fosdem-2014-and-config-management-camp/

The next confirmed event is: Open Source Data Center Conference, 8-10 April, Berlin, Germany: http://www.netways.de/osdc/, Giving a talk titled “Monitoring with syslog-ng, Riemann and Kibana” There will be two developers present, so if anyone attends, and has Incubator, Lua, Riemann, etc. related questions, we’ll be happy to answer.

SHORT NEWS

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-01: 3.6 pre-alpha ; incubator ; PCI-DSS

Dear syslog-ng users,

This is the 30th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng at conferences

This year, we plan to present syslog-ng in many conferences, or just participate conferences where many syslog-ng users are expected to show up. If you want to meet someone from the syslog-ng team, come to one of our presentations or drop an e-mail so we can find each other. We are also very interested to hear suggestions, where syslog-ng should be presented!
Here is a list of confirmed events, which will be updated regularly in later newsletters:

  • FOSDEM, 1-2 February, Brussels, Belgium: https://fosdem.org/2014/, giving a talk titled: “Babelfish for DevOps: syslog-ng”
  • Fedora, JBoss and RedHat developers conference, 7-9 February, Brno, Czech Republic: http://devconf.cz/, Participating only
  • Open Source Data Center Conference, 8-10 April, Berlin, Germany: http://www.netways.de/osdc/, Giving a talk titled “Monitoring with syslog-ng, Riemann and Kibana”

Check out syslog-ng 3.6 pre-alpha!

While syslog-ng 3.6 still did not even reach an alpha release, it received already a lot of development. So, while it might still eat your logs for lunch, those who are interested in where syslog-ng is heading should check out syslog-ng 3.6 from git. Unlike previous syslog-ng versions, it is developed in a unified git repository without a version string attached.

While most changes are under the hood, there are also some user visible features like pseudofile destination.

Nodejs support was also added to syslog-ng: use the widespread winston logging API, and syslog-ng will process its JSON formatted messages.

And if you look at the stats you can see a healthy growth of the code base and in the number of contributors. Thank you for your support!

syslog-ng incubator

The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository. It has some very interesting code in it, like a riemann or an RSS destination, but until now it was completely undocumented. Not any more: http://asylum.madhouse-project.org/blog/2013/12/29/the-incubator/.
The Incubator also includes a Lua destination, which makes it possible to write simple destination drivers without a line of C. It is still a work in progress, but is an important step into writing modules in other languages.
And to make your life easier packages are available in Debian testing, Ubuntu Trusty and for openS– USE in the 3rd party repositories.

PCI DSS 3.0 Continues to Emphasize the Importance of Log Management

The Payment Card Industry Security Standards Council recently released the Data Security Standard 3.0, three years after the prior version. As one of the most important international data security standards, the latest release was eagerly awaited by IT security practitioners. Clarifications make up the bulk of the changes but the standards council changed most of the 12 major requirements to include modified or additional sub-requirements. With PCI DSS 3.0 the standards council has reiterated that log management is a critical part of security best practices. You can read more about it.

SHORT NEWS

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2013-11: syslog-ng 3.5 released; Logging to Hadoop; EoL

Dear syslog-ng users,

This is the 29th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng 3.5 is released

Beginning last week syslog-ng 3.5 was finally declared stable, so 3.5.1 was released. Source is available from https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/download if you want to compile syslog-ng yourself. Packages and ports for different Linux distributions and FreeBSD are already available from our 3rd party page.

New features of 3.5 were detailed here many times, so here are just the headlines:

  • Redis and Stomp destinations
  • multiline messages
  • type hinting

For details, check the ChangeLog or algernon’s blog.

Logging to Hadoop

Big data is gaining a momentum also in the logging world, so we made some experiments how syslog-ng can log to Hadoop. Even without a dedicated Hadoop destination driver one can already send logs to Hadoop and analyze them using Hadoop tools. For details, check https://tiborbenke.blogs.balabit.com/2013/11/the-syslog-ng-in-the-hadoop-era/

syslog-ng EoL policy

The syslog-ng End of Life policy was updated recently, based on the experiences of syslog-ng stable maintainer, Gergely Nagy. Practically there are four different branches of syslog-ng maintained at any time, representing different level of maturity and support. More details are available at http://asylum.madhouse-project.org/blog/2013/10/22/syslog-ng-eol-dates/.

And following the URLs below you can read about how this affects different Linux distributions and FreeBSD:

SHORT NEWS

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2013-10: syslog-ng 3.5 beta releases; syslog-ng PE with Windows support; GSoC summaries

Dear syslog-ng users,

This is the 28th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng 3.5 beta2 and beta3 are released

The second and third beta of syslog-ng 3.5 were released during the past few weeks. These brought mostly bugfixes, but also some new features. Beta2 brought us a new redis destination and extended type hinting. Beta3 finally compiles again on FreeBSD.
As usual, testing is greatly appreciated as it helps to iron out problems before the final release! Sources are available from git or as tgz.
For third-party binary packages for various Linux and UNIX platforms, visit this page. SLES / openS– USE packages are already available and Debian/Ubuntu packages are expected to be available in the coming days.

syslog-ng PE with full Windows support

The Premium Edition of syslog-ng now offers full Windows support with its latest release, version 5LTS. Until now, only the syslog-ng Agent for Windows could be installed on Windows platforms. With the latest version, syslog-ng Premium Edition can be installed on client but also as server, so even a homogenous Windows environment can benefit from using syslog-ng. Installing a separate Linux machine as relay is no longer necessary in remote offices, as a Windows machine can do the job.
Compression support in RLTP was also introduced, which can save valuable bandwidth.

Read the product managers blog about the new release.

syslog-ng GSoC: code merged

The syslog-ng GSoC finished a few weeks ago, but it does not mean, that the work is over. The code had to be updated to work with the latest syslog-ng 3.5 sources. The new redis destination is already merged and available as part of the syslog-ng 3.5 beta2 sources. The new mysql destination is heading now to the “incubator” project and waits there for some additional polish, before it is merged syslog-ng master.

A summary blog post of their work is available at http://petrovicsgyula.blogspot.com/ and http://tichygsoc.blogspot.hu/ A blog post about the syslog-ng GSoC from the mentors (and algernon’s) point of view can also be read.

syslog-ng 3.5 documentation is now feature complete

The syslog-ng 3.5 documentation is now feature complete and available on our website.
It is available in many forms:

SHORT NEWS

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2013-09: syslog-ng 3.5 beta release; incubator project; GSoC updates

Dear syslog-ng users,

This is the 27th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng 3.5 beta1 is released

The first beta of syslog-ng 3.5 was released today. It has many internal rework and also many user visible new features. Here you can read just the headlines, for a complete list and details please visit the NEWS file or this blog post.

  • Stomp destination
  • Multiline support
  • Blacklist support
  • and many more

As usual, testing is greatly appreciated as it helps to iron out problems before the final release!
Sources are available from git or as tgz. For third-party binary packages for various Linux and UNIX platforms, visit https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/3rd-party. Debian/Ubuntu and openS– USE packages are expected to be available in the coming days.

syslog-ng incubator project

The syslog-ng module incubator (Incubator henceforth) is a collection of tools and modules for syslog-ng that for one reason or the other, are not part of the official repository. This serves both as a staging ground for experimental modules, and as a repository of plugins that are not aimed at upstream inclusion. It’s also an example of a third party syslog-ng module.
Sources are available at https://github.com/algernon/syslog-ng-incubator

syslog-ng GSoC: finished successfully!

The coding time for Google Summer of Code is over and both of our students finished their projects successfully. The redis destination is to merged to syslog-ng 3.5 before the beta2 release and the high performance mysql destination is expected to follow it soon. A summary blog post of their work is due to appear next week at http://petrovicsgyula.blogspot.com/ and http://tichygsoc.blogspot.hu/

syslog-ng 3.5 draft documentation

A draft version of syslog-ng 3.5 documentation was also released today. It contains some of the new features. It is available in many forms:

SHORT NEWS

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2013-08: New releases; GSoC updates; Dedicated OSE developer; Observe, Hack, Make 2013;

Dear syslog-ng users,

This is the 26th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng 3.4.3 and 3.3.11 are released

Both the current (3.4) and previous (3.3) stable syslog-ng releases received maintenance updates. Sources are available from git or as tgz from https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/download
For third-party binary packages for various Linux and UNIX platforms, visit https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/3rd-party for 3rd party binary packages for various distributions and UNIX variants.

syslog-ng Agent for Windows 5.0

There is a major new version of syslog-ng Agent available, version 5.0. Under the hood it builds now on the same code base as the UNIX version of syslog-ng. The new highlights include support for the Reliable Log Transfer Protocol (RLTP), flow control and support for the Windows Server 2012 and Windows 8 platforms.

Long-time syslog-ng contributor Gergely Nagy now working full-time on syslog-ng Open Source Edition

If you’ve been following the mailing list, the community around syslog-ng or the commits in git you most certainly recognize the name of Gergely Nagy. He’s been working at BalaBit for several years and we’re happy to announce that from now on his dedicated role is to work on syslog-ng OSE full-time.
You can read his thoughts about the change on his blog.

syslog-ng GSoC: it’s half-time!

The coding for Google Summer of Code is at half-time. Our students passed half-time evaluations and are working hard on a faster MySQL destination and on a brand new redis destination. You can follow their progress from their related blogs:

There is also a plan to provide test packages for those really adventurous in the coming weeks 🙂

syslog-ng JSON HowTo

JSON is gaining popularity, not just in Web 2.0 but in all fields of IT. As it’s an easy way of storing and transmitting name value pairs, JSON emitting and parsing is now part of syslog-ng.
The HowTo describing how it works in syslog-ng is available at http://asylum.madhouse-project.org/blog/2013/07/29/json-howto/

syslog-ng developers at Observe, Hack, Make 2013

Laser harp, Wikileaks, lockpicking, NSA, SIM card exploiting, LHC, counter-cryptanalysis, 24 pull requests, neutrino detectors, open source Bach, weird lights, quadrocopters, electric music cranked up to 11, workshops, lectures, discussions, and a lot more! We’ve been Observing, Hacking and Making!
You can read more about it at this post.

SHORT NEWS

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2013-07: syslog-ng 3.4.2 released; PatternDB update; GSoC; RSS destination

Dear syslog-ng users,

This is the 25th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng 3.4.2 is released

Last week syslog-ng 3.4.2 was released, the first bugfix release in the 3.4 series. The most important fix is for a hang when suppress() was used. A full list of changes is available in git.
Sources are available from git or as tgz.
For third-party binary packages for various Linux and UNIX platforms, visit this page for 3rd party binary packages for various distributions and UNIX variants.

PatternDB git moved and updated

The BalaBit patterndb git moved recently to github. It’s available now at https://github.com/balabit/syslog-ng-patterndb instead of our server. That way the patterns are available next to the syslog-ng sources, and are also faster to download. We have also updated, reorganized and extended the available patterns. For details, read this post.

syslog-ng is participating GSoC again

Just as last year, syslog-ng is participating in the Google Summer of Code under the umbrella of the openS– USE project. Two of our candidates have the opportunity to code all summer long as GsoC students. Gyula Petrovics will work on a faster MySQL destination, which works without libdbi. The other student, TihamĂ©r Petrovics, will add a redis destination, which can not only store logs but also provide counters and help with statistics.

Related blogs:

RSS destination

RSS destination is a neat little feature of syslog-ng. It can store up to 100 pieces log lines, and can serve them as an Atom feed.
This destination works as a FIFO, so if it is full, the last incoming log line kicks out the first one. Now you are able to read the alerts from your machines in your favorite RSS reader (well, unfortunately not in Google Reader).
Available in the feature/rss-destination branch.

SHORT NEWS

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2013-06: syslog-ng configurator on Android; Using syslog-ng with Splunk; EU data protection and logging

Dear syslog-ng users,

This is the 24th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

syslog-ng configurator app for Android

There is now a new configurator app available for Android, developed as a hobby project by one of the syslog-ng team members. The focus of the application is to create a syslog-ng.conf wich provides optimum performance based on a number of questions.
For more details and download locations read the authors blog.

syslog-ng and Splunk

We often receive questions, how to use syslog-ng and Splunk together in a logging infrastructure. We collected the most popular usage scenarios into a white paper, together with example configurations to make testing and integration even more easy.

syslog-ng is participating GSoC again

Just as last year, syslog-ng is participating Google Summer of Code under the umbrella of the openS– USE project. We have candidates for developing a native mysql destination, a redis destination, a XMPP (jabber) destination and log signing, which is a big improvement from last year, where we only had a single candidate for a similar number of development projects. The application process is closed now and there are still a couple of weeks to go, before the final list of approved students is announced. http://news.opensuse.org/2013/04/25/opensuse-hedgewars-and-owncloud-are-moving-gsoc-along-participate-and-submit-your-proposals-fast/

Big changes ahead for EU data protection regulation

This summer will most likely bring big changes in the regulation of Data Protection in the European Union. We collected these proposed changes and also how syslog-ng and proper central log management can help to comply with these regulations.

Compiling syslog-ng with MS SQL support on RHEL / CentOS & Co.

During the past few months many people asked, how to log from syslog-ng to MS SQL on RHEL or CentOS. If you cannot buy syslog-ng PE, follow these steps to compile all the necessary components yourself and configure the MS SQL part.

SHORT NEWS

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.