insider 2015-03: GsoC; Hadoop; 3.7 documentation; kafka; grok;

Dear syslog-ng users,

This is the 40th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng-related news.

NEWS

syslog-ng in Google Summer of Code 2015

This year we participate in GSoC again, with a focus on making
the life of our users and contributors easier. Our main projects for this year include:

  • Python and Java language bindings to make code contribution easier
  • Develop a Qt-based graphical configuration editor to help new users to get started
  • A data-flow visualization tool to help in debugging

If you are a student and willing to spend the summer coding syslog-ng, or know someone who could participate, check our detailed project and idea list.

You can reach mentors by direct e-mail, by IRC on channel #syslog-ng
on FreeNode, or on our mailing list.

syslog-ng Hadoop support

With the release of syslog-ng PE 5F3 support for Hadoop arrived. It will also be part of the upcoming syslog-ng OSE 3.7 release. This enables syslog-ng to write log messages to HDFS. https://jluby.blogs.balabit.com/2015/03/10/syslog-ng-and-big-data-streaming-into-your-data-lake/

syslog-ng 3.7 beta is coming

The first beta of syslog-ng OSE 3.7 is expected to arrive in the coming weeks. It has many smaller and larger changes, like the Java destination is migrated from incubator to syslog-ng core.

insider 2014-09: 3.6 beta; eCSI; DevOps; anonymization; GSoC;

Dear syslog-ng users,

This is the 37th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

Beta testing of syslog-ng 3.6 started

Beta testing period of syslog-ng 3.6 started two weeks ago with a beta1 release. It was quickly followed by a beta2 release, fixing mostly portability. Some of the new features are coming from the syslog-ng incubator, like Riemann support or the graphite template function. The journal source and many enhancements to existing features are coming from the syslog-ng PE team. Also, the @cim prefix can be used as a local “rich” log transport on /dev/log, and messages will be parsed by the JSON parser if available.

Beta2 is already available in the FreeBSD ports in sysutils/syslog-ng-devel. For RPM distributions, check https://czanik.blogs.balabit.com/2014/09/journal-source-in-syslog-ng-3-6-beta1/.

BalaBit syslog-ng team takes over the syslog-ng OSE development

algernon, the full-time syslog-ng OSE developer at BalaBit decided to look for new challenges.
First of all, thank you very much for all your hard work, bugfixes nifty new features, and everything else you did for syslog-ng OSE! We all wish you the best, and hope you’ll have a great time outside BalaBit as well 🙂
To keep the development, maintenance, and releases of syslog-ng OSE on track, the developer team of syslog-ng Premium Edition will take over the tasks related to syslog-ng OSE: they will manage bugfixes, patches, pull requests, and also the general development of syslog-ng OSE. Naturally, this does not affect the current or future openness of syslog-ng OSE in any way: Your contribution is as welcome as ever.
For more details about this change, see algernon’s blog post.

eCSI training

BalaBit provides now a free training, which provides a fresh insight on logmanagement. The first level introduces the listener to compliance, planning an infrastructure and logs in forensics situations. Once your knowledge is tested, you can go to the next level and learn about IT security and eCSI.

syslog-ng, riemann, collectd-notifications, elasticsearch

How to build an event-based infrastructure to push structured messages to different subsystems for alerting, reporting and storage. Using syslog-ng, each message is normalized into a structured event, optionally correlated with other messages, and conditionally routed to systems. Read more at:
http://devops.com/features/guide-modern-monitoring-alerting/

Data Privacy, Anonymization and Log Data

Strong data privacy laws are arriving slowly but surely in Europe, which also affects logging. Raw data contains too much information, on the other hand anonymized data does not have enough information to handle a security incident. Read about a possible solution and how syslog-ng can help.

Google Summer of Code: success

Google Summer of Code ended a few weeks ago. All of our students successfully completed their projects. We would like to thank for the hard work of students and their mentors, and Google for the opportunity!

You can read more about the completed projects.

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-08: EPEL; graphite; PCI DSS

Dear syslog-ng users,

This is the 36th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

EPEL 7 now contains syslog-ng

RHEL 7 was released over a month ago and CentOS 7 not much later, but one piece of software was still missing: syslog-ng. Not any more. EPEL, which stands for Extra Packages for Enterprise Linux, is a software collection containing additional packages for Enterprise Linux and derivatives. Now its latest version, EPEL 7 also contains syslog-ng, version 3.5.

Introducing syslog-ng PE 5F1

The latest version of the syslog-ng Premium Edition, 5F1, adds support for the popular NoSQL database MongoDB. Along with support for MongoDB, we have added support for Java Script Object Notation (JSON), a text-based open standard designed for human-readable data interchange.

Performance monitoring using syslog-ng and graphite

For most of its history, syslog-ng could only be used for collecting, processing and storing log messages. Not any more. The Redis and Riemann destinations are already a step into the direction of metrics-based monitoring, and the monitoring source combined with Graphite template support are the next.
https://czanik.blogs.balabit.com/2014/07/how-to-setup-syslog-ng-quickly-for-performance-monitoring-using-graphite-inside-docker/

Introducing syslog-ng store box 3F2

We recently released a new version of our log management appliance, the syslog-ng Store Box. 3F2 is the latest feature release and includes one major new feature and a major improvement to an existing one. First, we have added a RESTful API which opens up all sorts of possibilities for accessing log data in SSB. Second, we have revamped the search interface on the web-based user interface making searching and troubleshooting much easier.

syslog-ng incubator 0.3.3 released

The syslog-ng incubator is a set of tools and modules for syslog-ng, which are not (yet) available in the official release. This version of incubator works with the latest stable syslog-ng (v3.5.5+) and fixes many problems of the initial 0.3 incubator release.

Log management and the Verizon 2014 PCI Compliance Report

Recently, the eagerly anticipated Verizon Data Breach Investigations Report for 2014 was published. With more than 63,000 security incidents, 1,300 confirmed data breaches and 50 contributing global organizations, it provides the most comprehensive insight to state of IT security around the world. Drawing on data from the Data Breach Investigation Report, Verizon also publishes a lesser known but very interesting report on the state of compliance of with the Payment Card Industry Data Security Standard (PCI DSS), perhaps the most widely-adopted security standards globally. Read, what requirements PCI DSS has towards log management/.

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-06: syslog-ng PE news; ISO27001

Dear syslog-ng users,

This is the 35th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

Handing over syslog-ng maintenance

Bazsi (Balázs Scheidler) started the syslog-ng project many years ago and maintained it ever since. He handed over stable branches to algernon (Gergely Nagy) a few years ago and now also the master branch, where current development is happening.
You can read the original announcement at https://bazsi.blogs.balabit.com/2014/05/handing-over-syslog-ng-maintenance/

JSON and MongoDB support added to syslog-ng PE

The two major new features of the syslog-ng PE 5.1.1 release are JSON support (both parsing and emitting JSON formatted messages) and MongoDB destination. You might notice, that these features were available in OSE for a while. They were cleaned up, bug fixed and enhanced by the PE team, and many quality assurance tests were added. The resulting code is merged back to upcoming OSE 3.6 version with some of the bugfixes ported back to 3.5 and even to 3.4 in some cases.
You can read more about what is new.

Moving from BugZilla to GitHub Issues

As development of syslog-ng moved to GitHub a while ago and we started to use the more convenient GitHub Issues for bugtracking the current syslog-ng BugZilla will depreciated. No new issues will be allowed in Bugzilla from the 13th of June, but existing issues will remain there.
For more details check the announcement at https://lists.balabit.hu/pipermail/syslog-ng/2014-June/021458.html

Python and Perl support in incubator

Perl and python support were already introduced in last months newsletter, now it’s available as a release. Other new features include support for getent template function and enhanced graphite template.
Source code is available at https://github.com/balabit/syslog-ng-incubator and there are compiled packages available for Debian, Fedora, openS– USE, Ubuntu. FreeBSD will be updated after additional bugfixes.

ISO27001 and Log Management

PCI DSS wasn’t the only standard to be updated recently. A new version of ISO27001, an information security standard first published in 2005, was released last September. ISO27001:2013 provides a framework for implementing an Information Security Management System (ISMS). The new version has been modified to align better with other ISO standards. Ten new controls have been added with an emphasis on measuring the effectiveness of the ISMS. Just as with PCI DSS, we decided ISO27001’s importance and broad adoption merited a technical white paper dedicated exclusively to how log management and more specifically, the syslog-ng application and the syslog-ng Store Box, can meet the standard’s requirements.
You can download the whitepaper at https://pages.balabit.com/iso-27001-compliance-and-log-management.html

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-05: Web based user interfaces for syslog-ng; GsoC; perl & python; protecting logs

Dear syslog-ng users,

This is the 34th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

Web based user interfaces for syslog-ng

One of the most popular BalaBit blogs is about syslog-ng web based graphical user interfaces (web GUIs). It’s already three years old, and many things have changed. At that time, only a single Logging as a Service solution was available, now a new one pops up regularly. Also, there were a lot less logging-related GUIs, so some not strictly syslog-ng related solutions were included as well. You can read an updated version of the blog, focusing on syslog-ng based solutions.

Protecting log data against targeted attacks

BalaBit has been saying that SIEM and other analytic tools are only as good as the underlying data. Attackers are also aware of this, and often target log management and SIEMs to hide their presence. Read this blog post for some logging best practices and how syslog-ng can help to secure your logging infrastructure.

Four GsoC students are working on syslog-ng

Thanks to Google, there are four students working on extending syslog-ng with new features during the summer. These are features, which were often requested on the mailing list or at different conferences:

  • integration with configuration management systems
  • ZMQ transport, both source and destination
  • AMQP source driver
  • TLS support for the mongodb destination

Read more about GSoC.

Python and Perl support in incubator

It is still only available in git, as it needs some more polish, but the syslog-ng incubator gained Perl and Python support during the last month. Both the perl and python destinations use the value-pairs framework to get data transferred from syslog-ng to the script, and thus, work differently from the Lua destination. With value-pairs, one can select what parts of the message will be transferred to the script. The script will need to have a queue function (settable with the queue-func() option), which will receive a hash-map of values. Additionally, one can set an init and a deinit function too, to run whenever the driver starts or shuts down.

Check it out and let us know your experiences!

NEW RELEASES

  • Check git if you are impatient 🙂

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-03: GSoC; Incubator; Conferences

Dear syslog-ng users,

This is the 32th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

Google Summer of Code (GsoC)

This year the syslog-ng project was accepted to GSoC as a mentoring organization. We vould like to thank to the Debian and openS– USE projects for vouching for us and openS– USE to let us participate previous years under the openS– USE umbrella. We don’t know yet the number of students we can mentor, but we already have a nice number of project ideas and students who expressed interest.

It’s not yet too late to browse the ideas page and write a project proposal to the syslog-ng mailing list. Contact information is available at this page.

syslog-ng incubator updates

The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository. Version 0.2.1 was released recently with many bugfixes, advanced LUA related features, support for FreeBSD and a python script based ElasticSearch destination.

The syslog-ng incubator is available in many ways. First of all, the source is available in git.

There are also packages for openS– USE and Fedora with riemann support available in a subpackage. To use it, visit one of the below web pages and add the necessary repositories, which also have syslog-ng 3.5 and the riemann C client available.

A FreeBSD port is also work in progress, once ready, it will be committed to sysutils/syslog-ng-incubator. Riemann support is not included.

syslog-ng at conferences

The next confirmed events are:

  • Open Source Data Center Conference, 8-10 April, Berlin, Germany: http://www.netways.de/osdc/, Giving a talk titled “Monitoring with syslog-ng, Riemann and Kibana” There will be two developers present, so if anyone attends, and has Incubator, Lua, Riemann, etc. related questions, we’ll be happy to answer.
  • Linux Open Admin Days, 5-6 April, Antwerp, Belgium: http://loadays.org/, Giving a talk titled “Babel Fish for DevOps: syslog-ng”. Those, who visited FOSDEM will be familiar with the basis of the presentation, but this time with a lot more details and configuration examples thanks to the more available time.
  • Infosecurity Europe, 29 April – 1 May, London, United Kingdom: http://www.infosec.co.uk/, Giving a talk titled “Finding method in the madness: the challenges of the automatic classification of log messages”. The talk will be given by Balazs Scheidler (Bazsi) who you will also be able to find at the BalaBit booth at the event.
  • LinuxTag, 8-10 May, Berlin, Germany: http://linuxtag.org, Giving a talk titled “Finding method in the madness: the challenges of the automatic classification of log messages”. BalaBit will also be sponsoring the event so you’ll be able to find us and get T-shirts and talk with our engineers. The talk will be a revised version of the talk given a week before at Infosecurity Europe.

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.

insider 2014-02: FIPS; GSoC; Incubator; Conferences

Dear syslog-ng users,

This is the 31th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.

FEATURED NEWS

FIPS

Certain departments of the US government and those conducting business on certain regulated markets are required to use tools utilizing cryptography certified according to the FIPS 140-2 standard to handle sensitive data. As logs often contain sensitive information, this regulation can apply to the log management system, too.

Starting with the 5.0.4 release, we are offering a version of syslog-ng Premium Edition to customers that is built and shipped with the OpenSSL FIPS Object Module and as thus usable in environments where such validation is required. (For more information about the OpenSSL FIPS Object Module, see http://openssl.com/fips/.) This is a separate version and the original version is available that uses the standard OpenSSL library. It is currently available on major Linux platforms and has the same functionality as the non-compliant version apart from a few limitations listed in the documentation.
Please visit https://www.balabit.com/lp/fips if for more information.

Google Summer of Code (GsoC)

Based on the success of last years syslog-ng Gsoc participation (in co-operation with the openS– USE project), this year we plan to participate in GsoC again with a number of syslog-ng related projects. The current list of ideas range from new sources and destination through under the hood improvements to integration with configuration management systems.

syslog-ng incubator updates

The syslog-ng incubator is a collection of tools and modules which are not (yet) part of the official repository.

Since last month there are many smaller changes and brand new features. The master branch goes now hand in hand with syslog-ng’s master branch, which is currently 3.6 pre-alpha code. If you are working on syslog-ng 3.5, use the 3.5/master branch of syslog-ng-incubator.

On the feature side the Lua destination was enhanced, a monitoring source was added, together with a Graphite output. Among the Lua examples one can also find a script to push logs into ElasticSearch.

To try the new features, check out https://github.com/balabit/syslog-ng-incubator from git. You can read more about the Lua destination at https://talien.blogs.balabit.com/2014/02/lua-the-undiscovered-country/

syslog-ng at conferences

In the past few weeks we visited three conferences. While there we talked about syslog-ng to many people, some during organized presentations, and many more between presentations and social events of the conferences.

A look back at FOSDEM and Config Management Camp: https://czanik.blogs.balabit.com/2014/02/fosdem-2014-and-config-management-camp/

The next confirmed event is: Open Source Data Center Conference, 8-10 April, Berlin, Germany: http://www.netways.de/osdc/, Giving a talk titled “Monitoring with syslog-ng, Riemann and Kibana” There will be two developers present, so if anyone attends, and has Incubator, Lua, Riemann, etc. related questions, we’ll be happy to answer.

SHORT NEWS

NEW RELEASES

Your feedback and news tips about the next issue is welcome at documentation(at)balabit.com.