syslog-ng Insider – March 2012

Dear syslog-ng users,

This is the 12th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

GSoC wants you to code syslog-ng in the summer

GSoC is a nice opportunity for higher education students to spend their summers productively by coding in open source software projects. This time BalaBit participates in GSoC with the help of the openS– USE project. If you are interested in enhancing syslog-ng or Zorp, please see our project ideas on the openS– USE ideas page:

Alpha1 of syslog-ng 3.4 is released

The first alpha version of syslog-ng 3.4 is released. Major new features are junctions & channels which add even more flexibility to the syslog-ng configuration. There is now also a json parser, smtp destination and modules are now loaded automatically. For more details on what is new, please check Bazsi’s blog.

Instead of using the release, it is recommended to use sources from git, which have some major stability fixes.

If you intend to package syslog-ng 3.4, it’s recommended to check the mailing list for patches from Algernon, which make packaging easier.
There are already packages for openS– USE and an updated syslog-ng-devel port for FreeBSD.

The (r)evolution of name value pairs

Name value pairs were at the heart of syslog-ng even before PatternDB made it obvious. And now the CEE board and the Lumberjack project also push into this direction: instead of free form text messages, use name value pairs for logging. Recent developments in syslog-ng also serve this purpose: v3.3 can output name value pairs in JSON and v3.4 will be able to parse these logs and turn them into name value pairs again.

syslog-ng community forum

BalaBit has had a very positive experience with the syslog-ng community and we believe that your feedback has played a key role in the success of syslog-ng. We have decided to adopt this community model to our other products so that we can have our customers involved in product design. It could be interesting for you to visit this brand new community site. You are one of the first of our friends to be invited.

OTHER SHORT NEWS

NEW RELEASES

ARCHIVE

http://insider.blogs.balabit.com/

syslog-ng Insider – February 2012

Dear syslog-ng users,

This is the 11th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

Brno: Fedora, CEE, journal and syslog-ng

Last week the Brno Red Hat office hosted two conferences, one small about logging and the Fedora Developer conference. The logging miniconf covered some very hot topics: CEE, journal, auditd and some lesser known projects, like ELAPI. After the formal program, we had some very good discussions about the future of logging.
You can check the diagram drawn up as conclusion here:
http://czanik.blogs.balabit.com/2012/02/brno-fedora-cee-journal-and-syslog-ng/
And read more about how syslog-ng supports CEE: http://algernon.blogs.balabit.com/2012/02/cee-handling-with-syslog-ng/

BalaBit has just released the latest version of its leading log management tool, syslog-ng 4 F2

Adding to the existing, rich feature set which includes high-performance multi-thread processing, encrypted and timestamped log files, disk-based buffering, direct database access, native TLS support, the syslog-ng 4 F2 now supports Application-level Acknowledgement via Reliable Log Transport Protocol (RLTP)™ , a new transport protocol that prevents message loss during connection breaks. In addition, the latest version of syslog-ng can now natively collect and process log messages from SQL databases enabling users to easily manage log messages from a wide variety of enterprise software and custom applications.

syslog-ng 3.3.4 is released

It is a bugfix release, which fixes all previously known problems in the 3.3 series. There is only one change in 3.3 sources since the last release: manual pages were put under the GPL, and XML sources are now also available, so that the entire source code of syslog-ng is free from this point onwards.

Sources are available at https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/download.

Packages for some distributions are available from https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/downloads/3rd_party

Detailed changelog is available at https://www.balabit.com/files/syslog-ng/open-source-edition/3.3.4/changelog-en.txt

Documentation was also updated: https://www.balabit.com/support/documentation/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/bk01-toc.html

EU Data Protection Directive – How a single regulation could boost the transparency in IT security?

Personal opinion from Balázs Scheidler, CEO of BalaBit

Overall, the EU Data Protection Directive can be a milestone in boosting the transparency of IT security at organizations – similarly to the regulatory compliance regulations after the Enron case. If adopted, the new directive could bring about a change in the implementation of IT security policies so that the current focus on audits could shift to the deeper integration of IT security processes into business processes.
As logging and log management are the base of every monitoring method, technologies with high-speed and zero message loss capabilities, like syslog-ng, will come to the front. Encrypting log files, in which companies store user names, passwords and other sensitive company data is also key to prevent data loss. http://bscheidler.blogs.balabit.com/2012/02/eu-data-protection-directive-how-a-single-regulation-could-boost-the-transparency-in-it-security/

OTHER SHORT NEWS

NEW RELEASES

ARCHIVE

http://insider.blogs.balabit.com/

syslog-ng Insider – January 2012

Dear syslog-ng users,

This is the 10th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

POLL

We would like to know, which features of syslog-ng do you regard as most important. Please check our survey.

FEATURED NEWS

syslog-ng 3.3.4 is being prepared

The release of 3.3.3 brought many new users to the latest syslog-ng version, which helped to uncover some more bugs in advanced configurations. Until a new release arrives, please check the git tree ( git://git.balabit.hu/bazsi/syslog-ng-3.3 ) and the mailing list archives, if your problem is addressed by a patch.

If working directly from git is problematic in your environment, a daily shapshot of 3.3 development is available at http://packages.madhouse-project.org/syslog-ng/3.3/syslog-ng-3.3-HEAD.tar.gz.

Meet us at FOSDEM 2012 in Brussels!

There will be one or two presentations held by BalaBit guys this year. As schedule and rooms are not yet final, please check http://www.fosdem.org/ if you want to meet us or drop me an e-mail!

Junction

Version 3.4 of syslog-ng is under intensive development. One of the new features, the “junction”, which makes syslog-ng.conf even more flexible, was just introduced on the syslog-ng mailing list: https://lists.balabit.hu/pipermail/syslog-ng/2012-January/018074.html.

OTHER SHORT NEWS

WHITE PAPERS

  • SIEM solutions are the core of enterprise security. But they are not much use without data integrity and high performance centralized log collection. Learn how to enhance your SIEM with syslog-ng from our latest white paper.

ARCHIVE

http://insider.blogs.balabit.com/

syslog-ng Insider – December 2011

Dear syslog-ng users,

This is the 9th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

syslog-ng 3.3.3 is released!

A new version of syslog-ng is released! There are no new features to announce, but most problems reported since 3.3.1 should be fixed by now! Thank you for all of those, who helped us to hunt bugs with detailed reports and many testing!
The release of 3.3.3 brought many new users to the latest syslog-ng version, which helped to uncover some more bugs in advanced configurations. Until a new release arrives, please check the git tree ( git://git.balabit.hu/bazsi/syslog-ng-3.3 ) and the mailing list archives, if your problem is addressed by a patch.

Sources are available in git or as a snapshot:

Binary packages are available are available for several Linux distributions. Please check availability at

syslog-ng and patterns

Patterndb is one of the most important features of syslog-ng, still not many people are using it. So we are very happy to see, that patterndb was the focus point in many recent syslog-ng mailing list threads.
First of all, thanks for Evan Rempel for providing many useful ideas and feedback about patterndb on the syslog-ng mailing list.
ELSA (Enterprise Log and Search Archive), which uses patterndb heavily, had some major updates recently, which make it a lot more easy to install on a couple of different systems. It is available at http://code.google.com/p/enterprise-log-search-and-archive/
We plan to use CEE for our patterns in the long term. But even until this standard is available, please share your patterns in any form to lower the entry barrier for your fellow syslog-ng users. If you send them to the list or directly to me, I’ll make them available.

syslog-ng and the journal

There’s an ongoing project to create a new logging subsystem for Linux, called the journal, by Lennart Poettering of PulseAudio & systemd fame. It is implemented as a core component of systemd, thus has a good chance to be integrated to all distributions that carry systemd. Since syslog-ng is also in the logging sphere, the logical question arises: how does this new project affect syslog-ng in the long run?
For the answer, read Bazsi’s blog.

OTHER SHORT NEWS

  • An interesting article about extracting useful information from log messages was published in Free BSD Magazine (not only for BSD users 🙂 ): where you also can read about several up-to-date topics, like “Rolling Your Own FreeBSD Kernel”, “Hardening BSD with Security Levels” and so on. The whole Free BSD Magazine can be downloaded at http://bsdmag.org.

NEW RELEASES

ARCHIVE

http://insider.blogs.balabit.com/

syslog-ng Insider – November 2011

Dear syslog-ng users,

This is the 8th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

syslog-ng 3.3.2 is about to be released!

A new version of syslog-ng is about to be released! There are no new features to announce, but all problems reported since 3.3.1 should be fixed by now! To make it the best syslog-ng ever, please test it to make sure, that all your problems are fixed.

Sources are available in git or as a snapshot:

  • git://git.balabit.hu/bazsi/syslog-ng-3.3

Binary packages are available are available for several Linux distributions:

syslog-ng and CEE

The latest syslog-ng release, version 3.3 can be used to implement part of the “CEE over syslog” standard. BalaBit’s patterndb technology was able to extract information from syslog messages already for a long time. With this release JSON output was added, meaning the extracted information can be output as JSON data. What it means in practice, that syslog-ng is able to parse log messages, and output the extracted fields in the form required by CEE.
To see, how it works, check http://czanik.blogs.balabit.com/2011/10/cee-and-syslog-ng/

Development of syslog-ng 3.4 started

While 3.3 was just released, development of 3.4 is already started. The first version of a JSON parser is already merged. There are some pending fixes and enhancements, which add boolean, array and nested JSON parsing. Value-pairs key rewrite is work in progress and nested JSON output is also planned.
The above features among others help us to better support CEE. With key rewriting we could use a “.cee.” prefix in CEE related patterns and rewrite it later. It also makes parsing of messages possible.
All the current code is available for testing in Algernon’s 3.4 sandbox project.
To download it, use git:
$ git clone -b sandbox/3.4 git@github.com:algernon/syslog-ng

OTHER SHORT NEWS

NEW RELEASES

WHITE PAPERS

A longer paper about the “Future of logging tools”, which also provides some background information about HSRL, as used in syslog-ng.

It is available at http://andrea.blogs.balabit.com/files/2011/10/HSRL_backgrounder_english_final1.pdf

ARCHIVE

http://insider.blogs.balabit.com/

syslog-ng Insider – October 2011

Dear syslog-ng users,

This is the 7th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

syslog-ng 3.3.1 is released!

A new version of syslog-ng is released! It took about a year to develop as a collaboration of over 50 individuals. Major new features include:

  • mongodb support
  • JSON support
  • Multi-threaded architecture

For a complete list of changes, please check the NEWS file

Documentation is also available: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/index.html-single.html

Binary packages are available are already available for several Linux distributions:

Many other distributions have already experimental ports, ebuilds, etc. including Arch, Fedora, FreeBSD, Gentoo.

Regulatory compliance and system logging

We are happy to announce our latest technical white paper titled “Regulatory compliance and system logging”. The document introduces the advantages of using syslog-ng Premium Edition to collect system log (syslog) and eventlog messages for policy compliance. The white paper includes a detailed list of policy requirements, including the requirements of the Payment Card Industry Data Security Standard (PCI-DSS), COBIT 4.1, ISO 27001, and the Health Insurance Portability and Accountability Act (HIPAA) that you can address with syslog-ng Premium Edition. The document is recommended for technical experts and decision makers working on implementing centralized logging solutions, and want to improve their compliance with industry standards. In addition, the white paper discusses further features of syslog-ng PE that can come handy for you when designing and implementing your system logging architecture.
While the document focuses on syslog-ng PE, part of it also applies to syslog-ng OSE. It is available at https://www.balabit.com/support/documentation/syslog-ng-v3.0-whitepaper-compliance-en_0.pdf

OTHER SHORT NEWS

NEW RELEASES

WHITE PAPERS

Download our latest white paper titled “Business benefits of logging“. From this White Paper you can learn:

  • How logging-based benefits can make the work of managers more successful.
  • How logging contributes to lowering the operational costs of organizations while making them more efficient.
  • How syslog-ng logging technology can contribute to organizations’ business requirements.

It is available at https://www.balabit.com/whitepaper?wp=46820069482870468259863

ARCHIVE

http://insider.blogs.balabit.com/

syslog-ng Insider – September 2011

Dear syslog-ng users,

This is the 6th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

POLL

Please take a minute to answer three syslog-ng performance related questions at https://www.surveymonkey.com/s/6ZQDVH6

FEATURED NEWS

Please test 3.3 git!

Version 3.3 release is just around the corner. There were many smaller fixes since the last beta release, so please check out the latest source code from git and help us to make sure the release is free from known problems!

Your feedback is very valuable, especially if you could test it in real world situations with logs and configurations we could never imagine ourselves. As threading is a major new feature, which is not enabled by default, please try it by adding “threaded(yes)” to your options in syslog-ng.conf

You can download it by “git clone git://git.balabit.hu/bazsi/syslog-ng-3.3”

Documentation is also available: target=&https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/index.html-single.html

Binary packages from latest git are available:

Logs for patterns, patterns for logs

We also would like to extend our UNIX/Linux patterns. Creating logs in a “lab” environment just for pattern creation is very time consuming. It would be very helpful for us, if you could send logs in exchange for patterns. Just make sure, that there is no sensitive data left in the logs, as the result will be published to make it available for the whole syslog-ng community.
I published a blog ( http://czanik.blogs.balabit.com/2010/11/log-sample-collecting-project/ ), how I collect logs for pattern creation, but of course, any logs are welcome!

Syslog clients for Windows

Central logging using syslog is long part of the UNIX / Linux infrastructure. But if someone also happens to have Windows machines, it is still possible to use the proven syslog-ng servers. There are many clients available, both open and closed source, ranging from simple event forwarders to complex logging solutions. Here is a collection of them:
http://czanik.blogs.balabit.com/2011/09/syslog-clients-for-windows/.

OTHER SHORT NEWS

WHITE PAPERS

Download our latest white paper titled “Logging, the Pillar of Compliance”.

  • How can you avoid a breach of compliance, and ensure your business continuity.
  • What are the key IT security requirements of the most frequently applied standards, such as the ISO 27001, PCI, SOX or COBIT.
  • How advanced logging technology can contribute to cost-effective compliance and successful accomplishment of audits.

It is available at https://www.balabit.com/compliance-and-logging

syslog-ng PE Case Study – DataPath Inc.

DataPath, founded in 1984, is a management-owned, privately held company based in Little Rock, Arkansas, that produces software solutions for administering employee benefit plans. They implemented a SYSLOG-NG AGENT FOR WINDOWS-based logging infrastructure to meet HIPAA and PCI DSS requirements in their Microsoft-based environment.
It is available at https://www.balabit.com/support/documentation/PE_DataPath_en.pdf

ARCHIVE

http://insider.blogs.balabit.com/

syslog-ng Insider – August 2011

Dear syslog-ng users,

This is the 5th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

Please test 3.3 beta2!

Version 3.3 is a major step in syslog-ng’s development, as it is now fully multi-threaded, and is also the first release with considerable amount of community developed features. A new beta was released this weekend! It received many changes and fixes since the beta1 release.
According to internal testing, it works nice and stable, but we already received some external problem reports. So your feedback is very valuable, especially if you could test it in real world situations with logs and configurations we could never imagine ourselves. As threading is a major new feature, which is not enabled by default, please try it by adding “threaded(yes)” to your options in syslog-ng.conf

Please download sources from https://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.0beta2 !
For a list of changes and fixes check the announcment at http://lists.balabit.hu/pipermail/syslog-ng-announce/2011-August/000117.html
If you use FreeBSD, ports is already updated to beta2: http://www.freshports.org/sysutils/syslog-ng3-devel/
Documentation is also available: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/index.html-single.html

Windows in focus

The syslog-ng application is often used in a Windows environment. There is now an ongoing blog series about syslog clients for Windows. BalaBit also released some patterns for Windows.
The patterns are available from http://czanik.blogs.balabit.com/2011/07/patterns-for-windows-server-2008/
Blogs about syslog clients:

To be continued…

Performance

Both syslog-ng OSE and PE are now multi threaded, which brings performance to extreme. On the test machine HDD and multiple gigabit Ethernet lines were the limiting factor, not syslog-ng. The test were done using PE, but OSE performance should be similar:
http://pzolee.blogs.balabit.com/2011/07/do-you-want-to-process-800-000-messagessec/
BalaBit call the features and technologies aimed compliance at a high performance HSRL (High Spead Reliable Logging).

POLL

Please take a minute to answer three syslog-ng related questions at https://www.surveymonkey.com/s/6ZQDVH6

OTHER SHORT NEWS

NEW RELEASES

syslog-ng Insider – June 2011

Dear syslog-ng users,

This is the 4th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

syslog-ng FAQ moved and updated

The syslog-ng FAQ was maintained by Nate Campi for many years. Questions and answers for old releases are still available at the old URL. FAQ for recent releases is now maintained in-house and available at https://www.balabit.com/wiki/syslog-ng-faq

To place a question of yours into the syslog-ng FAQ, please don’t hesitate to contact us on the mailing list, which is available at http://lists.balabit.hu/mailman/listinfo/syslog-ng

syslog-ng OSE 3.3 beta1 released

The first beta version of syslog-ng 3.3 was released. This is a major step in syslog-ng’s development, as it is now fully multi-threaded, and is also the first release with considerable amount of community developed features.
For a full list of features, changes and fixes check the announcment at http://lists.balabit.hu/pipermail/syslog-ng/2011-May/016624.html

Development of syslog-ng OSE 3.4 started

As syslog-ng 3.3 is in feature freeze, new development goes on in the 3.4 repo. Looking at http://git.balabit.hu/?p=bazsi/syslog-ng-3.4.git shows some very interesting commit logs:

  • $(sanitize): add new template function useful to sanitize filenames
  • basicfuncs: Implement a $(substr STR START [LEN]) template function.
  • basicfuncs: Implement a few numeric template functions

While not yet merged, the following blog has a preview of a planned extension to the recently merged value-pairs() functionality

Blog series on web GUIs for syslog-ng

Web based GUIs for syslog-ng is a hot topic recently, so a new blog series was started a few month ago. This month I covered LogStash. If you have any suggestions what else to cover, please let us know!

CVE-2011-1951: problems, when syslog-ng is compiled with PCRE 8.12+

Under certain circumstances Versions 3.0, 3.1 and 3.2 of syslog-ng Open Source Edition (OSE) are vulnerable to a Denial of Service attack if the PCRE engine is enabled in syslog-ng and libpcre version 8.12 is installed.
The syslog-ng Premium Edition (PE) application is not affected, as it uses a different version of the libpcre package.
In libpcre version 8.12 a return value has been changed. This change causes an infinite loop in syslog-ng if a pcre filter is used and the global flag is enabled for the expression. If such a filter expression is used in the configuration of syslog-ng and a log message does not match the regular expression (which most probably happens within seconds of starting an affected version of syslog-ng), syslog-ng consumes the processor resources and denial of service occurs.
All 3.X branches are affected before 3.2.4
syslog-ng Open Source Edition (OSE):
Branch
3.0.X < 3.0.11
3.1.X < 3.1.5
3.2.X < 3.2.4

OTHER SHORT NEWS

NEW RELEASES

RECENT WHITEPAPERS

  • There is a new WP in preparation about syslog-ng GUIs. I published the basis of it as a blog already. Your comments are very welcome!

syslog-ng Insider – May 2011

Dear syslog-ng users,

This is the 3rd issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news.
Your feedback and news tips about the next issue is welcome at
documentation(at)balabit.com

FEATURED NEWS

syslog-ng OSE 3.3 development

The previous month did not see a new syslog-ng 3.3 alpha or beta release, but development still went on. There were many important bugfixes and performance improvements. Also, most of the patches from the community were merged, including value-pairs(), mongo-db updates and JSON support. There is now a feature freeze in effect, so new features go now into the freshly opened syslog-ng 3.4 repository.
The current version can be checked out out from git.

syslog-ng called open source project of the year

IDG hold its first Open Source conference on the 24th of February in Hungary. A new award was established for open source projects, and it was first given to syslog-ng OSE.

Blog series on web GUIs for syslog-ng

Web based GUIs for syslog-ng is a hot topic recently, so a new blog series was started. This month I’ll cover a new challenger: ELSA, which has patterndb at its heart.If you have any suggestions what else to cover, please let us know!

OTHER SHORT NEWS

NEW RELEASES

RECENT WHITEPAPERS