Skip to content

Log Courier, a lightweight log shipper with Logstash integration.

License

Notifications You must be signed in to change notification settings

Scratch-net/log-courier

 
 

Repository files navigation

Log Courier

Build Status Latest Release

Log Courier is a lightweight tool created to ship log files speedily and securely, with low resource usage, to remote Logstash instances. The project is an enhanced fork of Logstash Forwarder 0.3.1 with many fixes and behavioural improvements.

Table of Contents generated with DocToc

Main Features

  • Read events from a file or over a Unix pipeline
  • Follow log file rotations and movements
  • Close files after inactivity, reopening if they change
  • Add extra fields to events prior to shipping
  • Reload configuration without restarting
  • Ship events securely using TLS with server (and optionally client) certificate verification
  • Ship events securely to multiple Logstash instances using ZeroMQ with Curve security (requires ZeroMQ 4+)
  • Ship events in plaintext using TCP
  • Ship events in plaintext using ZeroMQ (requires ZeroMQ 3+)
  • Monitor shipping speed and status with the Administration utility
  • Pre-process events using codecs (e.g. Multiline, Filter)
  • Logstash Integration with an input and output plugin
  • Very low resource usage

Differences to Logstash Forwarder

Log Courier is an enhanced fork of Logstash Forwarder 0.3.1 with many fixes and behavioural improvements. The primary changes are:

  • The publisher protocol is rewritten to avoid many causes of "i/o timeout" which would result in duplicate events sent to Logstash
  • The prospector and registrar are heavily revamped to handle log rotations and movements far more reliably, and to report errors cleanly
  • The harvester is improved to retry if an error occurred rather than stop
  • The configuration can be reloaded without restarting
  • An administration tool is available which can display the shipping speed and status of all watched log files
  • Fields configurations can contain arrays and dictionaries, not just strings
  • Codec support is available which allows multiline processing at the sender side
  • A TCP transport is available which removes the requirement for SSL certificates
  • There is support for client SSL certificate verification
  • Peer IP address and certificate DN can be added to received events in Logstash to distinguish events send from different instances
  • Windows: Log files are not locked allowing log rotation to occur
  • Windows: Log rotation is detected correctly

Public Repositories

RPM

The author maintains a COPR repository with RedHat/CentOS compatible RPMs that may be installed using yum. This repository depends on the widely used EPEL repository for dependencies.

The EPEL repository can be installed automatically on CentOS distributions by running yum install epel-release. Otherwise, you may follow the instructions on the EPEL homepage.

To install the Log Courier repository, download the corresponding .repo configuration file below, and place it in /etc/yum.repos.d. Log Courier may then be installed using yum install log-courier.

NOTE: The RPM packages versions of Log Courier are built using ZeroMQ 3.2 and therefore do not support the encrypted zmq transport. They do support the unencrypted plainzmq transport.

DEB

A Debian/Ubuntu compatible PPA repository is under consideration. At the moment, no such repository exists.

Building From Source

Requirements:

  1. Linux, Unix, OS X or Windows
  2. The golang compiler tools (1.2-1.4)
  3. git
  4. GNU make

Linux/Unix: Most requirements can usually be installed by your favourite package manager.
OS X: Git and GNU make are provided automatically by XCode.
Windows: GNU make for Windows can be found here.

To build the binaries, simply run make as follows.

git clone https://github.com/driskell/log-courier
cd log-courier
make

The log-courier program can then be found in the 'bin' folder. This can be manually installed anywhere on your system. Startup scripts for various platforms can be found in the contrib/initscripts folder.

Note: If you receive errors whilst running make, try gmake instead.

Logstash Integration

Log Courier does not utilise the lumberjack Logstash plugin and instead uses its own custom plugin. This allows significant enhancements to the integration far beyond the lumberjack protocol allows.

Install using the Logstash 1.5+ Plugin manager.

cd /path/to/logstash
bin/logstash plugin install logstash-input-log-courier

Detailed instructions, including integration with Logstash 1.4.x, can be found on the Logstash Integration page.

Generating Certificates and Keys

Log Courier provides two commands to help generate SSL certificates and Curve keys, lc-tlscert and lc-curvekey respectively. Both are bundled with the packages provided by the public repositories.

When building from source, running make selfsigned will automatically build and run the lc-tlscert utility that can quickly and easily generate a self-signed certificate for the TLS shipping transport.

Likewise, running make curvekey will automatically build and run the lc-curvekey utility that can quickly and easily generate CurveZMQ key pairs for the CurveZMQ shipping transport. This tool is only available when Log Courier is built with ZeroMQ >=4.0.

ZeroMQ support

To use the 'plainzmq' or 'zmq' transports, you will need to install ZeroMQ (>=3.2 for 'plainzmq', >=4.0 for 'zmq' which supports encryption).

Linux\Unix: ZeroMQ >=3.2 is usually available via the package manager. ZeroMQ >=4.0 may need to be built and installed manually.
OS X: ZeroMQ can be installed via Homebrew.
Windows: ZeroMQ will need to be built and installed manually.

Once the required version of ZeroMQ is installed, run the corresponding make command to build Log Courier with the ZMQ transports.

# ZeroMQ >=3.2 - cleartext 'plainzmq' transport
make with=zmq3
# ZeroMQ >=4.0 - both cleartext 'plainzmq' and encrypted 'zmq' transport
make with=zmq4

Note: If you receive errors whilst running make, try gmake instead.

Please ensure that the versions of ZeroMQ installed on the Logstash hosts and the Log Courier hosts are of the same major version. A Log Courier host that has ZeroMQ 4.0.5 will not work with a Logstash host using ZeroMQ 3.2.4 (but will work with a Logstash host using ZeroMQ 4.0.4.)

Documentation

About

Log Courier, a lightweight log shipper with Logstash integration.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 66.7%
  • Ruby 31.4%
  • Other 1.9%