filelogger

A small Linux program that continously monitors one or more files and sends a copy of each new line to local or remote syslog.

Features

  • Very lightweight, fast, and stable
  • Extremely simple to use, requires no extra service setup or additional applications. Just launch a single command on the command line and leave it running.
  • Continues to run if log files are rotated or even if they disappear altogether: will simply wait for the file to reappear
  • supports long lines which means you can use it on your Apache logs (does not have the 400 character limit imposed by the system logger)
  • Can be run in an unprivileged account and can monitor any file as long as it has read-access
  • Will run forever or until the system is rebooted or it is unable to write to the log socket
  • write to any local or remote syslog, over TCP or UDP

Also a useful testing and troubleshooting tool: you can quickly and easily - within a few seconds - test whether your local system can log to a remote syslog or Splunk server, without going through the hassle of configuring your local syslogd.conf and restarting the syslog daemon. Just launch filelogger on the command line and feed it a log file.

Performance metrics

17,000 lines of text (1.8MB) in 3 seconds from first appearnce in local file to appearance on disk on remote server running syslog-ng, the two systems on different subnets separated via physical switch and VLAN.

Download and Installation

Download: https://github.com/yahoo/filelogger/releases/tag/v1.0

You may either download the binary called "filelogger". Chances are very good this will work out of the box on your 64-bit CentOS or Redhat 6.x system.

Or you may compile your own binary, following the instructions in the INSTALL file. You will need a copy of the source code for util-linux-2.20.1 and coreutils-8.9.

Although I haven't tested on other systems, chances are very good it will compile on any Unix system, if you can get util-linux and GNU coreutils to compile.

Licensing

tail.c is licensed under GPL, while logger.c is under old BSD. In this situation, the GPL supersedes and therefore filelogger is distributed under GPL. Please be aware of this.

Usage

This program combines "logger" functionality with "tail" functionality:

/* tail: all original command-line options removed
 * except q, v, and s. tail -F is assumed.
 * stdin is not allowed. must specify a file
 * by name.
 */

/* logger: an additional command line option
 * --add which allows additional text to be
 *  inserted at the beginning of the line before
 *  sent to the syslog server.
 *  The previous hardcoded 400 character limit
 *  for the log message has been increased to 8096.
 */

The program takes the same command line arguments as util-linux "logger" (ubiquitous on Linux systems), plus the additional arguments:

  • -a or --add: (optional) insert an extra text string of your choice at the beginning of each log line
  • -S: (optional) polling interval for the tail functionality. default 1 second
  • It also takes the "tail" options as described above.

    Run "man logger" and "man tail" on your Linux system to learn about the command-line options.

    filelogger is a great candidate for running under daemontools.

    Here is an example of how to launch it on the command line:

    ./filelogger -t 'access' -d -p local1.info -n loghost.mydomain.com -u /tmp/ignored -a "additional text string" /var/log/httpd/access
    

History

I wrote filelogger while working as a contractor at Yahoo.

To say that I "wrote it" is an overstatement. I hacked together tail.c and logger.c. The reason I did this was to address the following needs:

  • single binary program for ease of process supervision and monitoring of exit code (do not use shell pipes such as "tail | logger" as these pose difficulties)
  • does not need syslog-ng or any other log server software running on the local system
  • can be run as unprivileged user: you can capture log files and send them to remote syslog server even if you are not root
  • removes the silly 400-byte limit on the size of the log line

filelogger is a combination of "logger.c" from util-linux and "tail.c" from GNU coreutils. It retains all the logger functionality and enough of the tail functionality to do what it needs. It includes an additional command line option as described above.

On Additionally it removes the byte limit restriction imposed by the native logger program. On Linux systems, there is a hardcoded 400-character limit on the length of the log line. You can see this for yourself by running the following command:

strings $(which logger) | grep 400
The output from this is :
<%d>%.15s %.200s%s: %.400s

Sometimes it is desirable to capture longer lines, so I increased this to 8096.

filelogger Copyright 2014 Yahoo Inc. All Rights Reserved. See the copyright and License notices in the distribution package for details.