The One and the Many

On monitoring logs

Like many people, I have a personal server (actually several). Until recently I have not been good at monitoring logs. I was lazy and it was never very easy. I recently set up a new physical server, and I decided I should be more diligent about monitoring it. In this post I talk about the system I built to monitor logs.

I started out with these basic requirements:

Essentially I wanted to see only messages I might care about. There are a lot of regular status logs that are not interesting for me to look at.

I decided to write a program in Go. I knew there exist solutions to do this type of thing, such as Logwatch. I did not try existing solutions because of the last requirement. I wanted to have a complete understanding of the log gathering process.

My first version worked roughly like this:

As input the program took:

This let me see log messages I cared about since the last time I ran it. However, it was awkward to use:

There was also the up front cost of adding patterns to exclude log messages I did not care to see.

This did work for me. I monitored logs on one server this way for a few weeks. But I wanted to monitor the logs on all of my servers, not just one.

I decided to start using this program to monitor logs on two more servers. I found this to involve more manual effort than I wanted.

I updated the program to automatically track its starting time so it could know the date/time to start collecting logs from without me having to tell it. Then I ran the program from cron on each host.

It was time consuming: I was receiving emails from three hosts that I needed to look at. I needed to take each report and add patterns to exclude messages I didn't care about. Once I had added the patterns, I needed to update the configuration file on each host.

At this point I started looking at solutions others had built. I read an interesting post about Fluentd, so I read more about whether Fluentd could work the way I wanted. That is, could I make it monitor arbitrary raw log files? At first I thought it was not able to do this, but today I found it has a core plugin called in_tail which would work.

I also looked into Logstash. There is a product from the same company called Filebeat which I thought would do what I needed. I found today that there is a core Logstash plugin called file that would work too, so in theory Filebeat would not be needed. I could run a Logstash instance on each host.

There are drawbacks with the above though:

Both have capabilities to deal with log rotation, but it's a worry. My program avoids the log rotation problem by looking at all log files when it runs, so it does not need to track a file position or whether a log rotated. Instead it decides depending on each log line's time whether it should report the line. This has the drawback that it usually must read/parse messages that could be skipped entirely using the tail method of course.

I tried out using Logstash using the file input plugin and the Elasticsearch output plugin to gather logs to a central location. It did work well, but I thought to myself: If I can write something similar myself that would work exactly how I want, why not?

I decided to improve how my program worked to reduce the work load on me. I split it up into three programs:

  1. logauditsubmit is a log submission client and runs via cron (daily). It has a configuration file that tells it what files to monitor. It reads them in, skips any older than the last run time, and then submits them via an HTTP request to logauditd.
  2. logauditd acts as a central log ingestion point. It receives log lines from logauditsubmit clients (HTTP) and inserts them into a PostgreSQL database.
  3. logaudit queries for logs from the PostgreSQL database. It asks for lines since its last run time, and then applies regular expressions to each line to know whether to report it or not.

This means I don't need to make regular changes on each host (assuming there are no new log files to configure). Each day, each host reports its new logs to the logauditd server. Afterwards, I run a single report to see those logs. If there are lines I don't want to see any more, I have a single configuration file to update (logaudit's).

This system is fairly rudimentary but it is working well for me so far since I don't need realtime logs. I only care to look at the logs periodically so having them once a day works fine.

I have the project up on Github.