[Xymon] Help with very large log file - not getting the right lines

[Jeremy Laidman]

On Wed, Nov 23, 2011 at 10:06 AM, Elizabeth Schwartz
<betsy.schwartz at gmail.com> wrote:
> I've got to monitor some very large log files. They're up to a couple

You can use the GNU tail "--retry" and "--follow=name" options to tail
a file even through log rotation.  Then pipe it into grep (set
bufferring to per-line if more than one grep is used) and send the
output via "logger" to syslog.  If you simply redirect to another file
rather than logger/syslog, you have problems when you want to
zero/rotate the output file, so using logger/syslog is easier that
way.

For example, add this into /etc/inittab:

tail --follow=MMRequest.log | grep "ERROR servicename LotsOfText" |
grep --line-buffered -v "LOG NNNNN servicename LotsOfHTML" | logger -p
user2.info -t logwatch

Note that some versions of init can't handle inittab lines that are
too long, so you'd put it into a script.

> but I'm just not getting the ERROR lines in the log. Is this file just
> too large and too full of HTML to parse? Any suggestions?

Yes, could be.  I've had problems using the "log:" stuff with
fast-moving logfiles.

Also, I've found bugs in the "logfetch"  program in some cases of
"trigger" and "ignore".

Cheers
Jeremy