[hobbit] Logfile monitoring - I'd like some comments

[Rolf Schrittenlocher]

Hi Henrik,

that sounds wonderful, your ideas are pretty good. I have one suggestion 
and one point to think about.

For monitoring application-logs the config file soon would become very 
complex and likely unreadable. There it would be nice if there is a 
possibilty for include-files. So general definitions -e.g. for 
OS-related logs - would be made in the general config  file, all the 
individual stuff in <host>.config files.

Second, there are logs which change their names once the server or the 
application stops. So a "message.log" might become "message<date>.log" 
while there is a new message.log now. Still the last informations in 
"message<date>.log" are relevant, especially if the reason for the new 
log was a crash of the application. I don't know how to deal with this 
situation as there are multiple ways how logs might change their names 
but perhaps others have an idea of how to do that.

kind regards
Rolf

>A few days ago, I mentioned that I would like to do logfile
>monitoring for the next Hobbit release.
>
>I've worked a bit on this and have a prototype solution for
>it, which you can test with the current snapshots. I'd like
>some comments on how it works to make sure I haven't overlooked
>something before committing myself.
>
>There are several objectives:
>- As far as is possible, logfile monitoring must be configured
>  centrally, on the Hobbit server. Having to go to each server
>  to (re)configure what logfiles to check and what to look for
>  simply doesn't work.
>- The amount of data sent from each client to Hobbit should be
>  small, but it must catch the "important" stuff.
>- You rarely know in advance what will be in the logs when you
>  need them the most. So the monitor should give you as much
>  of the log entries as possible, not just those lines that
>  match some pre-defined strings or regex'es.
>- Some systems log messages on multiple lines. The system must
>  be able to show all parts of a log entry.
>- Logfile entries must appear on the monitor for some time after
>  they show up in the logs, but should also disappear after a
>  while.
>
>In other words: The ideal solution would let you have the entire
>logfile available on the Hobbit server - but that obviously 
>won't work. So the client should - after weeding out the really 
>irrelevant stuff - send us as much of each logfile as possible.
>
>My proposed solution is this:
>- On the Hobbit server, there's a log-monitoring configuration
>  file for the Hobbit clients. This defines which logfiles are
>  monitored for a single client installation, or you can define
>  it for a group of clients. (The idea is to define at least 
>  one group for each operating system, since the standard
>  system logs are OS dependant). This configuration lists the
>  log filename, the maximum amount of data to send from this
>  logfile, a regex "noise" filter (i.e. lines that are stripped
>  from the logfile), and *optionally* a regex identifying really
>  interesting stuff in the logfile that should always be 
>  reported.
>- When a client connects to the Hobbit server and sends the
>  normal client message, the Hobbit server will respond with
>  the logfile configuration for this client. So the client
>  has a copy of the central configuration file, but only the
>  part that it needs for itself. The reason for sending this
>  as a response to the client message is to avoid an extra
>  round-trip from client to server; piggy-backing the config
>  push on the client message means that it is almost without
>  any performance cost on the server side.
>- When the client runs, it uses the local copy of the configuration
>  file to determine what logs to look at. For each logfile, it
>  maintains a "where-was-I-the-last-time" status, so it only
>  looks at the entries made to the logfile during the past 30
>  minutes. First, the client strips off any "noise" messages.
>  Then, if all of the entries fit into the maximum size that
>  can be reported, it sends all of the log to the Hobbit server.
>  If there is more than will fit, it first checks to see of the
>  regex defining the really interesting stuff is present in the
>  log - if it is, then it drops anything before the interesting
>  text. If there is still more than will fit, it keeps the
>  interesting text + a few lines after that (to allow for
>  multi-line log-entries which some OS'es have), and then
>  sends that together with as much of last part the log as will
>  fit inside the max. message size.
>
>This part has been implemented in the Hobbit daemon (hobbitd),
>and in the clients via a new "logfetch" utility. This utility
>uses standard regular expressions - not the Perl-compatible
>ones, because that would require you to install the PCRE
>library on all of your clients. The standard regex routines
>are included in all (I think) system libraries used today.
>
>The last part is what happens when the log data arrives on the
>Hobbit server. Currently, there's a simple processing of this
>data to just dump it into an always-green "msgs" column. What
>should happen once I get it coded is:
>- Data from each logfile is matched against a set of strings 
>  (regex'es) defined in the hobbit-clients.cfg file. Each string 
>  determines the color (red, yellow, green) and sets the color
>  of the msgs column accordingly.
>
>When the color has been decided, all of the normal alerting
>happens automatically. I do plan on making a more fine-grained
>alert mechanism (for the msgs, procs and disk statuses) so you
>can direct alerts to different groups depending on exactly 
>which log-message triggered the alert, but that will not be
>part of this release.
>
>
>So - how does that sound ? Anything I've missed ?
>
>
>Regards,
>Henrik
>
>
>To unsubscribe from the hobbit list, send an e-mail to
>hobbit-unsubscribe at hswn.dk
>
>
>
>  
>


-- 
Mit freundlichen Gruessen
Rolf Schrittenlocher

HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt 
Tel: (49) 69 - 798 28908   Fax: (49) 69 798 2881
LBS: lbs-f at mlist.uni-frankfurt.de
Persoenlich: schritte at rz.uni-frankfurt.de