Dealing with large log files

[Matthew Epp <matthew.epp (at) us.army.mil>]

I've researched this problem through the list archives, and haven't 
found an easy solution for this yet. Basically, we have a application 
that dumps roughly 25k of data into a log file every SECOND. Now, I'm 
trying to figure out if the client-local.cfg is able to do what I think 
the man page says.

It states that if you use a TRIGGER line, that the Hobbit client will 
parse and only match against lines that are in the trigger, and only 
send those to the server. Is that correct? If so, shouldn't it not 
matter that there's a lot of data being added to the log itself, as it's 
not all getting sent?

The other issue is, with the settings for this application, the log file 
ends up rotating every 10 minutes. This means that there is a chance of 
missing a match, if the file rotates in between polling cycles. How can 
I increase the interval of log file checking?

I thought if maybe I added an IGNORE line, and dumped all of the 
extraneous data, maybe then the server could handle it. But it doesn't 
seem to be paying any attention to it.

My client-local.cfg looks like this:
[server01]
log:/var/adm/messages:10240
log:/logs/http/http:10240
ignore General\sInformation|Account\sNotice|Account\sInformation
trigger DSA\sis\sbusy

And hobbit-clients.cfg:
HOST=%^server01
        LOG /logs/http/http %DSA\sis\sbusy COLOR=red