monitoring large log files--does SIZE setting in clients-local.cfg affect parsing?

[Tom Georgoulias]

I'm trying to monitor a log file that gets pretty big--right now it's 
14MB and won't be rotated until this evening, when it'll probably be 
35MB.  I would like to ensure that the hobbit client catches each and 
every time it runs across a line that contains WRN or ERR, but it seems 
like it keeps missing it and I'm wondering if the file just grows too 
quickly for the config I'm using.

The parsing should be pretty easy, since nearly every line that we don't 
need to worry about contains "MSG:" after the time stamp.  Here's a snippet:

[10/17/07 16:14:50] - MSG: ================================================
[10/17/07 16:14:50] - MSG: ================================================
[10/17/07 16:14:50] - MSG: Processing dispatched check queue [J: 0]
[10/17/07 16:14:50] - MSG: Handling 2 jobs
[10/17/07 16:14:51] - MSG: Handling 2 jobs
[10/17/07 16:14:51] - MSG: Handling 1 jobs
[10/17/07 16:14:52] - MSG: ================================================
[10/17/07 16:14:52] - MSG: Processing run queue [J: 2]
[10/17/07 16:14:52] - MSG: Generated available worker list [W: 4]

If a line had WRN, I'd like to get a yellow alert.  If ERR, red.

Here's what I've got in my clients-local.cfg for this class of servers:

[linux]
log:/secret/path/app.log:10240
ignore MSG:

Here's the hobbit-clients.cfg entry:

LOG /secret/path/app.log WRN color=yellow
LOG /secret/path/app.log ERR color=red

The logfetch.<hostname>.cfg file gets propagated to the client, so that 
works correctly.

I thought that my configuration would send every line from 
/secret/path/app.log that wasn't there the last time the hobbit client 
ran the test and that doesn't contain "MSG:" into the Hobbit server, as 
long as the total amount of log data doesn't exceet 10240 bytes.  Then 
the hobbit server would parse that incoming data and look for two regex, 
WRN and ERR, and change color accordingly.

However, I can grep through the logs and find instances of lines with 
ERR that weren't noticed by the Hobbit server.  Is the LOG size of 10240 
restricting the amount of data that the client processes or is there 
something else happening that I don't understand?  Any help is appreciated.

Thanks,
TOm