[Xymon] Help with very large log file - not getting the right lines

Elizabeth Schwartz betsy.schwartz at gmail.com
Wed Nov 23 00:26:18 CET 2011


TYPO there, 47M not 47G. The files are about 2G/day

On Tue, Nov 22, 2011 at 6:06 PM, Elizabeth Schwartz
<betsy.schwartz at gmail.com> wrote:
> I've got to monitor some very large log files. They're up to a couple
> gigs a day and individual lines can be 30800 characters or more ,
> including HTML.
> (changing the log file format is a project for another day)   So my
> last half hour of one of these files chosen at random is 21,000 lines,
> 47G.
>
> I want to look at all the lines that start with
>
> 2011-11-22 4:15:31 ERROR        servicename LotsOfText
>
> I want to ignore lines that start
> 2011-11-22 17:13:39 LOG NNNNN   servicename LotsOfHTML
>
> Ignoring all of those lines would  bring it to a manageable size (this
> particular file is 41 lines, 23k data)
>
> I've been playing around with rules in client-local.cfg like:
> [mmw2.example.com]
> log:/var/log/mmb1/MMRequest.log:10240
> trigger ERROR
> ignore LOG
>
> but I'm just not getting the ERROR lines in the log. Is this file just
> too large and too full of HTML to parse? Any suggestions?
>
> (we can write a custom script, of course, and I'm thinking of bringing
> in SEC. But it sure would be handy to be able to do this with out of
> the box xymon)
>



More information about the Xymon mailing list