[Xymon] Breaking msgs in multiple columns for various log files monitoring

Andy Smith abs at shadymint.com
Sun Jan 26 23:34:00 CET 2014


Vernon Everett wrote:
> Hi all
> 
> My DBAs at a previous client wanted their own alert.log column for their 
> database alerts logs.
> It was really simple. I no longer have access to it, but in pseudo-code, 
> it looked something like this.
> 
> LOGFILES=$(grep for files to check in clientlocal)
> #This makes it configurable at server side
> for LOGFILE in $LOGFILES
> do
>   colour=green
>   # If this must go to a specific alert column in the display
>   column_name=$(grep $LOGFILE.colname clientlocal)
>   if [ -z $column_name ] && column_name=alt_logs
> 
>   # the next bit makes sure we only check the last bit of the file.
>   if [ exist lengthfile.$LOGFILE ]
>   then
>      lastline=$(cat lengthfile.$LOGFILE)
>   else
>      lastline=0
>   fi
>  
>   # get the current length of the log file
>   file_len=$(wc $LOGFILE)
> 
>   if [ $lastline -gt $file_len ] && lastline=0 # Somebody cycled the log 
> file
>  
>   echo $file_len > lengthfile.$LOGFILE # save the length for next time 
> we run
> 
>   grepstringred=$(grep string_that_trigger_red_alerts.$LOGFILE clientlocal)
>   grepstringyel=$(grep string_that_trigger_yellow_alerts.$LOGFILE 
> clientlocal)
>   # The above lines makes it configurable from server side
>    tail -($file_len-$lastline)>$tmpfile
>    grep $grepstringred $tmpfile && colour=red
>   if [ $colour == green ] && grep $grepstringyel $tmpfile &&  colour=yellow
> 
> send results and $tmpfile to server
> 
> done
>  
> This only worked if we need one log file per column. You may have to add 
> some logic and come up with some clever html display to put more than 
> one file per column.
> For our DBAs it was easy, because they wanted one column per database 
> instance, which I gave them on their own page with some clever use of 
> the group-except and group-only.
> 
> Hope it helps.
> 
> Cheers
> Vernon
> 
> P.S. Maybe I will rewrite this properly, and pop it on xymonton in the 
> near future. I haven't contributed anything there for some time.
> 
> 
> 
> On 25 January 2014 17:56, Andy Smith <abs at shadymint.com 
> <mailto:abs at shadymint.com>> wrote:
> 
>     Neil Simmonds wrote:
> 
>         Possibly a better way would be to stick with the single msgs
>         column but in your analysis.cfg where you specifiy the log
>         strings you want to alert on, you can add GROUP=
> 
>          
>         i.e.  LOG /applogs/was/server/SystemOut.__log
>         "java.lang.OutOfMemoryError" COLOR=red GROUP=waslogs
> 
>          
>         That way when you set up the alerts you can specify the host,
>         the service and the group
> 
>          
>         HOST=server SERVICE=msgs GROUP=waslogs
> 
>                     MAIL etc…..
> 
>          
>         *From:* Xymon [mailto:xymon-bounces at xymon.__com
>         <mailto:xymon-bounces at xymon.com>] *On Behalf Of *Pierre Malenfant
>         *Sent:* 20 January 2014 13:54
>         *To:* xymon at xymon.com <mailto:xymon at xymon.com>
>         *Subject:* [Xymon] Breaking msgs in multiple columns for various
>         log files monitoring
> 
> 
>          
>         Hello All,
> 
>         I searched the Xymon Archive without success about this.  Found
>         a few inquiries but no solutions.
> 
>         I would like to have multiple columns for log file monitoring.
>          I need the exact same thing as msgs, but in many columns to
>         alert differently.
> 
>         Ex:
>         System logs -> Alert to sysadmins
>         Application logs -> Alert to Analysts
>         DB logs -> Alert to DBA.
> 
>         Having the columns named like msgs_sys / msgs_app / msgs_db
>          would make sense to me.
> 
>         There may be good ext scripts for this.  I tought it would be
>         simpler to have a central setup with the msgs "style".
> 
>         For thoses of you that had the issue, what's your recommendation?
> 
>         Thanks in advance!
> 
>         Pierre Malenfant
> 
>         Name & Registered Office: EXPRESS GIFTS LIMITED, 2 GREGORY ST,
>         HYDE, CHESHIRE, ENGLAND, SK14 4TH, Company No. 00718151.
> 
>         Express Gifts Limited is authorised and regulated by the
>         Financial Conduct Authority
> 
>         ------------------------------__------------------------------__------------
> 
> 
>         NOTE: This email and any information contained within or
>         attached in a separate file is confidential and intended solely
>         for the Individual to whom it is addressed. The information or
>         data included is solely for the purpose indicated or previously
>         agreed. Any information or data included with this e-mail
>         remains the property of Findel PLC and the recipient will
>         refrain from utilising the information for any purpose other
>         than that indicated and upon request will destroy the
>         information and remove it from their records. Any views or
>         opinions presented are solely those of the author and do not
>         necessarily represent those of Findel PLC. If you are not the
>         intended recipient, be advised that you have received this email
>         in error and that any use, dissemination, forwarding, printing,
>         or copying of this email is strictly prohibited. No warranties
>         or assurances are made in relation to the safety and content of
>         this e-mail and any attachments. No liability is accepted for
>         any consequences arising from it. Findel Plc reserves the right
>         to monitor all e-mail communications through its internal and
>         external networks. If you have received this email in error
>         please notify our IT helpdesk on +44(0) 1254 303030
>         <tel:%2B44%280%29%201254%20303030>
> 
>          
> 
>         ------------------------------__------------------------------__------------
> 
>         _________________________________________________
>         Xymon mailing list
>         Xymon at xymon.com <mailto:Xymon at xymon.com>
>         http://lists.xymon.com/__mailman/listinfo/xymon
>         <http://lists.xymon.com/mailman/listinfo/xymon>
> 
> 
>     Using GROUPS is fine if your notification mechanism was solely
>     e-mail based.  We have a 24x7 team following the critical view and
>     we need the ability to separate faults found in different log files
>     in exactly the same way as Pierre describes.  For the moment we get
>     round it as follows :-
>     1) define the logs as normal in client-local.cfg on the sever.
>     2) create a custom ext script on the server which :-
>        a) extracts the logs from the client data :-
>           xymon localhost "clientlog <myhost> section=msgs:/path/to/the/log"
>        b) decides if this is good or bad.
>        c) sends a status update to xymon for the new column :-
>           xymon localhost "status <myhost>.msgs_apps RED|YELLOW|GREEN"
> 
>     2b) is messy and hides stuff that should otherwise be in
>     analysis.cfg, so as a compromise, we use TRIGGER in
>     client-local.cfg, which means 2b) can be almost as simple as a
>     non-zero line count.  I would much prefer support for multiple msgs
>     columns directly in core xymon but have not yet had chance to see
>     how difficult this would be to implement.
> 
>     HTH
>     -- 
>     Andy
>     _________________________________________________
>     Xymon mailing list
>     Xymon at xymon.com <mailto:Xymon at xymon.com>
>     http://lists.xymon.com/__mailman/listinfo/xymon
>     <http://lists.xymon.com/mailman/listinfo/xymon>
> 
> 
> 
> 
> -- 
> "Accept the challenges so that you can feel the exhilaration of victory"
> - General George Patton

This coincided with some work I had to do this weekend anyway, so for 
what it's worth I cleaned up my script to the point that it can be 
shared, changed the approach slightly after seeing Vernon's psuedo code 
and I documented in in Xymonton here :-

https://wiki.xymonton.org/doku.php/monitors:msgs

See if it does what you need.
-- 
Andy



More information about the Xymon mailing list