<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I have recently migrated from a large BigBrother/bbgen installation (hosts.cfg 5300 lines) to xymon 4.3.12.
<o:p></o:p></p>
<p class="MsoNormal">Surprisingly there have been very few issues. Performance is very good compared to BigBrother/bbgen.<o:p></o:p></p>
<p class="MsoNormal">We have just experienced a potentially major issue wiith alerting.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Our issue seems to be with alerts not being generated for a rule if the initial event transition to red is not within the "TIME" range for an alerting rule.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">An example follows of the behaviour experienced:<o:p></o:p></p>
<p class="MsoNormal">The "http" service went down for a system "butterfly.soe.uq.edu.au" at 03:07am and recovered 3 days later:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Mon Jan 20 10:14:31 2014 green 1 days 4:50:51<o:p></o:p></p>
<p class="MsoNormal">Fri Jan 17 03:07:44 2014 red 3 days 7:06:47<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Alerting for this test is as follows:<o:p></o:p></p>
<p class="MsoNormal">============<o:p></o:p></p>
<p class="MsoNormal">alerts.cfg:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$AISMAILSVCS=cifs,cont,cpu,disk,fping,http,inode,login,loginc,memory,ssh,sslcert,rtmpe,rtmps,rtmpt,svcs,xfer_proxy_c,xfer_proxy_e,xfer_proxy_k<o:p></o:p></p>
<p class="MsoNormal">$AISSMSSVCS=cifs,cont,cpu,disk,fping,http,inode,login,loginc,memory,ssh,sslcert,rtmpe,rtmps,rtmpt,svcs<o:p></o:p></p>
<p class="MsoNormal">$AISTFHSVCS=fping,http,login<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#<o:p></o:p></p>
<p class="MsoNormal"># web/proxy/other/cert alerts<o:p></o:p></p>
<p class="MsoNormal">#<o:p></o:p></p>
<p class="MsoNormal">PAGE=%its-ais/ais-(web|proxy|other).*<o:p></o:p></p>
<p class="MsoNormal"> MAIL ais-web@domain SERVICE=$AISMAILSVCS DURATION>2m COLOR=red REPEAT=1w FORMAT=PLAIN RECOVERED<o:p></o:p></p>
<p class="MsoNormal"> MAIL ais-web-sms@domain SERVICE=$AISSMSSVCS DURATION>6m TIME=*:0701:2159 COLOR=red REPEAT=1w FORMAT=SMS RECOVERED<o:p></o:p></p>
<p class="MsoNormal">============<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The "info" test output displays alerting rules as:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Alerting:<o:p></o:p></p>
<p class="MsoNormal">Service Recipient 1st Delay Stop after Repeat Time of Day Colors<o:p></o:p></p>
<p class="MsoNormal">ais-web@domain (R) 2m 1s - 1w - red<o:p></o:p></p>
<p class="MsoNormal">ais-web-sms@domain (R) 6m 1s - 1w *:0701:2159 red<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">============<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The notification log displays only email alert/recovery for "ais-web@domain", nothing for "ais-web-sms@domain" recipient:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Time Host Service Recipient<o:p></o:p></p>
<p class="MsoNormal">Mon Jan 20 10:14:47 2014 butterfly.soe.uq.edu.au http ais-web@domain<o:p></o:p></p>
<p class="MsoNormal">Fri Jan 17 03:10:29 2014 butterfly.soe.uq.edu.au http ais-web@domain<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">No notification was sent to "ais-web-sms@domain" by the second "MAIL" rule above after it's start time of 07:01 the morning following the failure even though the "http" test was to remain red for 3 days.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Manually testing the alerting rules with:<o:p></o:p></p>
<p class="MsoNormal">~/server/bin/xymoncmd xymond_alert --test butterfly.soe.uq.edu.au http --duration=362<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">indicates syntax is ok and will send both emails when tested during the 0701:2159 TIME window of the second rule:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 Matching host:service:dgroup:page 'butterfly.soe.uq.edu.au:http: Linux Servers:its-usg/usg-linux,its-ais/ais-other' against rule line 1002<o:p></o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 *** Match with 'PAGE=%its-ais/ais-(web|proxy|other).*' ***<o:p></o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 Matching host:service:dgroup:page 'butterfly.soe.uq.edu.au:http: Linux Servers:its-usg/usg-linux,its-ais/ais-other' against rule line 1003<o:p></o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 *** Match with 'MAIL ais-web@domain SERVICE=$AISMAILSVCS DURATION>2m COLOR=red REPEAT=1w FORMAT=PLAIN RECOVERED' ***<o:p></o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 Mail alert with command '/usr/bin/mutt -s "Xymon [12345] butterfly.soe.uq.edu.au:http CRITICAL (RED)" ais-web@domain'<o:p></o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 Matching host:service:dgroup:page 'butterfly.soe.uq.edu.au:http: Linux Servers:its-usg/usg-linux,its-ais/ais-other' against rule line 1004<o:p></o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 *** Match with 'MAIL ais-web-sms@domain SERVICE=$AISSMSSVCS DURATION>6m TIME=*:0701:2159 COLOR=red REPEAT=1w FORMAT=SMS RECOVERED' ***<o:p></o:p></p>
<p class="MsoNormal">00029580 2014-01-17 11:31:30 Mail alert with command '/usr/bin/mutt ais-web-sms@domain'<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is there anything wrong with the alerting logic I have used in alerts.cfg or am I mis-understanding how it works?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The BigBrother behaviour would have been to send the alert after the rule settle time at the start of the time window for the rule if an event happened prior to the start of the alerting time window.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Contriving a dummy test in the hosts.cfg and alerts.cfg for an unpingable host “dummy.alerting.test” “fping”.<o:p></o:p></p>
<p class="MsoNormal">Event log for “dummy.alerting.test” “fping”:<o:p></o:p></p>
<p class="MsoNormal">Tue Jan 21 15:47:43 2014 red 0:16:12<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">alerts.cfg:<o:p></o:p></p>
<p class="MsoNormal">HOST=dummy.alerting.test<o:p></o:p></p>
<p class="MsoNormal"> MAIL g.stone-tolcher@its.uq.edu.au DURATION>2m TIME=*:1600:1700 COLOR=red REPEAT=1w FORMAT=PLAIN RECOVERED<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Notification:<o:p></o:p></p>
<p class="MsoNormal">Tue Jan 21 16:00:36 2014 dummy.alerting.test fping
<a href="mailto:g.stone-tolcher@its.uq.edu.au">g.stone-tolcher@its.uq.edu.au</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Seems to indicate that it is working similar to what is expected, i.e. send notification at start of TIME window if event is still current (ignore duration/settle time unlike bigbrother)?<o:p></o:p></p>
<p class="MsoNormal">I do not understand why the other alert would not have occurred.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any help with this issue would be appreciated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;mso-fareast-language:EN-AU">Gavin Stone-Tolcher, IT Support Officer, Network Operations and Incident Response<span style="color:red"><o:p></o:p></span></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">Information Technology Services<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">The University of Queensland<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">Level 4, Prentice Building, St Lucia 4072<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">T: +61 7 334 66645, M: +61 401 140 838<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">E:
<span style="color:red"><a href="mailto:g.stone-tolcher@its.uq.edu.au"><span style="color:blue">g.stone-tolcher@its.uq.edu.au</span></a>
</span>W: <a href="http://www.its.uq.edu.au"><span style="color:blue">www.its.uq.edu.au</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">ITS: Service. Team. Accountability. Results.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;mso-fareast-language:EN-AU">IMPORTANT:</span></b><span style="font-size:10.0pt;mso-fareast-language:EN-AU"> This email and any attachments are intended solely for the addressee(s), contain copyright material
and are confidential. We do not waive any legal privilege or rights in respect of copyright or confidentiality. Except as intended addressees are otherwise permitted, you do not have permission to use, disclose, reproduce or communicate any part of this email
or its attachments. Statements, opinions and information not related to the official business of The University of Queensland are neither given nor endorsed by us. By using this email (including accessing any attachments or links) you agree we are not liable
for any loss or damage of any kind arising in connection with any electronic defect, virus or other malicious code we did not intentionally include.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">Please consider the environment before printing this email.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-AU">CRICOS Code 00025B<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>