[Xymon] Alert transition yellow -> red with repeat problem

[john.r.rothlisberger at accenture.com]

This problem contributed to an outage last night - something is wrong.

Last night we had a disk that was in the warning state - warning email sent.

That disk then went into an alert state and an alert email was triggered (right away as the DURATION value took into account the time it was yellow) and then again 15 minutes later as designed.

THEN - another disk on that server went yellow.  It did NOT trigger any emails (as expected we should continue to focus on the alerts) but it somehow interfered with the REPEAT time of the alerts and those STOPPED.

There is a bug somewhere.

Thanks,
John
Upcoming PTO:
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
312.693.3136 office
_____________________________________________________________________

From: Rothlisberger, John R.
Sent: Tuesday, April 11, 2017 7:49 AM
To: 'xymon >> xymon at xymon.com' <xymon at xymon.com>
Subject: Alert transition yellow -> red with repeat problem

This is a problem I have seen for a long long time and have actually brought it up on the list before.
Xymon 4.3.21
Ubuntu 14.04LTS

The problem I have (in this instance) a warning which is to be repeated daily suddenly goes red and triggers a single alert but doesn't repeat again until the repeat time of the warning has passed.

>From the notification.log:
Tue Mar 28 05:31:28 2017 ServerA.disk (IP) disk_warn 1490697082 100 <- warning sets a repeat time of 1 day
Tue Mar 28 05:53:32 2017 ServerA.disk (IP) disk_alert 1490698405 100 <- minutes later it goes red (red repeat time is 15 minutes but no further alerts are generated)
Next alert comes out 1 day after the above warning:
Wed Mar 29 05:31:31 2017 ServerA.disk (IP) disk_alert 1490783486 100 <- 1 day after previous warning.  This should have been repeated every 15 minutes.
Wed Mar 29 05:46:40 2017 ServerA.disk (IP) disk_alert 1490784394 100 <- now, the repeat time is 15 minutes
... <- and continues every 15 minutes.

This alert went a full 24 hours with only a single notification.  :(
I have seen this before (not always) where the repeat time in a warning overrides a follow up alert until the warning repeat time has expired.

Alert rules:
   SCRIPT /home/xymon/server/ext/pg/exwarn_SCRIPT disk_warn DURATION>30 REPEAT=1d COLOR=yellow SERVICE=disk FORMAT=TEXT UNMATCHED
   SCRIPT /home/xymon/server/ext/pg/exalert_SCRIPT disk_alert DURATION>20 REPEAT=15 COLOR=red SERVICE=disk FORMAT=TEXT UNMATCHED

Ideas/thoughts?


Thanks,
John
Upcoming PTO:  4/3
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
312.693.3136 office
_____________________________________________________________________


________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20170428/9e58bd64/attachment.html>