[Xymon] [External] Re: Alert transition yellow -> red with repeat problem

Tue Nov 21 14:17:39 CET 2017

Is anyone looking to fix this issue?

It keeps popping up and folks are losing confidence in Xymon.  It’s hard not to when something goes from yellow->red and there are no alerts generated.

To recap:

  *   Test goes yellow
  *   Duration passes
  *   Warning sent
  *   Repeat time set to X (in my case 8 hours)
  *   Test goes red (within that 8 hour repeat time)
  *   There are no alerts generated even though the test has gone red

Thanks,
John

From: Josh Luthman [mailto:josh at imaginenetworksllc.com]
Sent: Thursday, June 1, 2017 1:21 PM
To: Rothlisberger, John R. <john.r.rothlisberger at accenture.com>
Cc: xymon at xymon.com
Subject: [External] Re: [Xymon] Alert transition yellow -> red with repeat problem

It's a known issue.  I don't believe I've ever seen any kind of resolution.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Thu, Jun 1, 2017 at 2:03 PM, <john.r.rothlisberger at accenture.com<mailto:john.r.rothlisberger at accenture.com>> wrote:
This problem continues and I can’t seem to get anybody’s attention.

Has anyone seen this problem before?????????

Thanks,
John
Upcoming PTO:
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
312.693.3136<tel:(312)%20693-3136> office
_____________________________________________________________________

From: Rothlisberger, John R.
Sent: Friday, April 28, 2017 1:40 PM
To: 'xymon >> xymon at xymon.com<mailto:xymon at xymon.com>' <xymon at xymon.com<mailto:xymon at xymon.com>>
Subject: RE: Alert transition yellow -> red with repeat problem

This problem contributed to an outage last night – something is wrong.

Last night we had a disk that was in the warning state – warning email sent.

That disk then went into an alert state and an alert email was triggered (right away as the DURATION value took into account the time it was yellow) and then again 15 minutes later as designed.

THEN – another disk on that server went yellow.  It did NOT trigger any emails (as expected we should continue to focus on the alerts) but it somehow interfered with the REPEAT time of the alerts and those STOPPED.

There is a bug somewhere.

Thanks,
John
Upcoming PTO:
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
312.693.3136<tel:(312)%20693-3136> office
_____________________________________________________________________

From: Rothlisberger, John R.
Sent: Tuesday, April 11, 2017 7:49 AM
To: 'xymon >> xymon at xymon.com<mailto:xymon at xymon.com>' <xymon at xymon.com<mailto:xymon at xymon.com>>
Subject: Alert transition yellow -> red with repeat problem

This is a problem I have seen for a long long time and have actually brought it up on the list before.
Xymon 4.3.21
Ubuntu 14.04LTS

The problem I have (in this instance) a warning which is to be repeated daily suddenly goes red and triggers a single alert but doesn’t repeat again until the repeat time of the warning has passed.

From the notification.log:
Tue Mar 28 05:31:28 2017 ServerA.disk (IP) disk_warn 1490697082 100 <- warning sets a repeat time of 1 day
Tue Mar 28 05:53:32 2017 ServerA.disk (IP) disk_alert 1490698405 100 <- minutes later it goes red (red repeat time is 15 minutes but no further alerts are generated)
Next alert comes out 1 day after the above warning:
Wed Mar 29 05:31:31 2017 ServerA.disk (IP) disk_alert 1490783486 100 <- 1 day after previous warning.  This should have been repeated every 15 minutes.
Wed Mar 29 05:46:40 2017 ServerA.disk (IP) disk_alert 1490784394 100 <- now, the repeat time is 15 minutes
… <- and continues every 15 minutes.

This alert went a full 24 hours with only a single notification.  ☹
I have seen this before (not always) where the repeat time in a warning overrides a follow up alert until the warning repeat time has expired.

Alert rules:
   SCRIPT /home/xymon/server/ext/pg/exwarn_SCRIPT disk_warn DURATION>30 REPEAT=1d COLOR=yellow SERVICE=disk FORMAT=TEXT UNMATCHED
   SCRIPT /home/xymon/server/ext/pg/exalert_SCRIPT disk_alert DURATION>20 REPEAT=15 COLOR=red SERVICE=disk FORMAT=TEXT UNMATCHED

Ideas/thoughts?

Thanks,
John
Upcoming PTO:  4/3
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform
TGP for Business Process Outsourcing
Accenture
312.693.3136<tel:(312)%20693-3136> office
_____________________________________________________________________

________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com<http://www.accenture.com>

_______________________________________________
Xymon mailing list
Xymon at xymon.com<mailto:Xymon at xymon.com>
http://lists.xymon.com/mailman/listinfo/xymon<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_mailman_listinfo_xymon&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=u6KtIBCRNAeN-AbgJjdZe5zZJVFEfq04dnWD-hYNPL_fxJIIFncbL8W6k0NMJtuq&m=SAgqVg6kYnRT4gTqLDhZ--XB9LO_3wl6luywEPyLj8s&s=AjoIDplLoBGVFu8ZNCy1vTPQCtZph0ENnqRkCzcz3z4&e=>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20171121/1aded175/attachment.html>