[Xymon] [External] Re: Alert transition yellow -> red with repeat problem

[Rothlisberger, John R.]

The "red" alert rules do precede the "yellow" rules.

This problem has to do with the duration and it not being overridden when it goes from yellow to red.

Thanks,
John

-----Original Message-----
From: Ford, Alan [mailto:Alan.FORD at stanwell.com] 
Sent: Tuesday, November 21, 2017 3:19 PM
To: Rothlisberger, John R. <john.r.rothlisberger at accenture.com>
Cc: xymon >> xymon at xymon.com <xymon at xymon.com>
Subject: Re: [Xymon] [External] Re: Alert transition yellow -> red with repeat problem

Have you tried putting the red alert before the yellow?

Sent from my iPhone

On 21 Nov 2017, at 11:18 pm, Rothlisberger, John R. <john.r.rothlisberger at accenture.com<mailto:john.r.rothlisberger at accenture.com>> wrote:

Is anyone looking to fix this issue?

It keeps popping up and folks are losing confidence in Xymon.  ItÂ's hard not to when something goes from yellow->red and there are no alerts generated.

To recap:

  *   Test goes yellow
  *   Duration passes
  *   Warning sent
  *   Repeat time set to X (in my case 8 hours)
  *   Test goes red (within that 8 hour repeat time)
  *   There are no alerts generated even though the test has gone red

Thanks,
John

From: Josh Luthman [mailto:josh at imaginenetworksllc.com]
Sent: Thursday, June 1, 2017 1:21 PM
To: Rothlisberger, John R. <john.r.rothlisberger at accenture.com<mailto:john.r.rothlisberger at accenture.com>>
Cc: xymon at xymon.com<mailto:xymon at xymon.com>
Subject: [External] Re: [Xymon] Alert transition yellow -> red with repeat problem

It's a known issue.  I don't believe I've ever seen any kind of resolution.


Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Thu, Jun 1, 2017 at 2:03 PM, <john.r.rothlisberger at accenture.com<mailto:john.r.rothlisberger at accenture.com>> wrote:
This problem continues and I canÂ't seem to get anybodyÂ's attention.

Has anyone seen this problem before?????????

Thanks,
John
Upcoming PTO:
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform TGP for Business Process Outsourcing Accenture 312.693.3136<tel:(312)%20693-3136> office _____________________________________________________________________

From: Rothlisberger, John R.
Sent: Friday, April 28, 2017 1:40 PM
To: 'xymon >> xymon at xymon.com<mailto:xymon at xymon.com>' <xymon at xymon.com<mailto:xymon at xymon.com>>
Subject: RE: Alert transition yellow -> red with repeat problem

This problem contributed to an outage last night Â- something is wrong.

Last night we had a disk that was in the warning state Â- warning email sent.

That disk then went into an alert state and an alert email was triggered (right away as the DURATION value took into account the time it was yellow) and then again 15 minutes later as designed.

THEN Â- another disk on that server went yellow.  It did NOT trigger any emails (as expected we should continue to focus on the alerts) but it somehow interfered with the REPEAT time of the alerts and those STOPPED.

There is a bug somewhere.

Thanks,
John
Upcoming PTO:
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform TGP for Business Process Outsourcing Accenture 312.693.3136<tel:(312)%20693-3136> office _____________________________________________________________________

From: Rothlisberger, John R.
Sent: Tuesday, April 11, 2017 7:49 AM
To: 'xymon >> xymon at xymon.com<mailto:xymon at xymon.com>' <xymon at xymon.com<mailto:xymon at xymon.com>>
Subject: Alert transition yellow -> red with repeat problem

This is a problem I have seen for a long long time and have actually brought it up on the list before.
Xymon 4.3.21
Ubuntu 14.04LTS

The problem I have (in this instance) a warning which is to be repeated daily suddenly goes red and triggers a single alert but doesnÂ't repeat again until the repeat time of the warning has passed.

>From the notification.log:
Tue Mar 28 05:31:28 2017 ServerA.disk (IP) disk_warn 1490697082 100 <- warning sets a repeat time of 1 day Tue Mar 28 05:53:32 2017 ServerA.disk (IP) disk_alert 1490698405 100 <- minutes later it goes red (red repeat time is 15 minutes but no further alerts are generated) Next alert comes out 1 day after the above warning:
Wed Mar 29 05:31:31 2017 ServerA.disk (IP) disk_alert 1490783486 100 <- 1 day after previous warning.  This should have been repeated every 15 minutes.
Wed Mar 29 05:46:40 2017 ServerA.disk (IP) disk_alert 1490784394 100 <- now, the repeat time is 15 minutes Â. <- and continues every 15 minutes.

This alert went a full 24 hours with only a single notification.  :( I have seen this before (not always) where the repeat time in a warning overrides a follow up alert until the warning repeat time has expired.

Alert rules:
   SCRIPT /home/xymon/server/ext/pg/exwarn_SCRIPT disk_warn DURATION>30 REPEAT=1d COLOR=yellow SERVICE=disk FORMAT=TEXT UNMATCHED
   SCRIPT /home/xymon/server/ext/pg/exalert_SCRIPT disk_alert DURATION>20 REPEAT=15 COLOR=red SERVICE=disk FORMAT=TEXT UNMATCHED

Ideas/thoughts?


Thanks,
John
Upcoming PTO:  4/3
_____________________________________________________________________
John Rothlisberger
IT Strategy, Infrastructure & Security - Technology Growth Platform TGP for Business Process Outsourcing Accenture 312.693.3136<tel:(312)%20693-3136> office _____________________________________________________________________


________________________________

This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________

www.accenture.com<http://www.accenture.com>

_______________________________________________
Xymon mailing list
Xymon at xymon.com<mailto:Xymon at xymon.com>
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_mailman_listinfo_xymon&d=DwIF-g&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=u6KtIBCRNAeN-AbgJjdZe5zZJVFEfq04dnWD-hYNPL_fxJIIFncbL8W6k0NMJtuq&m=zr9quBrMtyQNZ1Zgpy-rW0PqSKW8B-aJBCjfd3JNY_E&s=tVU96yWQSoNR7NZ2J5qR2Dty7hgihUfdKcwm-l9s6KE&e=<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_mailman_listinfo_xymon&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=u6KtIBCRNAeN-AbgJjdZe5zZJVFEfq04dnWD-hYNPL_fxJIIFncbL8W6k0NMJtuq&m=SAgqVg6kYnRT4gTqLDhZ--XB9LO_3wl6luywEPyLj8s&s=AjoIDplLoBGVFu8ZNCy1vTPQCtZph0ENnqRkCzcz3z4&e=>

_______________________________________________
Xymon mailing list
Xymon at xymon.com<mailto:Xymon at xymon.com>
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_mailman_listinfo_xymon&d=DwIF-g&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=u6KtIBCRNAeN-AbgJjdZe5zZJVFEfq04dnWD-hYNPL_fxJIIFncbL8W6k0NMJtuq&m=zr9quBrMtyQNZ1Zgpy-rW0PqSKW8B-aJBCjfd3JNY_E&s=tVU96yWQSoNR7NZ2J5qR2Dty7hgihUfdKcwm-l9s6KE&e=

This email is to be read subject to the email disclaimer located at https://urldefense.proofpoint.com/v2/url?u=http-3A__www.stanwell.com_site-2Dinformation_email-2Ddisclaimer_&d=DwIF-g&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=u6KtIBCRNAeN-AbgJjdZe5zZJVFEfq04dnWD-hYNPL_fxJIIFncbL8W6k0NMJtuq&m=zr9quBrMtyQNZ1Zgpy-rW0PqSKW8B-aJBCjfd3JNY_E&s=d7P_iyGszWLWJzsh88Hqg4IHWKO7pbZtCGNvYdatDwY&e=