[Xymon] Purple storm

[Poppy, Ben]

The DNS tests executed jumps to over 1500-2400 from the normal ~1 when those 4 DC's are down (which we are testing DNS, but are not the DNS servers set up in /etc/resolv.conf on the xymon servers). We are not doing any NTP tests against any hosts, nor do we do any special dns test, just the standard test.

-----Original Message-----
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Henrik Størner
Sent: Tuesday, March 20, 2012 2:11 AM
To: xymon at xymon.com
Subject: Re: [Xymon] Purple storm

On 19-03-2012 19:15, Poppy, Ben wrote:
> I have an interesting problem that happened last night. We are working 
> on a DR test. Part of that test includes shutting down some DC's in 
> our DR datacenter. When that happened, most tests that are initiated 
> from the xymon servers (http, dns, ssh, ftp, etc) to the monitored 
> server went purple. The servers that went purple were not all in our 
> DR datacenter, it was at all of our sites, and even included some 
> tests to the xymon server itself (we monitor the HTTP web page of 
> xymon itself as well).
>
> Both of our xymon servers point to 2 windows DC's in our production 
> datacenter in /etc/resolv.conf for DNS lookups.

Check the "xymonnet" status history. I suppose this status will show some yellow events during this, caused by the network tests taking too long to run.

The status will tell you more about what part of the network tests are taking too long.

This should also show up in the xymonnet.log file.

One likely culprit would be if you are doing "ntp" tests or custom DNS queries from Xymon against the DC's that are down. "ntp" tests use an external program (ntpdate) to perform the query, and it has a very long timeout when servers are not responding. DNS queries use the C-ARES library, and because I misunderstood how the timeout handling works in this library it can several minutes *per test* to timeout.

Fixes for both of these issues are "in the pipeline" for the next major Xymon version.


Regards,
Henrik
_______________________________________________
Xymon mailing list
Xymon at xymon.com
http://lists.xymon.com/mailman/listinfo/xymon

______________________________________________________________________
The contents of this message may contain private, protected and/or privileged information.  If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within.  Please contact the sender and advise of the erroneous delivery by return e-mail or telephone.  Thank you for your cooperation.