[Xymon] Purple storm

[Henrik Størner]

On 19-03-2012 19:15, Poppy, Ben wrote:
> I have an interesting problem that happened last night. We are working
> on a DR test. Part of that test includes shutting down some DC’s in our
> DR datacenter. When that happened, most tests that are initiated from
> the xymon servers (http, dns, ssh, ftp, etc) to the monitored server
> went purple. The servers that went purple were not all in our DR
> datacenter, it was at all of our sites, and even included some tests to
> the xymon server itself (we monitor the HTTP web page of xymon itself as
> well).
>
> Both of our xymon servers point to 2 windows DC’s in our production
> datacenter in /etc/resolv.conf for DNS lookups.

Check the "xymonnet" status history. I suppose this status will show 
some yellow events during this, caused by the network tests taking too 
long to run.

The status will tell you more about what part of the network tests are 
taking too long.

This should also show up in the xymonnet.log file.

One likely culprit would be if you are doing "ntp" tests or custom DNS 
queries from Xymon against the DC's that are down. "ntp" tests use an 
external program (ntpdate) to perform the query, and it has a very long 
timeout when servers are not responding. DNS queries use the C-ARES 
library, and because I misunderstood how the timeout handling works in 
this library it can several minutes *per test* to timeout.

Fixes for both of these issues are "in the pipeline" for the next major 
Xymon version.


Regards,
Henrik