[Xymon] Purple storm

[Don Kuhlman]

Would you be able to run a tcpdump or use a network sniffer to see what
the server is doing when you're getting the long response times?

Maybe that will help you see what it is trying to reach when that is
happening.


On 3/20/12 1:51 PM, "Poppy, Ben" <poppy.ben at marshfieldclinic.org> wrote:

>Yes, that's the strange part, we can still manually do digs and nslookups
>from the xymon server to other DNS servers.
>
>-----Original Message-----
>From: Phil Crooker [mailto:Phil.Crooker at orix.com.au]
>Sent: Tuesday, March 20, 2012 12:41 AM
>To: Poppy, Ben
>Cc: xymon at xymon.com
>Subject: Re: [Xymon] Purple storm
>
>So, can you do DNS queries from the xymon server when DC3 & 4 are down?
>
>
>>>> "Poppy, Ben"  03/20/12 11:50 AM >>>
>So they are pointing to 2 DC's that stay up this entire time, we'll call
>them DC1 and DC2. Then we shutdown DR-DC3 and DR-DC4. When those servers
>are down, we begin to have issues.
>
>-----Original Message-----
>From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au]
>Sent: Monday, March 19, 2012 7:46 PM
>To: Poppy, Ben
>Cc: xymon at xymon.com
>Subject: Re: [Xymon] Purple storm
>
>On Tue, Mar 20, 2012 at 5:15 AM, Poppy, Ben  wrote:
>> I have an interesting problem that happened last night. We are working
>
>> on a DR test. Part of that test includes shutting down some DC's in
>> our DR datacenter. When that happened, most tests that are initiated
>> from the xymon servers (http, dns, ssh, ftp, etc) to the monitored
>server went purple.
>
>For network tests, Xymon resolves the IP address from the servername
>(typically using DNS), and then uses that IP address to perform the test.
> The IP address in the hosts.cfg file is not normally used for network
>tests.  So if your DNS fails, Xymon's network tests fail also.
>
>You can prevent this, and use the IP address supplied in hosts.cfg, by
>adding "testip" to each hosts.cfg entry that requires it.  You can add it
>to a ".default." entry so that it applies to all hosts.
>
>J
>
>______________________________________________________________________
>The contents of this message may contain private, protected and/or
>privileged information.  If you received this message in error, you
>should destroy the e-mail message and any attachments or copies, and you
>are prohibited from retaining, distributing, disclosing or using any
>information contained within.  Please contact the sender and advise of
>the erroneous delivery by return e-mail or telephone.  Thank you for your
>cooperation.
>_______________________________________________
>Xymon mailing list
>Xymon at xymon.com
>http://lists.xymon.com/mailman/listinfo/xymon
>
>-- 
>
>This message from ORIX Australia might contain confidential and/or
>privileged information. If you are not the intended recipient, any use,
>disclosure or copying of this message (or of any attachments to it) is
>not authorised.
>
>If you have received this message in error, please notify the sender
>immediately and delete the message and any attachments from your system.
>Please inform the sender if you do not wish to receive future
>communications by email.
>
>ORIX handles personal information according to a Privacy Policy that is
>consistent with the National Privacy Principles. Please let us know if
>you would like a copy. It is also available at http://www.orix.com.au .
>
>
>
>______________________________________________________________________
>The contents of this message may contain private, protected and/or
>privileged information.  If you received this message in error, you
>should destroy the e-mail message and any attachments or copies, and you
>are prohibited from retaining, distributing, disclosing or using any
>information contained within.  Please contact the sender and advise of
>the erroneous delivery by return e-mail or telephone.  Thank you for your
>cooperation.
>_______________________________________________
>Xymon mailing list
>Xymon at xymon.com
>http://lists.xymon.com/mailman/listinfo/xymon