[Xymon] Purple storm

[Poppy, Ben]

So far so good, I have been unable to reproduce the purple storm. We'll find out in a couple weeks when we do our next DR isolation test. Thanks so much for your help!

-----Original Message-----
From: henrik at hswn.dk [mailto:henrik at hswn.dk] 
Sent: Thursday, April 12, 2012 3:43 AM
To: Poppy, Ben
Cc: xymon at xymon.com
Subject: RE: [Xymon] Purple storm

On Thu, 12 Apr 2012 06:27:01 +0000, "Poppy, Ben"
<poppy.ben at marshfieldclinic.org> wrote:
> I may have missed this in a past post, how do I apply this patch?

Ah - ok, my developer-mind assumed everyone knows how to do that :-)

Save the attachment to /tmp/dnstimeout.patch, then:

  cd xymon-4.3.7
  patch -p0 </tmp/dnstimeout.patch
  make clean
  make

You can run "make install" afterwards, but a safer option would be to just copy the "xymon-4.3.7/xymonnet/xymonnet" binary into your Xymon "bin"
directory, replacing the one that is already there.


> I do test DNS for sure on servers at our DR site (many of them). The
test
> you suggest below, is that to simulate the purple storm? 

It is to simulate that your Xymon server loses connectivity to the DNS server on the primary site.

> Should it
> essentially turn purple if I begin dropping all packets to a few DNS 
> servers I'm testing?

That is what I suspect, yes.
 
> Would I be able to run this same iptables on my backup xymon server in
our
> primary site to a few servers it checks DNS against in our DR site?
Should
> that effectively cause the purple storm?

What I'm trying to do is to simulate the situation you had which caused the purple storm, without actually pulling the plug and disrupting the network between the two sites. If I understand you correctly, then the purple storm happened when you lost the connection between your two datacenters. Since I suspect that this is related to DNS lookups taking a very long time with the stock 4.3.7 Xymon version, you can use iptables to just block traffic from Xymon to the DNS server(s) in the other datacenter.


Regards,
Henrik



______________________________________________________________________
The contents of this message may contain private, protected and/or privileged information.  If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within.  Please contact the sender and advise of the erroneous delivery by return e-mail or telephone.  Thank you for your cooperation.