[Xymon] New server causing issues with CONN test

[Poppy, Ben]

I got it figured out, turns out the systems were in multiple domains DNS wise, and I had my /etc/resolve.conf entries out of order a bit from the existing hobbit servers.. But they were both pointing to the same 2 DNS servers. So what was happening was one server would get the "wrong" IP, and cache it on the DNS servers, then get the right IP and cache it, and so on. And this would cause the flapping every 5-10 minutes..

Once I got it sync'd up, everything started working. one or 2 quirks, but I'll start a new thread for that..
thanks for your help!

-----Original Message-----
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Poppy, Ben
Sent: Monday, August 15, 2011 4:27 PM
To: Tim McCloskey; Josh Luthman
Cc: xymon at xymon.com
Subject: Re: [Xymon] New server causing issues with CONN test

I'll give this a try as well during my next testing phase.

-----Original Message-----
From: Tim McCloskey [mailto:tm at freedom.com] 
Sent: Monday, August 15, 2011 4:18 PM
To: Josh Luthman; Poppy, Ben
Cc: xymon at xymon.com
Subject: RE: [Xymon] New server causing issues with CONN test

Josh is correct.  Clear the arp cache, everywhere. (client and switch)

"When I bring the new replacement server online, I shutdown hobbit2, and have the new server assume it's IP."



________________________________________
From: xymon-bounces at xymon.com [xymon-bounces at xymon.com] On Behalf Of Josh Luthman [josh at imaginenetworksllc.com]
Sent: Monday, August 15, 2011 2:14 PM
To: Poppy, Ben
Cc: xymon at xymon.com
Subject: Re: [Xymon] New server causing issues with CONN test

CONN is done by the server, so it is best to look from the server's perspective.  Without knowing your network and details of the server it's tough to know where to start, but I would start by seeing when the server fails to ping the host (see if you get ARP, route to it, etc).

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Mon, Aug 15, 2011 at 5:11 PM, Poppy, Ben <poppy.ben at marshfieldclinic.org<mailto:poppy.ben at marshfieldclinic.org>> wrote:
The 2 servers are not using the same IP. They are tied to each other in the hobbit configs in that they point to each other.

My existing hobbit servers, hobbit1 and hobbit2, are the fail-over for each other. So they have the exact same configuration, and report data to each other in their client settings. All servers with hobbit or bbwin clients send data to both servers.

When I bring the new replacement server online, I shutdown hobbit2, and have the new server assume it's IP.

I will try the ping test to see exactly when it stops responding..

The strangest thing, like I said, is it's the exact same 4 hosts that show red for CONN. This was with every combination of centos/xymon/hobbit below.. I even cloned one of my existing centos 5 32-bit servers running hobbit 4.2 in another environment (our perimeter network that is firewalled off), such that the only thing that was different was the linux distro, and that also caused the same 4 servers to show red..

From: Josh Luthman [mailto:josh at imaginenetworksllc.com<mailto:josh at imaginenetworksllc.com>]
Sent: Monday, August 15, 2011 4:07 PM
To: Poppy, Ben
Cc: xymon at xymon.com<mailto:xymon at xymon.com>
Subject: Re: [Xymon] New server causing issues with CONN test


Are the two servers using the same IP?  Tied to one another in any way?  I would start a ping and turn the other server on and see when it goes down.
On Aug 15, 2011 5:03 PM, "Poppy, Ben" <poppy.ben at marshfieldclinic.org<mailto:poppy.ben at marshfieldclinic.org>> wrote:
> I'm having a pretty strange issue. We have our existing hobbit servers running on Fedora servers running hobbit 4.2.0. I'm working on installing brand new servers that will be running CentOS 6 64-bit and the latest version of xymon (4.3.3 before I saw 4.3.4 today). I did not see a fix to my issue in the 4.3.4 change log though, so figured I'd post here.
>
> In doing my update, I install the brand new server from scratch. Basically install CentOS 6 as a web server install, and then add in all the bits xymon needs (pcre, openssl, openldap, rrdtool, etc).. Then I compile and install xymon to /usr/lib/xymon. Next I copy over the bb-hosts file to the hosts.cfg, and follow the "migration" steps to get the data and configuration files over. Then I turn on xymon on the new server.
>
> Within a few minutes, 4 servers turn to red alerts on CONN on the existing Fedora based Hobbit servers. They begin flapping on and off of red alert until I shutdown the new CentOS xymon server. Within a few minutes of the new server being shut down, the alerts go away for good.
>
> I have tried going to Centos 5 32-bit, 64-bit, even trying xymon 4.2.3, or all the way back to hobbit 4.2.0 all with the same result, and the exact same 4 servers each time.
>
> I'm completely at a loss here. Does anyone know what may be causing these issues where the only difference is the OS being used (the distro, that is)?
>
> I just want to get our monitoring server upgraded to a stable OS, with updates, and get xymon up to date as well.
>
> Thanks,
> -Ben
>
> ______________________________________________________________________
> The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.
________________________________
The contents of this message may contain private, protected and/or privileged information. If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within. Please contact the sender and advise of the erroneous delivery by return e-mail or telephone. Thank you for your cooperation.

______________________________________________________________________
The contents of this message may contain private, protected and/or privileged information.  If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within.  Please contact the sender and advise of the erroneous delivery by return e-mail or telephone.  Thank you for your cooperation.
_______________________________________________
Xymon mailing list
Xymon at xymon.com
http://lists.xymon.com/mailman/listinfo/xymon

______________________________________________________________________
The contents of this message may contain private, protected and/or privileged information.  If you received this message in error, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained within.  Please contact the sender and advise of the erroneous delivery by return e-mail or telephone.  Thank you for your cooperation.