[Xymon] Xymon disruption every night!

[Greg Earle]

> On Apr 20, 2016, at 3:40 PM, cleaver at terabithia.org wrote:
> 
> Date: Wed, 20 Apr 2016 15:40:14 -0700
> From: "J.C. Cleaver" <cleaver at terabithia.org>
> To: "Xymon Mailing List" <xymon at xymon.com>
> Subject: Re: [Xymon] Xymon disruption every night!
> Message-ID:
> 	<ffe8d4b5d0aa3d63517f2d994cf03d5c.squirrel at mail.kkytbs.net>
> Content-Type: text/plain;charset=iso-8859-1
> 
> 
> On Wed, April 20, 2016 12:55 pm, Greg Earle wrote:
>> 
>>> On Feb 17, 2016, at 12:50 AM,"J.C. Cleaver" <cleaver at terabithia.org>
>>> wrote:
>>> 
>>>> - Why I have a "DNS error" here ?  I set up the IP yesterday to this
>>>> host
>>>> to solve the issue.  The "conn" error disappeared since yesterday
>>>> evening
>>>> but the http error still remains.
>>> 
>>> All signs do point to an issue with DNS resolution here.
>>> 
>>> Was this a custom compile or are you using a package?  If custom, what
>>> version of c-ares is on your system?  That's the underlying resolution
>>> library that xymonnet is using by default to handle DNS lookups.  The
>>> fact that the 'conn' test remained good after you added the local hosts
>>> entry matches that, since HTTP tests are performed using their own
>>> secondary DNS lookup (to deal with vhosts, etc.) unless the IP is
>>> specified there as well.
>> 
>> J.C.,
>> 
>> I just stumbled across this thread from 2 months ago.  We're having DNS
>> glitches at my work and it's causing a flood of <hostname>:http
>> "DNS error" alerts in Xymon, which is becoming a real problem.
>> 
>> But here's what I don't understand.  All of our HTTP-tested hosts are
>> in the "hosts.cfg" file with their short names (instead of FQHNs).  So I
>> couldn't understand why DNS was involved since the IP addresses and names
>> were right there in "hosts.cfg" for Xymon to use.
>> 
>> Your response - specifically "unless the IP is specified there as well" -
>> implies that there might be another location where I could load the
>> names and addresses of our HTTP-tested hosts, to avoid this problem.
>> 
>> (Yes, I know - hosts and IP addresses can change.  But I'm in control of
>> that so I can deal.)
>> 
>> If this is the case, where is that other location where I can specify the
>> short names/IP addresses for the HTTP tests?
> 
> Yep: https://xymon.com/help/manpages/man5/hosts.cfg.5.html#lbAR
> 
> 192.168.0.10 mywebserver # http://www.sample.com=192.168.0.10/
> 
> For HTTP tests, an IP override is put in for each URL you're using. This
> will prevent a DNS lookup at http-test time for this URL, and when
> combined with combined with testip (for any other TCP checks here) and
> noconn (for ping->fping resolution), it should prevent any DNS lookups
> from being done for the string "mywebserver".
> 
> Unless there's been a regression, that should be sufficient. It's
> definitely worked at scale for mass testing of "hosts" that are not being
> referred to by a valid DNS name and simply listen on a distinct port.

Thank you!!!  I had tried using "testip" in each of the Web server
entries and couldn't understand why we still kept getting DNS meltdowns.

Apparently I missed this subtle line in hosts.cfg(5):

--
xymonnet ignores the "testip" tag normally used to force a test to use the
IP-address from the hosts.cfg file instead of the hostname, when it performs
http and https tests.
--

D'oh!

	- Greg