RE: [hobbit] Which DNS query is causing delays

["Ward, Martin" <Martin.Ward (at) colt.net>]

Hey Henrik/all, I need more hand-holding.

I modified the command line for the bbtest-net command to include the dnslog option, the entry now looks like this:

[bbnet]
        ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
        NEEDS hobbitd
        CMD bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout=10 --dnslog=/var/log/hobbit/dns.log
        LOGFILE $BBSERVERLOGS/bb-network.log
        INTERVAL 10m


After restarting the whole Xymon application I looked at the open connections that the bbtest-net process had and saw this:

root (at) hbt0.lon:/home/mward>pfiles /proc/2944
2944:   bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout=
  Current rlimit: 256 file descriptors
   0: S_IFCHR mode:0666 dev:285,3 ino:41103 uid:0 gid:3 rdev:13,2
      O_RDONLY
      /dev/null
   1: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913
      O_WRONLY|O_APPEND|O_CREAT
      /var/log/hobbit/bb-network.log
   2: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913
      O_WRONLY|O_APPEND|O_CREAT
      /var/log/hobbit/bb-network.log
   3: S_IFREG mode:0644 dev:61,261 ino:42256 uid:1985 gid:1985 size:0
      O_WRONLY|O_CREAT|O_TRUNC
      /var/log/hobbit/dns.log
   4: S_IFSOCK mode:0666 dev:280,0 ino:25389 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK
        SOCK_STREAM
        SO_SNDBUF(49152),SO_RCVBUF(49640),IP_NEXTHOP(232.193.0.0)
        sockname: AF_INET6 ::ffff:10.44.107.48  port: 43062
        peername: AF_INET6 ::ffff:10.44.107.51  port: 389
   5: S_IFSOCK mode:0666 dev:280,0 ino:59068 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK FD_CLOEXEC
        SOCK_DGRAM
        SO_DGRAM_ERRIND,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.224.0.0)
        sockname: AF_INET 212.74.77.181  port: 50902
        peername: AF_INET 212.74.77.108  port: 53

So you can see that the bbtest-net process has the log file opened OK on file ID 3.
The problem is that the log file is empty :-(

root (at) hbt0.lon:/home/mward>ls -al /var/log/hobbit/dns.log
-rw-r--r--   1 hobbit   hobbit         0 Oct 12 12:05 /var/log/hobbit/dns.log

I did reduce the number of tests being made as there were a load of duplicates but I have plenty of DNS tests still configured, here you can see one configured entry:

root (at) hbt0.lon:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts /opt/hobbit/server/etc/bb-hosts-* | egrep -v ":#" | wc -l
79
root (at) hbt0.lon:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts | head -1
/opt/hobbit/server/etc/bb-hosts:212.121.128.1   ns0.de.colt.net         # ssh dns=ns:de.colt.net COMMENT:212.121.128.1

and all these take 1,351 seconds to complete testing.

So it seems I need to add another step in my process to resolve my DNS issues, how can I persuade bbtest-net to actually log something?

|\/|artin

> -----Original Message-----
> From: Ward, Martin [mailto:Martin.Ward (at) colt.net]
> Sent: 09 October 2009 14:26
> To: hobbit (at) hswn.dk
> Subject: RE: [hobbit] Which DNS query is causing delays
> 
> Thanks Henrik, I am not using the --no-ares option so have put the
> dnslog entry in as you recommend.
> 
> Thanks and I'll let you know how I get on.
> 
> |\/|artin
> 
> > -----Original Message-----
> > From: Henrik "Størner [mailto:henrik (at) hswn.dk]
> > Sent: 09 October 2009 12:26
> > To: hobbit (at) hswn.dk
> > Subject: Re: [hobbit] Which DNS query is causing delays
> >
> > In <1F7B01020EC4D04DA17703634B9E888E0A01CCD1 (at) ULPGCTMVMAI003.EU.COLT>
> > "Ward, Martin" <Martin.Ward (at) colt.net> writes:
> >
> > >I have noticed recently that my bb-test runs are taking a long time
> to
> > >complete, 6500 seconds instead of the maximum of 600. The info on
> the
> > >bb-test results page shows that the DNS tests are taking most of the
> > time,
> > >but I have around 200 of these, does anyone have any suggestions on
> > how
> > >I can find out which DNS tests are causing the delays without having
> > to
> > >click on every DNS test in all of my Xymon website?
> >
> > Are you using a local DNS cache ? I would strongly recommend that you
> > do - the resolver library in Xymon is pretty aggressive, and has been
> > known to knock out DNS servers if you have many hosts (although 200
> > doesn't sound like an awful lot).
> >
> > It shouldn't take that long unless you're using the standard resolver
> > library (i.e. you are running bbtest-net with the "--no-ares"
> option),
> > in which case Xymon cannot control the timeout for the DNS lookup.
> >
> > Try running with the option "--dnslog=FILENAME", this will log those
> > DNS requests that fail. It is probably one of those.
> >
> >
> > Regards,
> > Henrik
> >
> >
> > To unsubscribe from the hobbit list, send an e-mail to
> > hobbit-unsubscribe (at) hswn.dk
> >
> 
> 
> 
> ***********************************************************************
> **************
> The message is intended for the named addressee only and may not be
> disclosed to or used by anyone else, nor may it be copied in any way.
> 
> The contents of this message and its attachments are confidential and
> may also be subject to legal privilege.  If you are not the named
> addressee and/or have received this message in error, please advise us
> by e-mailing security (at) colt.net and delete the message and any
> attachments without retaining any copies.
> 
> Internet communications are not secure and COLT does not accept
> responsibility for this message, its contents nor responsibility for
> any viruses.
> 
> No contracts can be created or varied on behalf of COLT
> Telecommunications, its subsidiaries or affiliates ("COLT") and any
> other party by email Communications unless expressly agreed in writing
> with such other party.
> 
> Please note that incoming emails will be automatically scanned to
> eliminate potential viruses and unsolicited promotional emails. For
> more information refer to www.colt.net or contact us on +44(0)20 7390
> 3900.
> 
> 
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe (at) hswn.dk
> 



*************************************************************************************
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way. 

The contents of this message and its attachments are confidential and may also be subject to legal privilege.  If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security (at) colt.net and delete the message and any attachments without retaining any copies. 

Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses. 

No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.  

Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.