[hobbit] Which DNS query is causing delays
Ward, Martin
Martin.Ward at colt.net
Mon Oct 12 15:17:53 CEST 2009
Hey Henrik/all, I need more hand-holding.
I modified the command line for the bbtest-net command to include the dnslog option, the entry now looks like this:
[bbnet]
ENVFILE /opt/hobbit/server/etc/hobbitserver.cfg
NEEDS hobbitd
CMD bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout=10 --dnslog=/var/log/hobbit/dns.log
LOGFILE $BBSERVERLOGS/bb-network.log
INTERVAL 10m
After restarting the whole Xymon application I looked at the open connections that the bbtest-net process had and saw this:
root at hbt0.lon:/home/mward>pfiles /proc/2944
2944: bbtest-net --report --ping --checkresponse --timeout=20 --dns-timeout=
Current rlimit: 256 file descriptors
0: S_IFCHR mode:0666 dev:285,3 ino:41103 uid:0 gid:3 rdev:13,2
O_RDONLY
/dev/null
1: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913
O_WRONLY|O_APPEND|O_CREAT
/var/log/hobbit/bb-network.log
2: S_IFREG mode:0644 dev:61,261 ino:41257 uid:1985 gid:1985 size:1319913
O_WRONLY|O_APPEND|O_CREAT
/var/log/hobbit/bb-network.log
3: S_IFREG mode:0644 dev:61,261 ino:42256 uid:1985 gid:1985 size:0
O_WRONLY|O_CREAT|O_TRUNC
/var/log/hobbit/dns.log
4: S_IFSOCK mode:0666 dev:280,0 ino:25389 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK
SOCK_STREAM
SO_SNDBUF(49152),SO_RCVBUF(49640),IP_NEXTHOP(232.193.0.0)
sockname: AF_INET6 ::ffff:10.44.107.48 port: 43062
peername: AF_INET6 ::ffff:10.44.107.51 port: 389
5: S_IFSOCK mode:0666 dev:280,0 ino:59068 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
SOCK_DGRAM
SO_DGRAM_ERRIND,SO_SNDBUF(57344),SO_RCVBUF(57344),IP_NEXTHOP(0.224.0.0)
sockname: AF_INET 212.74.77.181 port: 50902
peername: AF_INET 212.74.77.108 port: 53
So you can see that the bbtest-net process has the log file opened OK on file ID 3.
The problem is that the log file is empty :-(
root at hbt0.lon:/home/mward>ls -al /var/log/hobbit/dns.log
-rw-r--r-- 1 hobbit hobbit 0 Oct 12 12:05 /var/log/hobbit/dns.log
I did reduce the number of tests being made as there were a load of duplicates but I have plenty of DNS tests still configured, here you can see one configured entry:
root at hbt0.lon:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts /opt/hobbit/server/etc/bb-hosts-* | egrep -v ":#" | wc -l
79
root at hbt0.lon:/home/mward>grep dns= /opt/hobbit/server/etc/bb-hosts | head -1
/opt/hobbit/server/etc/bb-hosts:212.121.128.1 ns0.de.colt.net # ssh dns=ns:de.colt.net COMMENT:212.121.128.1
and all these take 1,351 seconds to complete testing.
So it seems I need to add another step in my process to resolve my DNS issues, how can I persuade bbtest-net to actually log something?
|\/|artin
> -----Original Message-----
> From: Ward, Martin [mailto:Martin.Ward at colt.net]
> Sent: 09 October 2009 14:26
> To: hobbit at hswn.dk
> Subject: RE: [hobbit] Which DNS query is causing delays
>
> Thanks Henrik, I am not using the --no-ares option so have put the
> dnslog entry in as you recommend.
>
> Thanks and I'll let you know how I get on.
>
> |\/|artin
>
> > -----Original Message-----
> > From: Henrik "Størner [mailto:henrik at hswn.dk]
> > Sent: 09 October 2009 12:26
> > To: hobbit at hswn.dk
> > Subject: Re: [hobbit] Which DNS query is causing delays
> >
> > In <1F7B01020EC4D04DA17703634B9E888E0A01CCD1 at ULPGCTMVMAI003.EU.COLT>
> > "Ward, Martin" <Martin.Ward at colt.net> writes:
> >
> > >I have noticed recently that my bb-test runs are taking a long time
> to
> > >complete, 6500 seconds instead of the maximum of 600. The info on
> the
> > >bb-test results page shows that the DNS tests are taking most of the
> > time,
> > >but I have around 200 of these, does anyone have any suggestions on
> > how
> > >I can find out which DNS tests are causing the delays without having
> > to
> > >click on every DNS test in all of my Xymon website?
> >
> > Are you using a local DNS cache ? I would strongly recommend that you
> > do - the resolver library in Xymon is pretty aggressive, and has been
> > known to knock out DNS servers if you have many hosts (although 200
> > doesn't sound like an awful lot).
> >
> > It shouldn't take that long unless you're using the standard resolver
> > library (i.e. you are running bbtest-net with the "--no-ares"
> option),
> > in which case Xymon cannot control the timeout for the DNS lookup.
> >
> > Try running with the option "--dnslog=FILENAME", this will log those
> > DNS requests that fail. It is probably one of those.
> >
> >
> > Regards,
> > Henrik
> >
> >
> > To unsubscribe from the hobbit list, send an e-mail to
> > hobbit-unsubscribe at hswn.dk
> >
>
>
>
> ***********************************************************************
> **************
> The message is intended for the named addressee only and may not be
> disclosed to or used by anyone else, nor may it be copied in any way.
>
> The contents of this message and its attachments are confidential and
> may also be subject to legal privilege. If you are not the named
> addressee and/or have received this message in error, please advise us
> by e-mailing security at colt.net and delete the message and any
> attachments without retaining any copies.
>
> Internet communications are not secure and COLT does not accept
> responsibility for this message, its contents nor responsibility for
> any viruses.
>
> No contracts can be created or varied on behalf of COLT
> Telecommunications, its subsidiaries or affiliates ("COLT") and any
> other party by email Communications unless expressly agreed in writing
> with such other party.
>
> Please note that incoming emails will be automatically scanned to
> eliminate potential viruses and unsolicited promotional emails. For
> more information refer to www.colt.net or contact us on +44(0)20 7390
> 3900.
>
>
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe at hswn.dk
>
*************************************************************************************
The message is intended for the named addressee only and may not be disclosed to or used by anyone else, nor may it be copied in any way.
The contents of this message and its attachments are confidential and may also be subject to legal privilege. If you are not the named addressee and/or have received this message in error, please advise us by e-mailing security at colt.net and delete the message and any attachments without retaining any copies.
Internet communications are not secure and COLT does not accept responsibility for this message, its contents nor responsibility for any viruses.
No contracts can be created or varied on behalf of COLT Telecommunications, its subsidiaries or affiliates ("COLT") and any other party by email Communications unless expressly agreed in writing with such other party.
Please note that incoming emails will be automatically scanned to eliminate potential viruses and unsolicited promotional emails. For more information refer to www.colt.net or contact us on +44(0)20 7390 3900.
More information about the Xymon
mailing list