[hobbit] Runtime longer than time limit (300)

[David Baldwin]

Chris Wopat wrote:
> Unsure if this is a bug or not. I noticed that for a few days I've been 
> getting "Runtime 483 longer than time limit (300)" and the bbtest column 
> shows:
>
> TIME SPENT
> <snip>
> DNS tests executed	450.074868
> <snip>
> TIME TOTAL		482.741716
>
>
> I was able to successfully debug this by adding "--debug" to the [bbnet] 
> CMD in hobbitlaunch.cfg. This revealed:
>
>
> 2009-09-01 08:26:01 ares_search: tlookup='dns.server.com', class=1, type=1
> 2009-09-01 08:26:01 Processing 0 DNS lookups with ARES
> 2009-09-01 08:33:31 Finished ARES queue after loop 20
>
> Note that it took 7 minutes to complete this. Where 'dns.server.com' is 
> a server that we took offline on the day the issue began. We didn't 
> remove this from bb-hosts and instead marked it blue.
>
> The fix was to comment it out from bb-hosts. It says "loop 20" above so 
> I was searching for config options that listed 20 as a value but didn't 
> come up with anything.
>
> The issue is resolved on my end but I'd like to find out if this is a 
> bug or expected behavior. It seems that a single DNS check on a host 
> that's down shouldn't delay things by 7 minutes.
>
> I'm running Xymon v4.2.3 on FreeBSD 7.2-RELEASE.
>   
I've also tuned my DNS tests. The [bbnet] section in hobbitlaunch.cfg is
the place to do it.
See 'man bbtest-net' for options.
I've upped the general timeout because I have a slow web server. I might
shift known ones to urlplus because I can do custom timeouts on each server.
My settings are:
[bbnet]
        ENVFILE /usr/lib/hobbit/server/etc/hobbitserver.cfg
        NEEDS hobbitd
        CMD bbtest-net --report --ping --checkresponse --dns-timeout=15
--timeout=20 --concurrency=512
        LOGFILE $BBSERVERLOGS/bb-network.log
        INTERVAL 5m

Check the 'bbtest' column for your hobbit server for stats on bbtest-net
runtimes - there's an RRD graph in there too.

David.

-- 
David Baldwin - IT Unit
Australian Sports Commission          www.ausport.gov.au
Tel 02 62147830 Fax 02 62141830       PO Box 176 Belconnen ACT 2616
david.baldwin at ausport.gov.au          Leverrier Street Bruce ACT 2617


-------------------------------------------------------------------------------------
Keep up to date with what's happening in Australian sport visit http://www.ausport.gov.au

This message is intended for the addressee named and may contain confidential and privileged information. If you are not the intended recipient please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you receive this message in error, please delete it and notify the sender.
-------------------------------------------------------------------------------------