[hobbit] fping tuning

[Henrik Stoerner]

On Wed, Apr 26, 2006 at 03:24:15PM -0400, Schwimmer, Eric E *HS wrote:
> 
> We're monitoring 1420 IPs in hobbit, and it takes fping
> ~40 seconds to go through them all:

Is that a number you get from the "bbtest" status or from running
fping by hand?

Are you doing other network tests in Hobbit than just ping?
Hobbit does the ping tests in parallel with the other tests.

> <snip>
> [root at hobbit fping]# fping -i5 -b12 -f ips -r1 -t250 -B2 -q -s

Are you using those parameters also on the FPING command in
hobbitserver.cfg? Or is it just for your testing ?

> Now, this seems a bit lengthy to me.  I mean, if the avg
> round trip time is 5.83 ms, and there are 1430 hosts, 
> should the total time in transit for all hosts should
> be 8336ms, or 8 seconds... right?

No, it should be less - because fping pings several hosts in
parallel.

You have "-i5" which causes a 5 ms delay between each ping.
So that's (5/1000)*1430 = 7.15 seconds where it does nothing.
The default setting is "-i25" - i.e. 5 times higher - which
would actually match your ~40 seconds nicely.

Don't forget that there is probably also some time spent doing 
ARP lookups for all of these IP's. Unless you have "testip"
on all of the entries in bb-hosts (or run bbtest-net with "--dns=ip"), 
you'll also spend some time on DNS lookups (hint: use a local
caching DNS server on the Hobbit server).

> Even when I remove the hosts that aren't responding, the 
> results on are par with those above.
> 
> Our polling interval is once every 60 seconds (which we want 
> to maintain, because we like to know ASAP when something drops
> even one ping), so it's not a problem yet. We add hosts on a 
> daily basis, however, so it will be a problem some time in 
> the future and I'd like to fix it before it becomes a problem.

Well, the good news is that it probably won't become a problem.
Because fping pings multiple hosts in parallel, the runtime
doesn't change very much when you add more hosts.

If it does become an issue, spread the load. Setup an extra server
to do half the network tests, and configure your bb-hosts file 
with "NET:net-a" and "NET:net-b" tags on the hosts. Then you
set BBLOCATION="net-a" on one box, and "BBLOCATION=net-b" on the
other. Then they'll only test those hosts where the NET:... 
setting matches. Unless it's an OS limitation, you could probably
do that on a single box and just have two instances of the [bbnet]
task in hobbitlaunch.cfg - instead of running bbtest-net directly,
they would run a shell-script which sets the BBLOCATION environment 
just before running bbtest-net.


Regards,
Henrik