[hobbit] fping tuning

[Schwimmer, Eric E *HS]

> > 
> > We're monitoring 1420 IPs in hobbit, and it takes fping
> > ~40 seconds to go through them all:
> 
> Is that a number you get from the "bbtest" status or from running
> fping by hand?

Both.  The values are fairly consistent, falling between somewhere
in the 39-42 range.
 
> Are you doing other network tests in Hobbit than just ping?
> Hobbit does the ping tests in parallel with the other tests.

We are doing other tests, but not many.  Here's the relevent
lines from our servers bbtest report:

TIME SPENT
Event                                            Starttime
Duration
TCP tests completed                      1146086231.293585
1.211963 
PING test completed (1434 hosts)         1146086271.488185
40.194600 
PING test results sent                   1146086271.523332
0.035147 
TIME TOTAL
41.549643 

> > <snip>
> > [root at hobbit fping]# fping -i5 -b12 -f ips -r1 -t250 -B2 -q -s
> 
> Are you using those parameters also on the FPING command in
> hobbitserver.cfg? Or is it just for your testing ?

This is just what I've been using for testing (the -f flag is
root only and wouldn't work very well when used from within
hobbit).  The value of my FPING envvar in hobbitserver.cfg
is "/usr/sbin/fping -i10 -b12".  However the average difference 
in polling time betweeh the two is only 1 or 2 seconds.
 
> > Now, this seems a bit lengthy to me.  I mean, if the avg
> > round trip time is 5.83 ms, and there are 1430 hosts, 
> > should the total time in transit for all hosts should
> > be 8336ms, or 8 seconds... right?
> 
> No, it should be less - because fping pings several hosts in
> parallel.
> 
> You have "-i5" which causes a 5 ms delay between each ping.
> So that's (5/1000)*1430 = 7.15 seconds where it does nothing.
> The default setting is "-i25" - i.e. 5 times higher - which
> would actually match your ~40 seconds nicely.

Using the default delay interval (i.e. not specifying the -i flag
when calling fping) causes the test to take much longer, on
the order of 60 - 70 seconds.  However, values of 15 or less
passed to -i don't make much of a difference in polling time.
(FWIW, fping doesn't let you specific a value for -i less than
10 unless you are root.  I hacked the fping code to get around
this so I could run it under hobbit with -i1, but I saw no
difference in polling times using -i1 vs -i15).


> Don't forget that there is probably also some time spent doing 
> ARP lookups for all of these IP's. Unless you have "testip"
> on all of the entries in bb-hosts (or run bbtest-net with 
> "--dns=ip"), you'll also spend some time on DNS lookups 
> (hint: use a local caching DNS server on the Hobbit server).

Yep, I have --dns=ip in the bbtest-net stanza of my hobbitlaunch.cfg
(that makes a BIG difference), so I don't think it's a DNS resolution
problem.  In the testing fping command above, the -f flag specifies
a file that is a list of all the IP addresses from my bb-host file, 
with not DNS names included, so I don't think it's a DNS problem. 
I feel like its some sort of concurrency issue within fping, since
I can reproduce this latency completely outside of hobbit.

As a complete aside, we caching server for things outside of hobbit,
and I've written a little script that monitors the bb-hosts file (
and all filed included from bb-hosts) and when it detects any
changes, it will write a bind9 zone file to somewhere on disk.  Its
handy for making sure your bb-hosts is synced with your DNS.  If 
anybody is interested in it, drop me a line (I'll have to 'pretty it
up' first) and I'll posted it on my hobbit tools page for people to
use.

> > Even when I remove the hosts that aren't responding, the 
> > results on are par with those above.
> > 
> > Our polling interval is once every 60 seconds (which we want 
> > to maintain, because we like to know ASAP when something drops
> > even one ping), so it's not a problem yet. We add hosts on a 
> > daily basis, however, so it will be a problem some time in 
> > the future and I'd like to fix it before it becomes a problem.
> 
> Well, the good news is that it probably won't become a problem.
> Because fping pings multiple hosts in parallel, the runtime
> doesn't change very much when you add more hosts.

Ah, so you would think ;)  However, our graph in our bbtest column
says otherwise;  it has been climbing slowly but steadily since it
started graphing data.  You can also reproduce this by using
a newline delimited list of IP addresses in a file, like I did above,
and feeding it to fping.  As you increase the number of IPs in the
file, the poll time increases geometrically.  For instance,
when I poll 300 hosts:

<snip>
[root at hobbit fpingtest]# fping -i5 -b12 -f 300ips -r1 -t250 -B2 -q -s

     300 targets
     298 alive
       2 unreachable
       0 unknown addresses

      12 timeouts (waiting for response)
     310 ICMP Echos sent
     298 ICMP Echo Replies received
       0 other ICMP received

 0.24 ms (min round trip time)
 2.38 ms (avg round trip time)
 101 ms (max round trip time)
        8.856 sec (elapsed real time)
</snip>

vs when I poll 600 hosts:
<snip>
[root at hobbit fpingtest]# fping -i5 -b12 -f 600ips -r1 -t250 -B2 -q -s

     600 targets
     597 alive
       3 unreachable
       0 unknown addresses

      14 timeouts (waiting for response)
     611 ICMP Echos sent
     597 ICMP Echo Replies received
       0 other ICMP received

 0.21 ms (min round trip time)
 2.48 ms (avg round trip time)
 100 ms (max round trip time)
       16.144 sec (elapsed real time)
</snip>

You can see that the ping time roughly doubles.  This is bad :(

> If it does become an issue, spread the load. Setup an extra server
> to do half the network tests, and configure your bb-hosts file 
> with "NET:net-a" and "NET:net-b" tags on the hosts. Then you
> set BBLOCATION="net-a" on one box, and "BBLOCATION=net-b" on the
> other. Then they'll only test those hosts where the NET:... 
> setting matches. Unless it's an OS limitation, you could probably
> do that on a single box and just have two instances of the [bbnet]
> task in hobbitlaunch.cfg - instead of running bbtest-net directly,
> they would run a shell-script which sets the BBLOCATION environment 
> just before running bbtest-net.

I was thinking of doing something along these lines, however the
bb-hosts
file is maintained mostly by the (non-unix-savvy) staff here, using the 
bb-hostedit CGI script, and I'd rather not have them have to keep track 
of which host needed which NET tag, etc.  

I've tested the network capabilities of this box using iperf as well 
as several concurrent ping floods, and it can send upwards of 10000+ 
ICMP packets per second (with successful replies from another host on
the 
same 1000bT switch).

So this leads me to believe that it is a problem solely with fping;  if
they had a public forum or a mailing list, I'd be whining there instead
of here. :)  I can't say that I was expecting to find the 'magic bullet'
for this problem here, but I was hoping that there might be some fping
wizard out there some magic bullets to spare.  Anywho, thanks for your
thoughts, Henrik.  I'll poke some more at the fping code and see if
I can figure out whats going on (I doubt it);  if not, I'll start
working towards hacking together a load balancing script that will
auto-add NET: tags to bb-hosts entry, or something along those lines.

Thanks,
-Eric