[hobbit] randomizing execution of tests

[Henrik Størner]

On Thu, Feb 05, 2009 at 12:25:48PM -0500, David Paper wrote:
> I've got a bb-hosts file with 8 server process instances getting tested.  
> Each instance gets tested with 3 HTTP requests (2 GET, 1 POST).  All 8 
> server processes live on the same physical OS instance.  This results in 
> 24 HTTP requests getting sent from hobbit within 1/100th of a second.  
> This causes the load on the host to spike, and generates contention w/in 
> each server to satisfy the requests.  This same setup is repeated for 
> hundreds of hosts and hundreds of processes.

Yeah, Xymon can be pretty agressive about testing network services.
It's unfortunate that you hit the same server with multiple requests 
at once, although it is a bit unusual that a web/application server
has a problem with just 24 requests simultaneously.

But of course, it depends on what your web application does.

> Is there a way to tell hobbit to take all of the entries in bb-hosts and 
> test them in a random order w/in the 1 minute testing interval?  This 
> would end up staggering the arrival of each HTTP test somewhat and lessen 
> contention within each HTTP server and on each host.

I'm afraid not, at least not directly.

What you *could do was to setup more than one task to run the network
tests, using the "NET:foo" definition in bb-hosts to create groups of
tests than can be run simultaneously. From you description it sounds 
as if you have 8 "pseudo-hosts" defined in bb-hosts, each of them with
3 HTTP tests. I.e.

   0.0.0.0  web1   # http://web1/test1 http://web1/test2 \
                     post;http://web1/test3;blababla
   0.0.0.0  web2   # http://web2/test1 http://web2/test2 \
                     post;http://web2/test3;blababla

etc. for your 8 server processes. Is that correct ? Or would it at
least be possible to configure it that way ?

If you can do that, then you can assign each of web1, web2, web3 
and so on a different NET: setting, and then you run the checks 
for each of these NET's one at a time.

E.g. if you want to run the tests for "web1" and "web2" separately
from "web3" and "web4", then your bb-hosts file would be like this:

   0.0.0.0  web1   # http://web1/test1 http://web1/test2 \
                     post;http://web1/test3;blababla \
		     NET:testgroup1
   0.0.0.0  web2   # http://web2/test1 http://web2/test2 \
                     post;http://web2/test3;blababla
		     NET:testgroup1
   0.0.0.0  web3   # http://web3/test1 http://web3/test2 \
                     post;http://web3/test3;blababla
		     NET:testgroup2
   0.0.0.0  web4   # http://web4/test1 http://web4/test2 \
                     post;http://web4/test3;blababla
		     NET:testgroup2

So you have two "NET:" groups of tests - "testgroup1" and "testgroup2".
Then you change to [bbnet] task to run this script "runnetworktests.sh" 
instead of just launching bbtest-net:

   #!/bin/sh

   # Run each NET group of tests separately
   # This script passes all commandline options to bbtest-net

   BBLOCATION=testgroup1 bbtest-net $*
   BBLOCATION=testgroup2 bbtest-net $*

   # Finally, run the tests that have no NET definition
   bbtest-net --test-untagged $*

   exit 0

The [bbnet] task (in hobbitlaunch.cfg) would then have

   CMD runnetworktests.sh --report --ping --checkresponse

instead of the default "CMD bbtest-net --report --ping --checkresponse"


The only "problem" with this is that you get to do the configuration of
what tests can run simultaneously by hand. And each invocation of 
bbtest-net has to parse all of the bb-hosts file, there is a small
overhead in doing that.


Regards,
Henrik