[hobbit] Hobbit client and large networks

[Henrik Stoerner]

On Sat, Jan 27, 2007 at 06:41:36PM -0700, Alan Sparks wrote:
> Have used Hobbit with BB clients at a previous gig, but now looking to 
> set one up with Hobbit clients at a new gig.  This will be an 
> installation of close to 1000 nodes.  Having started testing using the 
> clients, have some concerns and questions:
> 
> 1) I am concerned that the server performs all the proc and disk 
> tests... I'm concerned that several hundred servers will overwhelm the 
> Hobbit server.  I see the client can be configured to run the checks 
> locally, provided the configuration is copied to the clients.  Has 
> anyone any practical experience with this many clients, and whether 
> local checks are better in this case?

The client handling on the Hobbit server is almost exclusively CPU
bound, so how many clients can be handled depends on the speed of your
processor.

I'm gradually converting all of our Unix servers to use Hobbit clients.
At the moment I have 295 hosts with the Hobbit client on them, and the
hobbitd_client program (which takes care of processing the client data)
is using ~0.3% CPU time. This is on a pretty old Hobbit server (a Sun
E220 with a single, 450 MHz CPU - an equivalent x86 processor would be
something like a 1 GHz PIII).

In other words, I doubt if it will be a problem. If I'm mistaken, then
the next Hobbit version will let you split the load over multiple
servers; this code is already in production use and is available in the 
current snapshots.


> 2) Most of the servers are Web and database servers... meaning that 
> port, proc, and netstat reports are very large.  I've seen so far that 
> the Hobbit client data sent to the server is almost 500k or larger, 
> causing truncation problems and problems with the server-side checks 
> (proc tests did not work until I used either client-local tests, or 
> disabled the ports client data).  Can client data be disabled without 
> affecting the ability to run checks?  Is there a way to streamline the 
> data so that effective monitoring is possible, without sending 1/2 MB 
> data for each client every 5 minutes?

The first thing to do would be to increase the max. size allowed for the 
"client" messages. See the hobbitserver.cfg(5) man-page, look for the
MAXMSG_* settings.

You can remove the "netstat -an" command from the client-side scripts,
or limit it to eg only listing the listener ports. Of course, you will
not be able to use the PORT checks on the Hobbit server, but that's the
trade-off you'll have to make.


Regards,
Henrik