Re: [hobbit] Is there a limit on the number of hosts that can polled?

[Henrik Størner <henrik (at) hswn.dk>]

On Thu, Jan 15, 2009 at 10:16:11AM -0500, Smith, Jonathan wrote:
> Once I start to get around 2700 hosts hobbit stops updating. Is there a
> cause for this in hobbit (memory or host limit)? Top shows my system
> with very little utilization of memory or cpu and all hosts go purple
> for no updates.  I can get hobbit updating again by commenting out hosts
> and restarting it but I still have roughly another fifteen hundred hosts
> to enter.

Which tests go purple - the network tests ("conn" status, since you
seem to be doing mostly ping tests), or all of them including the
client-side tests (cpu, disk, memory etc.) ?

You shouldn't have any problems with that number of hosts. 

You're nowhere near the number of hosts I have in my production setup;
I have about 5700 entries in bb-hosts, my main network probe tests
4100 of them. And it seems your network tests complete well within
the 300 second max. poll time. What does the "bbgen" status say about
the time it takes to build the webpages ? And what's the I/O load on
the Hobbit server - check out the "CPU utilization" graph in the
"trends" column (NOT the "CPU load" one - you want the multi-color
'stacked' graph).

Are there any errors logged in the various Hobbit logfiles ? Or any
"ressource" problems logged in the operating system logs - like,
out of sockets, network card issues, or other weird messages?


And what operating system is this on ?


The limitations I've seen over time mostly have to do with the amount
of disk I/O caused by the hobbitd_rrd RRD graph data collector (the
update caching added in the current development version solves that
problem completely), and with the network ressources used for testing
lots of hosts - some systems have fairly small ARP caches, and this
can cause all sorts of weird problems, because Hobbit will sporadically
lose contact with itself or with the systems it is testing.


Regards,
Henrik