[hobbit] Hobbid_channel crashing on me

[Henrik Stoerner]

On Tue, Aug 28, 2007 at 09:26:51AM -0400, Sean R. Clark wrote:
> I have 18,102 RRD's, 17,671 of which are controlled by the hobbitd_channel
> (the others are written/populated from other sources)
> 
> The slice I have the data on has a busy% between 16-88% depending on what's
> going on (so yes, high I/O as well)

OK, then I'd suggest that you pick up the current snapshot of Hobbit
from http://www.hswn.dk/beta/ and build that. The only parts you need
to replace in your current setup are these binaries:

  * hobbitd/hobbitd_channel
  * hobbitd/hobbitd_rrd 
  * web/hobbitgraph.cgi

After running "make", shutdown Hobbit and copy these files to your 
~hobbit/server/bin/ directory (it's probably wise to save the original 
ones first).  Then start Hobbit again, and everything should be working 
fine - with a lot less I/O load, and no memory leak in hobbitd_channel.

What's changed internally is that updates of the RRD files are now
cached for up to 30 minutes before being written to disk; the RRDtool
library can handle "batch" updates of the data, so instead of updating
the RRD file with 1 dataset every 5 minutes, it now gets 6 datasets in
one operation every 30 minutes.

This also means that when you shutdown Hobbit, you'll see that the
hobbitd_rrd process takes quite a long time to finish - it is busy
writing all of the cached updates to disk. On my work server, this 
takes about 5 minutes.


Regards,
Henrik