[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [hobbit] Hobbid_channel crashing on me
- To: hobbit (at) hswn.dk
- Subject: Re: [hobbit] Hobbid_channel crashing on me
- From: henrik (at) hswn.dk (Henrik Stoerner)
- Date: Tue, 28 Aug 2007 16:56:44 +0200
- References: <BAY138-F2415C2F398443E9550CAE39FD10 (at) phx.gbl> <000c01c7e7d7$c1662430$c4774f4a (at) rr.com> <00a901c7e8d2$ad983c80$04011818 (at) rr.com> <20070828024627.GA2172 (at) hswn.dk> <005a01c7e977$18621690$04011818 (at) rr.com>
- User-agent: Mutt/1.5.13 (2006-08-11)
On Tue, Aug 28, 2007 at 09:26:51AM -0400, Sean R. Clark wrote:
> I have 18,102 RRD's, 17,671 of which are controlled by the hobbitd_channel
> (the others are written/populated from other sources)
>
> The slice I have the data on has a busy% between 16-88% depending on what's
> going on (so yes, high I/O as well)
OK, then I'd suggest that you pick up the current snapshot of Hobbit
from http://www.hswn.dk/beta/ and build that. The only parts you need
to replace in your current setup are these binaries:
* hobbitd/hobbitd_channel
* hobbitd/hobbitd_rrd
* web/hobbitgraph.cgi
After running "make", shutdown Hobbit and copy these files to your
~hobbit/server/bin/ directory (it's probably wise to save the original
ones first). Then start Hobbit again, and everything should be working
fine - with a lot less I/O load, and no memory leak in hobbitd_channel.
What's changed internally is that updates of the RRD files are now
cached for up to 30 minutes before being written to disk; the RRDtool
library can handle "batch" updates of the data, so instead of updating
the RRD file with 1 dataset every 5 minutes, it now gets 6 datasets in
one operation every 30 minutes.
This also means that when you shutdown Hobbit, you'll see that the
hobbitd_rrd process takes quite a long time to finish - it is busy
writing all of the cached updates to disk. On my work server, this
takes about 5 minutes.
Regards,
Henrik