Re: [hobbit] performance help needed

[Buchan Milne <bgmilne (at) staff.telkomsa.net>]

On Monday, 26 October 2009 20:55:15 shea_greg (at) emc.com wrote:
> Hi all,
>
>
>
> First off, sorry for the long post, I'm trying to supply as much data as
> possible for analysis.
>
>
>
> I have a single Hobbit server with approximately 3500 hosts, a mixture
> of windows and unix, some DB tests,
>
> some BEA tests and a few custom tests.  I have over 70000 RRD files
> which seems to be causing Hobbit performance
>
> problems, most specifcally clock offset.  I have a cron job that
> restarts Hobbit every 30 minutes otherwise the offset
>
> grows so large it eats all memory and OOM kill starts.  NTP is fine, it
> seems to be the time it takes for Hobbit to process
>
> the client data.  OS resides on RAID1 146GB drives SAS 15K RPM, second
> drive for RRDs is a single 300GB SAS 15K RPM.
>
> At the end is a graph showing the clock offset.  What else can I try?

Add more spindles.

70 000 RRD files will result in a minimum of 233 IOPS (assuming they are all 
being  updated at 5-minute intervals). The EMC people I've spoken to say a 15k 
FC disk shouldn't really be averaging much more than 180 IOPS, 15k SAS or 15k 
SCSI wouldn't be any better. The 311 you seem to be doing isn't significant 
overhead for the minumum of 233, so it is unlikely that any tuning will help.

If you can't add spindles, you could look at the 4.3 branch, which has some 
features that allow scaling out to more hosts, or streamlining RRD writes 
(which may allow you to lose the clock offset, but will likely not reduce the 
load average much).

Regards,
Buchan