[hobbit] performance help needed

[shea_greg at emc.com]

>>On Monday, 26 October 2009 20:55:15 shea_greg at emc.com wrote:
>> Hi all,
>>
>>
>>
>> First off, sorry for the long post, I'm trying to supply as much data as
>> possible for analysis.
>>
>>
>>
>> I have a single Hobbit server with approximately 3500 hosts, a mixture
>> of windows and unix, some DB tests,
>>
>> some BEA tests and a few custom tests.  I have over 70000 RRD files
>> which seems to be causing Hobbit performance
>>
>> problems, most specifcally clock offset.  I have a cron job that
>> restarts Hobbit every 30 minutes otherwise the offset
>>
>> grows so large it eats all memory and OOM kill starts.  NTP is fine, it
>> seems to be the time it takes for Hobbit to process
>>
>> the client data.  OS resides on RAID1 146GB drives SAS 15K RPM, second
>> drive for RRDs is a single 300GB SAS 15K RPM.
>>
>> At the end is a graph showing the clock offset.  What else can I try?
>
>Add more spindles.
>
>70 000 RRD files will result in a minimum of 233 IOPS (assuming they are all 
>being  updated at 5-minute intervals). The EMC people I've spoken to say a 15k 
>FC disk shouldn't really be averaging much more than 180 IOPS, 15k SAS or 15k 
>SCSI wouldn't be any better. The 311 you seem to be doing isn't significant 
>overhead for the minumum of 233, so it is unlikely that any tuning will help.
>
>If you can't add spindles, you could look at the 4.3 branch, which has some 
>features that allow scaling out to more hosts, or streamlining RRD writes 
>(which may allow you to lose the clock offset, but will likely not reduce the 
>load average much).
>
>Regards,
>Buchan

Hi Buchan,

Thanks for your response.  I bounced around the idea of external storage, but even
here at EMC there is a cost associated with external storage, that's why I tried the
second drive.  I've read about the enhancements in 4.3, but thought I should upgrade
from RH 4.7 to RH 5.3 first (RH is the official supported Linux) as there were IO
improvements in the kernel.  I also tried a newer version of RRD 1.2.30 and 1.3.8.
RRD 1.3.8 doesn't work Hobbit 4.2.

On to the storage requisition process....

Thanks
-Grs-
Gregory R Shea
EMC Corporation