[hobbit] why rrd/do_disk.c couldn't store/extract correct CUR value to RRD?

[Henrik Stoerner]

On Tue, Jan 16, 2007 at 04:45:10PM -0500, Jerry Yu wrote:
> In order to piggy-back some database disk usage monitor onto Hobbit's
> default 'disk' check, I modifed hobbitclients-linux.sh to call a script
> under [df] section, so the following is appended to the regular df output.
> The rrd graph shows and alerts get generated when a threshold is
> overstepped.
> 
> However, for some reason MAX/MIN/CUR data is decimal, while the CUR is not
> closer to the AVERAGE than to the real data as reported.
> rrd/do_disk.c shows each line of the df output is tokenized using space
> character. I can't figure out what caused the CUR (as stored in
> disk,db_12_web_DATA.rrd as well) is not the real integer, nor even close to
> the real integer reported.
> 
> /db_12_web_DATA_dat 100 70 30 70% /db_12_web_DATA
> /db_12_web_LOG_dat 100 97 3 97% /db_12_web_LOG
> 
> Since the alert is generated, so I'd think Hobbit server does extract the
> correct 'pused' at some point
> 
> /db_12_web_LOG (97% used) has reached the PANIC level (95%)

The way RRD files work means that they will practically NEVER store the
exact value you push into them. Whenever you update an RRD file with a
new measurement, it looks at how long time has passed since the previous
update, and then computes what the new value "should" be, if the update
had occurred at exactly 300 seconds interval.

This means that when you have a measurement that changes suddenly, then
the data stored in the RRD file tends to lag behind slightly (for 10
minutes or so).

Hobbit feeds the percent-used measurement into the RRD files, and
rrdtool takes care of all the data manipulation needed to create graphs
from the data. BUT - Hobbit looks at the measurements BEFORE sending
them through rrdtool. So Hobbit will trigger an alert as soon as the
measurement says "97% used", regardless of what data the RRD-file has as
the CUR value.


Regards,
Henrik