[hobbit] Help with Graphing peak values

[Henrik Stoerner]

> Typically, our systems are very busy during the day yet at night are mostly idle (load 
> average of around 10 to 12. When you look at the graphs produced from a capacity planning 
> perspective, for the last 48 hours, you get accurate values, but from a 576 day graph, 
> the load average is much lower as it is taking into account the out of hours period.

That's how rrdtool works - the further back in time you go, the more
the data is averaged out over the entire time period.

> I was thinking it would be nice to see a line plotted on the same graph showing the peak 
> value for the sample rather than the average.

You *can* do it, but it will require that you create custom RRD's for
the data that you want to measure like this.

RRDtool stores data from one dataset in one or more "archives" called
"RRA's". An RRA uses a "consolidation function" to convert the raw data
into the data that it stores - typically by computing an average value
over the time interval that is used for the data. But you need not use
the "average" method - there are others, including MIN and MAX to store
the minimum and maximum values for a timeperiod.

When Hobbit creates a new RRD file, it defines 4 archives for each dataset.
All of them use the AVERAGE function, but averages the data over
different time periods, which is why there are 4 graphs for each data
showing the 48-hour, one-week, 4 weeks, and 576 days data.

So what you can do is to create the RRD files by hand using the "rrdtool
create" command. As long as the dataset names match, Hobbit will happily
update your custom RRD files - rrdtool takes care of that. So for e.g. 
the cpu load (the "la.rrd" file): "rrdtool dump" tells you that there is
one dataset called "la" of type GAUGE and a minimum value of 0. So you
could create such an rrd file with the command

   rrdtool create la.rrd \
      DS:la:GAUGE:600:0:U \
      RRA:AVERAGE:0.5:1:576 \
      RRA:AVERAGE:0.5:6:576 \
      RRA:AVERAGE:0.5:24:576 \
      RRA:AVERAGE:0.5:288:576 \
      RRA:MAX:0.5:288:576

The first 4 RRA's are the standard ones Hobbit uses. The last one is a
custom RRA that stores the maximum reading over 288 measurements (that's
24 hours since we measure once every 5 minutes), and keeps 576 of those
measurements in the database. So it gives you the max. load seen over
the past 576 days.

If you create such a file in the ~hobbit/data/rrd/HOSTNAME/ directory
where Hobbit puts its RRD files, it should be updated automatically.

Then to graph the data, you just have to setup a custom graph definition
that uses the MAX RRA instead of the AVERAGE one: 

[maxla]
        TITLE MAX CPU Load
        YAXIS Load
        DEF:avg=la.rrd:la:MAX
        CDEF:la=avg,100,/
        AREA:la#00CC00:CPU Load Average
        -u 1.0

Compare it to the existing [la] definition in hobbitgraph.cfg, and
you'll see they are almost identical.

I haven't tried this at all, so if you do - please let us know what 
your experiences are.


Regards,
Henrik