[hobbit] Client interval question

[Henrik Stoerner]

On Wed, Dec 14, 2005 at 05:31:28PM -0600, Jeff Newman wrote:
> On 12/13/05, Scott Walters <scott at packetpushers.com> wrote:
> 
> >> The graph DB's that vmstat feeds data into (the RRD files) are
> > >> constructed in such a way that a 5-minute interval is what makes
> > >> sense. So running them with anything else really just a waste of
> > >> ressources.
> >
> > > With the stock larrd/hobbit RRD definitions you are correct.  He'll
> > > only use one of the five, and whine about the timestamp of the other
> > > four.
> 
> Firstly, can you explain your comment in more detail? Secondly,
> im confused as to why you would state that I would "whine" about anything
> when you have no basis for a conclusion to that effect. It seems to be a
> rather pointed comment in a discussion that hasn't involved the use 
> of language that would dictate a response like that.

Jeff, I think you misunderstood what Scott wrote. The "he" that is doing
the whining is the rrdtool library; if you feed data into an rrd file
more often than the minimum interval between updates, it will complain
about this in the logs and just ignore the extra updates.

> > > The design goal of larrd, (I can't speak for Henrik and hobbit/RRD)
> > > was capacity planning and trending.  5m samples are  more than
> > > adequate for that activity.

[interesting discussion about using hobbit for capacity-planning vs.
 real-time analysis snipped]

The Hobbit design - as far as the graphing and trending is concerned -
was really just to re-implement the LARRD features. So in that respect
I have adopted Scott's design goals - even though I wasn't aware what
they were.

However, that doesn't mean hobbit cannot be leveraged to support other
uses for the data we collect about our systems. As Jeff writes:

> part of having a community such as this is to foster ideas and innovation.
[snip]
> What if you could add a high-frequency tag to a server and it generates a 
> seperate high-frequncey graph for that

I've picked up quite a few ideas from the discussions that have occurred
here, and Hobbit wouldn't be as good as it is without it. So please - 
keep those ideas coming, even though they might seem to be "off-topic"
or just plain weird. There's no guarantee that I'll use any of it, but
it is still interesting to discuss.

I actually think that Hobbit could support both uses, e.g. in the way
that Jeff suggests with a special high-frequency graph, in addition to
the normal ones. Hobbit does have several building-blocks that you could
use to implement this:
  - a method for the client to send data to the Hobbit server for
    processing without affecting the status display (the "data"
    messagetype)
  - a plugin-mechanism where hobbit "worker modules" can pick up these
    data and process them
  - a simple unix-pipe can be used to feed data into the normal graph
    handling module

E.g. one way Jeff could get his real-time graph goes like this:
  - On the client, run a job every minute to grab the data and send it
    to the Hobbit server using 
        $BB $BBDISP "data $MACHINE.xdata ...
  - On the Hobbit server, write a Hobbit module that grabs messages
    off the Hobbit "data" channel. This really just means reading
    messages from stdin - each message begins with a "@@data" line,
    and ends with a "@@" line. You can easily then pick out those
    messages that are "xdata" sent by the once-a-minute job.
  - This module then feeds the message into an RRD file. If it's
    one of the standard tests (e.g. disk), you can just change
    the "xdata" into "disk" and feed it into a child process running
    the normal hobbitd_rrd program. Start hobbitd_rrd for this
    purpose with a different BBRRDS setting, so the RRD files go
    into a separate directory (perhaps on a RAM disk, if you are
    really updating once a minute).

What's missing then is to get the RRD file created in a way so that it
will accept such frequent updates, and perhaps only store the last 
1 or 2 hours of data. So you'll have to dig into the "rrdtool create"
command to get the RRD file setup correctly, before you start feeding
data into it.


Regards,
Henrik