[hobbit] Client interval question

[Adam Goryachev]

On Wed, 2005-12-14 at 17:31 -0600, Jeff Newman wrote:
> 
> 
> On 12/13/05, Scott Walters <scott at packetpushers.com> wrote: 

>         >> The graph DB's that vmstat feeds data into (the RRD files)
>         are
>         >> constructed in such a way that a 5-minute interval is what
>         makes 
>         >> sense. So running them with anything else really just a
>         waste of
>         >> ressources.
>         
>         > With the stock larrd/hobbit RRD definitions you are
>         correct.  He'll
>         > only use one of the five, and whine about the timestamp of
>         the other 
>         > four.
>  
> Firstly, can you explain your comment in more detail? Secondly, 
> im confused as to why you would state that I would "whine" about
> anything
> when you have no basis for a conclusion to that effect. It seems to be
> a rather
> pointed comment in a discussion that hasn't involved the use of
> language that
> would dictate a response like that.

I think he was referring to the error message that would be generated
because of the extra data compared to the interval configured in the rrd
files.
>         > To become flexible enough
>         > to handle different sampling rates, the server would need to
>         know the 
>         > frequency of the tests.  And then changing the RRD in the
>         future is
>         > 'almost' impossible (very difficult at the least).  And I've
>         never
>         > seen what happens to 1.5 years of data when you start
>         messing with 
>         > the RRD.
>         
>         > In the end, I think you'd get the worst of both worlds.

Well, we could take this the other way, and say that we only wanted to
run our tests once every 10 minutes, because it was causing too much
overhead to run the tests every 5 minutes. How would we deal with that?

I think there are benefits, on the client side, the client should pass
the frequency which it is calling the tests at, to them, so that for
example, the vmstat test can adjust how long it will run for to either
600 seconds, or 60 seconds, or whatever else is needed.

Further to that, there is some additional work (which I feel is the real
place that all the work is involved, the client side stuff would be
quite simple, or so it sounds). On the server, we either need to
re-create/adjust the rrd file so that we can insert data more
frequently, or else we need to somehow summarise the data before
insertion (which means there was no benefit in collecting the data more
frequently in the first place).
So, the question becomes, how difficult is it to convert an rrd file,
which was initially created to store data-points every 300 seconds, such
that we can now store data-points every X seconds?
The second part to this question, is how does hobbit know how frequently
you want to send your reports? ie, it can't be based on 'however often
they are received', because that value would change very frequantly, ie,
the reports are done every 300 seconds, but by the time the report is
submitted/processed by hobbit, it might be 1 or 2 seconds late/early
compared to last time..... Could hobbit server 'learn' the frequency
from the client (which is where this is configured anyway), because the
client would report that value to the server as a part of the vmstat
output?

Of course, even once both of those questions are satisfactorily
answered, you (yes, you) need to convince somebody to take the effort to
actually do it. Simply seeing the methods, and the interest some people
have taken in the possibility might be enough for someone with the
coding skills to do it, or, sometimes you might need to provide other
incentives (even paid).  Let me state clearly right now before I go on,
no, don't pay me, I don't have that level of coding skills in C to do it
for you, nor the knowledge of RRD. No, I don't know who you might pay,
or anything else, I have no interest in any of it, I'm just stating a
simple fact of life (you want something done, you can't do it, so find
someone who can do it, and motivate them to the point where they will do
it whether they want to or not)...

>         > I disagree.  If real-time performance analysis is needed, I
>         would
>         > pick other tools --  "vmstat 5"  works for me;)  Or
>         construct/fork 
>         > the client agent specifically designed for such a task, and
>         run it on
>         > an as-needed basis.
>  
> There are other tools yes. I am trying to leverage hobbit. If it's not
> possible
> and nobody wants to do it, then yes, ill look into other tools. On the
> same token
> I don't want to kill my performance by running lots of different
> monitoring on the server.
> Hobbit is extrodinarily lightweight on the client (as opposed to other
> solutions out there)
> so I think something like this is possible without overloading a
> client.

Agreed, it would be nice to not have to run hobbit plus something else
when they are both collecting the same data (just a different
frequency).

> Just my two cents. 

Here's a couple of mine also :)

Regards,
Adam