[hobbit] some larrd issues on hobbit 4.0.3 rc1

[Kauffman, Tom]

Well, I see what is going on, and I don't want to try to explain it --

The vmstat rrd for all my AIX systems has the data elements defined in
alpha sequence: cpu_b, cpu_csw, cpu_idl, cpu_int, cpu_r, cpu_syc,
cpu_sys, cpu_usr, cpu_wait, mem_avm, mem_cy, mem_fr, mem_free, mem_pi,
mem_po, mem_re, and mem_sr.

Hmm. Seems that the vmstat rrds for my two SuSE linux boxes are also in
alpha sequence. I'm not real strong in perl, but I suspect this is the
culprit: foreach $col ( sort keys %{$htovm{$bbhosttype}}) {
                push @ds,"DS:$col:GAUGE:600:0:U";

The definitions in $htovm are field name followed by index number.

Short term, I'm going to change do_vmstat.c, while I research writing
some code to re-work the rrd xml (too many changes and too many rrds to
try it by hand).

The disk patch is much appreciated; our primary R3 DB server should drop
from 31 disk graphs down to three just by excluding
'/oracle/PRD/sapdata*'.

Tom

-----Original Message-----
From: Henrik Stoerner [mailto:henrik at hswn.dk] 
Sent: Sunday, May 01, 2005 2:12 AM
To: hobbit at hswn.dk
Subject: Re: [hobbit] some larrd issues on hobbit 4.0.3 rc1

On Sat, Apr 30, 2005 at 06:22:56PM -0500, Kauffman, Tom wrote:
> These vmstat rrds are from back on larrd 42; just after the change to
> accumulate cpu wait. So I'm trying the vmstat recreate to see if the
> definitions I've got are severely non-standard (I'd almost bet on it)
> 

Use the "rrdtool dump FILENAME.rrd" to dump the old data into a
text file (XML) format. When you look at this file, at the top you'll 
find the data-sets definitions that LARRD has setup in this RRD file;
these come from the "aix" definition in the old LARRD vmstat-larrd.pl
script. So there should be (in sequence): cpu_r, cpu_b, mem_avm,
mem_free, mem_re, mem_pi, mem_po, mem_fr, sr, mem_cy, cpu_int, cpu_syc,
cpu_csw, cpu_usr, cpu_sys, cpu_idl, cpu_wait - at least, that's what 
Hobbit would generate, and therefore it assumes this layout when
updating the RRD-file.

Since the files are being updated by Hobbit, but the data collected is
wrong, my guess is that you have these in a different sequence than
Hobbit expects.

There are two way of tackling that problem. 

One way is to change the Hobbit layout to match your current RRD files.
This layout is defined in the hobbit-4.0.3rc1/hobbitd/larrd/do_vmstat.c
file - just look for "aix" and you'll see it. Only problem with this is
that you'll need to repeat this change whenever you upgrade Hobbit.

The other way is to modify the dumped RRD-file, then use "rrdtool
restore" 
to convert the modified XML-file back to an RRD file.

You need to change the sequence of the dataset definitions at the
beginning of the file, and also change each of the data "rows" that make
up the bulk of the file. These look like this:

<!-- 2005-05-01 02:00:00 CEST / 1114905600 --><row><v> 1.5896990741e-01
</v><v>2.1686840278e+02 </v><v> 9.5610891204e+01 </v><v>
3.5725331019e+02</v><v> 1.0420138889e-01 </v><v>8.3974537037e-01 </v><v>
3.3892245370e+00</v><v> 3.3494723380e+02 </v><v>9.9369259259e+01 </v><v>
1.0934771532e+05</v><v> 3.8053798435e+05 </v><v> 8.1690244444e+03
</v><v> 2.7122800926e+00 </v><v>1.2084837963e+00 </v><v>
2.1852577870e+05</v></row>

Each of the "<v> VALUE </v>" appear in the sequence that the datasets
are defined. So you must swap values around to match the new layout.


> On the disk space rrds -- this is a lot of wasted activity for us; we
> have about 8 filesystems we care about, and my production R3 DB server
> currently has 95 filesystems that have been 100% full since creation
--
> and we add another 10 every 13 months (150 GB -- SAP just *eats*
disk).

I see - perhaps something like the attached patch could be used. With
this, you can setup two environment variables that are regexp patterns
that the filesystem name is matched against before they get graphed;
NORRDDISKS is an "exclude" pattern - any filesystem name matching this 
do not get a graph, RDDISKS is an "include" pattern - only filesystem 
names matching this pattern get graphed. You can use none of them (the
current behaviour), one of them or both.

E.g. if all of your SAP filesystems are mounted below "/sap", you would
just put
  NORRDDISKS="^/sap"
in hobbitserver.cfg, and they won't get graphed.

This doesn't affect any of the RRD files that have already been created,
so you must manually clean out the unwanted disk*.rrd files from the
~hobbit/data/rrd/HOSTNAME/ directory to get rid of the graphs you don't
want.


Henrik