[hobbit] Netstat rrd error updating

[Henrik Stoerner]

On Fri, Oct 13, 2006 at 04:09:59PM -0400, Epp, Matthew Contractor PEO EIS AKO wrote:
> Searched the archives and found other mentions of this error but no
> solution. I have a Solaris 8 server with the Hobbit client reporting to the
> Hobbit server, and everything works except the netstat trend graph. I made
> sure Hobbit generates the netstat.rrd file itself, and verified that it has
> the correct fields:
> 
> $ rrdtool dump netstat.rrd |grep "<name>"
> 
>                 <name> udpInDatagrams </name>
>                 <name> udpOutDatagrams </name>
>                 <name> udpInErrors </name>
>                 <name> tcpActiveOpens </name>
>                 <name> tcpPassiveOpens </name>
>                 <name> tcpAttemptFails </name>
>                 <name> tcpEstabResets </name>
>                 <name> tcpCurrEstab </name>
>                 <name> tcpOutDataBytes </name>
>                 <name> tcpInInorderBytes </name>
>                 <name> tcpInUnorderBytes </name>
>                 <name> tcpRetransBytes </name>
>                 <name> tcpOutDataPackets </name>
>                 <name> tcpInInorderPackets </name>
>                 <name> tcpInUnorderPackets </name>
>                 <name> tcpRetransPackets </name>
> 
> But viewing a debug output of rrd-data.log shows that something is making it
> think there should only be 11 fields:

> 2006-10-13 19:18:22 RRD update param 03:
> 'udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActiveOpens:tcpPassiveOpens:t
> cpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataBytes:tcpInInorderBytes
> :tcpInUnorderBytes'

11 items here, yes.

> 2006-10-13 19:18:22 RRD update param 04:
> '1160767102:1580411:5594723:0:113952:19256340:17330:36082:7:653169856:440119
> 035:9398449:159635980:26380336:13186666:30559:612744'

And 16 values there. Most odd.

> Where is "param 03" coming from? From some configuration on the client side?

Param 03 is the "-t" option, which tells rrd_update() that param 04
defines the sequence of data set values.


Hmm ... could you try attaching gdb to the running hobbitd_rrd process.
There are two of them - one is handling "status" messages, the other
"data" messages. You'll need to find the process-ID of the one handling
"data" messages. Do something like

  $ ps -u hobbit -f|egrep "hobbitd_rrd|hobbitd_channel"

and look for the lines with

hobbit   25196 25150  0 Oct04 ?        00:00:02 hobbitd_channel --channel=data --log=/var/log/hobbit/rrd-data.log hobbitd_rrd --rrddir=/var/lib/hobbit/rrd
hobbit   25219 25196  0 Oct04 ?        00:00:53 hobbitd_rrd --rrddir=/var/lib/hobbit/rrd

where the PID of the hobbitd_channel process (25196 here) matches the 
parent-PID of the hobbitd_rrd process (25219). The "25219" is what you
need. Now run gdb like this (as the hobbit user):

  $ gdb ~/server/bin/hobbitd_rrd 
  (gdb) attach 25219
  [... lots of gdb printout ...]
  (gdb) p netstat_tpl

I'm interested in what the netstat_tpl output shows, since that is what
is used to build that "param 04" value. Here it looks like

(gdb) p netstat_tpl
$1 = 0x8076a68
"udpInDatagrams:udpOutDatagrams:udpInErrors:tcpActiveOpens:tcpPassiveOpens:tcpAttemptFails:tcpEstabResets:tcpCurrEstab:tcpOutDataBytes:tcpInInorderBytes:tcpInUnorderBytes:tcpRetransBytes:tcpOutDataPack"...
(gdb) p netstat_tpl+200
$2 = 0x8076b30
"ets:tcpInInorderPackets:tcpInUnorderPackets:tcpRetransPackets"

After that, just ctrl-d to exit gdb. If it asks if you want to "detach",
answer "y".


Also, could you run this command (as the hobbit user)

  $ bbcmd hobbitd_channel --channel=data hobbitd_capture --hosts=localhost

(replace "localhost" with the hostname of your solaris 8 client) ?
Let it run for 10 minutes and send me the output.



Thanks,
Henrik