[hobbit] hobbitd_rrd crash in trunk

[Olivier Beau]

Henrik,


Here are 2 other extracts from crashes :


2009-01-20 16:36:06 hobbitd_rrd: Got message 517875 
@@status#517875/sw01.courrierinternational|1232465766.838715|192.168.255.32
||sw01.courrierinternational|if_load|1232467566|green||green|1225102669|0||0||0|0||network/switch-dedie
2009-01-20 16:36:06 startpos 162634, fillpos 166552, endpos -1
2009-01-20 16:36:06 Want msg 517876, startpos 162634, fillpos 166552, 
endpos -1, usedbytes=3918, bufleft=361831
2009-01-20 16:36:06 Want msg 517876, startpos 162634, fillpos 170333, 
endpos -1, usedbytes=7699, bufleft=358050
2009-01-20 16:36:06 hobbitd_rrd: Got message 517876 
@@status#517876/sw01.ctoutvert|1232465766.838761|192.168.255.32||sw01.ctout
vert|memory|1234247285|blue||blue|1231828085|0||1234247285|Disabled by
2009-01-20 16:36:06 startpos 172884, fillpos 172884, endpos -1
2009-01-20 16:36:06 Peer at 0.0.0.0:0 failed: Broken pipe
2009-01-20 16:36:06 Peer not up, flushing message queue
2009-01-20 16:36:06 Opening file 
/data/hobbit/server/etc/hobbit-rrddefinitions.cfg
2009-01-20 16:36:06 Want msg 1, startpos 0, fillpos 0, endpos -1, 
usedbytes=0, bufleft=528383
2009-01-20 16:36:06 hobbitd_rrd: Got message 517913 
@@status#517913/sw01.excenteurofac|1232465766.929692|192.168.255.32||sw01.e
xcenteurofac|if_err|1232467566|green||green|1231866461|0||0||0|0||network/switch-dedie


if_load and if_err are status from devmon, that i do not graph using 
ncv/extra-test..
memory is also generate from devmon, and is graphes by default in xymon




2009-01-22 17:14:20 hobbitd_rrd: Got message 343666 
@@status#343666/logicimmo-netapp2|1232640859.848737|127.0.0.1||logicimmo-ne
tapp2|disk|2147483647|blue||blue|1232479545|0||-1|Disabled by
2009-01-22 17:14:20 startpos 417512, fillpos 419047, endpos -1
2009-01-22 17:14:20 Peer at 0.0.0.0:0 failed: Broken pipe
2009-01-22 17:14:20 Peer not up, flushing message queue
2009-01-22 17:14:20 Opening file 
/data/hobbit/server/etc/hobbit-rrddefinitions.cfg
2009-01-22 17:14:20 Want msg 1, startpos 0, fillpos 0, endpos -1, 
usedbytes=0, bufleft=528383
2009-01-22 17:14:20 hobbitd_rrd: Got message 343677 
@@status#343677/tif-netapp1|1232640860.884630|127.0.0.1||tif-netapp1|disk|1
232644460|green||green|1230710616|0||0||0|0|stockage|unix/infrasys/stockage
2009-01-22 17:14:20 startpos 1335, fillpos 3954, endpos 2589

disk is generate by netapp.pl (from the hobbit-client-perl)




-> i noticed that in my 3 extracts, the last log before the crash is 
disabled. Looks like this could be a problem ?
(i've check 2 other crashes, and there again, the last log is a disabled 
status)

i checked those 3 disabled status : those hosts are up and running (so 
normal status are sent to hobbitd) we have disabled them for migration 
purpose, that might happen in a few days, or weeks...




For your mysql question :
yes i do graph mysql using NVC
NCV_mysql="Questions:DERIVE,Threadsconnected:GAUGE,*:NONE"	



Olivier



On 22/01/2009 15:29, Henrik Størner wrote:
> In <4974AE8B.80706 at gmail.com> Olivier Beau <obeau79 at gmail.com> writes:
> 
>> It happened again today at 17:00:22.
>> Nothing new when doing a bt on the coredump.
>> An extract of rrd-status.log from 16h55 to 17h05 is available at 
>> http://www.qalpit.com/~olivier/tmp/rrd-status.log.gz
> 
> OK, the interesting part is here when it crashes:
> 
> 2009-01-19 17:00:22 hobbitd_rrd: Got message 181436 @@status#181436/cedratnet-bdd1|1232380822.602633|127.
> 0.0.1||cedratnet-bdd1|mysql|1232398822|green||green|1231215890|0||0||1232380812|0|linuxmysql|unix/mysql
> 2009-01-19 17:00:22 startpos 342639, fillpos 378880, endpos 342991
> 2009-01-19 17:00:22 hobbitd_rrd: Got message 181437 @@status#181437/moniteur-ora2|1232380822.618847|10.12
> .0.67||moniteur-ora2|cpu|1255363113|blue||blue|1228751913|0||1255363113|Disabled by
> 2009-01-19 17:00:22 startpos 342995, fillpos 378880, endpos -1
> 2009-01-19 17:00:22 Peer at 0.0.0.0:0 failed: Broken pipe
> 2009-01-19 17:00:22 Peer not up, flushing message queue
> 2009-01-19 17:00:22 Opening file /data/hobbit/server/etc/hobbit-rrddefinitions.cfg
> 2009-01-19 17:00:22 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=528383
> 2009-01-19 17:00:22 hobbitd_rrd: Got message 181450 @@status#181450/nurun-etam-bdd1|1232380822.807004|127
> .0.0.1||nurun-etam-bdd1|mysql|1232398822|green||green|1231768476|0||0||1232380582|0|linuxmysql|unix/mysql
> 2009-01-19 17:00:22 startpos 17100, fillpos 19357, endpos 17846
> 2009-01-19 17:00:22 Opening file /data/hobbit/server/etc/bb-hosts
> 
> 
> It appears to be a "mysql" status from either cedratnet-bdd1 or nurun-etam-bdd1 
> that causes the crash (I cannot tell exactly, because output buffering comes
> into play when there's a crash). It *could* also be the cpu-report from
> moniteur-ora2, but I doubt that - the cpu-status is tested a lot more than 
> the mysql-status.
> 
> In fact, "mysql" isn't part of hobbitd_rrd by default. So is this something
> you've added ? Is it something that you generate graphs for ? Or is it just
> a status that hobbitd_rrd should ignore ?
> 
> 
> Regards,
> Henrik
> 
> 
> To unsubscribe from the hobbit list, send an e-mail to
> hobbit-unsubscribe at hswn.dk
> 
> 
>