[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [hobbit] hobbitd_rrd crash in trunk

i enabled all disabled status that are used from graphing
(cpu,disk,procs,...)
and i have not had a single crash in the last 36 hours
(before, crashes would happen at least twice per day)

-> from my user point of view, it looks like disabled status can crash
hobbitd_rrd


olivier


On 22/01/2009 21:28, Olivier Beau wrote:

Henrik,


Here are 2 other extracts from crashes :
2009-01-20 16:36:06 hobbitd_rrd: Got message 517875 @@status#517875/sw01.courrierinternational|1232465766.838715|192.168.255.32 ||sw01.courrierinternational|if_load|1232467566|green||green|1225102669|0||0||0|0||network/switch-dedie 
2009-01-20 16:36:06 startpos 162634, fillpos 166552, endpos -1
2009-01-20 16:36:06 Want msg 517876, startpos 162634, fillpos 166552, endpos -1, usedbytes=3918, bufleft=361831 2009-01-20 16:36:06 Want msg 517876, startpos 162634, fillpos 170333, endpos -1, usedbytes=7699, bufleft=358050 2009-01-20 16:36:06 hobbitd_rrd: Got message 517876 @@status#517876/sw01.ctoutvert|1232465766.838761|192.168.255.32||sw01.ctout 
vert|memory|1234247285|blue||blue|1231828085|0||1234247285|Disabled by
2009-01-20 16:36:06 startpos 172884, fillpos 172884, endpos -1
2009-01-20 16:36:06 Peer at 0.0.0.0:0 failed: Broken pipe
2009-01-20 16:36:06 Peer not up, flushing message queue
2009-01-20 16:36:06 Opening file /data/hobbit/server/etc/hobbit-rrddefinitions.cfg 2009-01-20 16:36:06 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=528383 2009-01-20 16:36:06 hobbitd_rrd: Got message 517913 @@status#517913/sw01.excenteurofac|1232465766.929692|192.168.255.32||sw01.e xcenteurofac|if_err|1232467566|green||green|1231866461|0||0||0|0||network/switch-dedie
if_load and if_err are status from devmon, that i do not graph using ncv/extra-test.. 
memory is also generate from devmon, and is graphes by default in xymon
2009-01-22 17:14:20 hobbitd_rrd: Got message 343666 @@status#343666/logicimmo-netapp2|1232640859.848737|127.0.0.1||logicimmo-ne 
tapp2|disk|2147483647|blue||blue|1232479545|0||-1|Disabled by
2009-01-22 17:14:20 startpos 417512, fillpos 419047, endpos -1
2009-01-22 17:14:20 Peer at 0.0.0.0:0 failed: Broken pipe
2009-01-22 17:14:20 Peer not up, flushing message queue
2009-01-22 17:14:20 Opening file /data/hobbit/server/etc/hobbit-rrddefinitions.cfg 2009-01-22 17:14:20 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=528383 2009-01-22 17:14:20 hobbitd_rrd: Got message 343677 @@status#343677/tif-netapp1|1232640860.884630|127.0.0.1||tif-netapp1|disk|1 
232644460|green||green|1230710616|0||0||0|0|stockage|unix/infrasys/stockage
2009-01-22 17:14:20 startpos 1335, fillpos 3954, endpos 2589

disk is generate by netapp.pl (from the hobbit-client-perl)
-> i noticed that in my 3 extracts, the last log before the crash is disabled. Looks like this could be a problem ? (i've check 2 other crashes, and there again, the last log is a disabled status)
i checked those 3 disabled status : those hosts are up and running (so normal status are sent to hobbitd) we have disabled them for migration purpose, that might happen in a few days, or weeks... 




For your mysql question :
yes i do graph mysql using NVC
NCV_mysql="Questions:DERIVE,Threadsconnected:GAUGE,*:NONE" 


Olivier



On 22/01/2009 15:29, Henrik Størner wrote:

In <4974AE8B.80706 (at) gmail.com> Olivier Beau <obeau79 (at) gmail.com> writes:


It happened again today at 17:00:22.
Nothing new when doing a bt on the coredump.
An extract of rrd-status.log from 16h55 to 17h05 is available at http://www.qalpit.com/~olivier/tmp/rrd-status.log.gz 

OK, the interesting part is here when it crashes:
2009-01-19 17:00:22 hobbitd_rrd: Got message 181436 @@status#181436/cedratnet-bdd1|1232380822.602633|127. 0.0.1||cedratnet-bdd1|mysql|1232398822|green||green|1231215890|0||0||1232380812|0|linuxmysql|unix/mysql 
2009-01-19 17:00:22 startpos 342639, fillpos 378880, endpos 342991
2009-01-19 17:00:22 hobbitd_rrd: Got message 181437 @@status#181437/moniteur-ora2|1232380822.618847|10.12 .0.67||moniteur-ora2|cpu|1255363113|blue||blue|1228751913|0||1255363113|Disabled by 
2009-01-19 17:00:22 startpos 342995, fillpos 378880, endpos -1
2009-01-19 17:00:22 Peer at 0.0.0.0:0 failed: Broken pipe
2009-01-19 17:00:22 Peer not up, flushing message queue
2009-01-19 17:00:22 Opening file /data/hobbit/server/etc/hobbit-rrddefinitions.cfg 2009-01-19 17:00:22 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=528383 2009-01-19 17:00:22 hobbitd_rrd: Got message 181450 @@status#181450/nurun-etam-bdd1|1232380822.807004|127 .0.0.1||nurun-etam-bdd1|mysql|1232398822|green||green|1231768476|0||0||1232380582|0|linuxmysql|unix/mysql 
2009-01-19 17:00:22 startpos 17100, fillpos 19357, endpos 17846
2009-01-19 17:00:22 Opening file /data/hobbit/server/etc/bb-hosts
It appears to be a "mysql" status from either cedratnet-bdd1 or nurun-etam-bdd1 that causes the crash (I cannot tell exactly, because output buffering comes 
into play when there's a crash). It *could* also be the cpu-report from
moniteur-ora2, but I doubt that - the cpu-status is tested a lot more than the mysql-status.
In fact, "mysql" isn't part of hobbitd_rrd by default. So is this something you've added ? Is it something that you generate graphs for ? Or is it just 
a status that hobbitd_rrd should ignore ?


Regards,
Henrik


To unsubscribe from the hobbit list, send an e-mail to
hobbit-unsubscribe (at) hswn.dk