hobbitfetch crashing

[Bill Arlofski]

I am moving from having DMZ clients sending reports into the Xymon server on
internal LAN to the "pulldata" method using hobbitfetch on the Xymon server
and msgcache on the DMS system(s).

Everything seems to be working. That is, data from the clients is being
logged, graphed and monitored and alerted on. So far so good.

BUT...

I saw hobbitfetch crashing and hobbitfetch column going red when using version
4.2.2 of Xymon, so I upgraded to 4.3.0-0.beta2 and am seeing the same symptoms.

I can not see anything odd in the hobbitfetch.log file (with --debug enabled).

There is a core file in ~xymon/server/tmp/core dated 18:40 today - I am
assuming it s a core from hobbitfetch - but the hobbitfetch log for that time
frame show only the following:

--[snip]--
2010-03-25 18:40:27 Queuing request 21 to 192.168.253.1:1984 for
zimbra.xxxxxx.com: 'pullclient 1


'
2010-03-25 18:40:27 Sent 14 bytes to 192.168.253.1:1984 (req 21)
2010-03-25 18:40:27 Done reading data from 192.168.253.1:1984 (req 21)
2010-03-25 18:40:27 Doing cleanup
2010-03-25 18:40:27 Next poll of zimbra.xxxxxx.com in 55 seconds
2010-03-25 18:40:27 Request completed: req 21, peer 192.168.253.1:1984, action
was 2, type was 0
2010-03-25 18:40:32 Queuing request 22 to 192.168.253.2:1984 for
helpdesk.xxxxxxx.com: 'pullclient 1
dir:/var/www/localhost/htdocs/admin/storage/mail/fail


'
2010-03-25 18:40:32 Sent 68 bytes to 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Got 1448 bytes of data from 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Got 1448 bytes of data from 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Got 1448 bytes of data from 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Got 1448 bytes of data from 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Got 1448 bytes of data from 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Got 1448 bytes of data from 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Got 8192 bytes of data from 192.168.253.2:1984 (req 22)
2010-03-25 18:40:32 Opening file /usr/local/xymon/server/etc/bb-hosts
2010-03-25 18:40:32 Queuing request 1 to 192.168.253.2:1984 for
helpdesk.xxxxxxx.com: 'pullclient 1

'
2010-03-25 18:40:32 Queuing request 2 to 192.168.253.1:1984 for
zimbra.xxxxxxxx.com: 'pullclient 1

'
2010-03-25 18:40:32 Sent 13 bytes to 192.168.253.2:1984 (req 1)
2010-03-25 18:40:32 Sent 13 bytes to 192.168.253.1:1984 (req 2)
2010-03-25 18:40:32 Done reading data from 192.168.253.2:1984 (req 1)
2010-03-25 18:40:32 Doing cleanup
2010-03-25 18:40:32 Next poll of helpdesk.xxxxx.com in 44 seconds
2010-03-25 18:40:32 Request completed: req 1, peer 192.168.253.2:1984, action
was 2, type was 0
2010-03-25 18:40:32 Done reading data from 192.168.253.1:1984 (req 2)
2010-03-25 18:40:32 Doing cleanup
2010-03-25 18:40:32 Next poll of zimbra.xxxxx.com in 55 seconds
2010-03-25 18:40:32 Request completed: req 2, peer 192.168.253.1:1984, action
was 2, type was 0
--[snip]--


In hobbitlaunch.cfg on the server I have enabled the hobbitfetch task:

[hobbitfetch]
        # waa - 20100321 - Need to pull data from DMZ clients
        # DISABLED
        ENVFILE /usr/local/xymon/server/etc/hobbitserver.cfg
        CMD $BBHOME/bin/hobbitfetch --debug --server=xymonIPaddress \
        --no-daemon --pidfile=$BBSERVERLOGS/hobbitfetch.pid
        LOGFILE $BBSERVERLOGS/hobbitfetch.log


I have enabled the the msgcache task on the clients in the
~hobbit/client/etc/clientlaunch.cfg file:

[msgcache]
        # waa 20100321
        # DISABLED
        ENVFILE $HOBBITCLIENTHOME/etc/hobbitclient.cfg
        CMD $HOBBITCLIENTHOME/bin/msgcache --no-daemon \
        --pidfile=$HOBBITCLIENTHOME/logs/msgcache.pid
        LOGFILE $HOBBITCLIENTHOME/logs/msgcache.log

And on the clients I have also modified the BBDISP to point to 127.0.0.1
instead of the actual Xymon server.

Firewall now allows connections OUT from Xymon server to DMZ clients, and
disallows DMZ clients to send in Xymon reports.


Any thoughts on this would be appreciated!

--
Bill Arlofski
Reverse Polarity, LLC
http://www.revpol.com/