[Xymon] xymon hostdata module going rogue

[John Thurston]

I was bit by this in the middle of November, and didn't notice it until 
a customer alerted me today to a shortage of email messages.

To recap:

Some alerts get sent correctly, but in other cases the alert daemon 
aborts message processing and no alert is sent. In the cases where the 
daemon stops processing, my debug log begins to accumulate messages of 
the sort:
> 1730 2015-12-01 07:58:39.501785 Checking criteria for host 'upsjdc.state.ak.us', which is not defined

There is sometimes a <defunct> process left hanging around. At other 
times there is not.

Performing a "xymon.sh restart" makes it all work again.

Today, I had a process tree something like:
> 29118 /opt/xymon/server/bin/xymonlaunch --config=/opt/xymon/server/etc/tasks.cfg --en
>   29119 xymond --pidfile=/var/log/xymon/xymond.pid --restart=/opt/xymon/server/tmp/xymo
>   29120 /opt/xymon/server/bin/xymonfetch --id=1 --interval=79 --no-daemon --pidfile=/va
>   29144 xymond_channel --channel=stachg --log=/var/log/xymon/history.log xymond_history
>     29201 xymond_history --pidfile=/var/log/xymon/xymond_history.pid
>   29145 xymond_channel --channel=page --log=/var/log/xymon/alert.log xymond_alert --deb
>     29307 xymond_alert --debug --checkpoint-file=/opt/xymon/server/tmp/alert.chk --checkp
>       1588  <defunct>

I killed off PID 29145, it was recreated, and the alerts began flowing 
again.

In this occurrence, it does not appear to be related to a "drop" 
message. My last recorded "drop" was at 20151103-0846 and the alert 
process didn't start logging "which is not defined" until 20151120-0007

The only thing I can think to do now is make my xymon client monitor the 
alert.log and warn me when "which is not defined" start appearing so I 
can manually kill/restart the process.
-- 
    Do things because you should, not just because you can.

John Thurston    907-465-8591
John.Thurston at alaska.gov
Enterprise Technology Services
Department of Administration
State of Alaska