[Xymon] xymond_history: program crashed

[J.C. Cleaver]

On Thu, March 26, 2015 1:05 pm, Paul Grondahl wrote:
> Hi,
>
> First post. Thanks for supporting this fantastic app!
>
> In order to reclaim some disk space I ran:
>
> "su xymon -c '/home/xymon/server/bin/trimhistory --drop --cutoff=`date +%s
> --date="1 Oct 2013"` —droplogs’"
>
> Afterwards xymond_history went purple, with the message "program crashed -
> fatal error”
>
> I then ran "xymon localhost "drop <hostname> xymond_history” and now
> xymond_history has disappeared.
>
> I should add that the first time I ran trimhistory as root which messed up
> permissions on the allevents file. chmod xymon:xymon appears to have fixed
> it.
>
> How to get xymond_history working again?
>
> Also, for some long-running hosts, the hostdata directory remains at over
> 2GB. Is there a way to prune the hostdata directory?
>
> Can I safely delete hostdata with "rm <hostfile>" for hosts that are no
> longer monitored?


Welcome! We're glad you like it! :)


In this case, it seems like xymond_history crashed for some reason while
the trimhistory script was running; possibly a bug with how we handle
cases where files disappear underneath us. If a backtrace or core dump
file was left by the process when it happened, or anything unusual in the
history.log, it would be very help for us to be able to track things down.


In terms of "getting xymond_history working again", it should have been
re-launched right away by xymonlaunch -- you should see it now in your
'ps' listing. The current model of sending crashes like this as a dot
requiring a manual drop more or less ensures a conscious action will be
taken to acknowledge the issue. xymond_history in its normal operations
doesn't send any status in, which is why the status you saw eventually
turned purple in color.


You may safely delete unneeded files out of the /hostdata/ directory
without operational impact. (This is probably something that trimhistory
should take care of, actually.) The only impact would be to someone
actually trying to read the file from the web pages at that moment, since
xymond_hostdata doesn't write out to that timestamp after initially saving
it.



Regards,

-jc