Definitely good to see your effort on Devmon, I thought it had since been a forgotten project. The effort is much appreciated!<br><br>Hopefully someone that see this problem can come forward and help everyone by testing the SVN!<br>
<br clear="all">Josh Luthman<br>Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373<br>
<br><br><div class="gmail_quote">On Wed, Jun 1, 2011 at 10:39 AM, Buchan Milne <span dir="ltr"><<a href="mailto:bgmilne@staff.telkomsa.net">bgmilne@staff.telkomsa.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="im">On Tuesday, 31 May 2011 03:24:05 <a href="mailto:kconnell@ryerson.ca">kconnell@ryerson.ca</a> wrote:<br>
</div><div class="im">> I've had issues with devmon not updating the bb-display and everything<br>
> going purple.<br>
<br>
</div>Firstly, I don't think this is Josh's problem, as he didn't have a devmon<br>
process, whereas this behaviour is typically that devmon hangs (but the<br>
process is still running).<br>
<br>
If you have different behaviour to the I discuss below, please log a new<br>
tracker item.<br>
<br>
The 'hang' issue is covered in this tracker item:<br>
<br>
<a href="http://sourceforge.net/tracker/?func=detail&aid=2897345&group_id=160720&atid=816977" target="_blank">http://sourceforge.net/tracker/?func=detail&aid=2897345&group_id=160720&atid=816977</a><br>
<br>
(Unfortunately, it was logged anonymously, and I have had no feedback on<br>
improvements in devmon svn for this issue, either via the tracker, or the<br>
mails on the mailing list)<br>
<br>
Discussion of the issue also occurred on the devmon-support mailing list:<br>
<br>
<a href="http://sourceforge.net/mailarchive/forum.php?thread_name=201102021424.30555.bgmilne%40staff.telkomsa.net&forum_name=devmon-" target="_blank">http://sourceforge.net/mailarchive/forum.php?thread_name=201102021424.30555.bgmilne%40staff.telkomsa.net&forum_name=devmon-</a><br>
support<br>
<br>
The status has not changed, my failure logs still die at:<br>
<br>
[11-05-05@15:54:02] DEBUG: Printing single combo message size 13390<br>
[11-05-05@15:54:02] DEBUG: Finished printing single combo message<br>
[11-05-05@15:55:42] Fork 3 timed out waiting for data from parent: Timeout at<br>
/usr/share/devmon/modules/<a href="http://dm_snmp.pm" target="_blank">dm_snmp.pm</a> line 516, <$__ANONIO__> line 30203.<br>
<br>
The printing code is wrapped in an eval'd alarm subroutine which should return<br>
within 10 seconds, and log that the printing had completed or that it had<br>
timed out. Instead, the fork has noticed that it hasn't seen anything from the<br>
'master' process within the poll period for some time 40s later.<br>
<br>
The question is, what should be done in this case? Should the forks attempt to<br>
kill the master devmon process?<br>
<br>
Anyway, I would be grateful if someone could reproduce this on a different<br>
platform. I currently see this on RHEL5 x86_64 with perl-5.8.8-27.el5. Other<br>
environments have been green since 25 Jan ( since they were upgraded to rev<br>
214:<br>
<a href="http://devmon.svn.sourceforge.net/viewvc/devmon?view=revision&revision=214" target="_blank">http://devmon.svn.sourceforge.net/viewvc/devmon?view=revision&revision=214</a>).<br>
<div class="im"><br>
> I created a "devmon watchdog" script that's runs every 5 min using lynx<br>
> (txt base html browser) which checks if the status of devmon (shows as dm<br>
> test) on bb-monitor. If its purple then I kill the devmon process and<br>
> start it up again....band-aid solution, but it does the trick.<br>
><br>
> I no script expert, but can share the bash script if you want/need.<br>
<br>
</div>Here is mine, but I am *not* going to add it to svn and the next release<br>
unless I have had some feedback on the changes to prevent this occurring at<br>
all, preferable with the failure logs the script keeps.<br>
<br>
I run mine from hobbitlaunch.cfg (the problematic box is still running 4.2.2<br>
for now):<br>
<br>
[devmon]<br>
ENVFILE /usr/lib64/hobbit/server/etc/hobbitserver.cfg<br>
CMD /usr/local/bin/restart-devmon-if-purple<br>
INTERVAL 1m<br>
LOGFILE /var/log/hobbit/devmon-restart.log<br>
<br>
I have a sudo rule in place to allow the hobbit user to call 'sudo<br>
/etc/init.d/devmon stop'<br>
<br>
<br>
#!/bin/bash<br>
if [ "$BB" == "" ]<br>
then<br>
echo "This script must be run under a Hobbit or Xymon environment" >&2<br>
echo "e.g. by: bbcmd $0" >&2<br>
exit 1<br>
fi<br>
if [ "$BBDISPLAYS" != "" ]<br>
then<br>
BBDISP=${BBDISPLAYS#,*}<br>
fi<br>
COLOR=$($BB $BBDISP "hobbitdboard host=$HOSTNAME test=dm" | cut -d'|' -f3)<br>
<br>
if [ "`id -u`" -eq 0 ]<br>
then<br>
DEVMON="/etc/init.d/devmon"<br>
PKILL="pkill"<br>
else<br>
DEVMON="sudo /etc/init.d/devmon"<br>
PKILL="sudo pkill"<br>
fi<br>
<br>
if [ "$COLOR" == "purple" ]<br>
then<br>
LOGSAVE=/var/log/devmon/failures/devmon-failure-`date +%Y-%m-%d-%H:%M:<br>
%S`.log<br>
echo "Devmon is purple, saving last 200 lines of log to $LOGSAVE"<br>
tail -n200 /var/log/devmon/devmon.log > $LOGSAVE<br>
$DEVMON stop<br>
NUM=$(pgrep -u devmon|wc -l)<br>
if [ "$NUM" -ne 0 ]<br>
then<br>
echo "Devmon failed to stop cleanly, terminating manually"<br>
$PKILL -u devmon<br>
sleep 5<br>
fi<br>
NUM=$(pgrep -u devmon|wc -l)<br>
if [ "$NUM" -ne 0 ]<br>
then<br>
echo "Devmon failed to terminate cleanly, killing manually"<br>
$PKILL -9 -u devmon<br>
fi<br>
$DEVMON start<br>
else<br>
[ "$DEBUG" == 1 ] && echo "Devmon isn't purple, it is $COLOR"<br>
fi<br>
<br>
<br>
<br>
Regards,<br>
<font color="#888888">Buchan<br>
</font></blockquote></div><br>