[Xymon] jboss monitorin
Scot Kreienkamp
SKreien at la-z-boy.com
Wed Sep 12 17:25:52 CEST 2012
I didn't think it was really appropriate to post because it's heavily customized for my different environments and such. You'll have to do some work before it's suitable for yours.
Basically I pull a list of queues using curl then loop back through each queue with curl to grab the queue stats. It sends those in a status message then sends the stats in a second data message so they can be graphed. It will also clear any queue with DLQ in the name once it gets above 25. Otherwise the DLQ's never clear. I started doing that because the DLQ's got so big they brought the server to almost a standstill after several months of nobody paying attention to it.
The entries in your hosts file are like: jbossjms:8080
(jbossjms being the name of the test, and 8080 being the port the jmx console is on. This assumes no auth on the jmx web console, which there isn't by default.)
Graphs.cfg entry:
[jboss]
FNPATTERN ^jboss.(.+).rrd
TITLE Queue sizes
YAXIS Queue size
DEF:p at RRDIDX@=@RRDFN@:lambda:MAX
LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@
GPRINT:p at RRDIDX@:LAST: \: %5.0lf (cur)
GPRINT:p at RRDIDX@:MAX: \: %5.0lf (max)
GPRINT:p at RRDIDX@:MIN: \: %5.0lf (min)
GPRINT:p at RRDIDX@:AVERAGE: \: %5.0lf (avg)\n
I have mine setup to graph the max value, you probably need to change that to average.
You need to use splitncv in the xymonserver.cfg also. Otherwise anytime you add or remove a queue the graph won't show up or the RRD updater will error out as the new data point won't exist in the RRD.
Works very well for me. Here it is, hope this helps.
#!/bin/bash
function GETSTATS ()
{
COLOR=green
RETURNQUEUEMESSAGE="Queue_Name Message_Count Consumer_Count"
RETURNQUEUEMONITOR=""
ENVIRONMENTPREFIX=`echo $1 | cut -c 1-4`
RETURNMESSAGE=""
CONSUMERCOUNTER=""
case $1 in
retv3040.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3041.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3042.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3043.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3044.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3045.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
esac
while read QUEUENAME ; do
curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.messaging.destination%3Aservice%3DQueue%2Cname%3D$QUEUENAME" > /tmp/$$
MESSAGECOUNT=`cat /tmp/$$ |grep -A 3 "The number of messages in the queue"|tail -1 | awk '{print $1}'`
#setup some environment variables for prod VS dev and for specific queues
if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then
case $QUEUENAME in
PricingUploadResponseQueue)
MSGCOUNTRED=5 #the number of messages in the queue to turn the test red
MSGCOUNTYELLOW=3 # the number of messages in the queue to turn the test yellow
MINCONSUMERCOUNT=12 # the expected number of consumers for the PricingUploadResponseQueue and UnMeteredReportResponseQueue
;;
*)
MSGCOUNTRED=500
MSGCOUNTYELLOW=250
MINCONSUMERCOUNT=0
;;
esac
else
case $QUEUENAME in
PricingUploadResponseQueue)
MSGCOUNTRED=10
MSGCOUNTYELLOW=5
MINCONSUMERCOUNT=0
;;
*)
MSGCOUNTRED=500
MSGCOUNTYELLOW=250
MINCONSUMERCOUNT=0
;;
esac
fi
if [ "$MESSAGECOUNT" -gt "$MSGCOUNTRED" ] ; then
COLOR=red
RETURNSTATUS="Not OK"
RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is very high!"
elif [ "$MESSAGECOUNT" -gt "$MSGCOUNTYELLOW" ] ; then
RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is higher than normal"
RETURNSTATUS="Caution"
if [ ! "$COLOR" = "red" ] ; then
COLOR=yellow
fi
fi
CONSUMERCOUNT=`cat /tmp/$$ |grep -A 3 "The number of consumers on the queue"|tail -1 | awk '{print $1}'`
if [ "$CONSUMERCOUNT" -lt "$MINCONSUMERCOUNT" ] ; then
COLOR=yellow
RETURNSTATUS="Not OK"
RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME has less than expected $MINCONSUMERCOUNT queue consumers!"
fi
RETURNQUEUEMESSAGE="$RETURNQUEUEMESSAGE\n$QUEUENAME $MESSAGECOUNT $CONSUMERCOUNT"
RETURNQUEUEMONITOR="$RETURNQUEUEMONITOR\n$QUEUENAME : $MESSAGECOUNT"
if [ "$MESSAGECOUNT" -gt "25" ] ; then
if echo $QUEUENAME | grep "DLQ" ; then
curl -m 10 -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=5&name=jboss.messaging.destination%3Aservice%3DQueue%2Cname%3D$QUEUENAME" >/dev/null 2>&1
RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME was cleared"
COLOR=yellow
fi
fi
done < <(curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=displayMBeans&filter=jboss.messaging.destination" |grep -o ">name=.*,"|sed -e 's/>name=//g' -e 's/,$//g')
#set how many queue consumers we should expect
if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then
MINQUEUECONSUMERSCOUNT=7 # the minimum number of queue consumers registered with JBoss before the test turns red for prod
else
MINQUEUECONSUMERSCOUNT=1 # the minimum number of queue consumers registered with JBoss before the test turns red for dev
fi
#get list of registered queue consumers
LISTEDCONSUMERS="Queue Consumers:<table>"
while read CONSUMERIPADDRESS ; do
let CONSUMERCOUNTER+=1
HOSTNAME=`host $CONSUMERIPADDRESS | awk '{print $5}'`
LISTEDCONSUMERS="$LISTEDCONSUMERS<tr><td>$HOSTNAME</td><td>$CONSUMERIPADDRESS</td></tr>"
done < <(curl -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=19&name=jboss.messaging%3Aservice%3DServerPeer" |grep '<td>10' |sed -e 's/<td>//g' -e 's/<\/td>//g' |sort |uniq)
LISTEDCONSUMERS="$LISTEDCONSUMERS </table>"
if [ "$CONSUMERCOUNTER" -lt "$MINQUEUECONSUMERSCOUNT" ] ; then
RETURNMESSAGE="$RETURNMESSAGE\nSome queue consumers are missing!"
COLOR=red
elif [ "$CONSUMERCOUNTER" -eq "$MINQUEUECONSUMERSCOUNT" ] ; then
LISTEDCONSUMERS=""
fi
RETURNMESSAGE="$RETURNMESSAGE\nExpected $MINQUEUECONSUMERSCOUNT, found $CONSUMERCOUNTER queue consumers currently registered."
if [ "$RETURNQUEUEMONITOR" = "" ] ; then
RETURNQUEUEMESSAGE="JBoss is down or unreachable!!!"
COLOR=red
fi
if [ "$RETURNSTATUS" = "" ] ; then
RETURNSTATUS="OK"
fi
if [ "$RETURNMESSAGE" = "" ] ; then
RETURNMESSAGE="OK"
fi
/home/hobbit/client/bin/xymon retv6100.na.lzb.hq "status $1.jboss $COLOR `date` $RETURNSTATUS
`echo -e "Status message:\n $RETURNMESSAGE"`
$LISTEDCONSUMERS
Queue List:
`echo -e $RETURNQUEUEMESSAGE|column -t`
"
/home/hobbit/client/bin/xymon retv6100.na.lzb.hq "data $1.jboss green `date` OK
`echo -e $RETURNQUEUEMONITOR|sed -e 1d -e 's/\(.*\)/\U\1/' -e 's/QUEUE//g' -e 's/ //g' -e 's/INVENTORY/INVTRY/g' -e 's/RETAIL/RETL/g' -e 's/PRICE/PRC/g' -e 's/RESPONSE/RESPNS/g' -e 's/STATUS/STS/g' -e 's/FREIGHT/FRGT/g' -e 's/REQUEST/REQST/g' | awk -F ":" '{print substr($1,1,18)" : "$2}'`
"
rm -f /tmp/$$
}
while read HOSTLINE ; do
SERVERNAME=`echo $HOSTLINE | awk '{print $2}'`
SERVERPORT=`echo $HOSTLINE | awk -F ":" '{print $2}'`
GETSTATS $SERVERNAME $SERVERPORT
done < <(/home/hobbit/server/bin/xymongrep jbossjms:*)
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com<mailto:scot.kreienkamp at la-z-boy.com> | www.la-z-boy.com<http://www.la-z-boy.com/>
From: Larry Barber [mailto:lebarber at gmail.com]
Sent: Wednesday, September 12, 2012 10:59 AM
To: Scot Kreienkamp
Cc: xymon at xymon.com
Subject: Re: [Xymon] jboss monitorin
Could you send me a copy, or post it on Xymonton?
Thanks,
Larry Barber
On Wed, Sep 12, 2012 at 9:50 AM, Scot Kreienkamp <SKreien at la-z-boy.com<mailto:SKreien at la-z-boy.com>> wrote:
I wrote a jboss client that pulls message and consumer counts from the jmx web console. That way I can tell if it's up, how full the queues are, and whether or not consumers are registered. That's about all the useful info I could find in the web console.
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com<mailto:scot.kreienkamp at la-z-boy.com> | www.la-z-boy.com<http://www.la-z-boy.com/>
From: xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com> [mailto:xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com>] On Behalf Of Larry Barber
Sent: Wednesday, September 12, 2012 10:43 AM
To: xymon at xymon.com<mailto:xymon at xymon.com>
Subject: [Xymon] jboss monitorin
Has anybody found a good way to monitor jboss servers? I have a bunch of jboss installations and need some way to monitor their performance. I checked Xymonton but couldn't find anything useful (the jmxstat project appears to be dead, at least the link leading to the download is dead). Any help would be much appreciated.
Thanks,
Larry Barber
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20120912/72200479/attachment.html>
More information about the Xymon
mailing list