[Xymon] jboss monitorin

Scot Kreienkamp SKreien at la-z-boy.com
Wed Sep 12 17:25:52 CEST 2012


I didn't think it was really appropriate to post because it's heavily customized for my different environments and such.  You'll have to do some work before it's suitable for yours.

Basically I pull a list of queues using curl then loop back through each queue with curl to grab the queue stats.  It sends those in a status message then sends the stats in a second data message so they can be graphed.  It will also clear any queue with DLQ in the name once it gets above 25.  Otherwise the DLQ's never clear.  I started doing that because the DLQ's got so big they brought the server to almost a standstill after several months of nobody paying attention to it.

The entries in your hosts file are like: jbossjms:8080
(jbossjms being the name of the test, and 8080 being the port the jmx console is on.  This assumes no auth on the jmx web console, which there isn't by default.)

Graphs.cfg entry:
[jboss]
        FNPATTERN ^jboss.(.+).rrd
        TITLE Queue sizes
        YAXIS Queue size
        DEF:p at RRDIDX@=@RRDFN@:lambda:MAX
        LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@
        GPRINT:p at RRDIDX@:LAST: \: %5.0lf (cur)
        GPRINT:p at RRDIDX@:MAX: \: %5.0lf (max)
        GPRINT:p at RRDIDX@:MIN: \: %5.0lf (min)
        GPRINT:p at RRDIDX@:AVERAGE: \: %5.0lf (avg)\n

I have mine setup to graph the max value, you probably need to change that to average.

You need to use splitncv in the xymonserver.cfg also.  Otherwise anytime you add or remove a queue the graph won't show up or the RRD updater will error out as the new data point won't exist in the RRD.

Works very well for me.  Here it is, hope this helps.



#!/bin/bash





function GETSTATS ()

{

COLOR=green

RETURNQUEUEMESSAGE="Queue_Name Message_Count Consumer_Count"

RETURNQUEUEMONITOR=""

ENVIRONMENTPREFIX=`echo $1 | cut -c 1-4`

RETURNMESSAGE=""

CONSUMERCOUNTER=""



case $1 in

                retv3040.na.lzb.hq)

                                ENVIRONMENTPREFIX=train

                ;;

                retv3041.na.lzb.hq)

                                ENVIRONMENTPREFIX=train

                ;;

                retv3042.na.lzb.hq)

                                ENVIRONMENTPREFIX=train

                ;;

                retv3043.na.lzb.hq)

                                ENVIRONMENTPREFIX=train

                ;;

                retv3044.na.lzb.hq)

                                ENVIRONMENTPREFIX=train

                ;;

                retv3045.na.lzb.hq)

                                ENVIRONMENTPREFIX=train

                ;;

esac



while read QUEUENAME ; do

                curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.messaging.destination%3Aservice%3DQueue%2Cname%3D$QUEUENAME" > /tmp/$$

                MESSAGECOUNT=`cat /tmp/$$ |grep -A 3 "The number of messages in the queue"|tail -1 | awk '{print $1}'`





                #setup some environment variables for prod VS dev and for specific queues

                if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then

                                case $QUEUENAME in

                                                PricingUploadResponseQueue)

                                                                MSGCOUNTRED=5 #the number of messages in the queue to turn the test red

                                                                MSGCOUNTYELLOW=3 # the number of messages in the queue to turn the test yellow

                                                                MINCONSUMERCOUNT=12 # the expected number of consumers for the PricingUploadResponseQueue and UnMeteredReportResponseQueue

                                                ;;

                                                *)

                                                                MSGCOUNTRED=500

                                                                MSGCOUNTYELLOW=250

                                                                MINCONSUMERCOUNT=0

                                                ;;

                                esac

                else

                                case $QUEUENAME in

                                                PricingUploadResponseQueue)

                                                                MSGCOUNTRED=10

                                                                MSGCOUNTYELLOW=5

                                                                MINCONSUMERCOUNT=0

                                                ;;

                                                *)

                                                                MSGCOUNTRED=500

                                                                MSGCOUNTYELLOW=250

                                                                MINCONSUMERCOUNT=0

                                                ;;

                                esac

                fi





                if [ "$MESSAGECOUNT" -gt "$MSGCOUNTRED" ] ; then

                                COLOR=red

                                RETURNSTATUS="Not OK"

                                RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is very high!"

                elif [ "$MESSAGECOUNT" -gt "$MSGCOUNTYELLOW" ] ; then

                                RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is higher than normal"

                                RETURNSTATUS="Caution"

                                if [ ! "$COLOR" = "red" ] ; then

                                                COLOR=yellow

                                fi

                fi

                CONSUMERCOUNT=`cat /tmp/$$ |grep -A 3 "The number of consumers on the queue"|tail -1 | awk '{print $1}'`

                if [ "$CONSUMERCOUNT" -lt "$MINCONSUMERCOUNT" ] ; then

                                COLOR=yellow

                                RETURNSTATUS="Not OK"

                                RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME has less than expected $MINCONSUMERCOUNT queue consumers!"

                fi

                RETURNQUEUEMESSAGE="$RETURNQUEUEMESSAGE\n$QUEUENAME $MESSAGECOUNT $CONSUMERCOUNT"

                RETURNQUEUEMONITOR="$RETURNQUEUEMONITOR\n$QUEUENAME : $MESSAGECOUNT"

                if [ "$MESSAGECOUNT" -gt "25" ] ; then

                                if echo $QUEUENAME | grep "DLQ" ; then

                                                curl  -m 10 -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=5&name=jboss.messaging.destination%3Aservice%3DQueue%2Cname%3D$QUEUENAME" >/dev/null 2>&1

                                                RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME was cleared"

                                                COLOR=yellow

                                fi

                fi



done < <(curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=displayMBeans&filter=jboss.messaging.destination" |grep -o ">name=.*,"|sed -e 's/>name=//g' -e 's/,$//g')





#set how many queue consumers we should expect

if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then

                MINQUEUECONSUMERSCOUNT=7 # the minimum number of queue consumers registered with JBoss before the test turns red for prod

else

                MINQUEUECONSUMERSCOUNT=1 # the minimum number of queue consumers registered with JBoss before the test turns red for dev

fi



#get list of registered queue consumers

LISTEDCONSUMERS="Queue Consumers:<table>"

while read CONSUMERIPADDRESS ; do

                let CONSUMERCOUNTER+=1

        HOSTNAME=`host $CONSUMERIPADDRESS | awk '{print $5}'`

        LISTEDCONSUMERS="$LISTEDCONSUMERS<tr><td>$HOSTNAME</td><td>$CONSUMERIPADDRESS</td></tr>"

done < <(curl -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=19&name=jboss.messaging%3Aservice%3DServerPeer" |grep '<td>10' |sed -e 's/<td>//g' -e 's/<\/td>//g' |sort |uniq)

LISTEDCONSUMERS="$LISTEDCONSUMERS </table>"

if [ "$CONSUMERCOUNTER" -lt "$MINQUEUECONSUMERSCOUNT" ] ; then

                RETURNMESSAGE="$RETURNMESSAGE\nSome queue consumers are missing!"

                COLOR=red

elif [ "$CONSUMERCOUNTER" -eq "$MINQUEUECONSUMERSCOUNT" ] ; then

                LISTEDCONSUMERS=""

fi



RETURNMESSAGE="$RETURNMESSAGE\nExpected $MINQUEUECONSUMERSCOUNT, found $CONSUMERCOUNTER queue consumers currently registered."



if [ "$RETURNQUEUEMONITOR" = "" ] ; then

        RETURNQUEUEMESSAGE="JBoss is down or unreachable!!!"

        COLOR=red

fi



if [ "$RETURNSTATUS" = "" ] ; then

        RETURNSTATUS="OK"

fi



if [ "$RETURNMESSAGE" = "" ] ; then

        RETURNMESSAGE="OK"

fi





/home/hobbit/client/bin/xymon retv6100.na.lzb.hq "status $1.jboss $COLOR `date` $RETURNSTATUS



`echo -e "Status message:\n $RETURNMESSAGE"`



$LISTEDCONSUMERS





Queue List:

`echo -e $RETURNQUEUEMESSAGE|column -t`



"



/home/hobbit/client/bin/xymon retv6100.na.lzb.hq "data $1.jboss green `date` OK



`echo -e $RETURNQUEUEMONITOR|sed -e 1d -e 's/\(.*\)/\U\1/' -e 's/QUEUE//g' -e 's/ //g' -e 's/INVENTORY/INVTRY/g' -e 's/RETAIL/RETL/g' -e 's/PRICE/PRC/g' -e 's/RESPONSE/RESPNS/g' -e 's/STATUS/STS/g' -e 's/FREIGHT/FRGT/g' -e 's/REQUEST/REQST/g' | awk -F ":" '{print substr($1,1,18)" : "$2}'`



"

rm -f /tmp/$$

}





while read HOSTLINE ; do

                SERVERNAME=`echo $HOSTLINE | awk '{print $2}'`

                SERVERPORT=`echo $HOSTLINE | awk -F ":" '{print $2}'`

                GETSTATS $SERVERNAME $SERVERPORT

done < <(/home/hobbit/server/bin/xymongrep jbossjms:*)




Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com<mailto:scot.kreienkamp at la-z-boy.com> | www.la-z-boy.com<http://www.la-z-boy.com/>

From: Larry Barber [mailto:lebarber at gmail.com]
Sent: Wednesday, September 12, 2012 10:59 AM
To: Scot Kreienkamp
Cc: xymon at xymon.com
Subject: Re: [Xymon] jboss monitorin

Could you send me a copy, or post it on Xymonton?

Thanks,
Larry Barber
On Wed, Sep 12, 2012 at 9:50 AM, Scot Kreienkamp <SKreien at la-z-boy.com<mailto:SKreien at la-z-boy.com>> wrote:
I wrote a jboss client that pulls message and consumer counts from the jmx web console.  That way I can tell if it's up, how full the queues are, and whether or not consumers are registered.  That's about all the useful info I could find in the web console.

Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com<mailto:scot.kreienkamp at la-z-boy.com> | www.la-z-boy.com<http://www.la-z-boy.com/>

From: xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com> [mailto:xymon-bounces at xymon.com<mailto:xymon-bounces at xymon.com>] On Behalf Of Larry Barber
Sent: Wednesday, September 12, 2012 10:43 AM
To: xymon at xymon.com<mailto:xymon at xymon.com>
Subject: [Xymon] jboss monitorin

Has anybody found a good way to monitor jboss servers? I have a bunch of jboss installations and need some way to monitor their performance. I checked Xymonton but couldn't find anything useful (the jmxstat project appears to be dead, at least the link leading to the download is dead). Any help would be much appreciated.

Thanks,
Larry Barber



This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.




This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20120912/72200479/attachment.html>


More information about the Xymon mailing list