[Xymon] jboss monitorin
Nico
nicolas at lienard.name
Wed Sep 12 21:30:00 CEST 2012
hi
you can also use the jvm check done by olivier (sent yesterday)
it uses jps/jinfo and jstat.
maybe you have to adapt the jps grep but it is minor.
#!/usr/bin/perl
use Data::Dumper;
my @jps = split /\n/, `sudo /usr/java/jdk1.6/bin/jps -v | grep Bootstrap`;
my %res;
foreach my $jvm (@jps) {
my @t = split /\s+/, $jvm;
my $pid = shift @t;
if ( my @tt = grep /-Dcatalina.base/, @t) {
my $tid = shift @tt;
my @ttid = split /\//, $tid;
my $id = pop @ttid;
if ( `sudo /usr/java/jdk1.6/bin/jinfo $pid 2>&1 ` =~ /java.specification.version/ ) {
$res{"$id"}{"version"} = "jdk1.6";
$res{"$id"}{"pid"} = $pid;
} elsif ( `sudo /usr/java/jdk1.5/bin/jinfo $pid 2>&1 ` =~ /java.specification.version/ ) {
$res{"$id"}{"version"} = "jdk1.5";
$res{"$id"}{"pid"} = $pid;
} else { print "Cannot identify jvm version for $pid\n"; }
}
}
foreach my $id (keys %res) {
my $pid = $res{"$id"}{"pid"};
my $bin = $res{"$id"}{"version"};
$str = `sudo /usr/java/$bin/bin/jstat -gccapacity -t $pid 1 1 | tail -n 1`;
($t,$timestamp,$NGCMN,$NGCMX,$NGC,$S0C,$S1C,$EC,$OGCMN,$OGCMX,$OGC,$OC,$PGCMN,$PGCMX,$PGC,$PC,$YGC,$FGC) = split /\s+/, $str;
$str = `sudo /usr/java/$bin/bin/jstat -gc -t $pid 1 1 | tail -n 1`;
($t,$timestamp,$s0c,$s1c,$s0u,$s1u,$ec,$eu,$oc,$ou,$pc,$pused,$ygc,$ygct,$fgc,$fgct,$gct) = split /\s+/, $str;
my $perused = sprintf ("%.2f",( $pused * 100 ) / $PGCMX );
if ($perused > 99) {
$color = "red";
} elsif ( $perused > 98 ) {
$color = "yellow";
} else { $color = "green"; }
$res{"$id"}{"gccap"}{"label"} = "Gc Capacity";
$res{"$id"}{"gccap"}{"value"} = $perused;
$res{"$id"}{"gccap"}{"alert"} = 0;
$str = `sudo /usr/java/$bin/bin/jstat -gc -t $pid 1 1 | tail -n 1`;
($t,$timestamp,$s0c,$s1c,$s0u,$s1u,$ec,$eu,$oc,$ou,$pc,$pu,$ygc,$ygct,$fgc,$fgct,$gct) = split /\s+/, $str;
$s0c *= 1024; $s1c *= 1024; $s0u *= 1024; $s1u *= 1024; $ec *= 1024; $eu *= 1024; $oc *= 1024; $ou *= 1024;
$pc *= 1024; $pu *= 1024;
$res{"$id"}{"gccap"}{"trends"} = "[jvm.$id.rrd]
DS:SOC:GAUGE:600:0:U $s0c
DS:S1C:GAUGE:600:0:U $s1c
DS:S0U:GAUGE:600:0:U $s0u
DS:S1U:GAUGE:600:0:U $s1u
DS:EC:GAUGE:600:0:U $ec
DS:EU:GAUGE:600:0:U $eu
DS:OC:GAUGE:600:0:U $oc
DS:OU:GAUGE:600:0:U $ou
DS:PC:GAUGE:600:0:U $pc
DS:PU:GAUGE:600:0:U $pu
DS:GCAP:GAUGE:600:0:U $perused";
}
$line = "Gc Capacity and VM memory reporting\n";
#$line = "$ENV{'BB'} $ENV{'BBDISP'} \"status $ENV{'MACHINE'}.jvm";
$mcolor = 0;
@colors = ("green","yellow","red");
$trends = "";
foreach my $id (sort keys %res) {
$line .= "&". at colors[$res{"$id"}{"gccap"}{"alert"}]." $id : ".$res{"$id"}{"gccap"}{"value"}."%\n";
if ( $res{"$id"}{"gccap"}{"alert"} > $mcolor ) {
$mcolor = $res{"$id"}{"gccap"}{"alert"} ;
}
$trends .= $res{"$id"}{"gccap"}{"trends"}."\n";
}
$date = `/bin/date`;
$color = $colors[$mcolor];
$line .= "\n<!-- linecount=6 -->";
system qq/ $ENV{'BB'} $ENV{'BBDISP'} "status $ENV{'MACHINE'}.jvm $color $line $date <!-- linecount=6 --> "/;
system qq/ $ENV{'BB'} $ENV{'BBDISP'} "data $ENV{'MACHINE'}.trends\n$trends"/;
graphs.cfg
[jvm]
FNPATTERN jvm.(.*).rrd
TITLE Garbage Collection
YAXIS Bytes
DEF:EC at RRDIDX@=@RRDFN@:EC:AVERAGE
DEF:EU at RRDIDX@=@RRDFN@:EU:AVERAGE
DEF:OC at RRDIDX@=@RRDFN@:OC:AVERAGE
DEF:OU at RRDIDX@=@RRDFN@:OU:AVERAGE
DEF:PC at RRDIDX@=@RRDFN@:PC:AVERAGE
DEF:PU at RRDIDX@=@RRDFN@:PU:AVERAGE
DEF:GCAP at RRDIDX@=@RRDFN@:GCAP:AVERAGE
LINE1:EC at RRDIDX@#@COLOR@:@RRDPARAM@ Current eden space capacity.\n
GPRINT:EC at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:EC at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:EC at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:EC at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:EU at RRDIDX@#@COLOR@:@RRDPARAM@ Eden space utilization.\n
GPRINT:EU at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:EU at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:EU at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:EU at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:OC at RRDIDX@#@COLOR@:@RRDPARAM@ Current old space capacity.\n
GPRINT:OC at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:OC at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:OC at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:OC at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:OU at RRDIDX@#@COLOR@:@RRDPARAM@ Old space utilization.\n
GPRINT:OU at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:OU at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:OU at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:OU at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:PC at RRDIDX@#@COLOR@:@RRDPARAM@ Current permanent space capacity.\n
GPRINT:PC at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:PC at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:PC at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:PC at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:PU at RRDIDX@#@COLOR@:@RRDPARAM@ Permanent space utilization.\n
GPRINT:PU at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:PU at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:PU at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:PU at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
cheers
nico
Le 12 sept. 2012 à 17:25, Scot Kreienkamp a écrit :
> I didn't think it was really appropriate to post because it's heavily customized for my different environments and such. You'll have to do some work before it's suitable for yours.
>
> Basically I pull a list of queues using curl then loop back through each queue with curl to grab the queue stats. It sends those in a status message then sends the stats in a second data message so they can be graphed. It will also clear any queue with DLQ in the name once it gets above 25. Otherwise the DLQ's never clear. I started doing that because the DLQ's got so big they brought the server to almost a standstill after several months of nobody paying attention to it.
>
> The entries in your hosts file are like: jbossjms:8080
> (jbossjms being the name of the test, and 8080 being the port the jmx console is on. This assumes no auth on the jmx web console, which there isn't by default.)
>
> Graphs.cfg entry:
> [jboss]
> FNPATTERN ^jboss.(.+).rrd
> TITLE Queue sizes
> YAXIS Queue size
> DEF:p at RRDIDX@=@RRDFN@:lambda:MAX
> LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@
> GPRINT:p at RRDIDX@:LAST: \: %5.0lf (cur)
> GPRINT:p at RRDIDX@:MAX: \: %5.0lf (max)
> GPRINT:p at RRDIDX@:MIN: \: %5.0lf (min)
> GPRINT:p at RRDIDX@:AVERAGE: \: %5.0lf (avg)\n
>
> I have mine setup to graph the max value, you probably need to change that to average.
>
> You need to use splitncv in the xymonserver.cfg also. Otherwise anytime you add or remove a queue the graph won't show up or the RRD updater will error out as the new data point won't exist in the RRD.
>
> Works very well for me. Here it is, hope this helps.
>
>
> #!/bin/bash
>
>
> function GETSTATS ()
> {
> COLOR=green
> RETURNQUEUEMESSAGE="Queue_Name Message_Count Consumer_Count"
> RETURNQUEUEMONITOR=""
> ENVIRONMENTPREFIX=`echo $1 | cut -c 1-4`
> RETURNMESSAGE=""
> CONSUMERCOUNTER=""
>
> case $1 in
> retv3040.na.lzb.hq)
> ENVIRONMENTPREFIX=train
> ;;
> retv3041.na.lzb.hq)
> ENVIRONMENTPREFIX=train
> ;;
> retv3042.na.lzb.hq)
> ENVIRONMENTPREFIX=train
> ;;
> retv3043.na.lzb.hq)
> ENVIRONMENTPREFIX=train
> ;;
> retv3044.na.lzb.hq)
> ENVIRONMENTPREFIX=train
> ;;
> retv3045.na.lzb.hq)
> ENVIRONMENTPREFIX=train
> ;;
> esac
>
> while read QUEUENAME ; do
> curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.messaging.destination%3Aservice%3DQueue%2Cname%3D$QUEUENAME" > /tmp/$$
> MESSAGECOUNT=`cat /tmp/$$ |grep -A 3 "The number of messages in the queue"|tail -1 | awk '{print $1}'`
>
>
> #setup some environment variables for prod VS dev and for specific queues
> if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then
> case $QUEUENAME in
> PricingUploadResponseQueue)
> MSGCOUNTRED=5 #the number of messages in the queue to turn the test red
> MSGCOUNTYELLOW=3 # the number of messages in the queue to turn the test yellow
> MINCONSUMERCOUNT=12 # the expected number of consumers for the PricingUploadResponseQueue and UnMeteredReportResponseQueue
> ;;
> *)
> MSGCOUNTRED=500
> MSGCOUNTYELLOW=250
> MINCONSUMERCOUNT=0
> ;;
> esac
> else
> case $QUEUENAME in
> PricingUploadResponseQueue)
> MSGCOUNTRED=10
> MSGCOUNTYELLOW=5
> MINCONSUMERCOUNT=0
> ;;
> *)
> MSGCOUNTRED=500
> MSGCOUNTYELLOW=250
> MINCONSUMERCOUNT=0
> ;;
> esac
> fi
>
>
> if [ "$MESSAGECOUNT" -gt "$MSGCOUNTRED" ] ; then
> COLOR=red
> RETURNSTATUS="Not OK"
> RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is very high!"
> elif [ "$MESSAGECOUNT" -gt "$MSGCOUNTYELLOW" ] ; then
> RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is higher than normal"
> RETURNSTATUS="Caution"
> if [ ! "$COLOR" = "red" ] ; then
> COLOR=yellow
> fi
> fi
> CONSUMERCOUNT=`cat /tmp/$$ |grep -A 3 "The number of consumers on the queue"|tail -1 | awk '{print $1}'`
> if [ "$CONSUMERCOUNT" -lt "$MINCONSUMERCOUNT" ] ; then
> COLOR=yellow
> RETURNSTATUS="Not OK"
> RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME has less than expected $MINCONSUMERCOUNT queue consumers!"
> fi
> RETURNQUEUEMESSAGE="$RETURNQUEUEMESSAGE\n$QUEUENAME $MESSAGECOUNT $CONSUMERCOUNT"
> RETURNQUEUEMONITOR="$RETURNQUEUEMONITOR\n$QUEUENAME : $MESSAGECOUNT"
> if [ "$MESSAGECOUNT" -gt "25" ] ; then
> if echo $QUEUENAME | grep "DLQ" ; then
> curl -m 10 -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=5&name=jboss.messaging.destination%3Aservice%3DQueue%2Cname%3D$QUEUENAME" >/dev/null 2>&1
> RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME was cleared"
> COLOR=yellow
> fi
> fi
>
> done < <(curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=displayMBeans&filter=jboss.messaging.destination" |grep -o ">name=.*,"|sed -e 's/>name=//g' -e 's/,$//g')
>
>
> #set how many queue consumers we should expect
> if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then
> MINQUEUECONSUMERSCOUNT=7 # the minimum number of queue consumers registered with JBoss before the test turns red for prod
> else
> MINQUEUECONSUMERSCOUNT=1 # the minimum number of queue consumers registered with JBoss before the test turns red for dev
> fi
>
> #get list of registered queue consumers
> LISTEDCONSUMERS="Queue Consumers:<table>"
> while read CONSUMERIPADDRESS ; do
> let CONSUMERCOUNTER+=1
> HOSTNAME=`host $CONSUMERIPADDRESS | awk '{print $5}'`
> LISTEDCONSUMERS="$LISTEDCONSUMERS<tr><td>$HOSTNAME</td><td>$CONSUMERIPADDRESS</td></tr>"
> done < <(curl -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=19&name=jboss.messaging%3Aservice%3DServerPeer" |grep '<td>10' |sed -e 's/<td>//g' -e 's/<\/td>//g' |sort |uniq)
> LISTEDCONSUMERS="$LISTEDCONSUMERS </table>"
> if [ "$CONSUMERCOUNTER" -lt "$MINQUEUECONSUMERSCOUNT" ] ; then
> RETURNMESSAGE="$RETURNMESSAGE\nSome queue consumers are missing!"
> COLOR=red
> elif [ "$CONSUMERCOUNTER" -eq "$MINQUEUECONSUMERSCOUNT" ] ; then
> LISTEDCONSUMERS=""
> fi
>
> RETURNMESSAGE="$RETURNMESSAGE\nExpected $MINQUEUECONSUMERSCOUNT, found $CONSUMERCOUNTER queue consumers currently registered."
>
> if [ "$RETURNQUEUEMONITOR" = "" ] ; then
> RETURNQUEUEMESSAGE="JBoss is down or unreachable!!!"
> COLOR=red
> fi
>
> if [ "$RETURNSTATUS" = "" ] ; then
> RETURNSTATUS="OK"
> fi
>
> if [ "$RETURNMESSAGE" = "" ] ; then
> RETURNMESSAGE="OK"
> fi
>
>
> /home/hobbit/client/bin/xymon retv6100.na.lzb.hq "status $1.jboss $COLOR `date` $RETURNSTATUS
>
> `echo -e "Status message:\n $RETURNMESSAGE"`
>
> $LISTEDCONSUMERS
>
>
> Queue List:
> `echo -e $RETURNQUEUEMESSAGE|column -t`
>
> "
>
> /home/hobbit/client/bin/xymon retv6100.na.lzb.hq "data $1.jboss green `date` OK
>
> `echo -e $RETURNQUEUEMONITOR|sed -e 1d -e 's/\(.*\)/\U\1/' -e 's/QUEUE//g' -e 's/ //g' -e 's/INVENTORY/INVTRY/g' -e 's/RETAIL/RETL/g' -e 's/PRICE/PRC/g' -e 's/RESPONSE/RESPNS/g' -e 's/STATUS/STS/g' -e 's/FREIGHT/FRGT/g' -e 's/REQUEST/REQST/g' | awk -F ":" '{print substr($1,1,18)" : "$2}'`
>
> "
> rm -f /tmp/$$
> }
>
>
> while read HOSTLINE ; do
> SERVERNAME=`echo $HOSTLINE | awk '{print $2}'`
> SERVERPORT=`echo $HOSTLINE | awk -F ":" '{print $2}'`
> GETSTATS $SERVERNAME $SERVERPORT
> done < <(/home/hobbit/server/bin/xymongrep jbossjms:*)
>
>
>
> Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
> 1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com | www.la-z-boy.com
>
> From: Larry Barber [mailto:lebarber at gmail.com]
> Sent: Wednesday, September 12, 2012 10:59 AM
> To: Scot Kreienkamp
> Cc: xymon at xymon.com
> Subject: Re: [Xymon] jboss monitorin
>
> Could you send me a copy, or post it on Xymonton?
>
> Thanks,
> Larry Barber
>
> On Wed, Sep 12, 2012 at 9:50 AM, Scot Kreienkamp <SKreien at la-z-boy.com> wrote:
> I wrote a jboss client that pulls message and consumer counts from the jmx web console. That way I can tell if it's up, how full the queues are, and whether or not consumers are registered. That's about all the useful info I could find in the web console.
>
> Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
> 1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com | www.la-z-boy.com
>
> From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Larry Barber
> Sent: Wednesday, September 12, 2012 10:43 AM
> To: xymon at xymon.com
> Subject: [Xymon] jboss monitorin
>
> Has anybody found a good way to monitor jboss servers? I have a bunch of jboss installations and need some way to monitor their performance. I checked Xymonton but couldn't find anything useful (the jmxstat project appears to be dead, at least the link leading to the download is dead). Any help would be much appreciated.
>
> Thanks,
> Larry Barber
>
>
>
> This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
>
>
>
>
> This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you._______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20120912/6ef35573/attachment.html>
More information about the Xymon
mailing list