<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Jeremy,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I got 7, one every 5 minutes.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ALERTREPEAT is set to 30 in hobbitserver.cfg.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Our hobbit-alerts.cfg file has “DURATION>1m REPEAT=5m” for the msgs test for that machine.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">As far as I could tell, the messages status is yellow and it is staying yellow, not flapping. When I click on history in the GUI, it shows that it was yellow
for 35 minutes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It looks like it’s the same message that we keep getting an alert for. We had an incident on Friday, where we got 7 email alerts. Below are examples of the
portion of the email that showed the yellow alert. The timestamp in the log is 21:00:16 for all of the alerts, so it’s the same message. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Email alert 1:<o:p></o:p></span></p>
<p class="MsoPlainText">yellow System logs at Fri Oct 31 21:01:10 EDT 2014 <pre> </pre><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">&yellow Warnings in <a href="/xymon-cgi/bb-hostsvc.sh?CLIENT=bbgroupa-web4.syr.edu&SECTION=msgs:/usr/local/blackboard/logs/tomcat/activemq.txt">/usr/local/blackboard/logs/tomcat/activemq.txt</a><o:p></o:p></p>
<p class="MsoPlainText"><pre><o:p></o:p></p>
<p class="MsoPlainText">&yellow WARN 2014-10-31 21:00:16,480 ActiveMQ NIO Worker 30057 org.apache.activemq.broker.TransportConnection.Transport - Transport Connection to: tcp://128.230.126.194:49464 failed: java.io.EOFException </pre><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Email alert 2<o:p></o:p></span></p>
<p class="MsoPlainText">yellow System logs at Fri Oct 31 21:06:10 EDT 2014 <pre> </pre><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">&yellow Warnings in <a href="/xymon-cgi/bb-hostsvc.sh?CLIENT=bbgroupa-web4.syr.edu&SECTION=msgs:/usr/local/blackboard/logs/tomcat/activemq.txt">/usr/local/blackboard/logs/tomcat/activemq.txt</a><o:p></o:p></p>
<p class="MsoPlainText"><pre><o:p></o:p></p>
<p class="MsoPlainText">&yellow WARN 2014-10-31 21:00:16,480 ActiveMQ NIO Worker 30057 org.apache.activemq.broker.TransportConnection.Transport - Transport Connection to: tcp://128.230.126.194:49464 failed: java.io.EOFException </pre><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Email alert 7<o:p></o:p></span></p>
<p class="MsoPlainText">yellow System logs at Fri Oct 31 21:31:11 EDT 2014 <pre> </pre><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">&yellow Warnings in <a href="/xymon-cgi/bb-hostsvc.sh?CLIENT=bbgroupa-web4.syr.edu&SECTION=msgs:/usr/local/blackboard/logs/tomcat/activemq.txt">/usr/local/blackboard/logs/tomcat/activemq.txt</a><o:p></o:p></p>
<p class="MsoPlainText"><pre><o:p></o:p></p>
<p class="MsoPlainText">&yellow WARN 2014-10-31 21:00:16,480 ActiveMQ NIO Worker 30057 org.apache.activemq.broker.TransportConnection.Transport - Transport Connection to: tcp://128.230.126.194:49464 failed: java.io.EOFException </pre><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The Hobbit acknowledge code that appears in the subject of the emails is all the same code. Maybe we are getting multiple email messages because we did not acknowledge
the alert. But, if the string does not appear again in the file in the next cycle, shouldn’t it turn back to green?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">When it happens again, I will try to look at the “client data available” link .<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I hope this helps.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Nicole<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Jeremy Laidman [mailto:jlaidman@rebel-it.com.au]
<br>
<b>Sent:</b> Tuesday, October 28, 2014 9:37 PM<br>
<b>To:</b> Nicole Beck<br>
<b>Cc:</b> Bill Arlofski; xymon@xymon.com<br>
<b>Subject:</b> Re: [Xymon] duration of MSG red status<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">Nicole<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On 29 October 2014 05:16, Nicole Beck <<a href="mailto:nskyrca@syr.edu" target="_blank">nskyrca@syr.edu</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">What I’m seeing is that I get an alert for my trigger string (which has a timestamp on it), and then
I keep getting alerts for the same trigger string (with the same timestamp) for the next 30 minutes.
</span><o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">How often do you get the repeated alerts? Or how many in that 30 minutes?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I’m not sure if anything else was append to the log file in that 30 minutes. I stop getting the alerts
after 30 minutes and don’t have to wait until the log is rotated for the alert to clear. </span><o:p></o:p></p>
</blockquote>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Do you have ALERTREPEAT defined in xymonserver.cfg? The default is 30 seconds, but you may have it less than that.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Similarly, do you have "REPEAT" defined in alerts.cfg for the rule matching these alerts? (The "REPEAT" value in alerts.cfg defaults to the setting of ALERTREPEAT.)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal">Is your message status (red?) staying non-green for the 30 minutes, or non-green for only a short time, or flapping like red/green/red/green?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The way messages get to Xymon are via the client data. So during an "event" you can click on the "Client data available" link at the bottom of your "msgs" page for the host, and it should show you all of the client data, and you can search
for the logfilename to see what log lines the client sent to the server. Or you can click on the logfile name on the "msgs" page for a modified client data report showing just the log lines for that logfile.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What I'm trying to understand is whether you are getting the same messages sent multiple times from the client causing multiple events, or whether the one event is generating multiple alerts.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From what I can tell, a red "msgs" status will stay red for only one 5-minute client cycle. The next time the client sends its client data report, if the logfile in question has no new matching lines, it will actively generate a green
status.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">J<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>