<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:11pt;">
<div>Hi, All …</div>
<div> </div>
<div>The other day, our Xymon (4.3.3) started sending out notifications due to flapping on various hosts, various network-based tests which lasted for a rather sharply-defined period. It caused a fair bit of angst and I was on the hot-seat to prove Xymon was
functioning properly.</div>
<div> </div>
<div>Here are some of the summary facts:</div>
<div> </div>
<ul style="margin:0;padding-left:20.25pt;">
<li>The flapping is pretty well documented in Xymon as occurring due to connection times exceeding our 10-second threshold – most of the , as configured in tasks.cfg<br>
<br>
CMD xymonnet --report --ping --checkresponse \</li></ul>
<div style="padding-left:2.25pt;"> --timeout=10 --dns-timeout=2 \</div>
<div style="padding-left:2.25pt;"> --dnslog=/var/log/xymon-4.3.3/dns.log \</div>
<div style="padding-left:2.25pt;"> --concurrency=5</div>
<div style="text-indent:15.75pt;padding-left:20.25pt;">INTERVAL 3m</div>
<div style="text-indent:15.75pt;padding-left:20.25pt;"> </div>
<ul style="margin:0;padding-left:20.25pt;">
<li>Output from the “xymonnet” report (currently – not captured during the “storm”) shows:</li></ul>
<div> </div>
<div style="padding-left:72pt;"><font face="Courier New">xymonnet version 4.3.3</font></div>
<div style="padding-left:72pt;"><font face="Courier New">SSL library : OpenSSL 0.9.8l 5 Nov 2009</font></div>
<div style="padding-left:72pt;"><font face="Courier New">LDAP library: OpenLDAP 20416</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> </font></div>
<div style="padding-left:72pt;"><font face="Courier New">Statistics:</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> Hosts total : 2081</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> Hosts with no tests : 2</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> Total test count : 2864</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> Status messages : 2856</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> Alert status msgs : 0</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> Transmissions : 30</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> </font></div>
<div style="padding-left:72pt;"><font face="Courier New">DNS statistics:</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # hostnames resolved : 3337</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # succesful : 921</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # failed : 1266</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # calls to dnsresolve : 2850</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> </font></div>
<div style="padding-left:72pt;"><font face="Courier New">TCP test statistics:</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # TCP tests total : 1769</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # HTTP tests : 1244</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # Simple TCP tests : 525</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # Connection attempts : 1767</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # bytes written : 235845</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> # bytes read : 2514747</font></div>
<div style="padding-left:72pt;"><font face="Courier New"> </font></div>
<div style="padding-left:72pt;"><font face="Courier New"> </font></div>
<div style="padding-left:72pt;"><font face="Courier New">TIME SPENT</font></div>
<div style="padding-left:72pt;"><font face="Courier New">Event Start time Duration</font></div>
<div style="padding-left:72pt;"><font face="Courier New">xymonnet startup 1040654.310651 -</font></div>
<div style="padding-left:72pt;"><font face="Courier New">Service definitions loaded 1040654.319152 0.008501 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">Tests loaded 1040655.696733 1.377581 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">DNS lookups completed 1040656.213268 0.516534 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">Test engine setup completed 1040657.416739 1.203470 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">TCP tests completed 1040675.444183 18.027443 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">PING test completed (923 hosts) 1040699.991467 24.547283 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">PING test results sent 1040700.080247 0.088780 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">Test result collection completed 1040700.144033 0.063785 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">LDAP test engine setup completed 1040700.152852 0.008819 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">LDAP tests executed 1040700.360821 0.207968 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">LDAP tests result collection completed 1040700.360829 0.000007 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">DNS tests executed 1040700.441820 0.080991 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">NTP tests executed 1040722.413523 21.971702 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">Test results transmitted 1040723.295458 0.881935 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">xymonnet completed 1040723.313935 0.018476 </font></div>
<div style="padding-left:72pt;"><font face="Courier New">TIME TOTAL 69.003284<br>
</font></div>
<ul style="margin:0;padding-left:20.25pt;">
<li>Rather sharply defined start-up / cut-off for the “storm”: I can point to the 5-minute segment when it started / stopped</li><li>The Xymon server OS/NIC hardware check out diagnostically</li><li>According to our network team’s records, the network connection bandwidth utilization coming in / out of the Xymon server was < 1% capacity (i.e. we have lots of bandwidth)</li><li>According to our network team there were no significant loss of packets or congestion at the switch level (there’s only one hop between the Xymon server and the rest of the hosts)</li><li>The types of services affected seemed pretty random: mostly HTTP tests, but LOTs of SSH/ping/NTP/LDAP, etc. as well.</li></ul>
<div> </div>
<div>Any initial thoughts?</div>
<div> </div>
<div>Thanks!</div>
<div> </div>
<div>david</div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Arial" size="2"><span style="font-size:10pt;">~~~~~~~~~~~~~~~~~~~<font face="Times New Roman" size="3"><span style="font-size:12pt;">
<br>
</span></font><font face="Times New Roman" size="3" color="blue"><span style="font-size:12pt;"><b>David Mills</b></span></font><font face="Times New Roman" size="3"><span style="font-size:12pt;">
<br>
</span></font>Systems Administrator<font face="Times New Roman" size="3"><span style="font-size:12pt;">
<br>
</span></font><b><i>Northrop Grumman</i></b> </span></font></div>
<div><font face="Arial" size="2"><span style="font-size:10pt;">512-595-1238<font face="Calibri" size="2"><span style="font-size:11pt;">
<br>
</span></font>david.mills@ngc.com<font face="Calibri" size="2"><span style="font-size:11pt;"> </span></font></span></font></div>
<div> </div>
<div> </div>
<div> </div>
</span></font>
</body>
</html>