<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
p
{mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
p.arial, li.arial, div.arial
{mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman";}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:971398127;
mso-list-type:hybrid;
mso-list-template-ids:1707614656 1663834074 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-number-format:alpha-upper;
mso-level-text:"%1\)";
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Sorry if this post gets a bit long, I’ve read thru the
man pages and the archives and have failed to reach understanding and need some
help.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I have over 4300 Hobbit 4.2.0+all-patches clients currently
reporting to one server running Xymon 4.3.0-0.beta2. This server is also
used for other workloads, which normally do not produce much of a load but can
at times. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>We are adding more clients to this configuration at rate of
about 400/week for a final total of about 7200 clients.<o:p></o:p></span></font></p>
<pre><font size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>As of this week it seems that my Xymon server is unable to
keep up with the load and I suspect this is due to several reasons: the large number
of clients, file system type (ext3) used for the data/.. directories (, the
problem is.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I am trying to determine my best course to be able to handle
the 7200+ clients … <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Some options I’ve considered:<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>A)<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>simply split
the clients over 2 or more independent Xymon servers<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>a.<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>easiest to
configure<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>b.<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>lose ‘single
web page’ overview<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>c.<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>lose
combined statistics/reporting<br>
<br>
<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>B)<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Split
clients over multiple servers for data gathering/storage and (how?) use one
server to display a bb2.html page which combines all non green from all ‘data-gathering’
servers<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>a.<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Is this even
possible?<br>
<br>
<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>C)<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>Some other
method/configuration? Anyone on the list running this many clients? <o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>a.<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>How did you
set up your environment?<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1'><![if !supportLists]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><span
style='mso-list:Ignore'>b.<font size=1 face="Times New Roman"><span
style='font:7.0pt "Times New Roman"'> </span></font></span></span></font><![endif]><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'>What size/performance/type
of system are you using for your Xymon server?<o:p></o:p></span></font></p>
<p class=MsoNormal style='margin-left:.75in'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Some of my current symptoms:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The ‘top’ command on the server is indicating
high I/O load (at times %iowait > 60%).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The ‘procs’ column warns/alerts about missing
processes; closer examination shows that the client data received has been
truncated, usually somewhere in the ‘ps’ outout section.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The bb2.html page sometimes shows many ‘purple’
clients; the clients in purple change (eg, a client goes purple for a while 5-30+
minutes, then we get another message processed and the client goes green again.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>In /var/log/xymon/clientdata.log, I am seeing many (2212
yesterday alone) messages like:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 11:38:09 hobbitd_client: Got message
55294, expected 55277<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 12:08:12 Flushed 7 stale messages for
0.0.0.0:0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 12:08:13 Flushed 43 stale messages for
0.0.0.0:0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 12:08:13 hobbitd_client: Got message
81190, expected 81140<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 12:38:36 Flushed 16 stale messages for
0.0.0.0:0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 12:38:37 Flushed 43 stale messages for
0.0.0.0:0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 12:38:38 Flushed 26 stale messages for
0.0.0.0:0<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face="Courier New"><span style='font-size:10.0pt;
font-family:"Courier New"'>2010-05-27 12:38:38 hobbitd_client: Got message
107484, expected 107399<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The hobbitd page shows:<o:p></o:p></span></font></p>
<pre><font size=2 face="Courier New"><span style='font-size:10.0pt'>Statistics for Hobbit daemon<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>Up since 26-May-2010 14:55:38 (0 days, 22:00:02)<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'><o:p> </o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>Incoming messages : 16721697<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>- status : 11146704<o:p></o:p></span></font></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>- combo : 1122112<o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<pre><font size=2 face="Courier New"><span style='font-size:10.0pt'>Incoming messages/sec : 216 (average last 300 seconds)<o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The bbtest is taking 225 seconds to complete <o:p></o:p></span></font></p>
<pre><st1:place w:st="on"><font size=2 face="Courier New"><span
style='font-size:10.0pt'>PING</span></font></st1:place> test completed (4390 hosts) 5005592.360761 203.397989 <o:p></o:p></pre><pre><font
size=2 face="Courier New"><span style='font-size:10.0pt'>TIME TOTAL 225.391564 <o:p></o:p></span></font></pre>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hoping the group-mind can help me out,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks, <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Tom<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> <o:p></o:p></span></font></p>
<p><b><i><font size=1 color=gray face=Arial><span style='font-size:7.5pt;
font-family:Arial;color:gray;font-weight:bold;font-style:italic'>Thomas Brand </span></font></i></b><b><i><font
size=1 color=blue face=Arial><span style='font-size:7.5pt;font-family:Arial;
color:blue;font-weight:bold;font-style:italic'><o:p></o:p></span></font></i></b></p>
<p class=MsoNormal><i><font size=1 color="#999999" face=Arial><span
style='font-size:7.5pt;font-family:Arial;color:#999999;font-style:italic'>Disclaimer:
1) all opinions are my own, 2) I may be completely wrong, 3) my advice is worth
at least as much as what you are paying for it, or your money cheerfully
refunded.<o:p></o:p></span></font></i></p>
<p class=arial><font size=1 color="#999999" face=Arial><span style='font-size:
7.5pt;font-family:Arial;color:#999999'>CONFIDENTIALITY NOTICE: This
communication and any attachments may contain confidential and/or privileged
information for</span></font><font size=1 color=gray face=Arial><span
style='font-size:7.5pt;font-family:Arial;color:gray'> the use of the designated
recipients named above. If you are not the intended recipient, you are
hereby notified that you have received this communication in error and that any
review, disclosure, dissemination, distribution or copying of it or its
contents is prohibited. If you have received this communication in error,
please notify the sender immediately by telephone and destroy all copies of
this communication and any attachments.</span></font><font size=1 color=gray
face=Arial><span style='font-size:7.5pt;font-family:Arial;color:gray'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>