<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>So I have a setup where, in total, I monitor about 300,000 hosts.</div>
<div><br>
</div>
<div>It's regionalized though, with the largest site having roughly 34,000 hosts</div>
<div><br>
</div>
<div>The xymon daemon runs on a VM, with 16 GB of Ram, and 4x2.5 Ghz xeon processors. The virtual NIC is a 1000Mb </div>
<div><br>
</div>
<div>It currently handles 481 msgs/sec (the hosts.cfg is 1.2 Mb)</div>
<div><br>
</div>
<div>Here are some ideas:</div>
<div><br>
</div>
<div><br>
</div>
<div>(1) Obviously split pollers out into separate machines</div>
<div>(2) Change the history daemon (Henrick as some sample daemons in the source code) - I changed it to write stachg and clichg and such are written out to mysql instead of files, and the textblog is compressed. This helped with managing history</div>
<div>(3) Use JC Cleaver's patches <a href="http://terabithia.org/rpms/xymon/">http://terabithia.org/rpms/xymon/</a> -- he has several memory leak fixes, tuning optimizations. I would just say that if you use clientupdate, you will have to patch his things
a little bit</div>
<div>(4) An interesting thing you can do, if you have the means via a load balancer is --></div>
<div> (a) I have 10 pollers for this instance. All of them run xymonproxy on localhost (stuff they poll reports to localhost and gets forwarded up)</div>
<div> (b) Using a load balancer for they xymon IP address, it inspects the first 12 characters of a xymon message – if it's something that can be combo'd in proxy it passes to a poller in the pool, otherwise, it sends it to the real xymon IP</div>
<div>(5) JC has some nanomsg options for channels, and myself, I use AMQP for channels to communicate data outbound. Both of these help if you use xymondata outside of the stock app</div>
<div><br>
</div>
<div>For your setup, rather than running more xymond's on different ports, setup xymonproxy's on different machines, and point your clients to them. Then you won't have to migrate your history files or anything else. But I would look at changing your history
setup, as several million files gets very unwieldy to manage, but millions of rows in databases are much easer to do.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div><br>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"<a href="mailto:fmaillard.ext@orange.com">fmaillard.ext@orange.com</a>" <<a href="mailto:fmaillard.ext@orange.com">fmaillard.ext@orange.com</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, December 17, 2014 at 11:04 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:xymon@xymon.com">xymon@xymon.com</a>" <<a href="mailto:xymon@xymon.com">xymon@xymon.com</a>><br>
<span style="font-weight:bold">Subject: </span>[Xymon] xymon on a large architecture<br>
</div>
<div><br>
</div>
<div xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Arial","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1977366681;
mso-list-type:hybrid;
mso-list-template-ids:678178648 67895297 67895299 67895301 67895297 67895299 67895301 67895297 67895299 67895301;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 10pt; font-family: Arial, sans-serif;">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">We’re running a quite large xymon setup, and have been dealing with performance issue for quite a while. Here are some stats to give an idea about the setup:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">- We have 2 xymon servers per datacenter, on 3 datacenter (all messages are sent to both servers for a given site)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">- Each xymon server receives on average between 200msg/s and 250msg/s. We’re getting peaks at 400msg/sec.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">- Each site hosts about 3000 hosts / 30 000 services<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">We’ve been suspecting for a long time that we might be losing messages… and I think I finally tracked it down to xymond not fetching the messages quickly enough
so that the kernel’s buffer fill up and messages get discarded (by the kernel). On one of our servers, even though I have already increased net.ipv4.tcp_rmem and net.ipv4.tcp_wmem I got the following output from “netstat -s”:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">148909 packets pruned from receive queue because of socket buffer overrun<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">4453143 packets collapsed in receive queue due to low socket buffer<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">And here I come to the question I’m having:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">1/ Is 250msg/s too much for a single xymond instance? Is anyone running instances with a lot more traffic than that?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">2/ I’m starting to look into running several instances of xymond on the same machine, by binding them to different ports. Another option is to set up new machines,
but that would mean migrating history files (several million files), sorting out the firewalling issues (our xymon interfaces are deeply connected to our information system) so I’d rather like avoiding this option. Are there any guidelines on how to do this?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">3/ Are there any settings and best practice that could improve performance? For instance, should we move to a massive use of combo statuses in order to lessen the
number of messages received?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif;"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size: 10pt; font-family: Arial, sans-serif; color: black;">Francois Maillard</span></b><span lang="EN-US" style="font-size: 10pt; font-family: Helvetica, sans-serif; color: black;"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size: 10pt; font-family: Helvetica, sans-serif; color: black;"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black;">Pilote des plateformes Supervision, DNS & FTP - Sysadmin Infrastructure<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black;">Altran Méditerranée<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black;">pour
</span><span style="font-size: 10pt; font-family: Arial, sans-serif;">Orange/OF/DTSI/DSI/DFY/HBX</span><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black;"><br>
</span><span style="font-size: 10pt; font-family: Helvetica, sans-serif; color: black;">Sophia Antipolis<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10pt; font-family: Helvetica, sans-serif; color: black;">tél. 04 97 12 87 53<br>
<a href="mailto:fmaillard.ext@orange.com"><span style="color:#FF6600">fmaillard.ext@orange.com</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<pre>_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
</pre>
</div>
</div>
</span><br>
<hr>
<font face="Arial" color="Gray" size="1">This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely
for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to
this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.<br>
</font>
</body>
</html>