<font size=2 face="sans-serif">Hello,</font>
<br>
<br><font size=2 face="sans-serif">Two little questions on big environments
splitted with xymon proxies (to balance the flow of data to the xymon server):</font>
<br>
<br><font size=2 face="sans-serif">- How many clients do you advise
per xymonproxy ?</font>
<br><font size=2 face="sans-serif">- Do you advise a different configuration
on the xymonproxy according the type of data collected (xymon agents, bbwin
agents, status messages, data messages etc.)</font>
<br>
<br>
<br><font size=2 face="sans-serif">Cordialement, Regards,Mit freundlichen
Grüßen,<br>
<br>
Gautier BEGIN<br>
<br>
</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif"><fmaillard.ext@orange.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">"xymon@xymon.com"
<xymon@xymon.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">12/18/2014 06:24 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: [Xymon]
xymon on a large architecture</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">"Xymon"
<xymon-bounces@xymon.com></font>
<br>
<hr noshade>
<br>
<br>
<br><font size=2 face="Arial">Thanks for the answer, I’m happily surprised
to see that we’re still far from xymon’s limit.</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">I made more checks today, and realized that
some tools that I thought were sending their statuses over a few seconds
actually send batches of ~200 statuses in 200ms. These burst are definitely
far higher than 200msg/s… I had no idea that our traffic was so irregular.</font>
<br><font size=2 face="Arial">I guess that, as you pointed out, the next
step is to setup xymonproxy and have the few tools that generate so much
traffic use them. As the only thing they know about xymon is the status
command, that should be quite easy to set up.</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Regarding the history files, it is not an
issue as far as performance is concerned, but it definitely is a pain to
manage. Moving to database solution has been in our todo list for some
time now.</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Regards,</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Francois Maillard</font><font size=2 face="sans-serif">
</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Tahoma"><b>De :</b> Clark, Sean [</font><a href=mailto:sean.clark@twcable.com><font size=2 face="Tahoma">mailto:sean.clark@twcable.com</font></a><font size=2 face="Tahoma">]
<b><br>
Envoyé :</b> mercredi 17 décembre 2014 19:46<b><br>
À :</b> MAILLARD Francois Ext DTSI/DSI; xymon@xymon.com<b><br>
Objet :</b> Re: [Xymon] xymon on a large architecture</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri">So I have a setup where, in total, I monitor
about 300,000 hosts.</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri">It's regionalized though, with the
largest site having roughly 34,000 hosts</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri">The xymon daemon runs on a VM, with 16
GB of Ram, and 4x2.5 Ghz xeon processors. The virtual NIC is a 1000Mb </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri">It currently handles 481 msgs/sec (the
hosts.cfg is 1.2 Mb)</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri">Here are some ideas:</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri">(1) Obviously split pollers out into separate
machines</font>
<br><font size=2 face="Calibri">(2) Change the history daemon (Henrick
as some sample daemons in the source code) - I changed it to write stachg
and clichg and such are written out to mysql instead of files, and the
textblog is compressed. This helped with managing history</font>
<br><font size=2 face="Calibri">(3) Use JC Cleaver's patches </font><a href=http://terabithia.org/rpms/xymon/><font size=2 color=blue face="Calibri"><u>http://terabithia.org/rpms/xymon/</u></font></a><font size=2 face="Calibri">
-- he has several memory leak fixes, tuning optimizations. I would
just say that if you use clientupdate, you will have to patch his things
a little bit</font>
<br><font size=2 face="Calibri">(4) An interesting thing you can do, if
you have the means via a load balancer is --></font>
<br><font size=2 face="Calibri"> (a) I have 10 pollers for
this instance. All of them run xymonproxy on localhost (stuff they poll
reports to localhost and gets forwarded up)</font>
<br><font size=2 face="Calibri"> (b) Using a load balancer
for they xymon IP address, it inspects the first 12 characters of a xymon
message – if it's something that can be combo'd in proxy it passes to
a poller in the pool, otherwise, it sends it to the real xymon IP</font>
<br><font size=2 face="Calibri">(5) JC has some nanomsg options for channels,
and myself, I use AMQP for channels to communicate data outbound. Both
of these help if you use xymondata outside of the stock app</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri">For your setup, rather than running more
xymond's on different ports, setup xymonproxy's on different machines,
and point your clients to them. Then you won't have to migrate your history
files or anything else. But I would look at changing your history setup,
as several million files gets very unwieldy to manage, but millions of
rows in databases are much easer to do.</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Calibri"><b>From: </b>"</font><a href=mailto:fmaillard.ext@orange.com><font size=2 color=blue face="Calibri"><u>fmaillard.ext@orange.com</u></font></a><font size=2 face="Calibri">"
<</font><a href=mailto:fmaillard.ext@orange.com><font size=2 color=blue face="Calibri"><u>fmaillard.ext@orange.com</u></font></a><font size=2 face="Calibri">><b><br>
Date: </b>Wednesday, December 17, 2014 at 11:04 AM<b><br>
To: </b>"</font><a href=mailto:xymon@xymon.com><font size=2 color=blue face="Calibri"><u>xymon@xymon.com</u></font></a><font size=2 face="Calibri">"
<</font><a href=mailto:xymon@xymon.com><font size=2 color=blue face="Calibri"><u>xymon@xymon.com</u></font></a><font size=2 face="Calibri">><b><br>
Subject: </b>[Xymon] xymon on a large architecture</font>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Arial">Hello,</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">We’re running a quite large xymon setup,
and have been dealing with performance issue for quite a while. Here are
some stats to give an idea about the setup:</font>
<br><font size=2 face="Arial">- We have 2 xymon servers per datacenter,
on 3 datacenter (all messages are sent to both servers for a given site)</font>
<br><font size=2 face="Arial">- Each xymon server receives on average between
200msg/s and 250msg/s. We’re getting peaks at 400msg/sec.</font>
<br><font size=2 face="Arial">- Each site hosts about 3000 hosts / 30 000
services</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">We’ve been suspecting for a long time that
we might be losing messages… and I think I finally tracked it down to
xymond not fetching the messages quickly enough so that the kernel’s buffer
fill up and messages get discarded (by the kernel). On one of our servers,
even though I have already increased net.ipv4.tcp_rmem and net.ipv4.tcp_wmem
I got the following output from “netstat -s”:</font>
<br><font size=2 face="Arial">148909 packets pruned from receive queue
because of socket buffer overrun</font>
<br><font size=2 face="Arial">4453143 packets collapsed in receive queue
due to low socket buffer</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">And here I come to the question I’m having:</font>
<br><font size=2 face="Arial">1/ Is 250msg/s too much for a single xymond
instance? Is anyone running instances with a lot more traffic than that?</font>
<br><font size=2 face="Arial">2/ I’m starting to look into running several
instances of xymond on the same machine, by binding them to different ports.
Another option is to set up new machines, but that would mean migrating
history files (several million files), sorting out the firewalling issues
(our xymon interfaces are deeply connected to our information system) so
I’d rather like avoiding this option. Are there any guidelines on how
to do this?</font>
<br><font size=2 face="Arial">3/ Are there any settings and best practice
that could improve performance? For instance, should we move to a massive
use of combo statuses in order to lessen the number of messages received?</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial">Best regards,</font>
<br><font size=2 face="Arial"> </font>
<br><font size=2 face="Arial"><b>Francois Maillard</b></font>
<br><font size=2 face="sans-serif"> </font>
<br><font size=2 face="Arial">Pilote des plateformes Supervision, DNS &
FTP - Sysadmin Infrastructure</font>
<br><font size=2 face="Arial">Altran Méditerranée</font>
<br><font size=2 face="Arial">pour Orange/OF/DTSI/DSI/DFY/HBX</font><font size=2 face="sans-serif"><br>
Sophia Antipolis</font>
<br><font size=2 face="sans-serif">tél. 04 97 12 87 53</font><font size=2 color=blue face="sans-serif"><u><br>
</u></font><a href=mailto:fmaillard.ext@orange.com><font size=2 color=#ff8100 face="sans-serif"><u>fmaillard.ext@orange.com</u></font></a>
<br><font size=2 face="Calibri"> </font>
<br><font size=2 face="Courier New">_________________________________________________________________________________________________________________________</font>
<br><font size=2 face="Courier New"> </font>
<br><font size=2 face="Courier New">Ce message et ses pieces jointes peuvent
contenir des informations confidentielles ou privilegiees et ne doivent
donc</font>
<br><font size=2 face="Courier New">pas etre diffuses, exploites ou copies
sans autorisation. Si vous avez recu ce message par erreur, veuillez le
signaler</font>
<br><font size=2 face="Courier New">a l'expediteur et le detruire ainsi
que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,</font>
<br><font size=2 face="Courier New">Orange decline toute responsabilite
si ce message a ete altere, deforme ou falsifie. Merci.</font>
<br><font size=2 face="Courier New"> </font>
<br><font size=2 face="Courier New">This message and its attachments may
contain confidential or privileged information that may be protected by
law;</font>
<br><font size=2 face="Courier New">they should not be distributed, used
or copied without authorisation.</font>
<br><font size=2 face="Courier New">If you have received this email in
error, please notify the sender and delete this message and its attachments.</font>
<br><font size=2 face="Courier New">As emails may be altered, Orange is
not liable for messages that have been modified, changed or falsified.</font>
<br><font size=2 face="Courier New">Thank you.</font>
<br><font size=2 face="Calibri"> </font>
<div align=center>
<br>
<hr></div>
<br><font size=1 color=#808080 face="Arial">This E-mail and any of its
attachments may contain Time Warner Cable proprietary information, which
is privileged, confidential, or subject to copyright belonging to Time
Warner Cable. This E-mail is intended solely for the use of the individual
or entity to which it is addressed. If you are not the intended recipient
of this E-mail, you are hereby notified that any dissemination, distribution,
copying, or action taken in relation to the contents of and attachments
to this E-mail is strictly prohibited and may be unlawful. If you have
received this E-mail in error, please notify the sender immediately and
permanently delete the original and any copy of this E-mail and any printout.</font>
<br><font size=2 face="Courier New">_________________________________________________________________________________________________________________________<br>
<br>
Ce message et ses pieces jointes peuvent contenir des informations confidentielles
ou privilegiees et ne doivent donc<br>
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
recu ce message par erreur, veuillez le signaler<br>
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,<br>
Orange decline toute responsabilite si ce message a ete altere, deforme
ou falsifie. Merci.<br>
<br>
This message and its attachments may contain confidential or privileged
information that may be protected by law;<br>
they should not be distributed, used or copied without authorisation.<br>
If you have received this email in error, please notify the sender and
delete this message and its attachments.<br>
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.<br>
Thank you.<br>
</font><tt><font size=2>_______________________________________________<br>
Xymon mailing list<br>
Xymon@xymon.com<br>
</font></tt><a href=http://lists.xymon.com/mailman/listinfo/xymon><tt><font size=2>http://lists.xymon.com/mailman/listinfo/xymon</font></tt></a><tt><font size=2><br>
</font></tt>
<br>