<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
the recent talk on xymon-developer about rewriting xymonproxy to
support TLS, IPv6 and other good stuff made me think about other
ways of scaling Xymon across large installations.<br>
<br>
Which led me to the idea of having multiple independent Xymon
servers - a swarm, because no one Xymon server depends on the
others, but they can cooperate.<br>
<br>
Simply put, you have a number of independent Xymon installations.
Each of them handles a group of servers - it could be one in each of
your datacentres, one for each organisational unit, one in each
network segment, or just a because you have such a large
installation that a single Xymon server cannot cope with the load
(and that would be a really big installation, judging by the numbers
I hear). This all works just like the Xymon you have today.<br>
<br>
The only thing that is needed to have all of these independent Xymon
servers show up as a single (virtual) Xymon installation is to have
the Xymon webpages - generated by xymongen - to display a set of
webpages showing the status of all of the Xymon servers in the
swarm. When you click on the detailed status log, you are
transparently sent to the Xymon server that holds the data about
that server (the URL points to the Xymon server handling the
particular server you want to check on).<br>
<br>
The nice thing about this is that I think it can be implemented
fairly easily, i.e. without having to change anything fundamental in
the way the various Xymon programs work. Which means it will also be
easy to adapt into an existing Xymon installation, and with a good
chance of not introducing difficult-to-troubleshoot bugs (difficult
because bugs involving remote systems are always a headache to
reproduce).<br>
<br>
There are of course a few nitty-gritty details, e.g. "Find host"
really should be able to search across all of the servers in the
swarm. But those cases are rather few and fairly isolated to not be
too much of a headache.<br>
<br>
<h4> Multiple independent Xymon servers</h4>
<h4> </h4>
<ul>
<li>Each site runs just like today.</li>
<li>A new sites.cfg file lists the other sites (just a site ID and
how to contact xymond there)<br>
</li>
<li>Each site UI (the static webpages from xymongen) merges data
from all sites</li>
</ul>
<h4>Advantages<br>
</h4>
<ul>
<li>More resilient - if one site dies, the others will remain
operational<br>
</li>
<li>Less cross-site traffic (local data remain local except when
needed)<br>
</li>
<li>Less load on each site (updates only go to one Xymon server)<br>
</li>
<li>Horizontally scalable</li>
</ul>
<h4> Limitations</h4>
<h4> </h4>
<ul>
<li>Hostnames must be unique globally. Probably not a significant
problem.<br>
</li>
<li>Functions that fetch data directly from disk-files cannot be
cross-site (rrd-files, history-logs), unless you can retrieve
the data via a network request. In a standard Xymon installation
that would be:</li>
<ul>
<li>Availability reports</li>
<li>Event log reports (but see below)<br>
</li>
<li>Multi-host graphs, unless all of the hosts are local</li>
</ul>
<li>Alerts are always handled locally<br>
</li>
</ul>
<h4> xymongen</h4>
<h4> </h4>
<ul>
<li>hosts.cfg file for the page layout must be merged from all
sites. Can be a simple append-one-after-the-other (built-in) or
perhaps allow foran externally generated hosts.cfg - if you want
to have servers from multiple locations on one page.<br>
</li>
<li>How do we handle non-unique pagenames? Transparently prefix
them with the remote site-ID?<br>
</li>
<li>xymondboard data is fetched from multiple sites and combined
(appended) - handled in sendmessage()<br>
</li>
<li>cgi-URL's are generated with a prefix of /SITE/ - no change
otherwise. The local webserver then proxies /SITE/ requests to
the remote site.</li>
<li>Should there be both a local and a global "all non-green"
page? Maybe even a full set of local and global webpages? That
would be easy by running xymongen twice - one for the local and
one for the global set of pages.</li>
</ul>
<h4> sendmessage() function<br>
</h4>
<h4> </h4>
<ul>
<li>No changes for sending status- or data-updates (status, combo,
extcombo, client, data, modify)<br>
</li>
<li>Option to fetch data from multiple sites. This is already in
place for sending to multiple Xymon servers, so we just need to
combine the output response from multiple sites.</li>
<li>When processing host-related requests, we learn where the host
is located. Cache this for use by various tools. Must be
disk-based (e.g. SQLite file) so it can be shared.</li>
</ul>
<h4>xymond<br>
</h4>
<ul>
<li>hostinfo requests should only answer for the local hosts. No
need to consult the SQLite cache - no changes.<br>
</li>
</ul>
<ul>
</ul>
<h4>CGI programs</h4>
<ul>
<li>"Find host" must be cross-site</li>
<li>Ack-alert: Suggest making it local-only. Since alerts are only
generated locally, it makes sense to also only ack the local
alerts.<br>
</li>
<li>Enable/disable only on the local site? Use the "info" page
enable/disable (automatically local). Global enable/disable
needs some more looking into.<br>
</li>
<li>Critical systems - would probably be nice to be able to do
both a local and a global version.<br>
</li>
<li>Eventlog - would be nice to have both local and global, even
though that means fetching a (large) remote logfile. Will
probably require a new "eventlog" CGI interface for retrieving a
remote logfile. It is probably not something we want to do on
every critical-systems/all-nongreen webpage update. So those
could keep the local eventlog display (as-is), and then the
eventlog CGI could have the option of combining logs from all
sites (or maybe a selection of sites).<br>
</li>
</ul>
<h4>xymon commands</h4>
<u>Commands re. specific hosts</u><br>
First check via hostinfo cache (see below) if we know where the host
is (performance optimization). If not then simply broadcast the
message to all sites and combining any data that is returned - there
will only be data from one server.<br>
<ul>
<li>notify</li>
<li>disable</li>
<li>enable</li>
<li>query</li>
<li>xymondlog, xymondxlog, clientlog<br>
</li>
<li>hostinfo - sendmessage() will fetch the data for us, whether
from the local xymond or from the SQLite cache.<br>
</li>
</ul>
<p><u>Commands that collect data on multiple hosts</u><br>
</p>
<ul>
<li>xymondboard, xymondxboard - option from user whether to fetch
local or global info. Handled in sendmessage()<br>
</li>
</ul>
<p><u>Command that only work locally</u><br>
</p>
<ul>
<li>ghostlist<br>
</li>
<li>drop<br>
</li>
<li>rename<br>
</li>
<li>schedule. If done via web i/f it becomes automatically
transparent, but not for scripts. Probably only used for
disable/enable/drop/rename so makes most sense to do it locally.
Doing global would have to parse the message to detect which
host it is about.<br>
</li>
</ul>
<br>
Comments are very welcome.<br>
<br>
Regards,<br>
Henrik<br>
<br>
</body>
</html>