[Xymon] Scaling

[cleaver at terabithia.org]

> On Wed, Apr 10, 2013 at 5:51 PM, White, Bruce <bewhite at fellowes.com>
> wrote:
>
>> Over 1000 devices monitored here and only real issue is rrd keeping up.
>> I
>> have been told an ssd for the rrd files will solve this issue.
>>
>
>
> ~2000 hosts and that will double or triple in the next few weeks. I really
> don't see any IO issues in the slightest.
> 6 x 15k RPM SCSI drives in Raid 5 on a Dell PowerEdge 2950 with 8 gigs of
> ram and the thing is snoring (LA: 0.25)
>
> Regards,
> Cami


We're currently processing ~2K incoming messages a second on a single
xymond instance. This is a pretty beefy box, but it's also handling lots
of other concurrent monitoring tasks that we're slowly moving over to
xymon... including a non-fping-enabled Icinga install >.<

]# xymon localhost "xymondboard test=info fields=hostname" | wc -l
42459

(Not all of those are full hosts; some are application nodes with statuses
being generated server-side out of client-side jvm stats or the like.)


At these levels it's important to ensure you're using whatever NUMA
capabilities your system has properly, since message passing is basically
just shoveling incoming TCP data around within memory. Also, you might
want to tweak net.ipv4.ip_local_port_range and enable
net.ipv4.tcp_tw_reuse and/or net.ipv4.tcp_tw_recycle on Linux to eke more
simultaneous testing out of xymonnet.


One of the beauties of Xymon's architecture is the ability to cleanly
disconnect the components... Xymongen can run on some other box,
xymond_locator can be used to send rrd data off somewhere if IO becomes an
issue, xymonnet pollers can be distributed, and xymonproxy can be used as
needed to aggregate and smooth out incoming status reports, etc.

There are lots of different mechanisms for "scaling" efficiently depending
on your particular needs, but I'd bet that on decently modern server
hardware you'll probably want to scale for HA purposes long before you
actually /need/ the additional power.


HTH,

-jc
-------------- next part --------------
A non-text attachment was scrubbed...
Name: xymond-msgs-per-sec.gif
Type: image/gif
Size: 14082 bytes
Desc: not available
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20130411/0a13c492/attachment.gif>