[Xymon] Xymon, community, updates, and directions (was Re: Is this thing on?)

[J.C. Cleaver]

On Fri, August 18, 2023 01:31, Adam Thorn wrote:
> On 17/08/2023 19:21, J.C. Cleaver wrote:
>
>> As it currently stands, xymon is already very efficient, so it's worth
>> pointing out that the complexity really doesn't have to come in until
>> you
>> start hitting things at scale. If you're on a modern core and handling
>> 90
>> msg/s, xymond is going to be just fine. If you're trying to push 9000
>> msg/s, more needs to be taken into consideration.
>
> We see about ~100 msgs / second and CPU load on xymond has never even
> crossed our radar (we run xymond on a relatively low-spec VM with a
> single virtual CPU)
>
> The place where we have historically encountered performance-related
> issues was writing data to rrd files. We have a lot of custom tests and
> like being able to store and graph numerical data. My colleague thus
> implemented a custom xymond_channel listener which lets us take all our
> custom messages that come in a whole range of different formats, parse
> the messages to extract the numerical fields we're interested in, and
> then store them in a postgresql backend database rather than rrd files.
> This did also mean a moderate amount of custom development work for the
> web frontend (including some external js libs to draw graphs) to extract
> the data from the backend.
>
> That wouldn't have been possible without the xymond_channel architecture
> along with the simple and well-defined format of the xymon messages
> themselves.
>

Agreed. The simplicity of a streaming pipe of well-defined ASCII messages
combined with a bus architecture makes it incredibly easy to extend or (in
larger orgs) spew data out to another team to process in whatever manner
they see fit.

We had a bigdata team that wanted to do multi-dimensional analysis on
"metrics" and a data/DS feed into their cluster via xymond_channel solved
the problem completely. Often the only sticking point is that
xymond_channel's recipient needs to be able to handle the quantity of data
that's coming at it efficiently. The SysV IPC for registered channel
listeners flagging their message as taken is one of the very few pinch
points in the system.

-jc