[Xymon] localmode, got over-size message, truncating

[Jeremy Laidman]

Great work Christoph.

Sorry, it appears that I led you down the wrong path, asserting that it was
a server-only setting in xymond. It would appear to be a client-side
setting. This seems to be undocumented in the man page for xymonclient.cfg.

J

On Thu, 10 Mar 2022 at 21:18, Christoph Zechner <zechner at vrvis.at> wrote:

> I solved it!
>
> I had to add and set "MAXMSG_CLIENT=1024" in /etc/xymon/xymonclient.cfg,
> restarted xymon-client and all the errors were gone.
>
> Thanks again for your help!
>
> Cheers
> Christoph
>
>
> On 09/03/2022 06:42, Christoph Zechner wrote:
> > On 09/03/2022 00:04, Jeremy Laidman wrote:
> >> On Tue, 8 Mar 2022 at 18:52, Christoph Zechner <zechner at vrvis.at
> >> <mailto:zechner at vrvis.at>> wrote:
> >>
> >>     It seems I celebrated prematurely, the errors are back in exactly
> the
> >>     same way :-/
> >>
> >>     2022-03-08 08:47:19.321457 Got over-size message, truncating at
> >> 528383
> >>     bytes (max: 524288)
> >>     2022-03-08 08:47:19.339786 Dropping (more) garbled data
> >>
> >>     I don't understand where this limit 05 512 comes from, everything on
> >>     the
> >>     server checks out (2048 before, tried 4096 as well, no change).
> >>
> >>
> >> I'm at a loss. If the xymond process is proven to have this set at
> >> 2048, then I see no reason why it would give that error message with
> >> that number.
> >>
> >> Unless it's referring to another message type and hence a different
> >> maximum setting? Perhaps take a look at xymond's environment again,
> >> but search for all MAXMSG_ variables. See which one is set to 512, and
> >> that might be the culprit. The defaults for these max values are all
> >> different, with only two of them defaulting to 512: MAXMSG_CLIENT,
> >> MAXMSG_CLICHG (reference: lib/xymond_buffer.c). But it's possible one
> >> of them has been set to 512.
> >
> > Thanks, I tried that, but unfortunately, this did not help, since all
> > the values were set correctly, according to my config.
> >
> >>
> >> The only other thing I can think of is that you have two copies of
> >> xymond running, somehow with different values of MAXMSG_CLIENT. But I
> >> can't think how this could come about. And you've already killed off
> >> any rogue processes.
> >
> > Right, that's not it either. :-/
> >
> >>
> >> Maybe run xymond in debug mode for one round of updates, until you get
> >> the "Got over-size message" and review the debug logs. This might
> >> provide enough additional detail to find out what's going on.
> >>
> >> Another approach to solve the problem (truncated client data message)
> >> is to modify the client script (eg xymonclient-linux.sh) to truncate
> >> the ps command output, so that the total message size is less, and
> >> hopefully fits within the max message size. This will mean that PROC
> >> checks might not work anymore (which is likely the case now). But the
> >> current state is that monitoring of the sections that come after [ps]
> >> are likely broken now. On Linux this is notably the [top] and [vmstat]
> >> sections of the client data message, that are used for the "cpu"
> >> status and several metrics for graphing. Maybe something like adding
> >> "head -1000" will cut it down to a reasonable size:
> >>
> >> echo "[ps]"
> >> ps -Aww -o pid,ppid,user,start,state,pri,pcpu,time,pmem,rsz,vsz,cmd |
> >> head -1000
> >
> > That's actually a gread idea and I modified the [ports] section, because
> > I know this is the culprit (running a proxy there and all the active
> > client connections were too much for xymon to handle.
> >
> > I'm not interested in client connections anyway, I just want to monitor
> > my running programs and ports on that server, so I replaced the original
> >
> > netstat -antuW 2>/dev/null
> > netstat -antuT 2>/dev/null
> >
> > with
> >
> > netstat -tulpenW 2>/dev/null
> >
> > (adding your "| head 1000" suggestion did not work, because it cut off
> > the list before it could reach the IPv6 interfaces and thus the ports
> > check was always red).
> >
> > Now xymon works again, although this is just a workaround, because the
> > underlying problem of where exactly my messages got truncated, is still
> > to be found, but I can live with this solution.
> >
> > Anyway, I very much appreciate your time and efforts, thank you very
> much!
> >
> > Cheers
> > Christoph
> >
> >>
> >> Also, review the client data message before the [ps] section to see if
> >> there's actually something else pushing it over the limit, and [ps]
> >> just happens to be where the truncation happens.
> >>
> >> J
> >>
> > _______________________________________________
> > Xymon mailing list
> > Xymon at xymon.com
> > http://lists.xymon.com/mailman/listinfo/xymon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20220310/f2b77d03/attachment.htm>