<div dir="ltr">Great work Christoph.<div><br></div><div>Sorry, it appears that I led you down the wrong path, asserting that it was a server-only setting in xymond. It would appear to be a client-side setting. This seems to be undocumented in the man page for xymonclient.cfg.</div><div><br></div><div>J</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 10 Mar 2022 at 21:18, Christoph Zechner <<a href="mailto:zechner@vrvis.at">zechner@vrvis.at</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I solved it!<br>
<br>
I had to add and set "MAXMSG_CLIENT=1024" in /etc/xymon/xymonclient.cfg, <br>
restarted xymon-client and all the errors were gone.<br>
<br>
Thanks again for your help!<br>
<br>
Cheers<br>
Christoph<br>
<br>
<br>
On 09/03/2022 06:42, Christoph Zechner wrote:<br>
> On 09/03/2022 00:04, Jeremy Laidman wrote:<br>
>> On Tue, 8 Mar 2022 at 18:52, Christoph Zechner <<a href="mailto:zechner@vrvis.at" target="_blank">zechner@vrvis.at</a> <br>
>> <mailto:<a href="mailto:zechner@vrvis.at" target="_blank">zechner@vrvis.at</a>>> wrote:<br>
>><br>
>> It seems I celebrated prematurely, the errors are back in exactly the<br>
>> same way :-/<br>
>><br>
>> 2022-03-08 08:47:19.321457 Got over-size message, truncating at <br>
>> 528383<br>
>> bytes (max: 524288)<br>
>> 2022-03-08 08:47:19.339786 Dropping (more) garbled data<br>
>><br>
>> I don't understand where this limit 05 512 comes from, everything on<br>
>> the<br>
>> server checks out (2048 before, tried 4096 as well, no change).<br>
>><br>
>><br>
>> I'm at a loss. If the xymond process is proven to have this set at <br>
>> 2048, then I see no reason why it would give that error message with <br>
>> that number.<br>
>><br>
>> Unless it's referring to another message type and hence a different <br>
>> maximum setting? Perhaps take a look at xymond's environment again, <br>
>> but search for all MAXMSG_ variables. See which one is set to 512, and <br>
>> that might be the culprit. The defaults for these max values are all <br>
>> different, with only two of them defaulting to 512: MAXMSG_CLIENT, <br>
>> MAXMSG_CLICHG (reference: lib/xymond_buffer.c). But it's possible one <br>
>> of them has been set to 512.<br>
> <br>
> Thanks, I tried that, but unfortunately, this did not help, since all <br>
> the values were set correctly, according to my config.<br>
> <br>
>><br>
>> The only other thing I can think of is that you have two copies of <br>
>> xymond running, somehow with different values of MAXMSG_CLIENT. But I <br>
>> can't think how this could come about. And you've already killed off <br>
>> any rogue processes.<br>
> <br>
> Right, that's not it either. :-/<br>
> <br>
>><br>
>> Maybe run xymond in debug mode for one round of updates, until you get <br>
>> the "Got over-size message" and review the debug logs. This might <br>
>> provide enough additional detail to find out what's going on.<br>
>><br>
>> Another approach to solve the problem (truncated client data message) <br>
>> is to modify the client script (eg xymonclient-linux.sh) to truncate <br>
>> the ps command output, so that the total message size is less, and <br>
>> hopefully fits within the max message size. This will mean that PROC <br>
>> checks might not work anymore (which is likely the case now). But the <br>
>> current state is that monitoring of the sections that come after [ps] <br>
>> are likely broken now. On Linux this is notably the [top] and [vmstat] <br>
>> sections of the client data message, that are used for the "cpu" <br>
>> status and several metrics for graphing. Maybe something like adding <br>
>> "head -1000" will cut it down to a reasonable size:<br>
>><br>
>> echo "[ps]"<br>
>> ps -Aww -o pid,ppid,user,start,state,pri,pcpu,time,pmem,rsz,vsz,cmd | <br>
>> head -1000<br>
> <br>
> That's actually a gread idea and I modified the [ports] section, because <br>
> I know this is the culprit (running a proxy there and all the active <br>
> client connections were too much for xymon to handle.<br>
> <br>
> I'm not interested in client connections anyway, I just want to monitor <br>
> my running programs and ports on that server, so I replaced the original<br>
> <br>
> netstat -antuW 2>/dev/null<br>
> netstat -antuT 2>/dev/null<br>
> <br>
> with<br>
> <br>
> netstat -tulpenW 2>/dev/null<br>
> <br>
> (adding your "| head 1000" suggestion did not work, because it cut off <br>
> the list before it could reach the IPv6 interfaces and thus the ports <br>
> check was always red).<br>
> <br>
> Now xymon works again, although this is just a workaround, because the <br>
> underlying problem of where exactly my messages got truncated, is still <br>
> to be found, but I can live with this solution.<br>
> <br>
> Anyway, I very much appreciate your time and efforts, thank you very much!<br>
> <br>
> Cheers<br>
> Christoph<br>
> <br>
>><br>
>> Also, review the client data message before the [ps] section to see if <br>
>> there's actually something else pushing it over the limit, and [ps] <br>
>> just happens to be where the truncation happens.<br>
>><br>
>> J<br>
>><br>
> _______________________________________________<br>
> Xymon mailing list<br>
> <a href="mailto:Xymon@xymon.com" target="_blank">Xymon@xymon.com</a><br>
> <a href="http://lists.xymon.com/mailman/listinfo/xymon" rel="noreferrer" target="_blank">http://lists.xymon.com/mailman/listinfo/xymon</a><br>
</blockquote></div>