[Xymon] file check flapping and HTTP500 for xymoncgimsg.cgi

[Kii NODA]

Hi,

I have a problem where a file check for a machine hosting an ELK stack is
flapping between green and white, triggering alerts because of this. FYI,
the host is sending its status information over authenticated HTTPS POST
calls to xymoncgimsg.cgi.

I got to the point where I found out that the list of files that the client
is reporting on is received from the server on every call and then, on
subsequent runs, it is used to check the files and then report back. Upon
further investigation I observed that on some runs, this HTTP POST returns
an HTTP500 error, thus the client then points $LOGFETCHCFG to a file that
contains the HTTP500 page instead of the list of files to check. This means
that on the next run it will no longer report back to the server the status
of the files. It will recover when the POST returns HTTP200, hence looping
between green and white in between successful/unsuccessful POST requests.

Now, I would very much like to understand what happens BUT xymoncgimsg.cgi
which is receiving the HTTP POST is a binary and I'm not aware of how to
enable debugging to see what happens on those runs that fail with HTTP500.

FWIW, the calls only fail w/ HTTP500 for this particular host and I did
consider the size of the status message - it's just around the 1MB mark.
The MSG size set in the configs sits at 2MB currently so I thought I should
be covered here but maybe you guys know better than me.

For now, my quick/dirty hack was to `chattr +i` on a working
$LOGFETCHCFG just so that I no longer get the flapping behavior but I would
like to see this get a proper fix.

Would anyone be able to advise on a next step? Looking through C source
files is not exactly what I want to do since I don't have the expertise to
do so but would be willing to give it some love.

So... help?!? :)

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20190208/b490cde4/attachment.html>