<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Carl</div><div><br></div><div dir="ltr">On Tue, 15 Dec 2020 at 01:53, Carl Melgaard <<a href="mailto:Carl.Melgaard@stab.rm.dk">Carl.Melgaard@stab.rm.dk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DA">
<div class="gmail-m_1433118732593892724WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">After running for 5 hrs on my new installation on a RH 7.9, xymond has already allocated 11.5GB of memory…</span></p></div></div></blockquote><div><br></div><div>xymond using a lot of RAM could be something different from the core dumps. But I suspect they're related. For instance, if it's having to keep lots of large combo messages in RAM while other modules send or receive them, but the other modules keep crashing. It's not clear if the xymonnet and xymongen crashes are causing the high RAM usage, or the other way around. It might be worth checking log timestamps to work out what happened first.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DA"><div class="gmail-m_1433118732593892724WordSection1"><p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif"> Last night it core-dumped multiple times, and threw
“Cannot allocate memory” in multiple xymon logfiles, ala “newstrbuffer: Attempt to allocate failed (initialsize=1009956863): Cannot allocate memory”.</span></p></div></div></blockquote><div><br></div><div>"Cannot allocate memory" - do you have swap space? Is it being used?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DA"><div class="gmail-m_1433118732593892724WordSection1"><p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif"> Monitoring 1900 hosts currently – on my primary system I do this with only 4 GB of memory with no issues.</span></p></div></div></blockquote><div><br></div><div>What version of Xymon are you running on the primary system? Similar OS?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DA"><div class="gmail-m_1433118732593892724WordSection1"><p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:Verdana,sans-serif;font-size:10pt">Any idea where I should start to look – it’s a terabithia installation.</span><br></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">Heres a couple of the core-dumps gdb’ed:</span></p></div></div></blockquote><div><br></div><div>The two core dumps suggest the same cause.</div><div><br></div><div><p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">> #2 0x0000561f05bf6115 in sigsegv_handler (signum=<optimized out>) at sig.c:57<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">> #3 <signal handler called></span></p><p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif"><br></span></p></div><div>The sigsegv handler was called, which probably means there was a memory segment violation - typically using memory that hasn't been allocated.</div><div><br></div><div>I'm not a C programmer, but I'm guessing from this:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DA"><div class="gmail-m_1433118732593892724WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">#4 strbuf_addtobuffer (buf=0x0, newtext=0x561f0701db60 "extcombo", ' ' <repeats 192 times>..., newlen=2000) at strfunc.c:115</span></p></div></div></blockquote><div><br></div><div>that the code responsible is (in strfunc.c):</div><div><br></div><div><div><font face="monospace">void strbuf_addtobuffer(strbuffer_t *buf, char *newtext, size_t newlen)</font></div><div><font face="monospace">{</font></div><div><font face="monospace"> if (buf->s == NULL) {</font></div><div><font face="monospace"> buf->used = 0;</font></div><div><font face="monospace"> buf->sz = newlen + BUFSZINCREMENT;</font></div><div><font face="monospace"> buf->s = (char *) malloc(buf->sz);</font></div><div><font face="monospace"> *(buf->s) = '\0';</font></div><div><font face="monospace"> }</font></div></div><div><br></div><div>The "malloc()" operation may have failed due to running out of memory. Then the next line tries to store a "0" byte into unallocated RAM. I'd guess this would cause a sigsegv.</div><div><br></div><div>In other parts of the same file, malloc() is followed by a check for failure, before the memory is used:</div><div><br></div><div>For instance:</div><div><br></div><div><div><font face="monospace"> newbuf->s = (char *)malloc(initialsize);</font></div><div><font face="monospace"> if (newbuf->s == NULL) {</font></div><div><font face="monospace"> errprintf("newstrbuffer: Attempt to allocate failed (initialsize=%d): %s\n", initialsize, strerror(errno));</font></div><div><font face="monospace"> xfree(newbuf);</font></div><div><font face="monospace"> return NULL;</font></div><div><font face="monospace"> }</font></div><div><font face="monospace"> *(newbuf->s) = '\0';</font></div></div><div><br></div><div>The above error checking has been added to some of the code, but perhaps there are places it still needs to be added.</div><div><br></div><div>This appears to have happened during the addition of a combo message string to allocated memory, while creating a message to send to xymond (sendmsg.c):</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="DA"><div class="gmail-m_1433118732593892724WordSection1"><p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">#5 0x0000561f05bf79b5 in addtobufferraw (buf=<optimized out>, newdata=<optimized out>, bytes=<optimized out>) at strfunc.c:184<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10pt;font-family:Verdana,sans-serif">#6 0x0000561f05c00d32 in combo_start () at sendmsg.c:908</span></p></div></div></blockquote><div><br></div><div>Combo messages can be large, and this could a) cause increased RAM usage, or b) be affected by it. Again, it's not clear if the behaviour of xymongen and xymonnet are the cause of your problems or the result of them.</div><div><br></div><div>It looks like the call to combo_start() in xymongen is in code that runs as a result of the "--report" switch. In xymonnet, it's in common code, but there seems to be a modified code path available if you were to add "--bfq" (backfeed queue). I know nothing about the backfeed queue feature, but there's a little about it in the README.backfeed file.</div><div><br></div><div>So some things to consider, mostly just work-arounds and troubleshooting:</div><div><br></div><div>1. Make sure you have swap enabled, and monitor swap-in/swap-out.</div><div>2. See if anything else is using excessive RAM.</div><div>3. Play with combo message sizes. Perhaps a smaller size would help. You can set MAXMSGSPERCOMBO in xymonserver.cfg.<br></div><div>4. Run an older version of Xymon on your new installation, perhaps the same as your current installation. Or perhaps just copy the binaries for xymonnet and/or xymongen to the new server?</div><div>5. Patch the strfunc.c file to include the malloc error checking. You'd need to get the SRPM from Terabithia and build it yourself. Only the xymonnet and xymongen binaries would need to be replaced.</div><div>6. Profile the xymond process's memory usage. I'm not sure how to do this. Perhaps you can get it to dump core, then analyse the core (perhaps just run "strings" over it) to see what's using up all the memory. Perhaps there's some gdb techniques for this.<br></div><div>7. Try running xymongen without "--report", and xymonnet with "--bfq" or "--no-bfq".</div><div><br></div><div>Hope that helps.</div><div><br></div><div>Cheers</div><div>Jeremy</div><div></div></div></div></div></div></div></div></div></div>