<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
There's no need to rebuild the packages to enable this type of
testing. Just make sure the xymon-debuginfo RPM is installed (it's
in the same repo), as that contains all of the symbol information
on RH-type systems.<br>
<br>
As far as valgrind, all you really need is the base 'valgrind'
package. Simply modify the tasks.cfg as below and you should be
set. I also use "--track-origins=yes" typically.<br>
<br>
In terms of the overall problem, xymond_rrd will use a larg(ish)
amount of RAM as it spools up its cache of data points before
sending them out to rrdtool itself for writing. In theory, this
should hit a constant level once it's been running for an hour or
two (depending on your datapoints and hosts) and shouldn't grow
beyond that. The overall memory usage will scale linearly with
host x RRAs. <br>
<br>
I know it had been a source of leaks before, so it's possible
something is still in there. Are you adding and removing lots of
hosts at once by any chance? It's possible there's an incorrect
cleanup of previously cached data, but I'd thought those had been
resolved.<br>
<br>
HTH,<br>
-jc<br>
<br>
On 9/28/2016 1:27 AM, Peter Welter wrote:<br>
</div>
<blockquote
cite="mid:CAPitMFBbhwwxZd9mGinHdiHpeBtOtiPxk8oY1UZT2Y=RWZ22DA@mail.gmail.com"
type="cite">
<div dir="ltr">
<p class="gmail-p1"><span class="gmail-s1">Hi Henrik, J.C.,</span></p>
<p class="gmail-p1">Thanks for your response. </p>
<p class="gmail-p1">It seems that valgrind is available for RHEL
(see below) and now I wanted to ask J.C. the following: "What
do you want me to do?" </p>
<p class="gmail-p1">If I want to use the prebuild packages, and
YES that would be preferable, then can you supply me with a
pre-compiles binary for xymond_rrd that has all the options
Henrik talked about? So I can replace this with the currently
installed image?</p>
<p class="gmail-p1">Or should I build a package my self to debug
this issue?<br>
</p>
<p class="gmail-p1">Regards, Peter</p>
<p class="gmail-p1"><br>
</p>
<p class="gmail-p1"><span class="gmail-s1">[root@uhu-a xymon]#
yum search valgrind</span></p>
<p class="gmail-p1"><span class="gmail-s1">Loaded plugins:
product-id, search-disabled-repos, security,
subscription-manager</span></p>
<p class="gmail-p1"><span class="gmail-s1">==================================================================================================================================
N/S Matched: valgrind
==================================================================================================================================</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-devel.i686
: Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-devel.x86_64
: Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-openmpi.i686
: OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-openmpi.x86_64
: OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-eclipse-<b>valgrind</b>.noarch
: <b>Valgrind</b> Tools Integration for Eclipse</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-devel.i686
: Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-devel.x86_64
: Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-openmpi.i686
: OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-openmpi.x86_64
: OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">eclipse-<b>valgrind</b>.x86_64
: <b>Valgrind</b> Tools Integration for Eclipse</span></p>
<p class="gmail-p1"><span class="gmail-s1">perl-Test-<b>Valgrind</b>.noarch
: Generate suppressions, analyze and test any command with <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>-devel.i686
: Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>-devel.x86_64
: Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>-openmpi.x86_64
: OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>.i686
: Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>.x86_64
: Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>.i686
: Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>.x86_64
: Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>.i686
: Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>.x86_64
: Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">valkyrie.x86_64 :
Graphical User Interface for <b>Valgrind</b> Suite</span></p>
<p class="gmail-p2"><span class="gmail-s1"></span><br>
</p>
<p class="gmail-p1"><span class="gmail-s1"> Name and summary
matches <b>only</b>, use "search all" for everything.</span></p>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-09-24 14:18 GMT+02:00 Henrik
Størner <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:henrik@hswn.dk" target="_blank">henrik@hswn.dk</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hi,</p>
<p>memory leaks are the worst to troubleshoot.</p>
<p>If possible, then running xymond_rrd via the "valgrind"
tool is the best way to do it. valgrind comes with some
distributions, not sure about RHEL though. There might
be some CentOS packages that will work.</p>
<p>An important point is that the binaries must be
compiled with debugging info intact; i.e. "-g" as a
compile-time option, preferably only -O optimisation,
and not stripped. I guess Japheth can help you with
that, if necessary.</p>
<p>Then you change the tasks.cfg to run xymond_rrd via
valgrind: The CMD setting must then be</p>
<p>CMD valgrind --log-file=/tmp/valgrind-rrd.%<wbr>p
--leak-check=full \<br>
xymond_channel --channel=status
--log=$XYMONSERVERLOGS/rrd-<wbr>status.log xymond_rrd
--rrddir=$XYMONVAR/rrd<br>
</p>
Then run Xymon normally for some time, until hopefully it
starts logging memory leaks.<br>
<br>
<br>
This checking does have a significant performance impact,
so running it on a 4000-server system is probably not
possible.<br>
<br>
<br>
Regards,<br>
Henrik
<div>
<div class="h5"><br>
<br>
<br>
<div>Den 23-09-2016 kl. 13:38 skrev Peter Welter:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="h5">
<div dir="ltr">Hi Japheth,
<div><br>
</div>
<div>Probable one process (xymon_rrd) seems very
hungry for memory:</div>
<div><span style="font-family:monospace,monospace"><br>
</span></div>
<div><span style="font-family:monospace,monospace">[xymon]#
ps aux | egrep 'xymon|MEM'</span><br>
</div>
<div>
<p><span><font face="monospace, monospace">USER
PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND</font></span></p>
<p><span><font face="monospace, monospace">xymon
16889 0.0 0.0 4176 604 ? S
13:26 0:00 /bin/dash</font></span></p>
<p><span><font face="monospace, monospace">xymon
16892 0.0 0.0 6272 660 ? S
13:26 0:00 vmstat 300 2</font></span></p>
<p><span><font face="monospace, monospace">xymon
16986 0.0 0.0 4176 600 ? S
13:28 0:00 /bin/dash</font></span></p>
<p><span><font face="monospace, monospace">xymon
16989 0.0 0.0 6272 664 ? S
13:28 0:00 vmstat 300 2</font></span></p>
<p><span><font face="monospace, monospace">xymon
17060 0.0 0.0 4176 604 ? S
13:30 0:00 /bin/dash</font></span></p>
<p><span><font face="monospace, monospace">xymon
17063 0.0 0.0 6272 664 ? S
13:30 0:00 vmstat 300 2</font></span></p>
<p><span><font face="monospace, monospace">xymon
17107 0.5 0.1 <a
moz-do-not-send="true" href="tel:140340"
value="+31140340" target="_blank">140340</a>
10324 ? S 13:31 0:00
/usr/bin/perl -w -I/home/bbtest/server/ext
/etc/xymon/ext/netapp/<a
moz-do-not-send="true"
href="http://netapp.pl" target="_blank">netapp.<wbr>pl</a></font></span></p>
<p><span><font face="monospace, monospace">xymon
17110 0.2 0.1 142236 11108 ? S
13:31 0:00 /usr/bin/perl -w
-I/home/bbtest/server/ext
/etc/xymon/ext/netapp/<a
moz-do-not-send="true"
href="http://netapp.pl" target="_blank">netapp.<wbr>pl</a></font></span></p>
<p><span><font face="monospace, monospace">xymon
17160 0.0 0.0 106120 1248 ? S
13:31 0:00 sh -c /usr/bin/ssh -x -l
xymon xxx.xxx.xxx.xxx "environment status"
2>&1</font></span></p>
<p><span><font face="monospace, monospace">xymon
17161 0.0 0.0 60060 3440 ? S
13:31 0:00 /usr/bin/ssh -x -l xymon
10.10.1.30 environment status</font></span></p>
<p><span><font face="monospace, monospace">root
17163 0.0 0.0 103324 852 pts/1
S+ 13:31 0:00 egrep xymon|MEM</font></span></p>
<p><span><font face="monospace, monospace">xymon
27932 0.0 0.0 12648 592 ? Ss
Sep20 0:05 /usr/sbin/xymonlaunch
--log=/var/log/xymon/<wbr>xymonlaunch.log</font></span></p>
<p><span><font face="monospace, monospace">xymon
27992 0.0 0.1 25212804 8160 ? S
Sep20 1:57 xymond
--restart=/var/lib/xymon/tmp/<wbr>xymond.chk
--checkpoint-file=/var/lib/<wbr>xymon/tmp/xymond.chk
--checkpoint-interval=600
--admin-senders=127.0.0.1,132.<wbr>229.61.140
--store-clientlogs=!msgs</font></span></p>
<p><span><font face="monospace, monospace">xymon
27996 0.0 0.0 12624444 1452 ? S
Sep20 0:00 xymond_channel
--channel=stachg xymond_history</font></span></p>
<p><span><font face="monospace, monospace">xymon
27997 0.0 0.0 12624444 1244 ? S
Sep20 0:00 xymond_channel
--channel=page xymond_alert
--checkpoint-file=/var/lib/<wbr>xymon/tmp/alert.chk
--checkpoint-interval=600</font></span></p>
<p><span><font face="monospace, monospace">xymon
27998 0.0 0.0 12624444 1340 ? S
Sep20 0:00 xymond_channel
--channel=client xymond_client</font></span></p>
<p><span><font face="monospace, monospace">xymon
27999 0.0 0.0 12624860 4328 ? S
Sep20 0:02 xymond_channel
--channel=status xymond_rrd
--rrddir=/var/lib/xymon/rrd</font></span></p>
<p><span><font face="monospace, monospace">xymon
28000 0.0 0.0 12625628 4712 ? S
Sep20 0:00 xymond_channel
--channel=data xymond_rrd
--rrddir=/var/lib/xymon/rrd</font></span></p>
<p><span><font face="monospace, monospace">xymon
28001 0.0 0.0 12624444 1320 ? S
Sep20 0:00 xymond_channel
--channel=clichg xymond_hostdata</font></span></p>
<p><span><font face="monospace, monospace">xymon
28007 0.0 0.0 41788 1168 ? S
Sep20 0:00 xymond_channel
--channel=user
--log=/var/log/xymon/vmware-<wbr>monitord.log
vmware-monitord</font></span></p>
<p><span><font face="monospace, monospace">xymon
28008 0.0 0.0 10527268 1688 ? S
Sep20 0:00 xymond_history</font></span></p>
<p><span><font face="monospace, monospace">xymon
28009 0.0 1.5 12624884 122508 ? S
Sep20 0:00 xymond_client</font></span></p>
<p><span><font face="monospace, monospace">xymon
28010 0.0 0.0 106848 2176 ? S
Sep20 0:00 /bin/gawk -f
/usr/libexec/xymon/vmware-<wbr>monitord</font></span></p>
<p><span><font face="monospace, monospace">xymon
28011 0.0 0.0 10527252 1212 ? S
Sep20 0:00 xymond_hostdata</font></span></p>
<p><span><font face="monospace, monospace"><b>xymon
28012 0.0 9.4 12680832 765216 ?
S Sep20 0:08 xymond_rrd
--rrddir=/var/lib/xymon/rrd</b></font></span></p>
<p><span><font face="monospace, monospace"><b>xymon
28013 0.0 12.1 12689484 975908 ?
S Sep20 0:12 xymond_rrd
--rrddir=/var/lib/xymon/rrd</b></font></span></p>
<p><font face="monospace, monospace">xymon
28014 0.0 0.1 10527512 9980 ? S
Sep20 0:00 xymond_alert
--checkpoint-file=/var/lib/<wbr>xymon/tmp/alert.chk
--checkpoint-interval=600</font></p>
</div>
<div class="gmail_extra">I did one test migration,
were all hosts (about 4000 hosts) ran on this
system. So the directory /var/lib/xymon/rrd is
quite huge. However, currently there is only one
host (xymon server itself) running and it is
testing one netapp filer. So perhaps, xymon_rrd
and this large directory are somehow related. I
will have a try on the Accept environment which
I have installed by now. There are just a few
files in /var/lib/xymon/rrd on this Accept
system, and I check next monday how each system
will behave. </div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><So far an update;
will be continued. next week..></div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-09-21 13:18
GMT+02:00 Peter Welter <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:peter.welter@gmail.com"
target="_blank">peter.welter@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div dir="ltr">Hi Japheth,
<div><br>
</div>
<div>Thanks for your response. I'm looking
into this and will be back a.s.a.p. (a
few days or so, since I just restarted
Xymon ;-)</div>
<span><font color="#888888">
<div><br>
</div>
<div>Peter</div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-09-20
19:07 GMT+02:00 Japheth Cleaver <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:cleaver@terabithia.org"
target="_blank">cleaver@terabithia.org</a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div>On 9/20/2016 8:37 AM, Peter
Welter wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex"> Hi
J.C.,<br>
<br>
First of all: Thanks for
your work for Xymon!<br>
<br>
Second: I have a question
about the repository from
terabithia. I want to
install an Development,
Test Accept, Production
environment with the use of
this repository. I installed
first and are working on the
next phase.<br>
<br>
Over time however, I see
that my Xymon-server seems
to eat all the memory
available and starts
swapping until all memory is
consumed?!?<br>
<br>
This is for Development only
and there are no really any
tests. A very small
host.cfg. So, why is over
time, Xymon this hungry for
memory?<br>
<br>
Tue Sep 20 17:29:46 CEST
2016 - Memory CRITICAL<br>
<br>
Memory
Used Total Percentage<br>
green Real/Physical
7737M 7872M 98%<br>
yellow Actual/Virtual
7539M 7872M 95%<br>
red Swap/Page
3886M 4095M
94%<br>
<br>
After a Xymon restart, all
the swap is freed?<br>
<br>
I'm using Red Hat Enterprise
Linux Server release 6.8
(Santiago)<br>
<br>
Any suggestions what to do
next? Thanks in advance for
any help!<br>
<br>
Peter<br>
</blockquote>
<br>
</div>
</div>
Hi Peter,<br>
<br>
I'm not aware of any memory leaks
present in 4.3.27 itself that
would cause growth like that. Can
you provide the ps output for the
system's various xymon tools?
Which process seems to be running
out of control?<span><font
color="#888888"><br>
<br>
-jc<br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
Xymon mailing list
<a moz-do-not-send="true" href="mailto:Xymon@xymon.com" target="_blank">Xymon@xymon.com</a>
<a moz-do-not-send="true" href="http://lists.xymon.com/mailman/listinfo/xymon" target="_blank">http://lists.xymon.com/<wbr>mailman/listinfo/xymon</a>
</pre>
</blockquote>
</div>
______________________________<wbr>_________________
Xymon mailing list
<a moz-do-not-send="true" href="mailto:Xymon@xymon.com">Xymon@xymon.com</a>
<a moz-do-not-send="true" href="http://lists.xymon.com/mailman/listinfo/xymon" rel="noreferrer" target="_blank">http://lists.xymon.com/<wbr>mailman/listinfo/xymon</a>
</blockquote></div>
</div>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
Xymon mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Xymon@xymon.com">Xymon@xymon.com</a>
<a class="moz-txt-link-freetext" href="http://lists.xymon.com/mailman/listinfo/xymon">http://lists.xymon.com/mailman/listinfo/xymon</a>
</pre>
</blockquote><p>
</p></body></html>