<div dir="ltr">
<p class="gmail-p1"><span class="gmail-s1">Hi Henrik, J.C.,</span></p><p class="gmail-p1">Thanks for your response. </p><p class="gmail-p1">It seems that valgrind is available for RHEL (see below) and now I wanted to ask J.C. the following: "What do you want me to do?" </p><p class="gmail-p1">If I want to use the prebuild packages, and YES that would be preferable, then can you supply me with a pre-compiles binary for xymond_rrd that has all the options Henrik talked about? So I can replace this with the currently installed image?</p><p class="gmail-p1">Or should I build a package my self to debug this issue?<br></p><p class="gmail-p1">Regards, Peter</p><p class="gmail-p1"><br></p><p class="gmail-p1"><span class="gmail-s1">[root@uhu-a xymon]# yum search valgrind</span></p>
<p class="gmail-p1"><span class="gmail-s1">Loaded plugins: product-id, search-disabled-repos, security, subscription-manager</span></p>
<p class="gmail-p1"><span class="gmail-s1">================================================================================================================================== N/S Matched: valgrind ==================================================================================================================================</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-devel.i686 : Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-devel.x86_64 : Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-openmpi.i686 : OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>-openmpi.x86_64 : OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-eclipse-<b>valgrind</b>.noarch : <b>Valgrind</b> Tools Integration for Eclipse</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-devel.i686 : Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-devel.x86_64 : Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-openmpi.i686 : OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>-openmpi.x86_64 : OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">eclipse-<b>valgrind</b>.x86_64 : <b>Valgrind</b> Tools Integration for Eclipse</span></p>
<p class="gmail-p1"><span class="gmail-s1">perl-Test-<b>Valgrind</b>.noarch : Generate suppressions, analyze and test any command with <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>-devel.i686 : Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>-devel.x86_64 : Development files for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>-openmpi.x86_64 : OpenMPI support for <b>valgrind</b></span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>.i686 : Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-1.1-<b>valgrind</b>.x86_64 : Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>.i686 : Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">devtoolset-2-<b>valgrind</b>.x86_64 : Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>.i686 : Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1"><b>valgrind</b>.x86_64 : Tool for finding memory management bugs in programs</span></p>
<p class="gmail-p1"><span class="gmail-s1">valkyrie.x86_64 : Graphical User Interface for <b>Valgrind</b> Suite</span></p>
<p class="gmail-p2"><span class="gmail-s1"></span><br></p>
<p class="gmail-p1"><span class="gmail-s1"> Name and summary matches <b>only</b>, use "search all" for everything.</span></p></div><div class="gmail_extra"><br><div class="gmail_quote">2016-09-24 14:18 GMT+02:00 Henrik Størner <span dir="ltr"><<a href="mailto:henrik@hswn.dk" target="_blank">henrik@hswn.dk</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hi,</p>
<p>memory leaks are the worst to troubleshoot.</p>
<p>If possible, then running xymond_rrd via the "valgrind" tool is
the best way to do it. valgrind comes with some distributions, not
sure about RHEL though. There might be some CentOS packages that
will work.</p>
<p>An important point is that the binaries must be compiled with
debugging info intact; i.e. "-g" as a compile-time option,
preferably only -O optimisation, and not stripped. I guess Japheth
can help you with that, if necessary.</p>
<p>Then you change the tasks.cfg to run xymond_rrd via valgrind: The
CMD setting must then be</p>
<p>CMD valgrind --log-file=/tmp/valgrind-rrd.%<wbr>p --leak-check=full \<br>
xymond_channel --channel=status
--log=$XYMONSERVERLOGS/rrd-<wbr>status.log xymond_rrd
--rrddir=$XYMONVAR/rrd<br>
</p>
Then run Xymon normally for some time, until hopefully it starts
logging memory leaks.<br>
<br>
<br>
This checking does have a significant performance impact, so running
it on a 4000-server system is probably not possible.<br>
<br>
<br>
Regards,<br>
Henrik<div><div class="h5"><br>
<br>
<br>
<div>Den 23-09-2016 kl. 13:38 skrev Peter
Welter:<br>
</div>
</div></div><blockquote type="cite"><div><div class="h5">
<div dir="ltr">Hi Japheth,
<div><br>
</div>
<div>Probable one process (xymon_rrd) seems very hungry for
memory:</div>
<div><span style="font-family:monospace,monospace"><br>
</span></div>
<div><span style="font-family:monospace,monospace">[xymon]# ps
aux | egrep 'xymon|MEM'</span><br>
</div>
<div>
<p><span><font face="monospace, monospace">USER PID %CPU %MEM
VSZ RSS TTY STAT START TIME COMMAND</font></span></p>
<p><span><font face="monospace, monospace">xymon 16889 0.0 0.0
4176 604 ? S 13:26 0:00 /bin/dash</font></span></p>
<p><span><font face="monospace, monospace">xymon 16892 0.0 0.0
6272 660 ? S 13:26 0:00 vmstat 300 2</font></span></p>
<p><span><font face="monospace, monospace">xymon 16986 0.0 0.0
4176 600 ? S 13:28 0:00 /bin/dash</font></span></p>
<p><span><font face="monospace, monospace">xymon 16989 0.0 0.0
6272 664 ? S 13:28 0:00 vmstat 300 2</font></span></p>
<p><span><font face="monospace, monospace">xymon 17060 0.0 0.0
4176 604 ? S 13:30 0:00 /bin/dash</font></span></p>
<p><span><font face="monospace, monospace">xymon 17063 0.0 0.0
6272 664 ? S 13:30 0:00 vmstat 300 2</font></span></p>
<p><span><font face="monospace, monospace">xymon 17107 0.5 0.1
<a href="tel:140340" value="+31140340" target="_blank">140340</a> 10324 ? S 13:31 0:00 /usr/bin/perl -w
-I/home/bbtest/server/ext /etc/xymon/ext/netapp/<a href="http://netapp.pl" target="_blank">netapp.<wbr>pl</a></font></span></p>
<p><span><font face="monospace, monospace">xymon 17110 0.2 0.1
142236 11108 ? S 13:31 0:00 /usr/bin/perl -w
-I/home/bbtest/server/ext /etc/xymon/ext/netapp/<a href="http://netapp.pl" target="_blank">netapp.<wbr>pl</a></font></span></p>
<p><span><font face="monospace, monospace">xymon 17160 0.0 0.0
106120 1248 ? S 13:31 0:00 sh -c
/usr/bin/ssh -x -l xymon xxx.xxx.xxx.xxx "environment
status" 2>&1</font></span></p>
<p><span><font face="monospace, monospace">xymon 17161 0.0 0.0
60060 3440 ? S 13:31 0:00 /usr/bin/ssh -x
-l xymon 10.10.1.30 environment status</font></span></p>
<p><span><font face="monospace, monospace">root 17163 0.0 0.0
103324 852 pts/1 S+ 13:31 0:00 egrep xymon|MEM</font></span></p>
<p><span><font face="monospace, monospace">xymon 27932 0.0 0.0
12648 592 ? Ss Sep20 0:05
/usr/sbin/xymonlaunch
--log=/var/log/xymon/<wbr>xymonlaunch.log</font></span></p>
<p><span><font face="monospace, monospace">xymon 27992 0.0 0.1
25212804 8160 ? S Sep20 1:57 xymond
--restart=/var/lib/xymon/tmp/<wbr>xymond.chk
--checkpoint-file=/var/lib/<wbr>xymon/tmp/xymond.chk
--checkpoint-interval=600
--admin-senders=127.0.0.1,132.<wbr>229.61.140
--store-clientlogs=!msgs</font></span></p>
<p><span><font face="monospace, monospace">xymon 27996 0.0 0.0
12624444 1452 ? S Sep20 0:00 xymond_channel
--channel=stachg xymond_history</font></span></p>
<p><span><font face="monospace, monospace">xymon 27997 0.0 0.0
12624444 1244 ? S Sep20 0:00 xymond_channel
--channel=page xymond_alert
--checkpoint-file=/var/lib/<wbr>xymon/tmp/alert.chk
--checkpoint-interval=600</font></span></p>
<p><span><font face="monospace, monospace">xymon 27998 0.0 0.0
12624444 1340 ? S Sep20 0:00 xymond_channel
--channel=client xymond_client</font></span></p>
<p><span><font face="monospace, monospace">xymon 27999 0.0 0.0
12624860 4328 ? S Sep20 0:02 xymond_channel
--channel=status xymond_rrd --rrddir=/var/lib/xymon/rrd</font></span></p>
<p><span><font face="monospace, monospace">xymon 28000 0.0 0.0
12625628 4712 ? S Sep20 0:00 xymond_channel
--channel=data xymond_rrd --rrddir=/var/lib/xymon/rrd</font></span></p>
<p><span><font face="monospace, monospace">xymon 28001 0.0 0.0
12624444 1320 ? S Sep20 0:00 xymond_channel
--channel=clichg xymond_hostdata</font></span></p>
<p><span><font face="monospace, monospace">xymon 28007 0.0 0.0
41788 1168 ? S Sep20 0:00 xymond_channel
--channel=user --log=/var/log/xymon/vmware-<wbr>monitord.log
vmware-monitord</font></span></p>
<p><span><font face="monospace, monospace">xymon 28008 0.0 0.0
10527268 1688 ? S Sep20 0:00 xymond_history</font></span></p>
<p><span><font face="monospace, monospace">xymon 28009 0.0 1.5
12624884 122508 ? S Sep20 0:00 xymond_client</font></span></p>
<p><span><font face="monospace, monospace">xymon 28010 0.0 0.0
106848 2176 ? S Sep20 0:00 /bin/gawk -f
/usr/libexec/xymon/vmware-<wbr>monitord</font></span></p>
<p><span><font face="monospace, monospace">xymon 28011 0.0 0.0
10527252 1212 ? S Sep20 0:00 xymond_hostdata</font></span></p>
<p><span><font face="monospace, monospace"><b>xymon 28012 0.0 9.4
12680832 765216 ? S Sep20 0:08 xymond_rrd
--rrddir=/var/lib/xymon/rrd</b></font></span></p>
<p><span><font face="monospace, monospace"><b>xymon 28013 0.0 12.1
12689484 975908 ? S Sep20 0:12 xymond_rrd
--rrddir=/var/lib/xymon/rrd</b></font></span></p>
<p><font face="monospace, monospace">xymon
28014 0.0 0.1 10527512 9980 ? S Sep20 0:00
xymond_alert
--checkpoint-file=/var/lib/<wbr>xymon/tmp/alert.chk
--checkpoint-interval=600</font></p>
</div>
<div class="gmail_extra">I did one test migration, were all
hosts (about 4000 hosts) ran on this system. So the directory
/var/lib/xymon/rrd is quite huge. However, currently there is
only one host (xymon server itself) running and it is testing
one netapp filer. So perhaps, xymon_rrd and this large
directory are somehow related. I will have a try on the Accept
environment which I have installed by now. There are just a
few files in /var/lib/xymon/rrd on this Accept system, and I
check next monday how each system will behave. </div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><So far an update; will be
continued. next week..></div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-09-21 13:18 GMT+02:00 Peter
Welter <span dir="ltr"><<a href="mailto:peter.welter@gmail.com" target="_blank">peter.welter@gmail.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Japheth,
<div><br>
</div>
<div>Thanks for your response. I'm looking into this and
will be back a.s.a.p. (a few days or so, since I just
restarted Xymon ;-)</div>
<span><font color="#888888">
<div><br>
</div>
<div>Peter</div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-09-20 19:07 GMT+02:00
Japheth Cleaver <span dir="ltr"><<a href="mailto:cleaver@terabithia.org" target="_blank">cleaver@terabithia.org</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>On 9/20/2016 8:37 AM, Peter Welter wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi J.C.,<br>
<br>
First of all: Thanks for your work for
Xymon!<br>
<br>
Second: I have a question about the
repository from terabithia. I want to
install an Development, Test Accept,
Production environment with the use of
this repository. I installed first and are
working on the next phase.<br>
<br>
Over time however, I see that my
Xymon-server seems to eat all the memory
available and starts swapping until all
memory is consumed?!?<br>
<br>
This is for Development only and there are
no really any tests. A very small
host.cfg. So, why is over time, Xymon this
hungry for memory?<br>
<br>
Tue Sep 20 17:29:46 CEST 2016 - Memory
CRITICAL<br>
<br>
Memory Used
Total Percentage<br>
green Real/Physical 7737M
7872M 98%<br>
yellow Actual/Virtual 7539M
7872M 95%<br>
red Swap/Page 3886M
4095M 94%<br>
<br>
After a Xymon restart, all the swap is
freed?<br>
<br>
I'm using Red Hat Enterprise Linux Server
release 6.8 (Santiago)<br>
<br>
Any suggestions what to do next? Thanks in
advance for any help!<br>
<br>
Peter<br>
</blockquote>
<br>
</div>
</div>
Hi Peter,<br>
<br>
I'm not aware of any memory leaks present in
4.3.27 itself that would cause growth like that.
Can you provide the ps output for the system's
various xymon tools? Which process seems to be
running out of control?<span><font color="#888888"><br>
<br>
-jc<br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><pre>______________________________<wbr>_________________
Xymon mailing list
<a href="mailto:Xymon@xymon.com" target="_blank">Xymon@xymon.com</a>
<a href="http://lists.xymon.com/mailman/listinfo/xymon" target="_blank">http://lists.xymon.com/<wbr>mailman/listinfo/xymon</a>
</pre>
</blockquote>
<br>
</div>
<br>______________________________<wbr>_________________<br>
Xymon mailing list<br>
<a href="mailto:Xymon@xymon.com">Xymon@xymon.com</a><br>
<a href="http://lists.xymon.com/mailman/listinfo/xymon" rel="noreferrer" target="_blank">http://lists.xymon.com/<wbr>mailman/listinfo/xymon</a><br>
<br></blockquote></div><br></div>