<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Is this a shared memory issue (shm) - the process can't allocate shared memory? Maybe increase the SHMALL kernel parameter?<br>
</p>
<br>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"> </div>
<div>
<p><font size="2" face="Courier 10 Pitch">Hi Thomas,</font><br>
<br>
<font size="2" face="Courier 10 Pitch">thank you for your hints.</font><br>
<font size="2" face="Courier 10 Pitch">Lets go through your suggestions:</font><br>
<br>
<font size="2" face="Courier 10 Pitch">- you were right with the INTERVAL. This was added by accident, but doesn't harm (no more channels are started), removed, no effect</font><br>
<font size="2" face="Courier 10 Pitch">- yes, server process was restartet at 5:13am
</font><b><font size="2" face="Courier 10 Pitch">UTC</font></b><br>
<br>
<font size="2" face="Courier 10 Pitch">1.</font><br>
<font size="2" face="Courier 10 Pitch">- environment looks as expected:</font><br>
<font size="2" face="Courier 10 Pitch">$ xymoncmd /usr/bin/env | egrep -i 'xymsrv|xymonservers|bbdisp'</font><br>
<font size="2" face="Courier 10 Pitch">XYMONSERVERS=10.137.56.43</font><br>
<font size="2" face="Courier 10 Pitch">XYMSRV=10.137.56.43</font><br>
<font size="2" face="Courier 10 Pitch">BBDISP=10.137.56.43</font><br>
<font size="2" face="Courier 10 Pitch">BBDISPLAYS=10.137.56.43</font><br>
<br>
<font size="2" face="Courier 10 Pitch">2. <br>
- unfortunately i don't have root access here for a tcpdump</font><br>
<br>
<font size="2" face="Courier 10 Pitch">3. </font><br>
<font size="2" face="Courier 10 Pitch">- now it gets strange:<br>
if i try your suggestion i get this (doublechecked at other server and the syntax works ok):</font><br>
<font size="2" face="Courier 10 Pitch"><br>
$ xymoncmd xymond_channel --channel=client cat</font><br>
<font size="2" face="Courier 10 Pitch">2017-10-19 15:58:58.843500 execvp() failed: No such file or directory</font><br>
<font size="2" face="Courier 10 Pitch">(but of curse xymoncmd is in the path)<br>
</font><br>
<font size="2" face="Courier 10 Pitch">if i try with relative path i get this (run from bin path):<br>
$ ./xymoncmd ./xymond_channel --channel=client cat</font><br>
<font size="2" face="Courier 10 Pitch">2017-10-19 16:01:02.173601 Could not get shm of size 524288: No such file or directory</font><br>
<font size="2" face="Courier 10 Pitch">2017-10-19 16:01:02.173943 Channel not available</font><br>
<br>
<font size="2" face="Courier 10 Pitch">some of these messages i can find in some logs:</font><br>
<br>
<font size="2" face="Courier 10 Pitch">$ grep "Could not get shm" *<br>
alert.log:2017-10-19 10:38:00.733526 Could not get shm of size 1048576: No such file or directory</font><br>
<font size="2" face="Courier 10 Pitch">history.log:2017-10-19 10:38:00.730686 Could not get shm of size 1048576: No such file or directory</font><br>
<font size="2" face="Courier 10 Pitch">hostdata.log:2017-10-19 10:38:00.730538 Could not get shm of size 524288: No such file or directory</font><br>
<font size="2" face="Courier 10 Pitch">rrd-data.log:2017-10-19 10:38:00.734198 Could not get shm of size 524288: No such file or directory</font><br>
<br>
<font size="2" face="Courier 10 Pitch">I searched the web for those messages, and found some in hobbit/xymon mailing list, but no details or solution.</font><br>
<a href="http://lists.xymon.com/oldarchive/2008/06/msg00183.html"><font size="2" face="Courier 10 Pitch">http://lists.xymon.com/oldarchive/2008/06/msg00183.html</font></a><br>
<br>
<font size="2" face="Courier 10 Pitch">But: the client messages from server still arrive and the tests are all visible...</font><br>
<font size="2" face="Courier 10 Pitch">very strange</font><br>
<br>
<font size="2" face="Courier 10 Pitch">Norbert</font><br>
<br>
<br>
<br>
<img alt="Inactive hide details for Thomas Eckert ---19/10/2017 16:42:24---> On 19 Oct 2017, at 14:49, Norbert Kriegenburg <norbert.krieg" border="0" width="16" height="16" src="cid:1__=4EBB0B2DDFC114718f9e8a93df938690918c4EB@"><font size="2" color="#424282">Thomas
Eckert ---19/10/2017 16:42:24---> On 19 Oct 2017, at 14:49, Norbert Kriegenburg <norbert.kriegenburg@de.ibm.com> wrote: ></font><br>
<br>
<font size="2" color="#5F5F5F">From: </font><font size="2">Thomas Eckert <thomas.eckert@it-eckert.de></font><br>
<font size="2" color="#5F5F5F">To: </font><font size="2">xymon <xymon@xymon.com></font><br>
<font size="2" color="#5F5F5F">Date: </font><font size="2">19/10/2017 16:42</font><br>
<font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [Xymon] no client results arrive, lot of TIME_WAIT at server</font><br>
<font size="2" color="#5F5F5F">Sent by: </font><font size="2">"Xymon" <xymon-bounces@xymon.com></font><br>
</p>
<hr style="color:#8091A5" width="100%" noshade="" size="2" align="left">
<br>
<br>
<ul>
<ul>
On 19 Oct 2017, at 14:49, Norbert Kriegenburg <<a href="mailto:norbert.kriegenburg@de.ibm.com"><u><font color="#0000FF">norbert.kriegenburg@de.ibm.com</font></u></a>> wrote:<br>
<p><font size="2">Yes,<br>
that's what i checked at first. But i assume that also the info from the servers own client data are captured by the same client channel listener, and that obviously works.</font></p>
</ul>
</ul>
valid point
<ul>
<ul>
<font size="2">Nothing in the clientdata.log, even if i activate the DEBUG mode with a<br>
kill -USR2 PID<br>
<br>
My clientdata section:<br>
# The client back-end module. You need this if you are running the Xymon client on any system.<br>
[clientdata]<br>
ENVFILE $XYMONSERVERHOME/etc/xymonserver.cfg<br>
NEEDS xymond<br>
CMD /home/xymon/server/bin/xymond_channel --channel=client \<br>
--log=$XYMONSERVERLOGS/clientdata.log xymond_client \<br>
--no-clear-files \<br>
--no-clear-ports \<br>
--no-port-listing \<br>
--uptime-status<br>
INTERVAL 2m</font></ul>
</ul>
The `INTERVAL` here is not im my and <a href="http://xymon.com/"><u><font color="#0000FF">xymon.com</font></u></a>’s-version of tasks.cfg— not sure if it does harm (it _may_ try to start a new channel every 2 minutes).
<ul>
<ul>
<font size="2">ps listing:<br>
$ ps -ef|grep "xymond_channel --channel=client"|grep -v grep<br>
xymon 18865 18705 0 05:13 ? 00:00:00 /home/xymon/server/bin/xymond_channel --channel=client --log=/home/xymon/server/logs/clientdata.log xymond_client --no-clear-files --no-clear-ports --no-port-listing --uptime-status</font></ul>
</ul>
If this process does not change it should be fine; your xymon server was restarted 5:13 this morning?
<ul>
<ul>
<font size="2">I have a lot of TIME_WAITS from time to time, no idea why this goes up and down:</font><br>
<font size="2"><br>
$ netstat -atn|grep 1984<br>
tcp 0 0 0.0.0.0:1984 0.0.0.0:* LISTEN<br>
tcp 0 0 10.137.56.43:37752 10.137.56.43:1984 TIME_WAIT<br>
tcp 0 0 10.137.56.43:37822 10.137.56.43:1984 TIME_WAIT<br>
...</font><br>
<font size="2"><br>
10.137.56.43 is the Xymon server IP.<br>
<br>
$ netstat -atn|grep 1984|wc -l<br>
165<br>
(sometimes much much more)<br>
<br>
(But goes down to 7 after some mins)</font></ul>
</ul>
Looks like internal xymon-communication. As long as the TIME_WAIT-count does not go >20k that should not be the cause.<br>
<br>
Thing’s I’d try from here:<br>
<br>
1. double-check the XYMSRV / XYMONSERVERS setting on the client(s), e.g. with:<br>
<br>
xymoncmd /usr/bin/env | egrep -i 'xymsrv|xymonservers|bbdisp’<br>
<br>
2. just to double-check: tcpdump on xymon server and a client to check for tcp/1984 communication<br>
<br>
3. run a channel-listener an the server to see if the client messages (other than the ones from the local client) make it
<br>
<br>
xymoncmd xymond_channel --channel=client cat | grep ‘^client '<br>
xymoncmd xymond_channel --channel=client cat<br>
<br>
The latter might be very noisy — the former just prints the “header”/1st line of each connection.<br>
There should be one message for each client sending data every 5 minutes.<br>
<br>
HTH<br>
Thomas
<ul>
<ul>
<graycol.gif><font size="2" color="#424282">Thomas Eckert ---19/10/2017 11:14:56---Norbert, I assume your manual (BB way) is something like `xymon 1.2.3.4 “status example.com.test …”`</font><br>
<font size="2" color="#5F5F5F"><br>
From: </font><font size="2">Thomas Eckert <</font><a href="mailto:thomas.eckert@it-eckert.de"><u><font size="2" color="#0000FF">thomas.eckert@it-eckert.de</font></u></a><font size="2">></font><font size="2" color="#5F5F5F"><br>
To: </font><font size="2">xymon <</font><a href="mailto:xymon@xymon.com"><u><font size="2" color="#0000FF">xymon@xymon.com</font></u></a><font size="2">></font><font size="2" color="#5F5F5F"><br>
Date: </font><font size="2">19/10/2017 11:14</font><font size="2" color="#5F5F5F"><br>
Subject: </font><font size="2">Re: [Xymon] no client results arrive, lot of TIME_WAIT at server</font><font size="2" color="#5F5F5F"><br>
Sent by: </font><font size="2">"Xymon" <</font><a href="mailto:xymon-bounces@xymon.com"><u><font size="2" color="#0000FF">xymon-bounces@xymon.com</font></u></a><font size="2">></font>
<p></p>
<hr width="100%" noshade="" size="2" align="left">
<br>
<br>
<br>
Norbert,<br>
<br>
I assume your manual (BB way) is something like `xymon 1.2.3.4 “status example.com.test …”`.<br>
<br>
The xymon-client send `clientdata` that is processed by a “channel listener” and _that_ sends status-messages similar to above (a bit simplified).<br>
<br>
That channel-listener may not be running/crashing:<br>
Check our `tasks.cfg` for the section `[clientdata]` and verify, if the `xymond_channel`-process is running (I suspect it’s not). Maybe the `clientdata.log` helps with debugging. You may also run the task manually to get further information.<br>
<br>
What’s your problem with the "lot of TIME_WAIT”? Are these just for tcp/1984-connections?<br>
<br>
Cheers<br>
Thomas
<ul>
<ul>
<ul>
<ul>
On 19 Oct 2017, at 07:30, Norbert Kriegenburg <<a href="mailto:norbert.kriegenburg@de.ibm.com"><u><font color="#0000FF">norbert.kriegenburg@de.ibm.com</font></u></a>> wrote:
<p><font size="2" face="Courier 10 Pitch">experts,<br>
<br>
i have a new (not my first) Xymon installation (4.3.28 on RH 7.4, self compiled).<br>
All looks good, but the clients only show the network tests (conn, ssh). Only the Xymon server itself shows everything (disk, cpu,...).<br>
- all clients are RH 7.4, same install image used (i just copied over the client part with client config, create a cronjob for automatic start, so no root access nor extra install needed)<br>
- nothing suspicious in server or client logs (even in debug mode)<br>
- no ghosts<br>
- if i send a test manually (the old BB way) this shows at Xymon immediately (so the client settings are ok)<br>
- if i shutdown the xymond at server, the client logs complains:<br>
<br>
2017-10-19 05:09:18.051994 Whoops ! Failed to send message (Connection failed)<br>
2017-10-19 05:09:18.052104 -> Could not connect to Xymon </font><a href="mailto:daemon@10.137.56.43"><u><font size="2" color="#0000FF" face="Courier 10 Pitch">daemon@10.137.56.43</font></u></a><font size="2" face="Courier 10 Pitch">:1984 (Connection refused)<br>
<br>
(so obviously they communicate)<br>
<br>
Querying the xymond shows:<br>
xymon 127.0.0.1 "query clientname.ssh"<br>
green <!-- [flags:OrdastLe] --> Thu Oct 19 05:21:29 2017 ssh ok<br>
is ok, but<br>
xymon 127.0.0.1 "query clientname.disk"<br>
shows nothing<br>
<br>
Any ideas?<br>
<br>
Norbert</font></p>
<p>_______________________________________________<br>
Xymon mailing list<u><font color="#0000FF"><br>
</font></u><a href="mailto:Xymon@xymon.com"><u><font color="#0000FF">Xymon@xymon.com</font></u></a><u><font color="#0000FF"><br>
</font></u><a href="http://lists.xymon.com/mailman/listinfo/xymon"><u><font color="#0000FF">http://lists.xymon.com/mailman/listinfo/xymon</font></u></a></p>
</ul>
</ul>
</ul>
</ul>
<tt><font size="2">_______________________________________________<br>
Xymon mailing list</font></tt><tt><u><font size="2" color="#0000FF"><br>
</font></u></tt><a href="mailto:Xymon@xymon.com"><tt><u><font size="2" color="#0000FF">Xymon@xymon.com</font></u></tt></a><tt><u><font size="2" color="#0000FF"><br>
</font></u></tt><a href="http://lists.xymon.com/mailman/listinfo/xymon"><tt><u><font size="2" color="#0000FF">http://lists.xymon.com/mailman/listinfo/xymon</font></u></tt></a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Xymon mailing list<u><font color="#0000FF"><br>
</font></u><a href="mailto:Xymon@xymon.com"><u><font color="#0000FF">Xymon@xymon.com</font></u></a><br>
<a href="http://lists.xymon.com/mailman/listinfo/xymon">http://lists.xymon.com/mailman/listinfo/xymon</a></ul>
</ul>
<tt><font size="2">_______________________________________________<br>
Xymon mailing list<br>
Xymon@xymon.com<br>
</font></tt><tt><font size="2"><a href="http://lists.xymon.com/mailman/listinfo/xymon">http://lists.xymon.com/mailman/listinfo/xymon</a></font></tt><tt><font size="2"><br>
</font></tt><br>
<br>
<br>
</div>
</div>
</body>
</html>