[Xymon] xymonproxy crashing under the load

Jonathan Trott jtrott at dancrai.com
Tue May 31 08:22:50 CEST 2016 

Hi.

We have a large xymon setup that averages 25 messages/sec for xymond.
I am in the process of migrating to a new xymon server and was planning on 
using a front end xymonproxy process on a third box to keep traffic going 
to both old and new boxes while I migrated across the old data (primarily 
the rrd data as that requires conversion from 32bit to 64bit).
The xymonproxy is working fine under light load (20 or so client machines) 
but when I modify the firewall port forward to point the incoming port 
1984 traffic to it it crashes after a few seconds.

2016-05-31 16:08:19.793720 xymonproxy version 4.3.27-1.el7.terabithia 
starting
2016-05-31 16:08:19.793791 Listening on 0.0.0.0:1984
2016-05-31 16:08:19.793796 Sending to 2 Xymon server(s): 
172.16.100.103:1984 172.16.100.101:1984 
2016-05-31 16:09:23.010861  conn 0: server 1 (172.16.100.101:1984) not 
responding, message lost
2016-05-31 16:09:23.011363  conn 0, socket -1: write shutdown failed: Bad 
file descriptor
2016-05-31 16:09:24.012798  conn 0: server 1 (172.16.100.101:1984) not 
responding, message lost
2016-05-31 16:09:25.014274  conn 0, socket -1: write shutdown failed: Bad 
file descriptor
*** buffer overflow detected ***: xymonproxy terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f234c396597]
/lib64/libc.so.6(+0x10c750)[0x7f234c394750]
/lib64/libc.so.6(+0x10e507)[0x7f234c396507]
xymonproxy[0x403e11]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f234c2a9b15]
xymonproxy[0x406579]
======= Memory map: ========
00400000-0041d000 r-xp 00000000 fd:00 25459501 
/usr/libexec/xymon/xymonproxy
0061d000-0061e000 r--p 0001d000 fd:00 25459501 
/usr/libexec/xymon/xymonproxy
0061e000-0061f000 rw-p 0001e000 fd:00 25459501 
/usr/libexec/xymon/xymonproxy
0061f000-00646000 rw-p 00000000 00:00 0 
02472000-025f4000 rw-p 00000000 00:00 0 [heap]
7f234a874000-7f234a889000 r-xp 00000000 fd:00 18879360 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f234a889000-7f234aa88000 ---p 00015000 fd:00 18879360 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f234aa88000-7f234aa89000 r--p 00014000 fd:00 18879360 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f234aa89000-7f234aa8a000 rw-p 00015000 fd:00 18879360 
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f234aa8a000-7f234aaae000 r-xp 00000000 fd:00 16856000 
/usr/lib64/liblzma.so.5.0.99
7f234aaae000-7f234acad000 ---p 00024000 fd:00 16856000 
/usr/lib64/liblzma.so.5.0.99
7f234acad000-7f234acae000 r--p 00023000 fd:00 16856000 
/usr/lib64/liblzma.so.5.0.99
7f234acae000-7f234acaf000 rw-p 00024000 fd:00 16856000 
/usr/lib64/liblzma.so.5.0.99
7f234acaf000-7f234acd0000 r-xp 00000000 fd:00 16915048 
/usr/lib64/libselinux.so.1
7f234acd0000-7f234aed0000 ---p 00021000 fd:00 16915048 
/usr/lib64/libselinux.so.1
7f234aed0000-7f234aed1000 r--p 00021000 fd:00 16915048 
/usr/lib64/libselinux.so.1
7f234aed1000-7f234aed2000 rw-p 00022000 fd:00 16915048 
/usr/lib64/libselinux.so.1
7f234aed2000-7f234aed4000 rw-p 00000000 00:00 0 
7f234aed4000-7f234aeea000 r-xp 00000000 fd:00 16855963 
/usr/lib64/libresolv-2.17.so
7f234aeea000-7f234b0ea000 ---p 00016000 fd:00 16855963 
/usr/lib64/libresolv-2.17.so
7f234b0ea000-7f234b0eb000 r--p 00016000 fd:00 16855963 
/usr/lib64/libresolv-2.17.so
7f234b0eb000-7f234b0ec000 rw-p 00017000 fd:00 16855963 
/usr/lib64/libresolv-2.17.so
7f234b0ec000-7f234b0ee000 rw-p 00000000 00:00 0 
7f234b0ee000-7f234b0f1000 r-xp 00000000 fd:00 17026913 
/usr/lib64/libkeyutils.so.1.5
7f234b0f1000-7f234b2f0000 ---p 00003000 fd:00 17026913 
/usr/lib64/libkeyutils.so.1.5
7f234b2f0000-7f234b2f1000 r--p 00002000 fd:00 17026913 
/usr/lib64/libkeyutils.so.1.5
7f234b2f1000-7f234b2f2000 rw-p 00003000 fd:00 17026913 
/usr/lib64/libkeyutils.so.1.5
7f234b2f2000-7f234b2ff000 r-xp 00000000 fd:00 18086631 
/usr/lib64/libkrb5support.so.0.1
7f234b2ff000-7f234b4ff000 ---p 0000d000 fd:00 18086631 
/usr/lib64/libkrb5support.so.0.1
7f234b4ff000-7f234b500000 r--p 0000d000 fd:00 18086631 
/usr/lib64/libkrb5support.so.0.1
7f234b500000-7f234b501000 rw-p 0000e000 fd:00 18086631 
/usr/lib64/libkrb5support.so.0.1
7f234b501000-7f234b517000 r-xp 00000000 fd:00 16855957 
/usr/lib64/libpthread-2.17.so
7f234b517000-7f234b717000 ---p 00016000 fd:00 16855957 
/usr/lib64/libpthread-2.17.so
7f234b717000-7f234b718000 r--p 00016000 fd:00 16855957 
/usr/lib64/libpthread-2.17.so
7f234b718000-7f234b719000 rw-p 00017000 fd:00 16855957 
/usr/lib64/libpthread-2.17.so
7f234b719000-7f234b71d000 rw-p 00000000 00:00 0 
7f234b71d000-7f234b720000 r-xp 00000000 fd:00 17867018 
/usr/lib64/libdl-2.17.so
7f234b720000-7f234b91f000 ---p 00003000 fd:00 17867018 
/usr/lib64/libdl-2.17.so
7f234b91f000-7f234b920000 r--p 00002000 fd:00 17867018 
/usr/lib64/libdl-2.17.so
7f234b920000-7f234b921000 rw-p 00003000 fd:00 17867018 
/usr/lib64/libdl-2.17.so
7f234b921000-7f234b950000 r-xp 00000000 fd:00 17149883 
/usr/lib64/libk5crypto.so.3.1
7f234b950000-7f234bb4f000 ---p 0002f000 fd:00 17149883 
/usr/lib64/libk5crypto.so.3.1
7f234bb4f000-7f234bb51000 r--p 0002e000 fd:00 17149883 
/usr/lib64/libk5crypto.so.3.1
7f234bb51000-7f234bb52000 rw-p 00030000 fd:00 17149883 
/usr/lib64/libk5crypto.so.3.1
7f234bb52000-7f234bb53000 rw-p 00000000 00:00 0 
7f234bb53000-7f234bb56000 r-xp 00000000 fd:00 16915067 
/usr/lib64/libcom_err.so.2.1
7f234bb56000-7f234bd55000 ---p 00003000 fd:00 16915067 
/usr/lib64/libcom_err.so.2.1
7f234bd55000-7f234bd56000 r--p 00002000 fd:00 16915067 
/usr/lib64/libcom_err.so.2.1
7f234bd56000-7f234bd57000 rw-p 00003000 fd:00 16915067 
/usr/lib64/libcom_err.so.2.1
7f234bd57000-7f234be2c000 r-xp 00000000 fd:00 17166500 
/usr/lib64/libkrb5.so.3.3
7f234be2c000-7f234c02c000 ---p 000d5000 fd:00 17166500 
/usr/lib64/libkrb5.so.3.3
7f234c02c000-7f234c039000 r--p 000d5000 fd:00 17166500 
/usr/lib64/libkrb5.so.3.3
7f234c039000-7f234c03c000 rw-p 000e2000 fd:00 17166500 
/usr/lib64/libkrb5.so.3.3
7f234c03c000-7f234c085000 r-xp 00000000 fd:00 17177116 
/usr/lib64/libgssapi_krb5.so.2.2
7f234c085000-7f234c285000 ---p 00049000 fd:00 17177116 
/usr/lib64/libgssapi_krb5.so.2.2
7f234c285000-7f234c286000 r--p 00049000 fd:00 17177116 
/usr/lib64/libgssapi_krb5.so.2.2
7f234c286000-7f234c288000 rw-p 0004a000 fd:00 17177116 
/usr/lib64/libgssapi_krb5.so.2.2
7f234c288000-7f234c43f000 r-xp 00000000 fd:00 16855941 
/usr/lib64/libc-2.17.so
7f234c43f000-7f234c63f000 ---p 001b7000 fd:00 16855941 
/usr/lib64/libc-2.17.so
7f234c63f000-7f234c643000 r--p 001b7000 fd:00 16855941 
/usr/lib64/libc-2.17.so
7f234c643000-7f234c645000 rw-p 001bb000 fd:00 16855941 
/usr/lib64/libc-2.17.so
7f234c645000-7f234c64a000 rw-p 00000000 00:00 0 
7f234c64a000-7f234c6aa000 r-xp 00000000 fd:00 18095822 
/usr/lib64/libpcre.so.1.2.0
7f234c6aa000-7f234c8a9000 ---p 00060000 fd:00 18095822 
/usr/lib64/libpcre.so.1.2.0
7f234c8a9000-7f234c8aa000 r--p 0005f000 fd:00 18095822 
/usr/lib64/libpcre.so.1.2.0
7f234c8aa000-7f234c8ab000 rw-p 00060000 fd:00 18095822 
/usr/lib64/libpcre.so.1.2.0
7f234c8ab000-7f234ca69000 r-xp 00000000 fd:00 17026905 
/usr/lib64/libcrypto.so.1.0.1e
7f234ca69000-7f234cc69000 ---p 001be000 fd:00 17026905 
/usr/lib64/libcrypto.so.1.0.1e
7f234cc69000-7f234cc83000 r--p 001be000 fd:00 17026905 
/usr/lib64/libcrypto.so.1.0.1e
7f234cc83000-7f234cc8f000 rw-p 001d8000 fd:00 17026905 
/usr/lib64/libcrypto.so.1.0.1e
7f234cc8f000-7f234cc93000 rw-p 00000000 00:00 0 
7f234cc93000-7f234ccf6000 r-xp 00000000 fd:00 18093381 
/usr/lib64/libssl.so.1.0.1e
7f234ccf6000-7f234cef5000 ---p 00063000 fd:00 18093381 
/usr/lib64/libssl.so.1.0.1e
7f234cef5000-7f234cef9000 r--p 00062000 fd:00 18093381 
/usr/lib64/libssl.so.1.0.1e
7f234cef9000-7f234cf00000 rw-p 00066000 fd:00 18093381 
/usr/lib64/libssl.so.1.0.1e
7f234cf00000-7f234cf0b000 r-xp 00000000 fd:00 17867002 
/usr/lib64/liblz4.so.1.7.1
7f234cf0b000-7f234d10a000 ---p 0000b000 fd:00 17867002 
/usr/lib64/liblz4.so.1.7.1
7f234d10a000-7f234d10b000 r--p 0000a000 fd:00 17867002 
/usr/lib64/liblz4.so.1.7.1
7f234d10b000-7f234d10c000 rw-p 00000000 00:00 0 
7f234d10c000-7f234d121000 r-xp 00000000 fd:00 16915045 
/usr/lib64/libz.so.1.2.7
7f234d121000-7f234d320000 ---p 00015000 fd:00 16915045 
/usr/lib64/libz.so.1.2.7
7f234d320000-7f234d321000 r--p 00014000 fd:00 16915045 
/usr/lib64/libz.so.1.2.7
7f234d321000-7f234d322000 rw-p 00015000 fd:00 16915045 
/usr/lib64/libz.so.1.2.7
7f234d322000-7f234d343000 r-xp 00000000 fd:00 18043068 
/usr/lib64/ld-2.17.so
7f234d52e000-7f234d538000 rw-p 00000000 00:00 0 
7f234d540000-7f234d543000 rw-p 00000000 00:00 0 
7f234d543000-7f234d544000 r--p 00021000 fd:00 18043068 
/usr/lib64/ld-2.17.so
7f234d544000-7f234d545000 rw-p 00022000 fd:00 18043068 
/usr/lib64/ld-2.17.so
7f234d545000-7f234d546000 rw-p 00000000 00:00 0 
7ffdfcb9d000-7ffdfcbbf000 rw-p 00000000 00:00 0 [stack]
7ffdfcbdb000-7ffdfcbdd000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
2016-05-31 16:09:25.092174 xymonproxy version 4.3.27-1.el7.terabithia 
starting
2016-05-31 16:09:25.092364 Listening on 0.0.0.0:1984
2016-05-31 16:09:25.092379 Sending to 2 Xymon server(s): 
172.16.100.103:1984 172.16.100.101:1984 

Xymonproxy is running on CentOS 7 with all the latest patches installed.
172.16.100.103 is xymon 4.3.16 on CentOS 5, 172.16.100.101 is xymon 
4.3.27-1.el7.terabithia on CentOS 7.

Any ideas on how to troubleshoot this further?

Thanks,
JT
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 8172 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20160531/aa011828/attachment.bin>

More information about the Xymon mailing list