<div dir="ltr"><div><div><div>Just had a more careful look at the truss output.<br></div>It's trying to open every font or locale in existence.<br></div><div>Is this normal?<br></div>--- snip ---<br><br><span style="font-family:courier new,monospace">open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Italic.afm", O_RDONLY) Err#2 ENOENT<br>
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Italic.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Italic.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR<br>
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/rsrc", O_RDONLY) Err#20 ENOTDIR<br>open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Italic.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Italic.afm", O_RDONLY) Err#2 ENOENT<br>
munmap(0xFE0A0000, 15723) = 0<br>xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm", 0x0803F2C0) = 0<br>open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm", O_RDONLY) = 7<br>
fcntl(7, F_SETFD, 0x00000001) = 0<br>fxstat(2, 7, 0x0803F120) = 0<br>mmap(0x00000000, 15742, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000<br>close(7) = 0<br>open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Roman.afm", O_RDONLY) Err#2 ENOENT<br>
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Roman.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Roman.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR<br>
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/rsrc", O_RDONLY) Err#20 ENOTDIR<br>open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Roman.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Roman.afm", O_RDONLY) Err#2 ENOENT<br>
munmap(0xFE0A0000, 15742) = 0<br>xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", 0x0803F2C0) = 0<br>open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", O_RDONLY) = 7<br>fcntl(7, F_SETFD, 0x00000001) = 0<br>
fxstat(2, 7, 0x0803F120) = 0<br>mmap(0x00000000, 17286, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000<br>close(7) = 0<br>open("/usr/openwin/lib/X11/fonts/F3/afm/._Rockwell-Bold.afm", O_RDONLY) Err#2 ENOENT<br>
open("/usr/openwin/lib/X11/fonts/F3/afm/%Rockwell-Bold.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Rockwell-Bold.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR<br>
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/rsrc", O_RDONLY) Err#20 ENOTDIR<br>open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Rockwell-Bold.afm", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Rockwell-Bold.afm", O_RDONLY) Err#2 ENOENT</span><br>
<br><br></div>--- snip ---<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 29 April 2014 12:07, Vernon Everett <span dir="ltr"><<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi Jeremy<br><br></div>Thanks for the tips.<br></div>Manually, it runs in under a second.<br>
<br></div>And not using the "--no-cache" option?<br></div>--- snip ---<br><span style="font-family:courier new,monospace"># "rrdstatus" updates RRD files with information that arrives as "status" messages.<br>
[rrdstatus]<br> ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg<br> NEEDS xymond<br> CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd<br><br># "rrddata" updates RRD files with information that arrives as "data" messages.<br>
[rrddata]<br> ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg<br> NEEDS xymond<br> CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log xymond_rrd --rrddir=$XYMONVAR/rrd</span><br><br></div>--- snip ---<br>
<div><div><br></div><div>Something I did notice, that may be relevant.<br></div><div>As I mentioned earlier, viewing the trends column spawns a host of showgraph.cgi processes.<br></div><div>I ran a truss on one of them, and I see this group of errors repeated often.<br>
</div><div>--- snip ---<br></div><div><span style="font-family:courier new,monospace">open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT<br>
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT<br>open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc", O_RDONLY) Err#20 ENOTDIR<br>
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc", O_RDONLY) Err#20 ENOTDIR<br>open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT<br>
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps", O_RDONLY) Err#2 ENOENT<br>munmap(0xFE380000, 399) = 0<br>xstat(2, "/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", 0x0803F2D0) = 0<br>
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps", O_RDONLY) = 7<br>fcntl(7, F_SETFD, 0x00000001) = 0<br>fxstat(2, 7, 0x0803F130) = 0<br>mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000<br>
close(7) = 0</span><br>--- snip ---<br></div><div><br></div><div>This looks like it's trying to find locale related stuff. <br></div><div>However, I can find no reference to any <span style="font-family:courier new,monospace">zh_CN </span>locales anywhere on my system.</div>
<div><br></div><div>Any ideas?<br><br></div><div>Regards<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div>Vernon<br></div></font></span></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote"><div class="">On 29 April 2014 10:46, Jeremy Laidman <span dir="ltr"><<a href="mailto:jlaidman@rebel-it.com.au" target="_blank">jlaidman@rebel-it.com.au</a>></span> wrote:<br>
</div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Is it quick to run the showgraph.cgi binary manually? Like so:<div><br></div>
<div>SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=<a href="http://hostname.example.com" target="_blank">hostname.example.com</a>&service=la&graph=hourly' /path/to/showgraph.cgi<br>
</div><div><br></div><div>If that's slow, probably not an Apache problem.</div><div><br></div><div>Are you running xymond_rrd with the "--no-cache" option?</div><div><br></div><div>J</div><div> </div></div>
<div class="gmail_extra">
<br><br><div class="gmail_quote"><div><div>On 29 April 2014 12:20, Vernon Everett <span dir="ltr"><<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
<div dir="ltr"><div><div><div>Hi<br><br></div>Nope, no IPv6.<br></div>Did a little constructive Googling on this error, and it's a fairly well known one.<br></div><div>All the notes I can find tell me to add <br></div>
<div><span style="font-family:courier new,monospace">Listen <a href="http://0.0.0.0:443" target="_blank">0.0.0.0:443</a></span><br></div><div>to my httpd.conf file, and the error will go away.<br><br></div><div>But that would be too easy. Not working in my world. And I still get the error. :-/<br>
</div><div>In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and it just takes longer before showing the errors after a restart.<br></div><div><br></div><div>But, I think this error is a red herring, and is probably quite unrelated to the slow, high CPU utilisation of the graph rendering.<br>
</div><div>Lots of junk in the error.log file I can fix with an appropriate entry in logadm.<br></div><div>Taking forever to draw my graphs is a pain.<br><br></div><div>Has anybody experienced problems with the rrd graphs taking long to render?<br>
<br></div><div>Regards<span><font color="#888888"><br></font></span></div><span><font color="#888888"><div>Vernon<br></div><div><br></div></font></span></div><div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On 25 April 2014 03:02, Galen Johnson <span dir="ltr"><<a href="mailto:Galen.Johnson@sas.com" target="_blank">Galen.Johnson@sas.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div name="divtagdefaultwrapper" style="font-size:12pt;margin:0;font-family:Calibri,Arial,Helvetica,sans-serif">
That looks like IPv6. Is IPv6 enabled? Is it actually listening on IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on my systems since using it is a mixed bag currently...especially on solaris (when I was managing it).
<div><br>
</div>
<div>=G=</div>
<div><br>
<div style="color:rgb(40,40,40)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Vernon Everett <<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>><br>
<b>Sent:</b> Thursday, April 24, 2014 4:46 AM<br>
<b>To:</b> Galen Johnson<br>
<b>Cc:</b> Xymon mailinglist<br>
<b>Subject:</b> Re: [Xymon] High CPU Load Rendering Graphs</font>
<div> </div>
</div><div><div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>Yes, and no.<br>
<br>
</div>
Have just enabled the status page in the web config, and it appears to have got rid of the one error message.<br>
</div>
(Not sure how I missed the config change at initial install time)<br>
<br>
</div>
But, I still get the other error<br>
[warn] (128)Network is unreachable: connect to listener on [::]:443<br>
</div>
And my graphs still take way too long to render, and send my CPU utilisation through the roof.<br>
<br>
</div>
Public holiday here tomorrow, so only back at this client on Tuesday.<br>
<br>
</div>
Thanks<br>
</div>
Vernon<br>
<div>
<div><br>
<br>
<br>
<br>
<br>
<div>
<div>
<div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On 24 April 2014 10:27, Galen Johnson <span dir="ltr"><<a href="mailto:Galen.Johnson@sas.com" target="_blank">Galen.Johnson@sas.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div name="divtagdefaultwrapper" style="font-size:12pt;margin:0;font-family:Calibri,Arial,Helvetica,sans-serif">
Do you have apache trending graphs enabled? If so, did you enable the status page in your apache configs?
<div><br>
</div>
<div>=G=</div>
<div><br>
<div style="color:rgb(40,40,40)">
<hr style="display:inline-block;width:98%">
<div dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Xymon <<a href="mailto:xymon-bounces@xymon.com" target="_blank">xymon-bounces@xymon.com</a>> on behalf of Vernon Everett <<a href="mailto:everett.vernon@gmail.com" target="_blank">everett.vernon@gmail.com</a>><br>
<b>Sent:</b> Wednesday, April 23, 2014 10:17 PM<br>
<b>To:</b> Xymon mailinglist<br>
<b>Subject:</b> [Xymon] High CPU Load Rendering Graphs</font>
<div> </div>
</div>
<div>
<div>
<div>
<div dir="ltr">
<div>Hi all<br>
<br>
</div>
My Xymon server 4.3.10 is burning the CPU cycles when we view multiple graphs, like the trends page, and takes about 5 seconds to render a single graph in a single-graph page view.<br>
<br clear="all">
<div>
<div>It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5..<br>
<span style="font-family:courier new,monospace"><br>
Version Location Tag<br>
-------------------------------- --------------------------<br>
Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1<br>
</span><br>
</div>
<div>Not a very powerful box, and a bit dated, but I have seen significantly better performance on far lesser systems.<br>
</div>
<div>So I am not really thinking the issue is with the hardware.<br>
</div>
<div>It's been slow since it was installed. <br>
If I view the trends column, I can see the CPU load jump from below 1 to over 10 at times.<br>
</div>
<div>Running prstat or top in another window while viewing the trends column, the process ranking by CPU gets dominated by showgraph.cgi, owned by the web server user.
<br>
</div>
<div>Top under normal conditions.<br>
</div>
<div><font size="1"><span style="font-family:courier new,monospace">CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0% swap</span></font><br>
Top rendering the trends column.<br>
<font size="1"><span style="font-family:courier new,monospace">CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0% swap</span></font><br>
<br>
</div>
<div>Also getting this error<br>
<span style="font-family:courier new,monospace">(128)Network is unreachable: connect to listener on [::]:443</span><br>
</div>
<div>in my Apache error.log file, repeated every second while rendering the graphs.<br>
</div>
<div>And from time to time, I get this one. <br>
<span style="font-family:courier new,monospace">File does not exist: /opt/csw/apache2/share/htdocs/server-status</span><br>
</div>
<div><br>
</div>
<div>Anybody seen anything like this?<br>
</div>
<div>Perhaps know of somewhere I can look for more info?<br>
<br>
</div>
<div>I have looked at this <a href="http://lists.xymon.com/archive/2014-January/038780.html" target="_blank">
http://lists.xymon.com/archive/2014-January/038780.html</a><br>
</div>
<div>But it doesn't seem relevant. Only 2 errant files, and deleting them made absolutely no difference.<br>
</div>
<div><br>
</div>
<div>Other info that may be important....<br>
<span style="font-family:courier new,monospace">bash-3.00# ./httpd -v<br>
Server version: Apache/2.2.22 (Unix)<br>
Server built: Jun 1 2012 05:09:20<br>
bash-3.00# ./httpd -V<br>
Server version: Apache/2.2.22 (Unix)<br>
Server built: Jun 1 2012 05:09:20<br>
Server's Module Magic Number: 20051115:30<br>
Server loaded: APR 1.4.5, APR-Util 1.3.12<br>
Compiled using: APR 1.4.6, APR-Util 1.3.12<br>
Architecture: 32-bit<br>
Server MPM: Prefork<br>
threaded: no<br>
forked: yes (variable process count)<br>
Server compiled with....<br>
-D APACHE_MPM_DIR="server/mpm/prefork"<br>
-D APR_HAS_SENDFILE<br>
-D APR_HAS_MMAP<br>
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)<br>
-D APR_USE_FCNTL_SERIALIZE<br>
-D APR_USE_PTHREAD_SERIALIZE<br>
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT<br>
-D APR_HAS_OTHER_CHILD<br>
-D AP_HAVE_RELIABLE_PIPED_LOGS<br>
-D DYNAMIC_MODULE_LIMIT=128<br>
-D HTTPD_ROOT="/opt/csw/apache2"<br>
-D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec"<br>
-D DEFAULT_PIDLOG="/var/run/httpd.pid"<br>
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"<br>
-D DEFAULT_LOCKFILE="/var/run/accept.lock"<br>
-D DEFAULT_ERRORLOG="logs/error_log"<br>
-D AP_TYPES_CONFIG_FILE="etc/mime.types"<br>
-D SERVER_CONFIG_FILE="etc/httpd.conf"</span><br>
<br>
</div>
<div>Thanks<br>
</div>
<div>Vernon<br>
</div>
<div><br>
<br>
-- <br>
<span>"Accept the challenges so that you can feel the exhilaration of victory"</span>
<div><span>- General George Patton</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<span>"Accept the challenges so that you can feel the exhilaration of victory"</span>
<div><span>- General George Patton</span></div>
</div>
</div>
</div></div></div>
</div>
</div>
</div>
</blockquote></div><br><br clear="all"><br>-- <br><span>"Accept the challenges so that you can feel the exhilaration of victory"</span><div><span>- General George Patton</span></div>
</div>
</div></div><br></div></div>_______________________________________________<br>
Xymon mailing list<br>
<a href="mailto:Xymon@xymon.com" target="_blank">Xymon@xymon.com</a><br>
<a href="http://lists.xymon.com/mailman/listinfo/xymon" target="_blank">http://lists.xymon.com/mailman/listinfo/xymon</a><br>
<br></blockquote></div><br></div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><br>-- <br><span>"Accept the challenges so that you can feel the exhilaration of victory"</span><div><span>- General George Patton</span></div>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><span>"Accept the challenges so that you can feel the exhilaration of victory"</span><div><span>- General George Patton</span></div>
</div>