[Xymon] High CPU Load Rendering Graphs
Vernon Everett
everett.vernon at gmail.com
Tue Apr 29 06:21:51 CEST 2014
Just had a more careful look at the truss output.
It's trying to open every font or locale in existence.
Is this normal?
--- snip ---
open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Italic.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Italic.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
munmap(0xFE0A0000, 15723) = 0
xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm",
0x0803F2C0) = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm", O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001) = 0
fxstat(2, 7, 0x0803F120) = 0
mmap(0x00000000, 15742, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000
close(7) = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Roman.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Roman.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/rsrc", O_RDONLY)
Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
munmap(0xFE0A0000, 15742) = 0
xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", 0x0803F2C0)
= 0
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001) = 0
fxstat(2, 7, 0x0803F120) = 0
mmap(0x00000000, 17286, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000
close(7) = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/._Rockwell-Bold.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Rockwell-Bold.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/rsrc", O_RDONLY)
Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
--- snip ---
On 29 April 2014 12:07, Vernon Everett <everett.vernon at gmail.com> wrote:
> Hi Jeremy
>
> Thanks for the tips.
> Manually, it runs in under a second.
>
> And not using the "--no-cache" option?
> --- snip ---
> # "rrdstatus" updates RRD files with information that arrives as "status"
> messages.
> [rrdstatus]
> ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
> NEEDS xymond
> CMD xymond_channel --channel=status
> --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd
>
> # "rrddata" updates RRD files with information that arrives as "data"
> messages.
> [rrddata]
> ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
> NEEDS xymond
> CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log
> xymond_rrd --rrddir=$XYMONVAR/rrd
>
> --- snip ---
>
> Something I did notice, that may be relevant.
> As I mentioned earlier, viewing the trends column spawns a host of
> showgraph.cgi processes.
> I ran a truss on one of them, and I see this group of errors repeated
> often.
> --- snip ---
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc",
> O_RDONLY) Err#20 ENOTDIR
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc",
> O_RDONLY) Err#20 ENOTDIR
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> munmap(0xFE380000, 399) = 0
> xstat(2,
> "/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
> 0x0803F2D0) = 0
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
> O_RDONLY) = 7
> fcntl(7, F_SETFD, 0x00000001) = 0
> fxstat(2, 7, 0x0803F130) = 0
> mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000
> close(7) = 0
> --- snip ---
>
> This looks like it's trying to find locale related stuff.
> However, I can find no reference to any zh_CN locales anywhere on my
> system.
>
> Any ideas?
>
> Regards
> Vernon
>
>
> On 29 April 2014 10:46, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
>
>> Is it quick to run the showgraph.cgi binary manually? Like so:
>>
>> SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly'
>> /path/to/showgraph.cgi
>>
>> If that's slow, probably not an Apache problem.
>>
>> Are you running xymond_rrd with the "--no-cache" option?
>>
>> J
>>
>>
>>
>> On 29 April 2014 12:20, Vernon Everett <everett.vernon at gmail.com> wrote:
>>
>>> Hi
>>>
>>> Nope, no IPv6.
>>> Did a little constructive Googling on this error, and it's a fairly well
>>> known one.
>>> All the notes I can find tell me to add
>>> Listen 0.0.0.0:443
>>> to my httpd.conf file, and the error will go away.
>>>
>>> But that would be too easy. Not working in my world. And I still get the
>>> error. :-/
>>> In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and
>>> it just takes longer before showing the errors after a restart.
>>>
>>> But, I think this error is a red herring, and is probably quite
>>> unrelated to the slow, high CPU utilisation of the graph rendering.
>>> Lots of junk in the error.log file I can fix with an appropriate entry
>>> in logadm.
>>> Taking forever to draw my graphs is a pain.
>>>
>>> Has anybody experienced problems with the rrd graphs taking long to
>>> render?
>>>
>>> Regards
>>> Vernon
>>>
>>>
>>>
>>> On 25 April 2014 03:02, Galen Johnson <Galen.Johnson at sas.com> wrote:
>>>
>>>> That looks like IPv6. Is IPv6 enabled? Is it actually listening on
>>>> IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on
>>>> my systems since using it is a mixed bag currently...especially on solaris
>>>> (when I was managing it).
>>>>
>>>> =G=
>>>>
>>>> ------------------------------
>>>> *From:* Vernon Everett <everett.vernon at gmail.com>
>>>> *Sent:* Thursday, April 24, 2014 4:46 AM
>>>> *To:* Galen Johnson
>>>> *Cc:* Xymon mailinglist
>>>> *Subject:* Re: [Xymon] High CPU Load Rendering Graphs
>>>>
>>>> Yes, and no.
>>>>
>>>> Have just enabled the status page in the web config, and it appears to
>>>> have got rid of the one error message.
>>>> (Not sure how I missed the config change at initial install time)
>>>>
>>>> But, I still get the other error
>>>> [warn] (128)Network is unreachable: connect to listener on [::]:443
>>>> And my graphs still take way too long to render, and send my CPU
>>>> utilisation through the roof.
>>>>
>>>> Public holiday here tomorrow, so only back at this client on Tuesday.
>>>>
>>>> Thanks
>>>> Vernon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 24 April 2014 10:27, Galen Johnson <Galen.Johnson at sas.com> wrote:
>>>>
>>>>> Do you have apache trending graphs enabled? If so, did you enable
>>>>> the status page in your apache configs?
>>>>>
>>>>> =G=
>>>>>
>>>>> ------------------------------
>>>>> *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett <
>>>>> everett.vernon at gmail.com>
>>>>> *Sent:* Wednesday, April 23, 2014 10:17 PM
>>>>> *To:* Xymon mailinglist
>>>>> *Subject:* [Xymon] High CPU Load Rendering Graphs
>>>>>
>>>>> Hi all
>>>>>
>>>>> My Xymon server 4.3.10 is burning the CPU cycles when we view
>>>>> multiple graphs, like the trends page, and takes about 5 seconds to render
>>>>> a single graph in a single-graph page view.
>>>>>
>>>>> It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5..
>>>>>
>>>>> Version Location Tag
>>>>> -------------------------------- --------------------------
>>>>> Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1
>>>>>
>>>>> Not a very powerful box, and a bit dated, but I have seen
>>>>> significantly better performance on far lesser systems.
>>>>> So I am not really thinking the issue is with the hardware.
>>>>> It's been slow since it was installed.
>>>>> If I view the trends column, I can see the CPU load jump from below 1
>>>>> to over 10 at times.
>>>>> Running prstat or top in another window while viewing the trends
>>>>> column, the process ranking by CPU gets dominated by showgraph.cgi, owned
>>>>> by the web server user.
>>>>> Top under normal conditions.
>>>>> CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait,
>>>>> 0.0% swap
>>>>> Top rendering the trends column.
>>>>> CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0%
>>>>> swap
>>>>>
>>>>> Also getting this error
>>>>> (128)Network is unreachable: connect to listener on [::]:443
>>>>> in my Apache error.log file, repeated every second while rendering
>>>>> the graphs.
>>>>> And from time to time, I get this one.
>>>>> File does not exist: /opt/csw/apache2/share/htdocs/server-status
>>>>>
>>>>> Anybody seen anything like this?
>>>>> Perhaps know of somewhere I can look for more info?
>>>>>
>>>>> I have looked at this
>>>>> http://lists.xymon.com/archive/2014-January/038780.html
>>>>> But it doesn't seem relevant. Only 2 errant files, and deleting them
>>>>> made absolutely no difference.
>>>>>
>>>>> Other info that may be important....
>>>>> bash-3.00# ./httpd -v
>>>>> Server version: Apache/2.2.22 (Unix)
>>>>> Server built: Jun 1 2012 05:09:20
>>>>> bash-3.00# ./httpd -V
>>>>> Server version: Apache/2.2.22 (Unix)
>>>>> Server built: Jun 1 2012 05:09:20
>>>>> Server's Module Magic Number: 20051115:30
>>>>> Server loaded: APR 1.4.5, APR-Util 1.3.12
>>>>> Compiled using: APR 1.4.6, APR-Util 1.3.12
>>>>> Architecture: 32-bit
>>>>> Server MPM: Prefork
>>>>> threaded: no
>>>>> forked: yes (variable process count)
>>>>> Server compiled with....
>>>>> -D APACHE_MPM_DIR="server/mpm/prefork"
>>>>> -D APR_HAS_SENDFILE
>>>>> -D APR_HAS_MMAP
>>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>>> -D APR_USE_FCNTL_SERIALIZE
>>>>> -D APR_USE_PTHREAD_SERIALIZE
>>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>>> -D APR_HAS_OTHER_CHILD
>>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>>> -D DYNAMIC_MODULE_LIMIT=128
>>>>> -D HTTPD_ROOT="/opt/csw/apache2"
>>>>> -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec"
>>>>> -D DEFAULT_PIDLOG="/var/run/httpd.pid"
>>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>>> -D DEFAULT_LOCKFILE="/var/run/accept.lock"
>>>>> -D DEFAULT_ERRORLOG="logs/error_log"
>>>>> -D AP_TYPES_CONFIG_FILE="etc/mime.types"
>>>>> -D SERVER_CONFIG_FILE="etc/httpd.conf"
>>>>>
>>>>> Thanks
>>>>> Vernon
>>>>>
>>>>>
>>>>> --
>>>>> "Accept the challenges so that you can feel the exhilaration of
>>>>> victory"
>>>>> - General George Patton
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> "Accept the challenges so that you can feel the exhilaration of victory"
>>>> - General George Patton
>>>>
>>>
>>>
>>>
>>> --
>>> "Accept the challenges so that you can feel the exhilaration of victory"
>>> - General George Patton
>>>
>>> _______________________________________________
>>> Xymon mailing list
>>> Xymon at xymon.com
>>> http://lists.xymon.com/mailman/listinfo/xymon
>>>
>>>
>>
>
>
> --
> "Accept the challenges so that you can feel the exhilaration of victory"
> - General George Patton
>
--
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20140429/f9458f1c/attachment.html>
More information about the Xymon
mailing list