[Xymon] High CPU Load Rendering Graphs

Vernon Everett everett.vernon at gmail.com
Tue Apr 29 06:21:51 CEST 2014


Just had a more careful look at the truss output.
It's trying to open every font or locale in existence.
Is this normal?
--- snip ---

open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Italic.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Italic.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Italic.afm/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Italic.afm",
O_RDONLY) Err#2 ENOENT
munmap(0xFE0A0000, 15723)        = 0
xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm",
0x0803F2C0) = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm", O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001)    = 0
fxstat(2, 7, 0x0803F120)        = 0
mmap(0x00000000, 15742, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000
close(7)              = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/._Palatino-Roman.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Palatino-Roman.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Palatino-Roman.afm/rsrc", O_RDONLY)
Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Palatino-Roman.afm",
O_RDONLY) Err#2 ENOENT
munmap(0xFE0A0000, 15742)        = 0
xstat(2, "/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", 0x0803F2C0)
= 0
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm", O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001)    = 0
fxstat(2, 7, 0x0803F120)        = 0
mmap(0x00000000, 17286, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE0A0000
close(7)              = 0
open("/usr/openwin/lib/X11/fonts/F3/afm/._Rockwell-Bold.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/%Rockwell-Bold.afm", O_RDONLY)
Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.AppleDouble/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/Rockwell-Bold.afm/rsrc", O_RDONLY)
Err#20 ENOTDIR
open("/usr/openwin/lib/X11/fonts/F3/afm/resource.frk/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/X11/fonts/F3/afm/.resource/Rockwell-Bold.afm",
O_RDONLY) Err#2 ENOENT


--- snip ---


On 29 April 2014 12:07, Vernon Everett <everett.vernon at gmail.com> wrote:

> Hi Jeremy
>
> Thanks for the tips.
> Manually, it runs in under a second.
>
> And not using the "--no-cache" option?
> --- snip ---
> # "rrdstatus" updates RRD files with information that arrives as "status"
> messages.
> [rrdstatus]
>   ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
>   NEEDS xymond
>   CMD xymond_channel --channel=status
> --log=$XYMONSERVERLOGS/rrd-status.log xymond_rrd --rrddir=$XYMONVAR/rrd
>
> # "rrddata" updates RRD files with information that arrives as "data"
> messages.
> [rrddata]
>   ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
>   NEEDS xymond
>   CMD xymond_channel --channel=data   --log=$XYMONSERVERLOGS/rrd-data.log
> xymond_rrd --rrddir=$XYMONVAR/rrd
>
> --- snip ---
>
> Something I did notice, that may be relevant.
> As I mentioned earlier, viewing the trends column spawns a host of
> showgraph.cgi processes.
> I ran a truss on one of them, and I see this group of errors repeated
> often.
> --- snip ---
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc",
> O_RDONLY) Err#20 ENOTDIR
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc",
> O_RDONLY) Err#20 ENOTDIR
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps",
> O_RDONLY) Err#2 ENOENT
> munmap(0xFE380000, 399)         = 0
> xstat(2,
> "/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
> 0x0803F2D0) = 0
> open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
> O_RDONLY) = 7
> fcntl(7, F_SETFD, 0x00000001)    = 0
> fxstat(2, 7, 0x0803F130)        = 0
> mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000
> close(7)              = 0
> --- snip ---
>
> This looks like it's trying to find locale related stuff.
> However, I can find no reference to any zh_CN locales anywhere on my
> system.
>
> Any ideas?
>
> Regards
> Vernon
>
>
> On 29 April 2014 10:46, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
>
>> Is it quick to run the showgraph.cgi binary manually?  Like so:
>>
>> SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly'
>> /path/to/showgraph.cgi
>>
>> If that's slow, probably not an Apache problem.
>>
>> Are you running xymond_rrd with the "--no-cache" option?
>>
>> J
>>
>>
>>
>> On 29 April 2014 12:20, Vernon Everett <everett.vernon at gmail.com> wrote:
>>
>>> Hi
>>>
>>> Nope, no IPv6.
>>> Did a little constructive Googling on this error, and it's a fairly well
>>> known one.
>>> All the notes I can find tell me to add
>>> Listen 0.0.0.0:443
>>> to my httpd.conf file, and the error will go away.
>>>
>>> But that would be too easy. Not working in my world. And I still get the
>>> error. :-/
>>> In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and
>>> it just takes longer before showing the errors after a restart.
>>>
>>> But, I think this error is a red herring, and is probably quite
>>> unrelated to the slow, high CPU utilisation of the graph rendering.
>>> Lots of junk in the error.log file I can fix with an appropriate entry
>>> in logadm.
>>> Taking forever to draw my graphs is a pain.
>>>
>>> Has anybody experienced problems with the rrd graphs taking long to
>>> render?
>>>
>>> Regards
>>> Vernon
>>>
>>>
>>>
>>> On 25 April 2014 03:02, Galen Johnson <Galen.Johnson at sas.com> wrote:
>>>
>>>>  That looks like IPv6.  Is IPv6 enabled? Is it actually listening on
>>>> IPv6 (netstat -tan | grep -i listen | grep 443)  I typically disable it on
>>>> my systems since using it is a mixed bag currently...especially on solaris
>>>> (when I was managing it).
>>>>
>>>>  =G=
>>>>
>>>>  ------------------------------
>>>> *From:* Vernon Everett <everett.vernon at gmail.com>
>>>> *Sent:* Thursday, April 24, 2014 4:46 AM
>>>> *To:* Galen Johnson
>>>> *Cc:* Xymon mailinglist
>>>> *Subject:* Re: [Xymon] High CPU Load Rendering Graphs
>>>>
>>>>      Yes, and no.
>>>>
>>>>  Have just enabled the status page in the web config, and it appears to
>>>> have got rid of the one error message.
>>>>  (Not sure how I missed the config change at initial install time)
>>>>
>>>>  But, I still get the other error
>>>> [warn] (128)Network is unreachable: connect to listener on [::]:443
>>>>  And my graphs still take way too long to render, and send my CPU
>>>> utilisation through the roof.
>>>>
>>>>  Public holiday here tomorrow, so only back at this client on Tuesday.
>>>>
>>>>  Thanks
>>>>  Vernon
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 24 April 2014 10:27, Galen Johnson <Galen.Johnson at sas.com> wrote:
>>>>
>>>>>  Do you have apache trending graphs enabled?  If so, did you enable
>>>>> the status page in your apache configs?
>>>>>
>>>>>  =G=
>>>>>
>>>>>  ------------------------------
>>>>> *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett <
>>>>> everett.vernon at gmail.com>
>>>>> *Sent:* Wednesday, April 23, 2014 10:17 PM
>>>>> *To:* Xymon mailinglist
>>>>> *Subject:* [Xymon] High CPU Load Rendering Graphs
>>>>>
>>>>>    Hi all
>>>>>
>>>>>  My Xymon server 4.3.10 is burning the CPU cycles when we view
>>>>> multiple graphs, like the trends page, and takes about 5 seconds to render
>>>>> a single graph in a single-graph page view.
>>>>>
>>>>>  It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5..
>>>>>
>>>>> Version                          Location Tag
>>>>> -------------------------------- --------------------------
>>>>> Intel(R) Xeon(R) CPU           X5460  @ 3.16GHz CPU 1
>>>>>
>>>>>  Not a very powerful box, and a bit dated, but I have seen
>>>>> significantly better performance on far lesser systems.
>>>>>  So I am not really thinking the issue is with the hardware.
>>>>>  It's been slow since it was installed.
>>>>> If I view the trends column, I can see the CPU load jump from below 1
>>>>> to over 10 at times.
>>>>>  Running prstat or top in another window while viewing the trends
>>>>> column, the process ranking by CPU gets dominated by showgraph.cgi, owned
>>>>> by the web server user.
>>>>>  Top under normal conditions.
>>>>>  CPU states: 99.9% idle,  0.0% user,  0.1% kernel,  0.0% iowait,
>>>>> 0.0% swap
>>>>> Top rendering the trends column.
>>>>> CPU states:  0.0% idle, 93.8% user,  6.2% kernel,  0.0% iowait,  0.0%
>>>>> swap
>>>>>
>>>>>  Also getting this error
>>>>> (128)Network is unreachable: connect to listener on [::]:443
>>>>>  in my Apache error.log file, repeated every second while rendering
>>>>> the graphs.
>>>>>  And from time to time, I get this one.
>>>>> File does not exist: /opt/csw/apache2/share/htdocs/server-status
>>>>>
>>>>>  Anybody seen anything like this?
>>>>>  Perhaps know of somewhere I can look for more info?
>>>>>
>>>>>  I have looked at this
>>>>> http://lists.xymon.com/archive/2014-January/038780.html
>>>>>  But it doesn't seem relevant. Only 2 errant files, and deleting them
>>>>> made absolutely no difference.
>>>>>
>>>>>  Other info that may be important....
>>>>> bash-3.00# ./httpd -v
>>>>> Server version: Apache/2.2.22 (Unix)
>>>>> Server built:   Jun  1 2012 05:09:20
>>>>> bash-3.00# ./httpd -V
>>>>> Server version: Apache/2.2.22 (Unix)
>>>>> Server built:   Jun  1 2012 05:09:20
>>>>> Server's Module Magic Number: 20051115:30
>>>>> Server loaded:  APR 1.4.5, APR-Util 1.3.12
>>>>> Compiled using: APR 1.4.6, APR-Util 1.3.12
>>>>> Architecture:   32-bit
>>>>> Server MPM:     Prefork
>>>>>   threaded:     no
>>>>>     forked:     yes (variable process count)
>>>>> Server compiled with....
>>>>>  -D APACHE_MPM_DIR="server/mpm/prefork"
>>>>>  -D APR_HAS_SENDFILE
>>>>>  -D APR_HAS_MMAP
>>>>>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>>>  -D APR_USE_FCNTL_SERIALIZE
>>>>>  -D APR_USE_PTHREAD_SERIALIZE
>>>>>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>>>  -D APR_HAS_OTHER_CHILD
>>>>>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>>>  -D DYNAMIC_MODULE_LIMIT=128
>>>>>  -D HTTPD_ROOT="/opt/csw/apache2"
>>>>>  -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec"
>>>>>  -D DEFAULT_PIDLOG="/var/run/httpd.pid"
>>>>>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>>>  -D DEFAULT_LOCKFILE="/var/run/accept.lock"
>>>>>  -D DEFAULT_ERRORLOG="logs/error_log"
>>>>>  -D AP_TYPES_CONFIG_FILE="etc/mime.types"
>>>>>  -D SERVER_CONFIG_FILE="etc/httpd.conf"
>>>>>
>>>>>  Thanks
>>>>>  Vernon
>>>>>
>>>>>
>>>>> --
>>>>> "Accept the challenges so that you can feel the exhilaration of
>>>>> victory"
>>>>> - General George Patton
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> "Accept the challenges so that you can feel the exhilaration of victory"
>>>> - General George Patton
>>>>
>>>
>>>
>>>
>>> --
>>> "Accept the challenges so that you can feel the exhilaration of victory"
>>> - General George Patton
>>>
>>> _______________________________________________
>>> Xymon mailing list
>>> Xymon at xymon.com
>>> http://lists.xymon.com/mailman/listinfo/xymon
>>>
>>>
>>
>
>
> --
> "Accept the challenges so that you can feel the exhilaration of victory"
> - General George Patton
>



-- 
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20140429/f9458f1c/attachment.html>


More information about the Xymon mailing list