[Xymon] High CPU Load Rendering Graphs
Vernon Everett
everett.vernon at gmail.com
Tue Apr 29 06:07:23 CEST 2014
Hi Jeremy
Thanks for the tips.
Manually, it runs in under a second.
And not using the "--no-cache" option?
--- snip ---
# "rrdstatus" updates RRD files with information that arrives as "status"
messages.
[rrdstatus]
ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=status --log=$XYMONSERVERLOGS/rrd-status.log
xymond_rrd --rrddir=$XYMONVAR/rrd
# "rrddata" updates RRD files with information that arrives as "data"
messages.
[rrddata]
ENVFILE /usr/local/xymon/server/etc/xymonserver.cfg
NEEDS xymond
CMD xymond_channel --channel=data --log=$XYMONSERVERLOGS/rrd-data.log
xymond_rrd --rrddir=$XYMONVAR/rrd
--- snip ---
Something I did notice, that may be relevant.
As I mentioned earlier, viewing the trends column spawns a host of
showgraph.cgi processes.
I ran a truss on one of them, and I see this group of errors repeated often.
--- snip ---
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/._FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/%FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.AppleDouble/FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/..namedfork/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium-EUC.ps/rsrc",
O_RDONLY) Err#20 ENOTDIR
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/resource.frk/FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/.resource/FangSong-Medium-EUC.ps",
O_RDONLY) Err#2 ENOENT
munmap(0xFE380000, 399) = 0
xstat(2,
"/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
0x0803F2D0) = 0
open("/usr/openwin/lib/locale/zh_CN.GB18030/X11/Resource/Font/FangSong-Medium.ps",
O_RDONLY) = 7
fcntl(7, F_SETFD, 0x00000001) = 0
fxstat(2, 7, 0x0803F130) = 0
mmap(0x00000000, 383, PROT_READ, MAP_PRIVATE, 7, 0) = 0xFE380000
close(7) = 0
--- snip ---
This looks like it's trying to find locale related stuff.
However, I can find no reference to any zh_CN locales anywhere on my system.
Any ideas?
Regards
Vernon
On 29 April 2014 10:46, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
> Is it quick to run the showgraph.cgi binary manually? Like so:
>
> SCRIPT_NAME= REQUEST_METHOD=GET QUERY_STRING='host=hostname.example.com&service=la&graph=hourly'
> /path/to/showgraph.cgi
>
> If that's slow, probably not an Apache problem.
>
> Are you running xymond_rrd with the "--no-cache" option?
>
> J
>
>
>
> On 29 April 2014 12:20, Vernon Everett <everett.vernon at gmail.com> wrote:
>
>> Hi
>>
>> Nope, no IPv6.
>> Did a little constructive Googling on this error, and it's a fairly well
>> known one.
>> All the notes I can find tell me to add
>> Listen 0.0.0.0:443
>> to my httpd.conf file, and the error will go away.
>>
>> But that would be too easy. Not working in my world. And I still get the
>> error. :-/
>> In desperation, I upgraded Apache to version: Apache/2.2.26 (Unix), and
>> it just takes longer before showing the errors after a restart.
>>
>> But, I think this error is a red herring, and is probably quite unrelated
>> to the slow, high CPU utilisation of the graph rendering.
>> Lots of junk in the error.log file I can fix with an appropriate entry in
>> logadm.
>> Taking forever to draw my graphs is a pain.
>>
>> Has anybody experienced problems with the rrd graphs taking long to
>> render?
>>
>> Regards
>> Vernon
>>
>>
>>
>> On 25 April 2014 03:02, Galen Johnson <Galen.Johnson at sas.com> wrote:
>>
>>> That looks like IPv6. Is IPv6 enabled? Is it actually listening on
>>> IPv6 (netstat -tan | grep -i listen | grep 443) I typically disable it on
>>> my systems since using it is a mixed bag currently...especially on solaris
>>> (when I was managing it).
>>>
>>> =G=
>>>
>>> ------------------------------
>>> *From:* Vernon Everett <everett.vernon at gmail.com>
>>> *Sent:* Thursday, April 24, 2014 4:46 AM
>>> *To:* Galen Johnson
>>> *Cc:* Xymon mailinglist
>>> *Subject:* Re: [Xymon] High CPU Load Rendering Graphs
>>>
>>> Yes, and no.
>>>
>>> Have just enabled the status page in the web config, and it appears to
>>> have got rid of the one error message.
>>> (Not sure how I missed the config change at initial install time)
>>>
>>> But, I still get the other error
>>> [warn] (128)Network is unreachable: connect to listener on [::]:443
>>> And my graphs still take way too long to render, and send my CPU
>>> utilisation through the roof.
>>>
>>> Public holiday here tomorrow, so only back at this client on Tuesday.
>>>
>>> Thanks
>>> Vernon
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 24 April 2014 10:27, Galen Johnson <Galen.Johnson at sas.com> wrote:
>>>
>>>> Do you have apache trending graphs enabled? If so, did you enable
>>>> the status page in your apache configs?
>>>>
>>>> =G=
>>>>
>>>> ------------------------------
>>>> *From:* Xymon <xymon-bounces at xymon.com> on behalf of Vernon Everett <
>>>> everett.vernon at gmail.com>
>>>> *Sent:* Wednesday, April 23, 2014 10:17 PM
>>>> *To:* Xymon mailinglist
>>>> *Subject:* [Xymon] High CPU Load Rendering Graphs
>>>>
>>>> Hi all
>>>>
>>>> My Xymon server 4.3.10 is burning the CPU cycles when we view multiple
>>>> graphs, like the trends page, and takes about 5 seconds to render a single
>>>> graph in a single-graph page view.
>>>>
>>>> It's a Sun Fire X4150 with 4Gb of RAM, running Solaris 10 update 5..
>>>>
>>>> Version Location Tag
>>>> -------------------------------- --------------------------
>>>> Intel(R) Xeon(R) CPU X5460 @ 3.16GHz CPU 1
>>>>
>>>> Not a very powerful box, and a bit dated, but I have seen
>>>> significantly better performance on far lesser systems.
>>>> So I am not really thinking the issue is with the hardware.
>>>> It's been slow since it was installed.
>>>> If I view the trends column, I can see the CPU load jump from below 1
>>>> to over 10 at times.
>>>> Running prstat or top in another window while viewing the trends
>>>> column, the process ranking by CPU gets dominated by showgraph.cgi, owned
>>>> by the web server user.
>>>> Top under normal conditions.
>>>> CPU states: 99.9% idle, 0.0% user, 0.1% kernel, 0.0% iowait, 0.0%
>>>> swap
>>>> Top rendering the trends column.
>>>> CPU states: 0.0% idle, 93.8% user, 6.2% kernel, 0.0% iowait, 0.0%
>>>> swap
>>>>
>>>> Also getting this error
>>>> (128)Network is unreachable: connect to listener on [::]:443
>>>> in my Apache error.log file, repeated every second while rendering
>>>> the graphs.
>>>> And from time to time, I get this one.
>>>> File does not exist: /opt/csw/apache2/share/htdocs/server-status
>>>>
>>>> Anybody seen anything like this?
>>>> Perhaps know of somewhere I can look for more info?
>>>>
>>>> I have looked at this
>>>> http://lists.xymon.com/archive/2014-January/038780.html
>>>> But it doesn't seem relevant. Only 2 errant files, and deleting them
>>>> made absolutely no difference.
>>>>
>>>> Other info that may be important....
>>>> bash-3.00# ./httpd -v
>>>> Server version: Apache/2.2.22 (Unix)
>>>> Server built: Jun 1 2012 05:09:20
>>>> bash-3.00# ./httpd -V
>>>> Server version: Apache/2.2.22 (Unix)
>>>> Server built: Jun 1 2012 05:09:20
>>>> Server's Module Magic Number: 20051115:30
>>>> Server loaded: APR 1.4.5, APR-Util 1.3.12
>>>> Compiled using: APR 1.4.6, APR-Util 1.3.12
>>>> Architecture: 32-bit
>>>> Server MPM: Prefork
>>>> threaded: no
>>>> forked: yes (variable process count)
>>>> Server compiled with....
>>>> -D APACHE_MPM_DIR="server/mpm/prefork"
>>>> -D APR_HAS_SENDFILE
>>>> -D APR_HAS_MMAP
>>>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>> -D APR_USE_FCNTL_SERIALIZE
>>>> -D APR_USE_PTHREAD_SERIALIZE
>>>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>> -D APR_HAS_OTHER_CHILD
>>>> -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>> -D DYNAMIC_MODULE_LIMIT=128
>>>> -D HTTPD_ROOT="/opt/csw/apache2"
>>>> -D SUEXEC_BIN="/opt/csw/apache2/sbin/suexec"
>>>> -D DEFAULT_PIDLOG="/var/run/httpd.pid"
>>>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>> -D DEFAULT_LOCKFILE="/var/run/accept.lock"
>>>> -D DEFAULT_ERRORLOG="logs/error_log"
>>>> -D AP_TYPES_CONFIG_FILE="etc/mime.types"
>>>> -D SERVER_CONFIG_FILE="etc/httpd.conf"
>>>>
>>>> Thanks
>>>> Vernon
>>>>
>>>>
>>>> --
>>>> "Accept the challenges so that you can feel the exhilaration of victory"
>>>> - General George Patton
>>>>
>>>
>>>
>>>
>>> --
>>> "Accept the challenges so that you can feel the exhilaration of victory"
>>> - General George Patton
>>>
>>
>>
>>
>> --
>> "Accept the challenges so that you can feel the exhilaration of victory"
>> - General George Patton
>>
>> _______________________________________________
>> Xymon mailing list
>> Xymon at xymon.com
>> http://lists.xymon.com/mailman/listinfo/xymon
>>
>>
>
--
"Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20140429/7ceae34b/attachment.html>
More information about the Xymon
mailing list