[Xymon] /var/lib/xymon/rrd folders growing larger than anticipated

J.C. Cleaver cleaver at terabithia.org
Sat Sep 21 02:21:52 CEST 2019 

On Fri, September 20, 2019 5:42 am, Eliza Danyi wrote:
> Morning all,
>
> We are currently running Xymon 4.3.28-1.el7.terabithia (on CentOS 7.x) and
> I have been struggling with this since we switched to the RPMs from an
> older version of xymon built from source about a year ago. We've gone from
> a few GB on the old server to now almost 40GB just for this directory.
>
> I posed the question a little while back to the group and thought I had
> found the cause in the rrddefinitions.cfg file under the line:
>
> RRA:HWPREDICT:8064:0.1:0.0035:2016
>
> I tried a few different things but ended up deciding to comment out this
> line to see if it would slow the growth, but it does not appear to be
> working. Is there any reason these files would be increasing in number or
> size? Is there something I can do with rrdtool to clean up data? Is it
> normal that an rrd directory for a single host is upwards of 60M on the
> newer versions of Xymon?
>
> Here is an example of the files in a folder for a host:

Hi,

RRD files by themselves do not change size (unless intentionally
manipulated), however new RRAs would have been added in I believe when it
was first run.

The ones you're seeing below, though, are a result of two things; one
easier to fix than the other. Systemd creates "/run/user/xxxx" partitions
on user login, and if the client is running when a new UID is logged in,
you'll get a new RRD file dropped in. This was changed in 4.3.29+ I
believe, but you should be able to exclude them by adding this in to your
analysis.cfg file:

CLASS=linux
        # Ignore some usually uninteresting tmpfs mounts.
        DISK    /dev IGNORE
        DISK    /lib/init/rw IGNORE
        DISK    /sys/fs/cgroup IGNORE
        DISK    %^/run/user/\d+$ IGNORE
        INODE   /dev IGNORE
        INODE   /lib/init/rw IGNORE
        INODE   /sys/fs/cgroup IGNORE
        INODE   %^/run/user/\d+$ IGNORE

After that's in there, you should be able to confirm that the /run/user/*
ones aren't being updated any more and you can purge them.

The second issue, however, is with your network interfaces:
ifstat.veth08022d9.rrd, etc... It looks like something is giving you
highly variable transient interface names?

Xymon does not currently have a mechanism to exclude interfaces by name
like this, since they just look like new things to track. Are these
changing during known-events that might be able to trigger a cleanup?

As a workaround, you should be able to purge ones that have not been
updated after a certain length of time.


HTH,
-jc



>
> [root at xymon]# du -h --max-depth=1 /var/lib/xymon/rrd/server.example/*
> 216K    /var/lib/xymon/rrd/server.example/clock.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,boot.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,dev,shm.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,elasticsearch.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,home.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,root.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,0.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,1000.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,1001.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,1002.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,1003.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,1004.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,1014.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,run,user,1016.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,sys,fs,cgroup.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,tmpfs.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,tmp.rrd
> 424K    /var/lib/xymon/rrd/server.example/disk,var.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.docker0.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.eno16780032.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth0506ddf.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth08022d9.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth081fb4b.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth0828eba.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth0ed858f.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth164c573.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth1740a33.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth189a45e.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth197189c.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth19d7cca.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth1dfb69f.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth2090f52.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth2282b2c.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth258f7aa.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth273f639.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth28aa403.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth29630d9.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth2c50b73.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth2ccd63f.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth38ba0c9.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth38c413a.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth3a47851.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth3c5e33e.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth3cb1195.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth3e20beb.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth42dcce0.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth4311e0a.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth461d830.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth4855b73.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth5074706.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth51e84f6.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth525c545.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth569cd53.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth59a3d88.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth5c4439e.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth5f09582.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth5fa1dd2.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth67d3cbf.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth67dbb70.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth682379a.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth684bcf3.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth69ab431.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth6ad9701.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth6c80cac.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth6ce4f39.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth6e24fbf.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth6e59721.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth6fb0766.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth71f97e2.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth741d0d5.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth7c752f1.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth7f93b1f.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth8386dfb.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth8e3b4e4.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth8f68e08.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth9063983.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth91bc3c6.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth96c1fac.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth9a73f58.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.veth9ed52d1.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vetha09504c.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vetha184ef3.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vetha410e0c.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vetha460481.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vetha5a56d6.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vetha6cec13.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethad8eda4.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethad8faa1.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethaffd7df.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethb3c04b7.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethb426b52.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethb52aae4.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethb6937d3.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethba647c0.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethbd846e1.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethc01c9dd.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethc81bb06.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethc8ea13a.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethcc1a6d6.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethcc51afb.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethd01eb33.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethd105c62.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethd1b1789.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethd676e0b.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethd947e9e.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethdcb0028.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethed73659.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethf05d463.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethf4659bf.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethf84f9fc.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethf8efbee.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethfb569a5.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethfb5988e.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethfeec0dd.rrd
> 424K    /var/lib/xymon/rrd/server.example/ifstat.vethffc6c75.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,boot.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,dev,shm.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,elasticsearch.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,home.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,root.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,0.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,1000.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,1001.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,1002.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,1003.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,1004.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,1014.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,run,user,1016.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,sys,fs,cgroup.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,tmpfs.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,tmp.rrd
> 424K    /var/lib/xymon/rrd/server.example/inode,var.rrd
> 216K    /var/lib/xymon/rrd/server.example/la.rrd
> 216K    /var/lib/xymon/rrd/server.example/memory.actual.rrd
> 216K    /var/lib/xymon/rrd/server.example/memory.real.rrd
> 216K    /var/lib/xymon/rrd/server.example/memory.swap.rrd
> 3.3M    /var/lib/xymon/rrd/server.example/netstat.rrd
> 216K    /var/lib/xymon/rrd/server.example/procs.rrd
> 216K    /var/lib/xymon/rrd/server.example/tcp.conn.rrd
> 216K    /var/lib/xymon/rrd/server.example/users.rrd
> 3.5M    /var/lib/xymon/rrd/server.example/vmstat.rrd
>
>
> Do these look like reasonable file sizes/number of files compared to
> everyone else? How much space should we reasonably expect to use for this
> given we have roughly 1k hosts being monitored (mix of Windows and Linux).
>
> Thanks,
> *Eliza Danyi*
> Operating Systems Analyst
> VCU Computer Center
> Phone: 804-827-1132
> E-mail: danyiem at vcu.edu
>
> *Don't be a phishing victim -- VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information.  For more details,
> visit **https://phishing.vcu.edu/* <https://phishing.vcu.edu/>
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>

More information about the Xymon mailing list