[Xymon] /var/lib/xymon/rrd folders growing larger than anticipated
Matt Vander Werf
mvanderw at nd.edu
Sat Sep 21 14:20:21 CEST 2019
I often see those kind of interface RRD files on systems that have Docker
containers running on them. Every time a new container is started, a random
interface like that is created on the host for the container network (and
thus an RRD file is created). Every time a container is restarted, a new
random interface is created for it and the old interface isn't used
anymore, but the RRD file remains for it. For me at least, these are on
systems that we don't admin but still monitor with Xymon, so I never know
when the containers will be created/restarted/etc.
Depending on how often the containers are restarted/created/etc., the
number of random veth* interface RRD files per host can be A LOT. Normally,
they aren't very large by themselves, but I can see how combined it might
take up a considerable amount of space.
I run https://wiki.xymonton.org/doku.php/addons:finit periodically to find
inactive RRD files and remove the ones that I know won't be updated again.
The script will output rm commands you can use to remove the inactive RRD
files. You just need to change the HomeDir and Threshold (in seconds)
variables to what you want to use. This script works really well to find
and clean up the inactive random veth* interfaces (and other inactive RRD
files). It can also help to find hosts that can just be removed (dropped)
completely from Xymon instead ('xymon drop X').
Alternatively, if you don't care about these network interfaces, and know
the specific interfaces you do care about, you could use the INTERFACES
option on the host entries in hosts.cfg:
INTERFACES:REGEXP
On systems with multiple network interfaces, the operating
system may report a number of net‐
work interface where the statistics are of no interest. By
default Xymon tracks and graphs
the traffic on all network interfaces. This option
defines a regular expression, and only
those interfaces whose name matches the expression are
tracked.
Hope this helps!
--
Matt Vander Werf
HPC System Administrator
University of Notre Dame
Center for Research Computing - Union Station
506 W. South Street
South Bend, IN 46601
Phone: (574) 631-0692
On Fri, Sep 20, 2019 at 8:22 PM J.C. Cleaver <cleaver at terabithia.org> wrote:
>
> On Fri, September 20, 2019 5:42 am, Eliza Danyi wrote:
> > Morning all,
> >
> > We are currently running Xymon 4.3.28-1.el7.terabithia (on CentOS 7.x)
> and
> > I have been struggling with this since we switched to the RPMs from an
> > older version of xymon built from source about a year ago. We've gone
> from
> > a few GB on the old server to now almost 40GB just for this directory.
> >
> > I posed the question a little while back to the group and thought I had
> > found the cause in the rrddefinitions.cfg file under the line:
> >
> > RRA:HWPREDICT:8064:0.1:0.0035:2016
> >
> > I tried a few different things but ended up deciding to comment out this
> > line to see if it would slow the growth, but it does not appear to be
> > working. Is there any reason these files would be increasing in number or
> > size? Is there something I can do with rrdtool to clean up data? Is it
> > normal that an rrd directory for a single host is upwards of 60M on the
> > newer versions of Xymon?
> >
> > Here is an example of the files in a folder for a host:
>
> Hi,
>
> RRD files by themselves do not change size (unless intentionally
> manipulated), however new RRAs would have been added in I believe when it
> was first run.
>
> The ones you're seeing below, though, are a result of two things; one
> easier to fix than the other. Systemd creates "/run/user/xxxx" partitions
> on user login, and if the client is running when a new UID is logged in,
> you'll get a new RRD file dropped in. This was changed in 4.3.29+ I
> believe, but you should be able to exclude them by adding this in to your
> analysis.cfg file:
>
> CLASS=linux
> # Ignore some usually uninteresting tmpfs mounts.
> DISK /dev IGNORE
> DISK /lib/init/rw IGNORE
> DISK /sys/fs/cgroup IGNORE
> DISK %^/run/user/\d+$ IGNORE
> INODE /dev IGNORE
> INODE /lib/init/rw IGNORE
> INODE /sys/fs/cgroup IGNORE
> INODE %^/run/user/\d+$ IGNORE
>
> After that's in there, you should be able to confirm that the /run/user/*
> ones aren't being updated any more and you can purge them.
>
> The second issue, however, is with your network interfaces:
> ifstat.veth08022d9.rrd, etc... It looks like something is giving you
> highly variable transient interface names?
>
> Xymon does not currently have a mechanism to exclude interfaces by name
> like this, since they just look like new things to track. Are these
> changing during known-events that might be able to trigger a cleanup?
>
> As a workaround, you should be able to purge ones that have not been
> updated after a certain length of time.
>
>
> HTH,
> -jc
>
>
>
> >
> > [root at xymon]# du -h --max-depth=1 /var/lib/xymon/rrd/server.example/*
> > 216K /var/lib/xymon/rrd/server.example/clock.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,boot.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,dev,shm.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,elasticsearch.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,home.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,root.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,0.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,1000.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,1001.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,1002.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,1003.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,1004.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,1014.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,run,user,1016.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,sys,fs,cgroup.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,tmpfs.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,tmp.rrd
> > 424K /var/lib/xymon/rrd/server.example/disk,var.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.docker0.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.eno16780032.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth0506ddf.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth08022d9.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth081fb4b.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth0828eba.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth0ed858f.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth164c573.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth1740a33.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth189a45e.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth197189c.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth19d7cca.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth1dfb69f.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth2090f52.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth2282b2c.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth258f7aa.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth273f639.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth28aa403.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth29630d9.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth2c50b73.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth2ccd63f.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth38ba0c9.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth38c413a.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth3a47851.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth3c5e33e.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth3cb1195.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth3e20beb.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth42dcce0.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth4311e0a.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth461d830.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth4855b73.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth5074706.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth51e84f6.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth525c545.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth569cd53.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth59a3d88.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth5c4439e.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth5f09582.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth5fa1dd2.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth67d3cbf.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth67dbb70.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth682379a.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth684bcf3.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth69ab431.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ad9701.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth6c80cac.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ce4f39.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e24fbf.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e59721.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth6fb0766.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth71f97e2.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth741d0d5.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth7c752f1.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth7f93b1f.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth8386dfb.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth8e3b4e4.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth8f68e08.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth9063983.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth91bc3c6.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth96c1fac.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth9a73f58.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.veth9ed52d1.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vetha09504c.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vetha184ef3.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vetha410e0c.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vetha460481.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vetha5a56d6.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vetha6cec13.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8eda4.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8faa1.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethaffd7df.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethb3c04b7.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethb426b52.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethb52aae4.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethb6937d3.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethba647c0.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethbd846e1.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethc01c9dd.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethc81bb06.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethc8ea13a.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc1a6d6.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc51afb.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethd01eb33.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethd105c62.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethd1b1789.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethd676e0b.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethd947e9e.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethdcb0028.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethed73659.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethf05d463.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethf4659bf.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethf84f9fc.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethf8efbee.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb569a5.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb5988e.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethfeec0dd.rrd
> > 424K /var/lib/xymon/rrd/server.example/ifstat.vethffc6c75.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,boot.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,dev,shm.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,elasticsearch.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,home.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,root.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,0.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,1000.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,1001.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,1002.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,1003.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,1004.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,1014.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,run,user,1016.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,sys,fs,cgroup.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,tmpfs.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,tmp.rrd
> > 424K /var/lib/xymon/rrd/server.example/inode,var.rrd
> > 216K /var/lib/xymon/rrd/server.example/la.rrd
> > 216K /var/lib/xymon/rrd/server.example/memory.actual.rrd
> > 216K /var/lib/xymon/rrd/server.example/memory.real.rrd
> > 216K /var/lib/xymon/rrd/server.example/memory.swap.rrd
> > 3.3M /var/lib/xymon/rrd/server.example/netstat.rrd
> > 216K /var/lib/xymon/rrd/server.example/procs.rrd
> > 216K /var/lib/xymon/rrd/server.example/tcp.conn.rrd
> > 216K /var/lib/xymon/rrd/server.example/users.rrd
> > 3.5M /var/lib/xymon/rrd/server.example/vmstat.rrd
> >
> >
> > Do these look like reasonable file sizes/number of files compared to
> > everyone else? How much space should we reasonably expect to use for this
> > given we have roughly 1k hosts being monitored (mix of Windows and
> Linux).
> >
> > Thanks,
> > *Eliza Danyi*
> > Operating Systems Analyst
> > VCU Computer Center
> > Phone: 804-827-1132
> > E-mail: danyiem at vcu.edu
> >
> > *Don't be a phishing victim -- VCU and other reputable organizations will
> > never use email to request that you reply with your password, social
> > security number or confidential personal information. For more details,
> > visit **https://phishing.vcu.edu/* <https://phishing.vcu.edu/>
> > _______________________________________________
> > Xymon mailing list
> > Xymon at xymon.com
> > http://lists.xymon.com/mailman/listinfo/xymon
> >
>
>
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com
> http://lists.xymon.com/mailman/listinfo/xymon
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20190921/3561f1fd/attachment.htm>
More information about the Xymon
mailing list