[Xymon] flushing rrd files - now possible via rrdcached
Martin Sperl
Martin.Sperl at amdocs.com
Thu Sep 4 10:54:00 CEST 2014
Well - he is talking about starting 1.5 in the next few weeks...
We are already using the head-branch in our production environment and it works without a lot of hickups.
BUT: from all my testing it also requires a few patches to xymon to use fully qualified pathnames for rrdfiles to work properly.
For us it now works fine and we are updating 32k rrd files via rrdcached every 5 minutes and with minimal IOWAITs...
Ciao,
Martin
p.s: the config-settings we use now are:
tasks.d/rrdcached:
[rrdcache]
LOGFILE $XYMONSERVERLOGS/rrdcached.log
NEEDS xymond
CMD /usr/bin/rrdcached -w 1800 -z 900 -f 3600 -l $XYMONVAR/rrdcached/rrdcached.socket -j $XYMONVAR/rrdcached -p $XYMONVAR/rrdcached/rrdcached.pid -g -l 0.0.0.0:41984 -b $XYMONVAR/ -B
[rrdstatus]
ENVFILE /usr/lib/xymon/server/etc/xymonserver-rrdcached.cfg
CMD +--no-cache
[rrddata]
ENVFILE /usr/lib/xymon/server/etc/xymonserver-rrdcached.cfg
CMD +--no-cache
And /usr/lib/xymon/server/etc/xymonserver-rrdcached.cfg looks like this:
include /usr/lib/xymon/server/etc/xymon/xymonserver.cfg
RRDCACHED_ADDRESS="$XYRRDCACHED_ADDRESS"
And the following added to /usr/lib/xymon/server/etc/xymon/xymonserver.cfg:
XYRRDCACHED_ADDRESS="127.0.0.1:41984"
RRDCACHED_STRIPPATH="$XYMONVAR"
The reason why there is RRDCACHED only in the /usr/lib/xymon/server/etc/xymonserver-rrdcached.cfg is that otherwise it would do the flushing several times:
Once for the global flushing of the host via rrd_cached_flush and then another time when creating the graph.
This could possibly get consolidated if the existing caching solution in xymon is removed...
Here the patch for the code:
diff -ru xymon-4.3.17.orig/web/perfdata.c xymon-4.3.17/web/perfdata.c
--- xymon-4.3.17.orig/web/perfdata.c 2012-07-23 04:47:20.000000000 -0700
+++ xymon-4.3.17/web/perfdata.c 2014-07-07 02:03:25.562153926 -0700
@@ -102,9 +102,11 @@
char *rrdargs[10];
int result;
+ char rrdfullname[2048];
+ snprintf(rrdfullname,sizeof(rrdfullname),"%s/%s/%s",xgetenv("XYMONRRDS"),hostname,rrdname);
rrdargs[0] = "rrdfetch";
- rrdargs[1] = rrdname;
+ rrdargs[1] = rrdfullname;
rrdargs[2] = "AVERAGE";
rrdargs[3] = "-s"; rrdargs[4] = starttimedate; rrdargs[5] = starttimehm;
rrdargs[6] = "-e"; rrdargs[7] = endtimedate; rrdargs[8] = endtimehm;
diff -ru xymon-4.3.17.orig/web/showgraph.c xymon-4.3.17/web/showgraph.c
--- xymon-4.3.17.orig/web/showgraph.c 2014-02-23 01:32:44.000000000 -0800
+++ xymon-4.3.17/web/showgraph.c 2014-07-07 05:19:39.314095689 -0700
@@ -119,6 +119,36 @@
exit(1);
}
+void rrd_cached_flush(char *hostname) {
+ /* the dir containing all the files */
+ DIR *dirHandle;
+ struct dirent * dirEntry;
+ char dirName[PATH_MAX];
+ char fileName[PATH_MAX];
+ /* flushing cache */
+ char *flush_args[4];
+ flush_args[1]="--daemon";
+ flush_args[2]=xgetenv("XYRRDCACHED_ADDRESS");
+ flush_args[3]=fileName;
+ /* create fqdn */
+ snprintf(dirName,sizeof(dirName),"%s/%s",xgetenv("XYMONRRDS"),hostname);
+ /* now iterate all the rrd files in this directory */
+ dirHandle = opendir(dirName);
+ if (dirHandle) {
+ while((dirEntry = readdir(dirHandle))) {
+ int len=strlen(dirEntry->d_name);
+ if (strcmp(dirEntry->d_name+len-4,".rrd")==0) {
+ /* create final filename */
+ snprintf(fileName,sizeof(fileName),"%s/%s",
+ dirName,dirEntry->d_name);
+ /* and flush */
+ rrd_flushcached(4,flush_args);
+ }
+ }
+ closedir(dirHandle);
+ }
+}
+
void request_cacheflush(char *hostname)
{
/* Build a cache-flush request, and send it to all of the $XYMONTMP/rrdctl.* sockets */
@@ -128,6 +158,9 @@
struct dirent *d;
int ctlsocket = -1;
+ if (xgetenv("XYRRDCACHED_ADDRESS"))
+ return rrd_cached_flush(hostname);
+
ctlsocket = socket(AF_UNIX, SOCK_DGRAM, 0);
if (ctlsocket == -1) {
errprintf("Cannot get socket: %s\n", strerror(errno));
diff -ru xymon-4.3.17.orig/xymond/do_rrd.c xymon-4.3.17/xymond/do_rrd.c
--- xymon-4.3.17.orig/xymond/do_rrd.c 2013-08-25 07:03:44.000000000 -0700
+++ xymon-4.3.17/xymond/do_rrd.c 2014-07-07 04:41:33.635681424 -0700
@@ -245,9 +246,10 @@
/*
* RRDtool 1.2+ uses mmap'ed I/O, but the Linux kernel does not update timestamps when
* doing file I/O on mmap'ed files. This breaks our check for stale/nostale RRD's.
- * So do an explicit timestamp update on the file here.
+ * So do an explicit timestamp update on the file here if we are not using rrdcache
*/
- utimes(filedir, NULL);
+ if (!xgetenv("RRDCACHED_ADDRESS"))
+ utimes(filedir, NULL);
#endif
/* Clear the cached data */
> -----Original Message-----
> From: J.C. Cleaver [mailto:cleaver at terabithia.org]
> Sent: Mittwoch, 03. September 2014 18:28
> To: Martin Sperl
> Subject: Re: [Xymon] flushing rrd files - now possible via rrdcached
>
> Hello!
>
> Not sure if you'd been following along on your original commit at
> https://github.com/oetiker/rrdtool-1.x/pull/462, but I'd expressed
> interest to T. Oetiker on possibly getting this into 1.4.x for the next
> possible release. After he requested a pull, I noticed a few build
> problems (aside from the patch not directly applying) with it on the
> current 1.4.x branch.
>
> If you have a chance, I was wondering if you might be able to take look at
> setting up a pull for him? Figure since you wrote it you'd be more
> familiar with what exactly's going wrong... :)
>
>
> Let me know if I can be of any help!
>
> Regards,
> -jc
>
>
> On 7 May 2014 08:54, Martin Sperl <Martin.Sperl at amdocs.com> wrote:
>
> There now exists a patch to rrdtool (which has already been merged) that
> allows xymon to work with the following config-changes:
>
> [rrdcache]
> LOGFILE $XYMONSERVERLOGS/rrdcached.log
> NEEDS xymond
> CMD $RRD_BASE/bin/rrdcached -w 1800 -z 900 -f 1800 -l
> $XYMONVAR/rrdcached/rrdcached.socket -j $XYMONVAR/rrdcached -p
> $XYMONVAR/rrdcached/rrdcached.pid -g -l 0.0.0.0:41984 -b
> $XYMONVAR/ -B
>
> [rrdstatus]
> CMD +--no-cache
> [rrddata]
> CMD +--no-cache
>
> And in the ENVFILE /etc/xymon/xymonserver.cfg add:
> RRDCACHED_ADDRESS="127.0.0.1:41984"
> RRDCACHED_STRIPPATH="$XYMONVAR"
> # these are there to override the default rrd install location
> # so that it uses the head build from rrdtool
> RRD_BASE="/opt/rrdtool-1.4.999"
> LD_LIBRARY_PATH="$RRD_BASE/lib/"
>
> The patch to rrdtool: https://github.com/oetiker/rrdtool-1.x/pull/462 has
> already been incorporated with the main development branch, so I hope it
> will get into the next rrdtool-1.4 release.
>
> Ciao,
> Martin
>
> From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Martin
> Sperl
> Sent: Dienstag, 29. April 2014 13:03
> To: Jeremy Laidman
> Cc: xymon at xymon.com
> Subject: Re: [Xymon] flushing rrd files
>
> Thanks - the problem is that those sockets change so you have to "guess"
> their path and see if it is functional (as with your perl example - in my
> case there are actually 16 socket files in temp and I have to iterate over
> all of them to find the 2 that are actually working - slightly stupid...)
>
> Obviously this also does not work from a remote node which only has access
> to the rrd files via NFS...
> Meaning I have to write a http proxy for that...
>
> So I was wondering if xymond_rrd with the --no-cache would use the
> RRDCache if set in the environment.
> My guess would be I need to use a different env file which has
> RRDCACHED_ADDRESS set correctly (which rrdtool makes use of automatically
> if the ENV is set) and then configure xymond_rrd to use that environment
> instead....
> If that works then that rrd-specific caching could get removed as a whole
> - with the exception of the need for
>
> I just want to avoid testing this on our live system, so I was asking if
> anyone has experience with this...
>
> Martin
>
> From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au]
> Sent: Dienstag, 29. April 2014 04:33
> To: Martin Sperl
> Cc: xymon at xymon.com<mailto:xymon at xymon.com>
> Subject: Re: [Xymon] flushing rrd files
>
> I think work would need to be done to integrate with rrdcached.
>
> The RRD stats are cached by xymond_rrd. The showgraph.cgi binary sends a
> flush command to the xymond_rrd instances, via UNIX sockets, requesting a
> cache flush, so that graphs are up-to-date. You could probably emulate
> this by running showgraph.cgi from the command-line. Like so:
>
> SCRIPT_NAME=showgraph.sh REQUEST_METHOD=GET
> QUERY_STRING="host=hostname.example.com<http://hostname.example.co
> m>&service=conn"
> /usr/lib/xymon/server/bin/showgraph.cgi >/dev/null
>
> In this case, the "conn" is not used for anything (all RRD files are
> flushed), but has to be a (I think) valid RRD filename (without the
> extension) or graph name (defined in graphs.cfg).
>
> Also, if you can send a string directly to the two UNIX sockets, you can
> cause a flush. The format is simply the hostname in slashes, like
> "/hostname.example.com<http://hostname.example.com>" and so you can
> probably achieve this using modern versions netcat or socat, or other
> things that can write to sockets.
>
> And just for fun, here's an implementation in Perl.
>
> Cheers
> Jeremy
>
> #!/usr/bin/perl -Tw
>
> my $hostname=shift;
> die "Specify hostname\n" unless defined($hostname);
>
> my $socketdir=$ENV{XYMONTMP};
> die "XYMONTMP not defined, run from xymoncmd\n" unless $socketdir;
>
> use Socket;
> socket(SOCK, AF_UNIX, SOCK_DGRAM, 0) or die "socket: $!\n";
> use Fcntl qw(F_GETFL F_SETFL O_NONBLOCK);
> my $flags = fcntl(SOCK, F_SETFL, O_NONBLOCK) or die "fcntl: $!\n";
>
> opendir(DIR,$socketdir) or die "$!: $socketdir\n";
> while(my $socketfile=readdir(DIR)) {
> next unless substr($socketfile,0,7) eq "rrdctl.";
> my $socketpath="$socketdir/$socketfile";
> if (! -e $socketpath) {
> warn "not found: $socketpath\n";
> next;
> }
> if (! -w $socketpath) {
> warn "not writeable: $socketpath\n";
> next;
> }
> if (! -S $socketpath) {
> warn "not a socket: $socketpath\n";
> next;
> }
>
> $socketaddr=sockaddr_un($socketpath);
>
> if (defined(send(SOCK, "/$hostname/", 0, $socketaddr))) {
> print "Flush command for '$hostname' sent to $socketfile\n";
> } else {
> warn "Flush command for '$hostname' failed to send to
> socket $socketfile\n";
> }
> }
>
>
>
> On 25 April 2014 22:29, Martin Sperl <Martin.Sperl at
> amdocs.com<mailto:Martin.Sperl at amdocs.com>> wrote:
> Hi!
>
> Is there a means to flush the rrd files filled in from xymond_rrd say via
> the "xymon" command?
>
> As an alternative is it possible to use the generic RRDCACHED as a
> replacement for the caching?
> Any experience with that?
>
> Thanks,
> Martin
>
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement, you may review at
> http://www.amdocs.com/email_disclaimer.asp
>
> _______________________________________________
> Xymon mailing list
> Xymon at xymon.com<mailto:Xymon at xymon.com>
> http://lists.xymon.com/mailman/listinfo/xymon
>
This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp
More information about the Xymon
mailing list