[Xymon] How to use Xymon to monitor if a file has not been updated in the last 24 hours

[Don Kuhlman]

Hi folks. I'm posting this to the list as a summary of what was done so
that it's all in one place.  Hopefully, this may help others trying to do
the same thing.  Much thanks to Larry B, Ryan N, Cynthia R, and Jeremy L
for their input on this problem.

Question:  How do I monitor if a file is less than 24 hours old in a Linux
Directory and alert if none are found using Xymon?
Purpose:  To validate that SQL Backups are running by checking for
existence of a SQL dump file being created daily by a cron job at various
times.

Description:  We run cron jobs to do mysql dumps on various hosts every
morning at 00:05 and 00:10 am.
We also run a cron job at 00:15 to delete files older than 8 days. Not
doing any monitoring for this event though.

Requirement: We need to have our Xymon server monitor the directories and
alert/email if there has not been a new file created within the past 24
hours.  There can be more than one file created since there can be more
than one database on a host.  This results in us needing to search for
multiple files in the same directories with the same timestamp requirement
being < 24 hour old.

Logic - We use the Xymon agent to check a directory for a file that is
created with a name of ???????_backup.YYYYmmdd.sql
There may be more than one file created during each dump. So we need to
look for any file that matches the date value.
Our logic is that in order to verify our dumps are working, we should
always see new files that are less than 24 hours old in the target
directories. We will use Xymon to watch for specific files and alert if
the file doesn't exist.
This is done using a combination of a find command in the
client-local.cfg, a regex in the analysis.cfg, and the corresponding
alerts.cfg to send the email to the right people.

Here are the values we used in our Xymon configurations to check for files
and then only return two entries to the analysis engine, and alert if
either of these two files are older than 24 hours. That would signal to us
that a dump did not occur the night before.

Client-local.cfg
# This will cause the Xymon client agent to do a find command on the
directory /Volumes/MySQL_Backups for any file ending in
# .sql. It will pipe the output to the sort command to reverse the order
(eg newest files on top) and then return only the
# first two lines of the results sorted by the YYYYmmdd values and not the
first part of the filename.

[hostservernamegoeshere]
file:`find /Volumes/MySQL_Backups/ -name '*.sql' |sort -r  -t"_" -k3 -k1
-k2 | head -2`

This returns two files as below because they are the newest files found:
/Volumes/MySQL_Backups/mysql_backup_20120923.sql - file date/time Sept 23
2012 00:10
/Volumes/MySQL_Backups/othersql_backup_20121008.sql - file date/time Oct 8
2012 00:10



Analysis.cfg


# This will cause the Xymon analysis engine to look at the results (from
above) for this host and
# if either of the two files returned have a modified time greater than 24
hours it will create a red alert condition.
HOST=hostservernamegoeshere
FILE %/Volumes/MySQL_Backups/.*\.sql red mtime<86400

Alerts.cfg
# This line will send an email to the recipient when the Xymon analysis
engine creates a files event for this host
# it will send it every 5 minutes using a custom perl email script
HOST=hostservernamegoeshere SERVICE=files
script /usr/lib64/xymon/server/ext/html_mail.pl alertrecipient at company.com
FORMAT=TEXT REPEAT=5m



Here is the resulting xymon webpage alert and email if either of the two
files does NOT exist:

red Mon Oct  8 11:22:43 CDT 2012 - Files NOT ok

 /Volumes/Backups/MySQL_Backups/mysql_backup_20120923.sql
CLIENT=hostservernamegoeshere&SECTION=file:/Volumes/MySQL_Backups/mysql_bac
kup_20120923.sql>
File was modified 850245 seconds ago - should be <86400


Green Mon Oct 8 11:22:43 CDT 2012 - File OK
 /Volumes/Backups/MySQL_Backups/othersql_backup_20121008.sql
CLIENT=twist01_dev&SECTION=file:/Volumes/MySQL_Backups/othersql_backup_2012
1008.sql>



Don K