[Xymon] Absolute free space value for DISK gone rogue?

[Mills,David (HHSC Contractor)]

New bit of data on this issue if that helps:

Speaking with my customer, he informs me this particular file system has been monitored / alarmed when full just fine for years(!!). The problems began when they emptied out the file system to (almost) zero content. That's when things went "red".

I'm thinking now that there's some bad data "stuck" in Xymon's state-full data that is yielding the false positives.

Any thoughts on that point?

david

From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Mills,David (HHSC Contractor)
Sent: Tuesday, January 31, 2017 4:54 PM
To: 'xymon at xymon.com'
Subject: [Xymon] Absolute free space value for DISK gone rogue?

All -

I have a file system on a host which is alarming "red" even though the file system is brand new / empty. In the notes below, the mis-reporting filesystem is "/ora_data_estdmprd"

The stanza in analysis.cfg that covers it is:

HOST=pwdu008                                    # Stand-alone prod DB hosts: won't have shared FS's
        DISK /ora_bin                     85     95     # 85%/95% = CRQ000000013038
        DISK /ora_data                    85     95     # 85%/95% = CRQ000000013038
        DISK /ora_admin                   85     95     # 85%/95% = CRQ000000013038
        DISK /ora_temp                    85     95     # 85%/95% = CRQ000000013038
        DISK /ora_redoa                   85     95     # 85%/95% = CRQ000000013038
        DISK /ora_redob                   85     95     # 85%/95% = CRQ000000013038
        DISK /ora_arch                    85     95     # 85%/95% = CRQ000000013038
        DISK /ora_data_finprod1           85     95     # 85%/95% = CRQ000000013038
        DISK /ora_finbin                  85     95     # 85%/95% = CRQ000000013038
        DISK %/ora_data_perpr             85     95     # 85%/95% = CRQ000000013038
        DISK %/ora_                     262144000U     209715200U       # I.e. Yellow: <= 256GB; Red <= 204.8GB

The output from the "Disk" detail page for that host was:

status pwdu008.disk yellow Tue Jan 31 14:06:20 CST 2017 - Filesystems NOT ok
&red /ora_data_estdmprd (3045500329 units free) has reached the CRITICAL level (209715200 units)

Filesystem            kbytes    used   avail capacity  Mounted on
/dev/md/dsk/d1       16525754 7879972 8480525    49%    /
/dev/md/dsk/d4       20655025 2723628 17724847    14%    /var
/dev/md/dsk/d7       17197130 2708305 14316854    16%    /export
/dev/odm                   0       0       0     0%    /dev/odm
sharefs                    0       0       0     0%    /etc/dfs/sharetab
swap                 147391344    1936 147389408     1%    /etc/svc/volatile
swap                 147396208    6840 147389368     1%    /tmp
swap                 147389344       0 147389344     0%    /dev/vx/dmp
swap                 147384744       0 147384744     0%    /dev/vx/rdmp
/dev/vx/dsk/ORA_DATA/ora_data 249925632 5610862 229045230     3%    /ora_data
/dev/vx/dsk/ORA_ARCH/ora_arch 249958400  685078 233695152     1%    /ora_arch
/dev/vx/dsk/ORA_BIN/ora_bin 124930048 54292389 66222935    46%    /ora_bin
/dev/vx/dsk/ORA_ADMIN/ora_admin 62465024  202884 58405611     1%    /ora_admin
/dev/vx/dsk/ORA_REDOA/ora_redoa 124962816  833493 116371374     1%    /ora_redoa
/dev/vx/dsk/ORA_ARCH_ESTDMPRD/ora_arch_estdmprd 1749710848 966260000 777330408    56%    /ora_arch_estdmprd
/dev/vx/dsk/ORA_TEMP/ora_temp 999961600  521070 936975632     1%    /ora_temp
/dev/vx/dsk/ORA_REDOB/ora_redob 124962816  423857 116755408     1%    /ora_redob
/dev/vx/dsk/ORA_DATA_DMSTGPRD/ora_data_dmstgprd 4999072768 1751121 4728890415     1%    /ora_data_dmstgprd
/dev/vx/dsk/ORA_DATA_ESTDMPRD/oradata_estdmprd01 3249462272  929825 3045500329     1%    /ora_data_estdmprd

In the report above, we see that the "critical" threshold (in KB on Solaris) is 209,715,200 free space (about 256 GB), but the actual free space for that file system is 3,045,500,329 (around 3 TB). Something is clearly not right!

Also, note the file system second-from-the bottom: /ora_data_dmstgprd. It has _even more_ free space and should be triggered by the same rule in analysis.cfg (DISK %/ora_ ...) yet it is not flagged as having a problem. As a stop-gap to keep the filesystem from alarming, I've had to change the "DISK" rule in analysis.cfg to percentages (85   95) which makes everything go green.

Particulars: Xymon v. 4.3.3 running on Solaris 10 host. (Have also tried restarting the server without impact.)

Any thoughts?

Thanks!

david

~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~
David Mills
Systems Administrator
Northrop Grumman
(512) 595-1238 (mobile)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20170203/47b2de87/attachment.html>