<font size=2 face="sans-serif">I would like to share some hints in resolving
history reporting problem after big time shift on monitoring server - about
4 hours.</font>
<br><font size=2 face="sans-serif">May be it will help anyone else.</font>
<br>
<br><font size=2 face="sans-serif">It was some month ago, but I have found
time to fix it only today.</font>
<br><font size=2 face="sans-serif">What happened:</font>
<br><font size=2 face="sans-serif">1. Time on monitoring host increased
for 4 hours.</font>
<br><font size=2 face="sans-serif">2. As result - all metrics reported
Purple status (it is intended functionality, but would be nice XyMon detect
big time shift and adopt reporting in some way)</font>
<br><font size=2 face="sans-serif">3. It was problem at virtual host provider,
I had reported the problem and time was fixed back to correct value</font>
<br><font size=2 face="sans-serif">4. To fix current reporting I had cleaned
some files under xymon/logs or acks (really I do not remember which ones
right now) - this has reset last status duration information, but current
values for all metrics become correct</font>
<br><font size=2 face="sans-serif">5. Everythig become OK, except
that when I check history for metric (</font><a href="https://dozor01.cominder.eu:9813/xymon-cgi/history.sh?"><font size=2 color=blue face="sans-serif">
...xymon-cgi/history.sh?</font></a><font size=2 face="sans-serif"> ...)
for some metrics.</font>
<br><font size=2 face="sans-serif">XyMon always reported Purple for last
event (since that incident time).</font>
<br>
<br>
<br><font size=2 face="sans-serif">It was just for some metrics (not all)
and I had second monitoring server with the same information (not having
time shift incident) and I was able to live with it some month.</font>
<br>
<br><font size=2 face="sans-serif">Solution: </font>
<br><font size=2 face="sans-serif">Today I have fixed that reporting problem
with the following steps, which should be executed for every host-metric
pair having the problem</font>
<br>
<br><font size=2 face="sans-serif">We should operate with 2 files:</font>
<br><font size=2 face="sans-serif">1) host history file like </font>
<br><font size=2 face="sans-serif"> hist/HOSTNAME </font>
<br><font size=2 face="sans-serif"># here we should find records with negative
duration values like:</font>
<br><font size=2 face="sans-serif">svcs 1435410898 1435426055 -15157 gr
pu 1</font>
<br><font size=2 face="sans-serif">who 1435410899 1435426055 -15156 gr
pu 1</font>
<br><font size=2 face="sans-serif">msgs 1435410899 1435426055 -15156 gr
pu 1</font>
<br><font size=2 face="sans-serif">netstat 1435410899 1435426055 -15156
gr pu 1</font>
<br><font size=2 face="sans-serif">memory 1435411034 1435426055 -15021
ye pu 2</font>
<br><font size=2 face="sans-serif">uptime 1435411140 1435426055 -14915
gr pu 1</font>
<br><font size=2 face="sans-serif">procs 1435411145 1435426055 -14910 gr
pu 1</font>
<br><font size=2 face="sans-serif">disk 1435411150 1435426055 -14905 ye
pu 2</font>
<br><font size=2 face="sans-serif">cpu 1435411222 1435426055 -14833 gr
pu 1</font>
<br>
<br><font size=2 face="sans-serif"># and drop them</font>
<br>
<br><font size=2 face="sans-serif">2) service history file like</font>
<br><font size=2 face="sans-serif"> hist/HOSTNAME.svc</font>
<br><font size=2 face="sans-serif"># again - find records with negative
duration values like:</font>
<br><font size=2 face="sans-serif">Sat Jun 27 20:27:35 2015 purple 1435426055
-15157</font>
<br>
<br><font size=2 face="sans-serif"># and drop record(s) - really
should be just one </font>
<br>
<br>
<br><font size=2 face="sans-serif">Really to fix just one service reporting
- it is enough to drop negative duration records from service history file
only (tested).</font>
<br><font size=2 face="sans-serif">But I do not see any reason to have
such records in host history file, so I delete from that file too.</font>
<br>
<br><font size=2 face="sans-serif">How to automate the process:</font>
<br><font size=2 face="sans-serif"># find hist files for </font>
<br><font size=2 face="sans-serif"># step 1: </font>
<br><font size=2 face="sans-serif">find hist/ -print0 -name "*.*"
| xargs -0 grep " -" | awk '{print $1" :"$4}' | grep
":-"</font>
<br>
<br><font size=2 face="sans-serif">#output like:</font>
<br><font size=2 face="sans-serif">...</font>
<br><font size=2 face="sans-serif">hist/idc-oracle03.msc-sh.local:ssh :-14862</font>
<br><font size=2 face="sans-serif">hist/idc-oracle03.msc-sh.local:dblock
:-15012</font>
<br><font size=2 face="sans-serif">hist/idc-oracle03.msc-sh.local:dbrec
:-15012</font>
<br><font size=2 face="sans-serif">hist/idc-oracle03.msc-sh.local:dbup
:-15011</font>
<br><font size=2 face="sans-serif">hist/idc-oracle03.msc-sh.local:dbext
:-14989</font>
<br><font size=2 face="sans-serif">...</font>
<br>
<br><font size=2 face="sans-serif"># step 2: find hist/ -print0
-name "*.*" | xargs -0 grep " -" | awk '{print $1"
:"$8}' | grep ":-"</font>
<br><font size=2 face="sans-serif"># output like:</font>
<br><font size=2 face="sans-serif">..</font>
<br><font size=2 face="sans-serif">hist/idc-oracle03,domain.com.dbrec:Sat
:-15012</font>
<br><font size=2 face="sans-serif">hist/gdc-oracle03,domain.com.dbup:Sat
:-15136</font>
<br><font size=2 face="sans-serif">hist/idc-oracle01,domain.com.disk:Sat
:-14961</font>
<br><font size=2 face="sans-serif">hist/gdc-oracle01,domain.com.dbaud:Thu
:-26793</font>
<br><font size=2 face="sans-serif">hist/gdc-oracle01,domain.com.dbaud:Sat
:-14940</font>
<br><font size=2 face="sans-serif">..</font>
<br>
<br><font size=2 face="sans-serif">Then can automate the records removal
too.</font>
<br>
<br>
<br><font size=2 face="Verdana">Best regards,</font>
<br>
<br><font size=2 face="Verdana">Andrey Chervonets</font>
<br><font size=2 face="Verdana">----------------------</font>
<br><font size=2 face="Verdana">SIA CoMinder</font>
<br><a href=http://www.cominder.eu/><font size=2 color=blue face="Verdana">http://www.cominder.eu/</font></a>
<br>
<br>
<br>