<div dir="ltr"><div>Max</div><div><br></div>It's certainly weird. This seems like some kind of integer overflow. If the log update rate high, I can imagine at some point, Xymon's calculations might cause an undetected overflow/underflow. Can you give an indication of the approximate increase in file size per 5 minutes, ideally around the time of an incidence of this fault?<div><br></div><div>You might be able to reproduce the problem by creating your own copy of the logfile and the logfetch.<hostname>.cfg and logfetch.<hostname>status files. If you can manipulate the logfile copy to have the size reported in your logfetch logs just prior to a fault (and confirm that it ends with the log lines reported in the Xymon alert), and then add the extra records to be the size at the time of the fault, you can run logfetch yourself to operate on the copies of the logfile. Once you can reproduce it in this way, we can work on troubleshooting the code.</div><div><br></div><div>It might be interesting to use the TRACK keyword in analysis.cfg to monitor the size of the file, and see if there's some correlation with the occurrence of the fault you're seeing.</div><div><br></div><div>Cheers</div><div>Jeremy</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 22 June 2017 at 10:04, Max Xu <span dir="ltr"><<a href="mailto:Max.Xu@servicenow.com" target="_blank">Max.Xu@servicenow.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:18px;font-family:Calibri,sans-serif">
<div>
<div>
<div>Thanks Jeremy.</div>
<div>The log file is created and updated by oracle Db. It was rotated by at 6/14 (note this ill alert was generated at 6/19).The file shrunk is not possible. I also checked (best effort) that no one manually touch the file.</div>
<div>
<div id="m_8706977057089294653MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
</div>
<div><br>
</div>
<span id="m_8706977057089294653OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri;font-size:12pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Jeremy Laidman <<a href="mailto:jlaidman@rebel-it.com.au" target="_blank">jlaidman@rebel-it.com.au</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, June 21, 2017 at 2:05 PM<br>
<span style="font-weight:bold">To: </span>Max Xu <<a href="mailto:Max.Xu@servicenow.com" target="_blank">Max.Xu@servicenow.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:xymon@xymon.com" target="_blank">xymon@xymon.com</a>" <<a href="mailto:xymon@xymon.com" target="_blank">xymon@xymon.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Xymon] logfetch issue - sending old data causing false alerts<br>
</div><div><div class="h5">
<div><br>
</div>
<div>
<div>
<div dir="auto">How is the log file being created and written to? When and how is it rotated? The logfetch output indicates that that file shrunk in size twice; is that possible?
<div dir="auto"><br>
</div>
<div dir="auto">J</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 22 Jun. 2017 05:20, "Max Xu" <<a href="mailto:Max.Xu@servicenow.com" target="_blank">Max.Xu@servicenow.com</a>> wrote:<br type="attribution">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:18px;font-family:Calibri,sans-serif">
<div>Hi,</div>
<div><br>
</div>
<div>Logfetch is sending old data causing false alerts.</div>
<div><br>
</div>
<div>The log file looks somewhat like this:</div>
<div>
<div>Error 2017-06-14 11:36:58.613343 39915 2184308576 Compare server: ……</div>
<div>Error 2017-06-14 11:36:58.613481 39913 1581872992 Command server: ……</div>
</div>
<div>…… (note. The above repeat about 780K times)</div>
<div>Info 2017-06-14 13:07:41.113163 1193 1036199776 Compare server exited normally, pid = 45494 [sp_desvr]</div>
<div>…..</div>
<div>Error 2017-06-15 02:42:22.820068 1761 2399766368 Command server:…..</div>
<div>……</div>
<div><br>
</div>
<div><br>
</div>
<div>At 6/19 and 6/20, msgs alert generated with all the old data of 6/14 and 6/15 etc. below is sniper of alert on 6/19</div>
<div>
<table align="CENTER" border="0" summary="Detail Status" style="color:rgb(216,216,191);font-family:Times;background-color:rgb(0,0,0)">
<tbody>
<tr>
<td align="LEFT">
<h3>Mon Jun 19 17:48:57 PDT 2017 - Log files NOT ok</h3>
<pre><img alt="red" height="16" width="16" border="0"> Critical entries in <a href="https://monitor01.lhr9.service-now.com/xymon-cgi/svcstatus.sh?CLIENT=ora164106.sjc4.service-now.com&SECTION=msgs:/u01/shareplex/var/log/event_log" style="color:rgb(255,255,68)" target="_blank">/u01/shareplex/var/log/event_l<wbr>og</a><img alt="red" height="16" width="16" border="0"> Error 2017-06-14 12:07:24.545252 9795 1581102944 Command server: ReconcileLog: failed to construct object-cache: Illegal state: Item 372354 already in the object id registry (connecting from <a href="http://ora164106.sjc4.service-now.com" target="_blank">ora164106.sjc4.service-now.com</a><wbr>) [module osp]
<img alt="red" height="16" width="16" border="0"> Error 2017-06-14 12:07:24.545499 9795 1581102944 Command server: ReconcileLog: failed to construct object-cache: Illegal state: Item 372356 already in the object id registry (connecting from <a href="http://ora164106.sjc4.service-now.com" target="_blank">ora164106.sjc4.service-now.com</a><wbr>) [module osp]</pre>
<pre><br></pre>
</td>
</tr>
</tbody>
</table>
</div>
<div><br>
</div>
<div>Meantime, see xymonclient.log:</div>
<div>
<div>2017-06-19 17:49:01.428381 logfetch: File /u01/shareplex/var/log/event_l<wbr>og shrank from >=173538314 to 48414720 bytes in size. Probably rotated; clearing position state</div>
<div>2017-06-19 17:49:01.428462 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 48414720 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-19 17:51:05.086815 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173538314 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-19 17:53:09.<a href="tel:13%2044%2069" value="+61134469" target="_blank">134469</a> logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173538314 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-19 17:55:12.647682 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173538314 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-19 17:57:16.163913 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173538314 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-19 17:59:19.662801 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173538314 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-19 18:01:23.180499 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173538453 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-19 18:03:26.777636 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 125123733 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:42:01.519481 logfetch: File /u01/shareplex/var/log/event_l<wbr>og shrank from >=173541482 to 74420224 bytes in size. Probably rotated; clearing position state</div>
<div>2017-06-20 06:42:01.519557 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 74420224 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:44:05.173606 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173541633 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:46:08.670466 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173541633 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:48:12.188216 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173541633 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:50:15.683455 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173541633 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:52:19.250727 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173541633 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:54:22.752463 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 173541633 bytes exceeds max buffer size 10485760; skipping some data</div>
<div>2017-06-20 06:56:23.426678 logfetch: /u01/shareplex/var/log/event_l<wbr>og delta 99121409 bytes exceeds max buffer size 10485760; skipping some data</div>
</div>
<div><br>
</div>
<div>Noted. </div>
<ol>
<li>The above 2m interval is my setup of xymon client. </li><li>It seems the logfetch status file is not successfully saved and source code shows no error check (so no direct evidence).</li><li>The behavior only last under 20 min. The server itself did not have disk and cpu alerts and no one report any issues related to disk and io.</li></ol>
<div><br>
</div>
<div>I was told that this behavior is not new although rarely happen. Is there any solution or work round?</div>
<div><br>
</div>
<div>My running version is: </div>
<div>Xymon version 4.3.25-1.el6.terabithia</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>-max</div>
<div><br>
</div>
<div>
<div id="m_8706977057089294653m_4084695072152890044MAC_OUTLOOK_SIGNATURE"></div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Xymon mailing list<br>
<a href="mailto:Xymon@xymon.com" target="_blank">Xymon@xymon.com</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.xymon.com_mailman_listinfo_xymon&d=DwMFaQ&c=Zok6nrOF6Fe0JtVEqKh3FEeUbToa1PtNBZf6G01cvEQ&r=_rRsvDpSvkhydGEmSgSLQ5J0DhQTISZJFuX6D4Oq8-A&m=jlInc0qH4D6d4VJhKVlklq7Q5yzJEljN-Xn1abaymNs&s=2sE4XpgdTb8MJw0cvVJQANiKNarwuYgReKpzXR9lL4k&e=" rel="noreferrer" target="_blank">http://lists.xymon.com/mailman<wbr>/listinfo/xymon</a><br>
<br>
</blockquote>
</div>
</div>
</div>
</div>
</div></div></span>
</div>
</blockquote></div><br></div>