<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
h5
{mso-style-priority:9;
mso-style-link:"Heading 5 Char";
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:10.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.Heading5Char
{mso-style-name:"Heading 5 Char";
mso-style-priority:9;
mso-style-link:"Heading 5";
font-family:"Calibri Light",sans-serif;
color:#2E74B5;
mso-fareast-language:EN-GB;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";
mso-fareast-language:EN-GB;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I have tried running xymon_rrd with the debug option and found some errors.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I took a sample host where the df output looks like this<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">Filesystem 1024-blocks Used Available Capacity Mounted on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">udev 1999540 0 1999540 0% /dev<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">tmpfs 403788 41084 362704 11% /run<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">/dev/mapper/rootvg-lv--root 57456900 16628168 37886988 31% /<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">tmpfs 2018924 0 2018924 0% /dev/shm<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">tmpfs 5120 0 5120 0% /run/lock<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">tmpfs 2018924 0 2018924 0% /sys/fs/cgroup<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">/dev/sda1 482922 249320 208668 55% /boot<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">tmpfs 403788 0 403788 0% /run/user/1000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I get errors like the ones below<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">31712 2017-08-10 10:56:57.362148 server/disk,root.rrd: Ignored duplicate (and identical) update timestamped 1502359017<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">31712 2017-08-10 10:56:57.362154 server/disk,run,lock.rrd: Ignored duplicate (and identical) update timestamped 1502359017<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">31712 2017-08-10 10:56:57.362158 server/disk,sys,fs,cgroup.rrd: Ignored duplicate (and identical) update timestamped 1502359017<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">31712 2017-08-10 10:56:57.362167 server/disk,run,user,1000.rrd: Ignored duplicate (and identical) update timestamped 1502359017<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">The timestamps on the RRD files with the errors do not update at the times when these errors appear. In the above example there is
no error for disk,boot.rrd, which relates to the /boot file system, and the timestamp on that file updated at 10:56 while the ones mentioned in the error messages did not. Each time xymond_rrd runs the errors relate to a slightly different set of files.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I also noticed the rrd_status.log file has messages like “server/memory.real.rrd: Bug - duplicate RRD data with same timestamp 1502336737,
different data”.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We only see these messages in relation to servers that have missing points in the graphs. This behaviour is similar to this problem
<a href="http://lists.xymon.com/archive/2016-April/043387.html">http://lists.xymon.com/archive/2016-April/043387.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I checked on what is suggested there but rrdstatus and rrddata are running on separate channels so I can’t see where the duplicate
data is coming from.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Jeremy Laidman [mailto:jlaidman@rebel-it.com.au]
<br>
<b>Sent:</b> 10 August 2017 02:34<br>
<b>To:</b> Stephen Barrie <sbarrie@brightsolid.com><br>
<b>Cc:</b> xymon@xymon.com<br>
<b>Subject:</b> Re: [Xymon] Graph lines for disk and memory showing gaps<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Barrie<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This is quite interesting. On the one hand, most of the gaps in the lines don't line up with gaps in other lines. On the other hand, on the graph with four lines, the red and pink lines appear to have gaps in the same place 100% of the
time. And the green line only appears when the blue line appears. It's as if there are two concurrent problems that are masking each others' symptoms.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Yes, you can run xymond_rrd with the "--debug" switch, to enable more output in the log file.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Rather than re-configuring the parameters for xymond_rrd in tasks.cfg and restarting, you can run a second copy of it, which runs independently of the main one, creating and updating its own copy of the RRD files, and logging to STDOUT
for you to peruse in real time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Typically there are two xymond_rrd processes running, one for status messages and the other for data messages. The one we're interested in is the one for the status messages, because the data are parsed from the "disk" message body. You
can run the following (as xymon user) to see what's going on with this:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">$ sudo -u xymon mkdir -p /tmp/myrrd</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">$ sudo -u xymon xymoncmd xymond_channel --channel=status \</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> --filter='^@@status.*/hostname.example.com.*\|disk\|' xymond_rrd \</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> --rrddir=/tmp/myrrd --debug</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This will create RRD files in /tmp/myrrd for the named host and for only the "disk" data. You can then use rrdtool to view the data and look for signs of gaps. Note that xymond_rrd will cache updates to the RRD file, and so you might not
get updates to the RRD file straight away, so you can either wait 30 minutes to review, or add the --no-cache switch. Also note that RRD requires more than just one sample for it to start trusting the data, so you would want to wait for at least two samples
before relying on the contents of the RRD file.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If that doesn't show any hints, then a more systematic approach may be warranted. There are several stages between the Xymon client creating the client data message and the RRD file being created, where messages could go missing. Here's
a rough summary:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">xymon-client.sh => df<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> => xymon [Client message]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> => xymond<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> => xymond_client -> [status message]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> => xymond<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> => xymond_channel --status<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> => xymond_rrd<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> => rrd<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What I'd be doing is checking each phase of this to determine where the messages are being dropped/corrupted. In most cases, you can impersonate the existing process and review the results to infer what's happening in the actual process.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">J<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 10 August 2017 at 01:41, Stephen Barrie <<a href="mailto:sbarrie@brightsolid.com" target="_blank">sbarrie@brightsolid.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Attached are sample memory and disk graphs. You will see the gaps in one line do not equate with the
gaps in others. The gaps vary in length and there is no clear pattern. Sometimes it is for minutes but they could be for more than an hour. The output from the df command in the client still updates the server every few minutes and this never seems to miss
any data. It is difficult to get the df output that coincides with gaps in the graphs as we don’t see the graphs are missing data until after the fact, but we can see that the gaps are not at the same times for all machines or all disks.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The server where we are not experiencing the problem is running 4.3.17 on Red Hat Linux. The others
are 4.3.25 on Ubuntu 16.04 and 4.3.7 on Ubuntu 14.04. There are no RRD or other related errors in the logs.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">A problem with the RRD parser seems a logical explanation. Is there a way of tracking what this is
doing?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Stephen</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Jeremy
Laidman [mailto:<a href="mailto:jlaidman@rebel-it.com.au" target="_blank">jlaidman@rebel-it.com.au</a>]
<br>
<b>Sent:</b> 09 August 2017 14:51<br>
<b>To:</b> Stephen Barrie <<a href="mailto:sbarrie@brightsolid.com" target="_blank">sbarrie@brightsolid.com</a>><br>
<b>Cc:</b> <a href="mailto:xymon@xymon.com" target="_blank">xymon@xymon.com</a><br>
<b>Subject:</b> Re: [Xymon] Graph lines for disk and memory showing gaps</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Can you show an example/screencap of a graph with the gaps? Do all servers and/or disks have gaps at the same time? How big are the gaps - one or two 5-minute sample size, or much
larger?<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The Windows disk usage message is parsed by different code to that which parses the UNIX and Linux "df" output. That might have something to do with it.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Is 4.3.25 in use on all Xymon servers?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Check the Xymon logs for error messages. Perhaps the RRD parser is crashing.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">It's possible that the "df" output is sometimes not parseable by the RRD parser, and so the data is ignored. Can you show the [df] section from the client data during one of the
gaps, and at another time when there are no gaps?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">J<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On 9 August 2017 at 22:13, Stephen Barrie <<a href="mailto:sbarrie@brightsolid.com" target="_blank">sbarrie@brightsolid.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We have a few Xymon servers running on Ubuntu 14.04 and 16.04 that show gaps in the graphs for disk and memory metrics. This specifically relates to Linux and Unix clients reporting
to these servers. The graph lines for Windows clients look ok.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">If the same Unix and Linux clients are pointed to another server running Red Hat we do not see the same gaps in the graphs. Upgrading the client version does not resolve this. So
it seems there is a problem with graph data being created on the Xymon server on Ubuntu. We are using the package version 4.3.25. Is there a known issue with this?<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Stephen Barrie<o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<h5>This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail
does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.<o:p></o:p></h5>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
__________________________________________________________________________<br>
<br>
This email has been checked for virus and other malicious content prior to leaving our network.<br>
__________________________________________________________________________<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><br>
_______________________________________________<br>
Xymon mailing list<br>
<a href="mailto:Xymon@xymon.com" target="_blank">Xymon@xymon.com</a><br>
<a href="http://lists.xymon.com/mailman/listinfo/xymon" target="_blank">http://lists.xymon.com/mailman/listinfo/xymon</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<h5>This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail
does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.<o:p></o:p></h5>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><br>
__________________________________________________________________________<br>
<br>
This email has been checked for virus and other malicious content prior to leaving our network.<br>
__________________________________________________________________________<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<hr>
<h5>This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail
does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.</h5>
<hr>
<br clear="both">
__________________________________________________________________________<BR>
<BR>
This email has been checked for virus and other malicious content prior to leaving our network.<BR>
__________________________________________________________________________<BR>
</body>
</html>