<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]--><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
h3
{mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:13.5pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:blue;
text-decoration:underline;}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=blue>
<div class=Section1>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>Thanks <st1:City w:st="on"><st1:place
w:st="on">Vernon</st1:place></st1:City>,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>This was a great summary. I was hoping to
run the server in the container,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>but I think I’ll move it to the global
zone.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>I will work on a separate client for
solaris zones. Has anyone out there<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>started writing one?<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'>James<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div>
<div class=MsoNormal align=center style='text-align:center'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>
<hr size=2 width="100%" align=center tabindex=-1>
</span></font></div>
<p class=MsoNormal><b><font size=2 face=Tahoma><span style='font-size:10.0pt;
font-family:Tahoma;font-weight:bold'>From:</span></font></b><font size=2
face=Tahoma><span style='font-size:10.0pt;font-family:Tahoma'> Vernon Everett
[mailto:everett.vernon@gmail.com] <br>
<b><span style='font-weight:bold'>Sent:</span></b> Tuesday, February 23, 2010
8:34 PM<br>
<b><span style='font-weight:bold'>To:</span></b> hobbit@hswn.dk<br>
<b><span style='font-weight:bold'>Subject:</span></b> Re: [hobbit] Problems
running on a Solaris Zone</span></font><o:p></o:p></p>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><font size=3
face="Times New Roman"><span style='font-size:12.0pt'>Hi James<br>
<br>
I put a lot of effort into this recently, and there does not appear to be any
real practical solution to the problem.<br>
The problem is caused by how zones use memory and kernel space.<br>
<br>
In sparse zones, all kernels are the same kernel. There is only one instance of
the kernel running, and as a result, only one chunk of memory visible to the
kernel. <br>
<br>
When you set a memory cap in your zone definition, and do a prtconf in the
zone, it reports the value of the memory cap as the available memory.<br>
So far, so good.<br>
<br>
However, to determine free memory, we have to interrogate the kernel. This can
be done a number of different ways. Xymon, by default uses vmstat.<br>
You can also use kstat -p unix:0:system_pages:freemem and I am sure there are
others.<br>
However, the kernel in question, is the kernel running in the global zone!<br>
It's all one kernel.<br>
So the reported memory free is the free memory available to the kernel. It
should be the same value in all the zones too.<br>
<br>
The error you are seeing occurs when free memory available to the global kernel
is more than the memory cap you have placed on the zone.<br>
In C (and many other programming languages), if you subtract big numbers from
smaller numbers, you sometimes get strange results depending on how your
variables are defined. I think that's where your multi-Petabyte memory is
coming from. Any programmers out there that can confirm this?<br>
<br>
The other problem this creates, is that any sane-looking zone memory percentages
are meaningless. They do not represent the true memory utilisation within the
zone. Your zone memory utilisation could be 100%, and you would not realise it,
because your kernel is still seeing heaps of free memory, and reporting lots
free.<br>
Imagine a 2gb cap, and the apps in the zone are using all 2gb.<br>
However, the kernel can see 1.8gb free. <br>
Do the maths. Xymon tells us your zone is only using 10% of memory, which is
far from the truth.<br>
<br>
The only real way round it might not fit with your policies and methods.<br>
You need to remove all memory caps.<br>
This floats all memory, meaning that the memory "seen" in the zone,
is the same as the kernel, and Solaris does the management of memory, ensuring
all zones get enough.<br>
It also means that all of the zones will show identical memory graphs.<br>
<br>
The other way, which I haven't had time to do yet, is to use prstat -Z in the
global zone.<br>
This gives a summary of what the zones are using, which might be worth
tracking.<br>
<br>
As a short-term workaround, because we need memory caps for certain apps, we
have skipped memory monitoring on the zones. (It's pretty meaningless anyway -
see above)<br>
We have the global zone, and below it, all the zones, with the NOCOLUMNS:memory
bb-hosts tag.<br>
<br>
It's not really ideal, but I hope to find time to revisit this in the near
future.<br>
<br>
It would be nice to be able to disable just the memory test on these, and only
keep an eye on swap. Swap is local to the zone, and if you start using heaps of
it in the zone, or are doing lots of paging, chances are you are maxing out
your memory allocation.<br>
So swap is probably a good indicator.<br>
<br>
Sorry I could not be of any more help.<br>
<br>
Regards<br>
<st1:City w:st="on"><st1:place w:st="on">Vernon</st1:place></st1:City><br>
<br>
<br>
<br>
<o:p></o:p></span></font></p>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'>On Wed, Feb 24, 2010 at 1:35 AM, James Wade <<a
href="mailto:jkwade@futurefrontiers.com">jkwade@futurefrontiers.com</a>>
wrote:<o:p></o:p></span></font></p>
<div link=blue vlink=purple>
<div>
<h3><b><font size=2 color=black face=Arial><span style='font-size:10.0pt;
font-family:Arial;color:black;font-weight:normal'>Has anyone see this problem.
I’ve just compiled 4.3.0.0.beta2 on a<br>
Solaris 10 system. I’m running on a Sun T5120 series in a Solaris<br>
sparse zone. </span></font></b><o:p></o:p></h3>
<h3><b><font size=2 color=black face=Arial><span style='font-size:10.0pt;
font-family:Arial;color:black;font-weight:normal'>When I run the server, I get
the following on the memory test.<br>
Fyi.. I don’t have 4.2 peta bytes of memory </span></font></b><font size=2
color=black face=Wingdings><span style='font-size:10.0pt;font-family:Wingdings;
color:black;font-weight:normal'>J</span></font><o:p></o:p></h3>
<h3><b><font size=2 color=black face=Arial><span style='font-size:10.0pt;
font-family:Arial;color:black;font-weight:normal'>Has anyone seen similar
problems. Running the client in the global zone works fine.</span></font></b><o:p></o:p></h3>
<h3><b><font size=4 color=black face="Times New Roman"><span style='font-size:
13.5pt;color:black'>Tue Feb 23 10:52:43 CST 2010 - Memory CRITICAL</span></font><o:p></o:p></b></h3>
<pre><font size=2 color=black face="Courier New"><span style='font-size:10.0pt;
color:black'> Memory Used Total Percentage</span></font><o:p></o:p></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt;color:black'><img
border=0 width=16 height=16 id="_x0000_i1025"
src="cid:image001.gif@01CAB56E.B9EF7AF0" alt=red> Physical 4294966186M 26624M 4294967292%</span></font><o:p></o:p></pre><pre><font
size=2 color=black face="Courier New"><span style='font-size:10.0pt;color:black'><img
border=0 width=16 height=16 id="_x0000_i1026"
src="cid:image002.gif@01CAB56E.B9EF7AF0" alt=green> Swap 148M 26623M 0%</span></font><o:p></o:p></pre><pre><font
size=2 color="#d8d8bf" face="Courier New"><span style='font-size:10.0pt;
color:#D8D8BF'> </span></font><o:p></o:p></pre>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'> </span></font><o:p></o:p></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=3 face="Times New Roman"><span style='font-size:12.0pt'> <o:p></o:p></span></font></p>
<p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><font
size=3 face="Times New Roman"><span style='font-size:12.0pt'>Thanks…James<o:p></o:p></span></font></p>
</div>
</div>
</div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>