<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3059" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007>I try to identify filesystem "space hogs" via
custom scripts I wrote a long time ago when using BB. 99% of my custom
stuff is done in PERL.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007>I use 'du -k' to get the size of all directories in the
filesystem. I then cut those results down to only the first and
second level directories (but you could go as deep as you want). I store
the size of each subdirectory in a small "database". I did this ages ago
and my code uses PERL's "Storable" module to store the accumulated
date into a file (called my "database"). These days I'd just use
Hobbit's easily accessed RRD files. I then use PERL's
Statistics::Descriptive::least_squares_fit() to calculate the slope and linear
correlation coefficient of the "best fit line". This allows me to see how
fast each subdirectory is growing/shrinking, and how linear that
growth/reduction is. I trigger yellow/red conditions based on rate of
growth and predicted fill time at current growth rate, in addition to
the standard "95% full = red" test.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007>The above makes it fairly easy to identify which
subdirectory is your problem, which is often times good enough to identify the
file/process that is killing you. When that's not, I have a seperate test
that tries to identify problem files a different way. BB/Hobbit uses
'top' to identify cpu-hogging processes. Many times you see files hogging
space are directly tied to processes hogging cpu (runaway process = runaway file
in many cases). 'top' identifies the process(es), then "lsof -p
<pid>" is used to identify the files that the suspect process has
open. Finding a cpu-hogger that has a filespace-hogger open is usually the
holy grail you seek.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007>As a "repair" action for Hobbit, I squirreled away 2Gb
of diskspace in 100Mb chunks for critical filesystems. "dd if=/dev/zero
of=/filesystem/DiskSpaceReserve/reserve01 bs=1024 count=102400", then "cp
reserve01 reserve02", etc. to build up the reserve. A seperate Hobbit
"notification script" is used to simply delete files from this reserve under
dire circumstances, after normal email/pager notifications have failed to
trigger action by developers/production support people.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=348265218-06082007>My BB/Hobbit custom scripts tend to get quite
involved. Probably too much so, but they're fun for me to
write!</SPAN></FONT></DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Gary Baluha [mailto:gumby3203@gmail.com]
<BR><B>Sent:</B> Monday, August 06, 2007 7:29 AM<BR><B>To:</B>
hobbit@hswn.dk<BR><B>Subject:</B> Re: [hobbit] Highlights of the 4.3.0
version<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=348265218-06082007><FONT face=Arial color=#0000ff
size=2> <FONT face=Tahoma color=#000000>< ... snip ...
></FONT></FONT></SPAN></DIV>
<DIV><SPAN class=348265218-06082007></SPAN> </DIV>
<DIV><EM>One <BR>example is disk space. A full filesystem would shut
many things down.<BR>Apps should not fill a filesystem, but sometimes they
do. So my custom<BR>Hobbit scripts first scream and scream about low
disk space, even<BR>analysing things down to specific subdirectories and fast
growing files<BR>and doing trend analysis. But if their call is not
answered, they start<BR>freeing up space from a "private reserve" I have set
aside to deal with <BR>emergencies. So if we experience a sudden
unexpected blowup in a<BR>filesystem at 3am, Hobbit keeps things running in
production until the<BR>appropriate people can look into and diagnose the
problem. This may not<BR>be Utopian behavior, but it sure is
practical at 3am in the morning!</EM></DIV>
<DIV>
<DIV><BR>What sort of trend analysis do your scripts perform? We have a
few boxes that are notorious for filling up their disk space, and I haven't yet
come up with an idea of how to neatly track exactly what it is that keeps
filling up the disk. <SPAN class=348265218-06082007><FONT face=Arial
color=#0000ff size=2> </FONT></SPAN></DIV>
<DIV><SPAN class=348265218-06082007></SPAN> </DIV>
<DIV><SPAN class=348265218-06082007><FONT face=Arial size=2>< ... snip
...></FONT> </SPAN></DIV></DIV></BODY></HTML>