[hobbit] Setting this up and monitoring VM Servers?

[Buchan Milne]

On Thursday 25 June 2009 13:36:10 T.J. Yang wrote:
> ________________________________
>
> > From: Vernon.Everett at woodside.com.au
> > To: hobbit at hswn.dk
> > Date: Thu, 25 Jun 2009 09:23:00 +0800
> > Subject: RE: [hobbit] Setting this up and monitoring VM Servers?
> >
> >
> >
> >
> >
> >
> >
> >
> > I started putting something together for Devmon to pull
> > infor from the VMWare host using SNMP.
> >
> > Then we did a reorg here at work, and suddenly we had way
> > more work than we could handle, and no time to play with things like
> > Devmon modules.
> >
> > I posted my (incomplete) work on the list, in the hope
> > somebody would take over, but I know of no takers.
> >
> > Search the archive for me, Devmon and ESX. You should find
> > it there somewhere.
>
> I did try out devmon some time ago but I don't have urgent need to monitor
> snmp devices. and stop putting time into devmon since Henrik is working on
> snmp query/monitoring in 4.(3/4).0 version.

Well, besides the most basic network interface utilisation graphs, NBAR 
protocol utilisation graphs, QoS graphs, and monitoring and graphing of CPU 
and memory usage on all network devices, we use devmon for:

-monitoring HP ProLiant:
--RAID array status
--Fan status
--Temperature (with graphs)
--Integrated Management Log
--Power supply status

We have replaced a number of power supplies which had intermittent problems 
that would not otherwise have been easily picked up.

-Monitoring for Dell PowerEdge:
--RAID array status
--Fan status
--Temperature (with graphs)
--Power supply status

-Radius servers:
--Monitoring per-RADIUS-client accounting and authentication requests

-Linux servers:
--Red Hat cluster status
--Per-Disk IO graphs

-Firewalls:
--Graph of connections
--Monitoring of cluster status

-Load balancer (templates still in progress)
--Connections to each real server
--Connections to the virtual server
--Server farm status

Anti-spam appliances:
--CPU utilisation
--Fans
--Keys for licensed features (e.g., warn when expiration imminent)
--Memory
--Power supplies
--RIAD arrays
--Resources  (graphs of disk io, open files, and threads)
--Temperatures (with graphs)

One of my colleagues is using devmon for agent-less monitoring of Windows 
servers (using microsoft-win2k3server template available in svn). Some 
agentless monitoring is also available for Linux (e.g., linux-netsnmp or 
linux-openwrt templates)

In most cases, templates can be developed quite easily. The VMWare case seems 
to be a bit difficult, as the IDs they use as indexes in some of the SNMP tables 
don't match up to the IDs they use as indexes in other tables.

Now, while MRTG can do some of this, most of the more complex templates 
(ProLiant IML, RAID for ProLiant and PowerEdge, Cisco CSM load balancer) I 
think can not be done with MRTG, and while they can be done with Cacti with 
some effort, there's no decent integration with any monitoring system.

AFAICT, most of the work done on Xymon for SNMP is following the (not so 
great) MRTG model. And besides, I get all the features Xymon's SNMP support 
will bring in future right now.

And, using the data, I can do things like this:
http://staff.telkomsa.net/~bgmilne/weathermap/
(every link that is not grey would have a pup-up to the Hobbit graph of e.g. 
network utilisation, and any host with a background colour to the text will 
have popup of a Hobbit service graph - conn by default - it works fine from 
inside our network, but won't work from outside it now).

I am trying to finish up this code so I can publish a final version.

Besides that, I think we are just about monitoring everything we can monitor 
with SNMP, at this stage the only think I can do to monitor more things via 
SNMP with devmon is to start filing bugs on SNMP agents (e.g. net-snmp can 
monitor mail queues, but only for sendmail, and we are running postfix on 
anything with more than a trivial queue on it).

Regards,
Buchan