Information about Hobbit architecture

[Henrik Stoerner]

Hi Wolfgang,

your mails were in my spam mailbox for some reason. I then started
writing a reply directly to You, but decided to send it to the list also
- so it can be referenced through the mail archive if anyone else needs
it.

>Now I need your help, because my professor at my university wants me
>to present the software architecture of Hobbit in my thesis. He wants
>some pictures and diagramms showing the 'inner mechanisms' and how the
>different modules work together. So my questions is, do you have any
>information about the architecture, that you can share with me?

I don't have any formal design documents; the Hobbit design has evolved
from a few basic principles and some ideas I had. I'll try to give you
a quick summary.

Hobbit should be portable across Unix architectures. This is obviously
important for the Hobbit client code, but I've done my best to use only
"standard" Unix and C for both the server- and client-code. This has turned 
out to be easier than I thought; there are some really old Unix systems 
that cannot run the Hobbit server, but any recent version of a Unix-like 
system (released within the past 5-10 years) runs Hobbit without problems.

Hobbit also has to be backwards compatible with Big Brother, in the
sense that you can use BB clients with a Hobbit server. I had over 1000
servers with a BB client on them (Unix and Windows), so doing a "Big
Bang" switch changing the servers and clients at once would be
impossible.

Hobbit must scale well. When I started using Big Brother I had 40 
servers to monitor. When I got to 100, I had to re-implement parts of BB 
and this became the "bbgen" toolkit. When I got to 500, my BB server was
getting overloaded, and I started to work on a replacement. Today I have
2500 servers, and before Christmas I will be monitoring nearly 4000
servers. That's a 100x increase in just 5 years.

Hobbit should not rely on a lot of other infrastructure to work. E.g. it
doesn't require a huge database backend; you can add one if you like,
but it is not needed. Keeping Hobbit simple makes it robust - and you
really, REALLY want your monitoring to work when everything else is
crashing. We once had a major power outage at our datacenter; the Hobbit 
server came up quickly, but there were a lot of systems that needed some 
manual intervention to get back online. It was quite interesting to see 
how the activity on the Hobbit server just sky-rocketed, because everyone 
was looking at Hobbit to see which systems were running, and which were down.


Analyzing data
==============
For me, a key principle of handling the data that is poured into Hobbit
is that as much as possible of the data analysis should take place on
the Hobbit server. I believe it is a huge benefit to keep configuration
settings in one place (the Hobbit server); also, by having access to the 
raw data you can also perform types of analysis that you didn't think of
when a script was initially created to collect some data.

E.g. the Hobbit client reports data about who is logged on to a system.
This information is not currently used by Hobbit, but I know that
someone wrote a custom backend utility to check this data and alert him
if there was someone logged in as "root". I had not thought of this, but
by making the raw data from the client available, it was very easy for
him to implement this check on all of his servers.

This is quite different from Big Brother, where the data is
pre-processed into status messages. The BB client can check if a
process is running, but then it just reports "process foo is OK".
When process "foo" is NOT running you get an error status - but you
cannot easily see if process "foo" has stopped because your backup is
running at the same time (and they cannot coexist) because you only get
part of the information, not the full process listing.

This may seem like a trivial example, but I realized early on that there
are far more ways of using these data than I could possibly imagine.
So instead of forcing my ideas of how to use the data upon others, it 
should be possible to just get the raw data and perform your own analysis 
of it.

Another example is that in Hobbit 4.2, I added a module which saves a
copy of the client data if a status goes red on a host. This has turned
out to be extremely helpful in diagnosing those "why did the webserver
crash at 4 AM last Tuesday" questions ... because you have access to a
lot of raw data collected by the client just before the crash happened,
including all of the data that Hobbit didn't analyze by itself but which
humans can use to put the whole picture together.

This is not implemented completely yet. The network test utility - which
was also carried over from the bbgen toolkit - works the "Big Brother" 
way. One thing on my agenda is to change that, so the network tester
just reports that the ping of host "foo" responded in 12.7 ms, the ping of
host "bar" failed and so on. Then a module on the Hobbit server can
decide if these should result in a red or yellow status, perhaps based
on other information it has (eg that the response time shouldn't exceed
10 ms during working hours, unless the primary network connection was
down so we were running on a backup line with less capacity).


The core daemons
================
I wanted to have a network daemon holding all of the "current state" 
information. This information changes all the time as new status
reports arrive, so it has to keep this in memory - writing it to disk
would be too slow (BB did this, and it doesn't scale). So a core
component of Hobbit would be this central daemon (hobbitd). The daemon
NEVER does any disk I/O; this would slow it down and I don't want that,
because Hobbit must support monitoring of thousands of servers. All
communication between hobbitd and the outside world goes via a network
connection; this is used both for in-band data (status updates and data
messages), but also for out-of-band data like control messages (drop a
host, disable a server and so on). Tools that need to fetch the entire
status of all servers, or just the detailed status of a host also do
this through a network connection to hobbitd.

However, some things must be stored on disk - RRD (graph) files, for
instance, or historical eventlogs. So this is handled by a bunch of
independant "worker" modules - hobbitd_rrd (RRD updates),
hobbitd_history (history logs), hobbitd_alert (sending out alerts).
These obviously have to be fed information about the data that flows
into the hobbitd daemon - e.g. hobbitd_rrd needs the full status message
to extract the data it puts into the RRD files, and hobbitd_history
needs information about the status changes from green->red and so on.
So I needed a fast inter-process communication mechanism between hobbitd
and the worker modules. Also, I wanted to be able to start/stop/restart
worker modules on-the-fly; this is extremely nice for testing and makes
the system much more robust. Finally, I wanted an interface that was
simple to use so that end-users can hook into the data stream if they
need to write some custom back-end script. The solution for this was
a mechanism that uses the System V "shared memory" IPC mechanism,
combined with a group of semaphores to control access to the shared
memory area. So hobbitd copies a message into the shared memory area
and up's a semaphore telling the workers that there is a new message.
The workers then pick up the message and down's another semaphore once
they have secured their copy of the message; hobbitd then knows when it
is safe to overwrite the shared-memory area with a new message. I call
this IPC mechanism a "channel", and there are in fact several of these:
One for each type of message. So there is a channel which receives all
of the raw "status" messages; another channel for the raw "data"
messages; a channel that receives messages about status changes (for
history logging); a channel that receives messages about critical
red/yellow statuses (for alerts) and so on. Recently a new channel was
added for the "client" messages that comes from the Hobbit client.

There are some early notes about this mechanism in the
hobbitd/new-daemon.txt file in the hobbit sources. Not all of the ideas
there have been implemented, e.g. the "streaming" protocol turned out
not to be particularly important.

To make sure that the semaphore stuff is handled correctly, I decided to
put a "buffer" module between hobbitd and the actual workers. This is
the hobbitd_channel module; it serves only one purpose, which is to grab
the messages that hobbitd sends out through the IPC mechanism, and queue
them for the real worker module (hobbitd_rrd, hobbitd_history etc). The
fact that hobbitd_channel acts as a message queue is useful to
accomodate spikes in the activity, e.g. the alert module sometimes gets
a huge spike of messages e.g. when a network switch dies.
hobbitd_channel also makes it easy to build your own backend modules,
because it forwards the messages via a simple text-based pipe; so your
custom backend modules can just read them from stdin.

Another benefit of having hobbitd_channel between hobbitd and the worker
modules showed up recently; I am currently working on a new version of
hobbitd_channel which can distribute the incoming messages between multiple
worker "clones" running on different servers, to perform some load
balancing of the heavy tasks (primarily RRD file updates). This has been
implemented almost exclusively by changing hobbitd_channel, instead of
having to modify all of the worker modules.

So the core design looks like this:


   Network tests --\
                    \
            TCP:1984 \           IPC
   Clients ----------> hobbitd --------> hobbitd_channel ---> worker module
                     /         Sh. mem.                 stdin
                    /
   Custom tests ---/


The Web interface
=================
The web interface is mostly carried over from Hobbit's predecessor, the
"bbgen" toolkit. I wrote this for Big Brother, to speed up the
generation of the Big Brother webpages, and by re-using this in Hobbit I
would quickly get a working web interface - all I had to do was to
change the programs to grab their data from the hobbitd daemon, instead
of reading through the status logfiles that Big Brother uses.

This also means that the web interface is not tied in with the core
daemons. Sure, they need to communicate and there are some things in the
core daemons that are closely related to how the web interface works -
e.g. disabling a host. But it should be possible for an adventurous
programmer to use the core Hobbit daemons with their own web front-end
tools and come up with a completely different user-interface.

So the web interface is probably the part of Hobbit that has evolved the
least from it's origins in Big Brother. Some new CGI programs have been
added, but nothing revolutionary new - it just picks up bits of
information from hobbitd and the configuration files and displays them.

One design criteria for the web interface is that it should be as
dynamic as possible; it must reflect the current status and
configuration as much as possible. That is why most of the web interface
is done with CGI programs; the only static webpages in Hobbit are the
overview pages generated by bbgen - and I hope to eliminate those soon.


The clients
===========
So with this background, it is obvious that the Hobbit client is really,
really dumb. It is basically just a shell script that runs some normal
OS commands - df, ps, who and so on - and then it's up to the Hobbit
server to analyze them and generate some status columns. Client data is
sent to hobbitd, which feeds it through a channel to the hobbitd_client
module. hobbitd_client has some parsers for each of the operating
systems it knows about, and uses those to grab the interesting data and
compare it to the client configuration rules. Then hobbitd_client
generates some "status" messages and sends them to hobbitd. The major
challenge with this design is logfiles; you cannot realistically send
entire logfiles - some of them are several GB of data - over to Hobbit
for analysis every 5 minutes. So some filtering must be done on the
client side; to keep all of the configuration data on the Hobbit server
this meant that the client has to pick up its filter-configuration from
the Hobbit server.


I hope this is enough of an overview for You. Good luck with your
thesis.


Regards,
Henrik