SSL Certificate checking

[Adam Goryachev]

I understand that hobbit (and bbgen) will check the validity of SSL
certificates on a HTTPS site, but I was wondering if hobbit (or bbgen)
would also check that a ssh certificate does NOT change?

Note, all the rest of this email is off-topic, so please don't respond
to it on the list. Feel free to send your comments offlist.

Reason being, this morning one of my servers was hacked, I found out
because:
*) BB noticed /var/log/messages was truncated
*) BB noticed sshd wasn't running any longer

I then noticed, because the SSH key had been changed, and basically
someone had compiled a new ssh and in the process changed the key. It
would have been nice had BB detected that as well (since a hacker might
not always truncate log files, nor change the process name of ssh, even
though it is still running).

For those that are interested, and I'd be keen to hear from people
(probably off-list) regarding their thoughts/suggestions.

This machine is running debian testing, and I have a BB ext which alerts
me if updates are available but not installed, so I install them daily,
so it is always up to date.
The machine runs a kernel which likely to have a local exploitable bug
(2.4.25)
The machine has open services to the internet of:
*) apache-perl (from debian)
*) DJB's tinydns (from debian source package)
*) DJB's qmail (from debian source package)
*) ssh (from debian)

apache-perl is serving up RT (from debian) and no other CGI/etc

qmail calls qmail-scanner-queue.pl which calls spamassassin + clamav
which are also both from debian.

The machine is listed as secondary MX for a load of domains, and also
primary NS for a load of domains.

The machine had 4 users with a password set (root + 3 admin users) all
the rest were disabled in /etc/shadow.

As for password brute-force, I've had john running for over an hour, and
it hasn't found anything yet, at 1221 attempts per second, I think that
comes to 1025640 passwords it has tried.....
guesses: 0  time: 0:01:10:13 (3)  c/s: 1221  trying: agig1

ie, the password for the 4 users are not easily guessable.... password
are never sent in cleartext either...

Basically, so far as I can tell, the person has set a password for user
games, compiled/installed openssh (into /usr/local/), and that's all I
can see so far.

The thing that bothers me most is that this is a debian (testing)
machine, with all the patches/updates etc, and yet it was still hacked.

My suspicion is that they gained access via ssh, since they went to the
trouble of replacing that....

My fear is that I won't find HOW they got in, and therefore can't put
the machine back online with any degree of confidence that it won't
happen again....

As above, please send comments/suggestions to me offline.

Regards,
Adam

-- 
 -- 
Adam Goryachev
Website Managers
Ph:  +61 2 8304 0000                        adam at websitemanagers.com.au
Fax: +61 2 8304 0001                        www.websitemanagers.com.au