[Xymon] pfSense monitoring

[Brian Scott]

Hi List,

I've just set up Xymon monitoring of a pfSense system and I thought I'd 
share a few details for others.

pfSense is a firewall system based on FreeBSD using the 'pf' packet 
filter system. The FreeBSD system is trimmed down to a smaller operating 
system than the normal distribution and has a very small number of 
available packages available in the default repository, all related to 
it's firewall functionality. I've seen reference to pulling in packages 
from the standard FreeBSD pkg system but have chosen not to go down that 
path.

My starting point is Jeremy Laidman's xymon-rclient.sh script from 
xymonton. This is a quite well thought out shell script (in contrast to 
many of the add-ons, including mine), that connects to the target system 
using ssh (or others that don't matter) and fires off a combination of 
the standard xymon client scripts and a few extra shell commands. The 
resulting output looks like the normal client log which is then handed 
back to xymon. The idea is that no special software is required on the 
target system.

After installing the script in the server/ext directory, you add a tasks 
entry (I use a tasks.d directory file) to run the script every 5 
minutes. All pretty standard stuff so far. The recommended task entry is:

[xymon-rclient]
         ENVFILE $XYMONHOME/etc/xymonserver.cfg
         CMD /$XYMONHOME/ext/xymon-rclient.sh
         LOGFILE $XYMONSERVERLOGS/xymon-rclient.log
         INTERVAL 5m

This looks for RCLIENT tags on entries in your hosts.cfg file. My tag 
looks like this:

     "RCLIENT:cmd(ssh -T -o BatchMode=yes admin at 10.0.1.54 
/bin/sh),ostype(freebsd)"

Key points include the ostype(freebsd) part which identifies the target 
system type. This identifies the client script to send to the target 
system, in this case client/bin/xymonclient-freebsd.sh.

The /bin/sh at the end of the command solves a particular problem. It 
tells the ssh daemon to run this program rather than the normal login 
shell. This is important because if you connect to a pfSense system by 
ssh (BTW this must be enabled first in the pfSense web interface), you 
are greeted by a menu rather than a command prompt. It would be possible 
to patch the rclient.sh script to send an '8' (the option for a shell) 
as the first thing it does, but that would mess up using the script for 
any other target systems. Including the /bin/sh command bypasses the 
menu and sets up the system in a shell to begin work. Note that this is 
a more or less standard bourne shell. Bash would only be available as an 
unsupported package but is also completely unnecessary.

SSH Setup
======

To use ssh you will need to generate some key pairs and transport the 
public key parts to the target system.

On the xymon server machine I did this (as root), assuming the xymon 
server runs as user called 'xymon':

     cd ~xymon

     mkdir .ssh

     chown xymon .ssh

     cd .ssh

     ssh-keygen -t rsa -f id_rsa -P ''

     ssh-keygen -t dsa -f id_dsa -P ''

     ssh-keygen -t ecdsa -f id_ecdsa -P ''

     ssh-keygen -t ed25519 -f id_ed25519 -P ''

     chown xymon *

Your ssh may have less or additional key types than mine, check your 
manpage for ssh-keygen. Also, that's two consecutive single quotes at 
the end of the line.

You then need to transfer your public keys over to the target system. To 
get your public keys, type:

     cat ~xymon/.ssh/*.pub

And copy the resulting 4 lines of output (some of which may be wrapped 
over a few lines).

We need to exchange public keys with the target server. Log on to the 
pfSense system as the local xymon user. This is important because we 
need this user to capture the remote pfSense host's public key in 
xymon's .ssh/known_hosts file.

     su - xymon -c "ssh admin at 10.0.1.54"

If you are a 'sudo' fan, adjust as necessary. Accept the remote system's 
key and then give the remote password. The pfSense menu should appear, 
select 8 to get a shell.

Create a .ssh directory:

     mkdir .ssh

And add our public keys to the system.

     cat >>.ssh/authorized_keys

Paste in our public keys from earlier and press control+D on a new line 
to end input. Type control+D again to exit the shell, then select option 
'0' to logout from the menu.

You should now be able to ssh to the pfSense system (as the xymon user) 
without accepting a new key or supplying a password. Test and verify.

     su - xymon -c "ssh admin at 10.0.1.54"

Fixes to xymon-rclient.sh
================

I've sent a diff file to Jeremy Laidman a couple of days ago. He may or 
may not change the code on xymonton. That's how these things work.

Essentially there are two essential changes to reflect the target system 
not being Linux. One is a problem with the sent command stream 
over-running a fresh shell invocation intended to set up proper 
environment variables. The other change is to some sed GNU extensions 
used. There shouldn't be any impact on other systems.

There are a couple of other minor fixes that I'd recommend but are not 
essential.

I'm including the patch in this email (not sure if the mail list server 
approves) incase you have an unpatched copy.

Fixes for Xymon
==========

The client script for a FreeBSD system includes a small C program to 
extract memory data. This is a problem since we don't want to install a 
program on the target machine. We can avoid including that section in 
the client data if the program isn't available but then we don't get 
memory tracking by xymon.

The reason for the program is that the installed memory on the system is 
available as a number of bytes. With recent systems this often excedes a 
32 bit number. The requirement is a memory size in MB and shell 
arithmetic is limited to 32 bit integers. I've implemented two 
workarounds for this:

1) A massive hack - drop the last three digits of the size and divide by 
1048 to get the size in MB. This should be surprisingly good up to 
memory sizes in TB. I've only tested to 32GB though.

2) Send the size in bytes back to xymon where it can properly process 
the numbers as unsigned longs.

Xymon already uses "memory free" from a later section (vmtotal) so we 
don't need to fake that up, although it also quite doable but commented 
out in my fix.

The other diff file includes patches for both the FreeBSD client script, 
and the xymond reader for freebsd client messages.

You have three choices depending on circumstances and how much you care:

1) Do nothing. You won't have a memory column. Remember to include a 
NOCOLUMNS:memory tag on your pfSense machine in this case.

2) Just patch the client script. This should be good to just short of 
2TB pfSense memory. pfSense doesn't need anything like that much memory.

3) Patch the script and the xymond code. Should be good up to a full 64 
bit used address space. No nasty hacks involved.

[Side note: netbsd and openbsd also have client programs for the same 
purpose. I don't have ready access to either of these systems to 
develop/test equivalent patches.]

Other systems
=========

I've also tested this on a nanobsd system (the ssh stuff is a little 
harder because of read-only file systems) and a dell server running my 
hypervisor on a recent version of FreeBSD. Also found that msdosfs file 
system on FreeBSD doesn't have inodes (I've normally handled this with 
exclusions) so added it to the inodes exclusion list.


I hope this helps someone in the future,


Brian

P.S. If this email doesn't have the diff files attached, check for 
another email following close behind with them included as the message body.
-------------- next part --------------
--- /home/brian/xymon-rclient (1).sh	2022-08-31 17:21:30.984701000 +1000
+++ xymon-rclient.sh	2022-08-31 21:42:17.063665000 +1000
@@ -140,13 +140,16 @@
 
 send_setup_commands() {
 	# sets up the environment
-	echo "/bin/sh -"
+	echo 'if [ -z "$HOME" ]; then echo "/bin/sh -"; fi'
+	sleep 1 # avoid sending any more commands until the new /bin/sh is ready for them
 	# for some reason, sunos shell sometimes corrupts the
 	# start of the command line so we need to pad it here
 	echo "################################################################"	# 64 chars
 	echo "################################################################"	# 64 chars
 	echo "XYMONTMP=$1"
 	echo "MACHINEDOTS=$2"
+	echo "XYMONHOME=/nonexistent"
+	echo "TOP=\"\`which top 2>/dev/null\`\""
 	# clean up any preamble
 	echo "echo"
 	echo "echo ---START---"
@@ -186,8 +189,8 @@
 	echo
 	echo [clock]
 	echo epoch: \${EPOCH}.000000
-	echo local: `date    "+%Y-%m-%d %H:%M:%S %Z"`
-	echo local: `date -u "+%Y-%m-%d %H:%M:%S %Z"`
+	echo local: \`date    "+%Y-%m-%d %H:%M:%S %Z"\`
+	echo local: \`date -u "+%Y-%m-%d %H:%M:%S %Z"\`
 fi
 
 -EOF-
@@ -235,7 +238,9 @@
 		echo ")"
 		send_clock_commands
 		send_finish_commands
-	) | eval $CMD $DUMPERR | sed '1,/---START---/d;$a[endmarker]\ndummy entry' &
+	) | eval $CMD $DUMPERR | sed -e '1,/---START---/d' -e '$a\
+[endmarker]\
+dummy entry' &
 	PROCPID=$!
 	[ "$PROCPID" ] || die "Failed to fork command $CMD"
 
@@ -645,9 +650,11 @@
  	[ 0$DEBUG -gt 3 ] && echo "Sending client script $XYMONSERVERROOT/client/bin/${XYMONCLIENT}-${SCRIPTOS}.sh"
 	[ 0$DEBUG -gt 4 ] && echo "Setup commands:" && send_setup_commands "$REM_XYMONTMP" "$MACHINEDOTS" | sed 's/^/  /'
  	[ 0$DEBUG -gt 4 -o "$STOPAFTERSCRIPT" ] && echo "Client script:" && (
-		send_setup_commands
+		send_setup_commands "$REM_XYMONTMP" "$MACHINEDOTS"
 		send_client_script "$EN" $XYMONSERVERROOT/client/bin/${XYMONCLIENT}-${SCRIPTOS}.sh
+		echo "("
 		send_logfetch_commands $LOGFETCHCFG
+		echo ")"
 		send_clock_commands
 		send_finish_commands
 	) | sed 's/^/  /'

-------------- next part --------------
--- client/xymonclient-freebsd.sh.orig	2015-12-17 02:13:03.000000000 +1100
+++ client/xymonclient-freebsd.sh	2022-09-02 16:50:49.868068000 +1000
@@ -28,7 +28,7 @@
 }'
 echo "[inode]"
 # The sed stuff is to make sure lines are not split into two.
-df -i -tnonfs,nullfs,cd9660,procfs,devfs,linprocfs,fdescfs | sed -e '/^[^ 	][^ 	]*$/{
+df -i -tnonfs,nullfs,cd9660,procfs,devfs,linprocfs,fdescfs,msdosfs | sed -e '/^[^ 	][^ 	]*$/{
 N
 s/[ 	]*\n[ 	]*/ /
 }' | awk '
@@ -37,7 +37,16 @@
 echo "[mount]"
 mount
 echo "[meminfo]"
-$XYMONHOME/bin/freebsd-meminfo
+if [ -x $XYMONHOME/bin/freebsd-meminfo ]; then
+	$XYMONHOME/bin/freebsd-meminfo
+else
+	physmem=`sysctl -n hw.physmem`
+	#vmfree=`sysctl vm.vmtotal | grep 'Free Memory:' | cut -w -f3 | cut -dK -f1`
+	total=`echo $physmem|sed -e 's/...$//'`
+	echo Total:$(($total/1048))
+	#echo Free:$(($vmfree/1024)) # the file reader gets this from the [vmtotal] section so we don't need to.
+	echo Physmem:$physmem # reader can avoid our dodgy maths completely
+fi
 echo "[swapinfo]"
 swapinfo -k
 echo "[vmtotal]"
--- xymond/client/freebsd.c.orig	2016-02-03 07:16:19.000000000 +1100
+++ xymond/client/freebsd.c	2022-09-02 09:59:12.485795000 +1000
@@ -82,6 +82,7 @@
 	if (meminfostr) {
 		p = strstr(meminfostr, "Total:"); if (p) { memphystotal = atol(p+6); found++; }
 		p = strstr(meminfostr, "Actual:"); if (p) memphysactual = atol(p+7);
+		p = strstr(meminfostr, "Physmem:"); if (p) { memphystotal = strtoul(p+8,NULL,10)/(1024 * 1024); found++; }
 	}
 
 	if (vmtotalstr) {