[Xymon] Encrypted Xymon reporting over SSL using stunnel

[Adam Goryachev]

On 14/3/19 16:00, Jeremy Laidman wrote:
>
> On Wed, 13 Mar 2019 at 10:57, Adam Goryachev 
> <mailinglists at websitemanagers.com.au 
> <mailto:mailinglists at websitemanagers.com.au>> wrote:
>
>     IMHO, when using "security" tools, we really need to make sure we
>     rely
>     on a third party "library" to do this, security/encryption is a fast
>     moving area, and we really don't want to be using something 5
>     years old
>     because nobody understood how to update it from TLS1.1 to TLS1.2
>     or from
>     using 3DES to using AES or whatever the issue becomes.
>
>
> I'm not sure I agree that encryption is fast-moving in all areas, or 
> even most of them. Yes, there are bugs found in encryption software, 
> short-cuts found in brute-forcing hashes, new algorithms every now and 
> then, and the occasional implementation problem (wep, wpa, ssl); all 
> of these scenarios require some kind of upgrade. However, the math(s) 
> behind the basic crypto primitives is fundamentally sound (well-vetted 
> and in some cases proven), particularly around simple use cases. Even 
> with deprecated hashes (MD5, SHA1) they're not considered insecure, 
> just that a well-resourced, and highly motivated and patient actor can 
> brute-force them in time. With careful selection of hashing, 
> encipherment and session key management algorithms, a good, strong 
> solution can be implemented using >20-year-old crypto.
>
> However, there's other value in being able to use a separate library, 
> and allow upgrades to the library to improve security or squash bugs, 
> without having to recompile Xymon.
>
>     Also, xymon may
>     not be updated as easily/quickly as what other system libraries
>     might,
>     so again, better to let the OS (system) provide the relevant tools
>     to do
>     this, and we just become another consumer.
>
>
> Agreed.
>
>
>     As such, an obvious choice would be openssl, or something similar,
>     where
>     it is widely supported across many platforms, well maintained, and
>     likely to be well maintained in the future.
>
>
> Not to mention, OpenSSL is already used by Xymon for things like https 
> checks.
>
>     I think there are two separate requirements when referring to
>     encryption:
>     1) Encrypt the content so anyone listening can't see your log file
>
>
> Yep.
>
>     2) Signature - ie, ensuring the message actually came from "one of my
>     clients" or taking this further it came from "this specific client".
>
>
> Yep.
>
> And there are situations where only signing is required. This not only 
> ensures no tampering, but it also ensures that a large message has not 
> been truncated by some process. On some systems, I often get a red 
> "procs" test because the [ps] or [netstat] sections were so long that 
> the data message was truncated. I'd prefer to skip a truncated message 
> entirely than to react to conclusions based on incomplete data, at 
> least in this case. A [signature] section at the end of the message 
> (or heck, even just an [end] section) and appropriate handling in 
> xymond, would fix this.
>
> Having encryption also gives us signing, and having signing also 
> ensures no tampering. If encryption and signing aren't required, the 
> overhead of managing keys makes it burdensome to detect truncated 
> messages. But if we bake in encryption, we get everything. If we use 
> x509 certificates, we get all of this, plus potentially PKI (federated 
> trust, revocation, expiry). It's a hierarchy, and the higher up we go, 
> the harder it is to implement (both in the code, and rolling it out to 
> nodes), but the more features we get.
>
> (As an aside I like the idea that an x509 certificate can be used to 
> tie a client data message to an entry in hosts.cfg, either allowing 
> the commonName field to override the value in the client data message, 
> or having a "CN" tag in the hosts.cfg file to match the client message 
> to.)
>
> Also, key management needs to be considered. It's a different task to 
> encrypt traffic between a xymonproxy server and xymond server, to 
> rolling out encryption to 500 devices each running a Xymon client.
>
>     - Do we need full PGP security for this? At least during setup, we
>     can
>     easily start with a known good/clean connection, or rely on the
>     admin to
>     ensure that some "shared secret" is valid on both ends.
>
>
> So, like bootstrapping the key rollout procedure. This could leverage 
> the existing "download" client message type. The client should know 
> when her certificate is soon to expire, and can just ask for a new one 
> with: "xymon download /meta/clientcert.x509" and the xymond process 
> could sign a new cert on-the-fly and send it down the wire.
>
>     - We can already send data back to the client after the client
>     successfully delivers data to the server. If the "current" data the
>     client is sending is considered "clean/secure" then we can easily
>     send
>     back a renewed "certificate".
>
>
> Yep, that's another option.
>
>     So...
>     a) Start with a simple shared secret that the admin can create (or is
>     auto-generated) on server install.
>     b) The admin configures this secret onto each client that is added,
>     along with adding the hostname to the server (xymon-hosts file).
>
> b1) admin configures ssh public key on client
>
>     c) When the client connects to the server, it uses the SSH
>     "encryption"
>     to protect the conversation, sends the shared key plus the normal
>     "1st
>     set of data reports". The server responds with a client "certificate".
>     d) When the client next connects to the server, it uses the SSH
>     "encryption" to protect the conversation, and signs the data with the
>     certificate it received from the server. If the certificate is due to
>     expire in X days (configurable on server) then the server will
>     provide
>     an updated certificate to the client. The client can continue to
>     use the
>     "old" certificate until expiry, but should allow plenty of time to
>     transition to the new one. In reality, this could be one or two days
>     only, since most clients report every 5 mins.
>
>
> e) Delete the shared secret.
>
> I think there's too much going on here, but not sure where. It seems 
> to me that there's already an authentication process going on using 
> asymmetric keys via the ssh authentication mechanism. It would seem 
> that an extra key to authenticate the client is redundant, because the 
> client has already authenticated itself.

Actually, I was suggesting the SSH style encryption which would be used 
for encryption only, not signing, in my mind that was a separate process 
which works with the contents. In my thought process, the SSH encryption 
is kind of like HTTPS, where any client can connect to the server 
(without authentication) but still ensure that the content is 
encrypted/protected.

Within this, there would be another signature to sign the content which 
is used to confirm it came from client abc and not random unknown, nor 
from known client xyz.

>
> Also, could the shared key be a problem if it gets out? Let's say I 
> have 50 devices all using certs, but still with the shared key still 
> in place, just in case of certificate corruption requiring a new 
> bootstrapping procedure. Now, let's say someone hacks into one of the 
> devices and retrieves the shared key. That key can now be used to 
> bootstrap any new device to trust a rogue certificate authority, or at 
> least to sign reports that are rejected by Xymon for being signed by 
> the wrong certificate. Perhaps this could be used as part of a 
> man-in-the-middle attack?
I think the shared key is only enough to "add" a client, and only if 
that client is already listed in the xymon-hosts file. So, the value of 
this shared key is pretty low. Worst case, it could allow a rogue client 
to get a valid certificate for a host that is listed but doesn't use 
encryption/certificates, either because it is a old client that doesn't 
support it, or it is a platform that isn't supported, etc...
>
> I think I'd be more comfortable if the shared key was unique to each 
> device. But ultimately, I think it shouldn't be required, because the 
> ssh keys are per-device keys, and should be sufficient to give the 
> same effect.
>
>     Or... we could configure a shared secret in the xymon-hosts file for
>     each client (different "password" for each client) and the client
>     simply
>     encrypts all data it returns with this shared secret. Advantage is
>     "simplicity" disadvantage is much less secure.
>
>
> Why less secure? I'm not disagreeing, just want to understand why.
 From my (limited) understanding, a lot of crypto attacks are based on 
dictionary attacks. Since pretty much 100% of our content is plain text, 
and a lot of that will be "recent dates, repeated a lot throughout the 
content", ie, log files... then it would make it easier to find out the 
"key" if you know (or can guess) the plain text that was encrypted. My 
understanding is that this is less relevant when you use some of the 
more modern encryption processes along with PKI. I guess everyone can 
know the content of the index.html that a user will get over https, so I 
assume that https encryption processes are able to overcome this issue.
>
> Another option is to do the ssh host key thing, where the first time a 
> connection is made to a host, everyone just accepts the new host key 
> into their known_hosts file (presumably after checking the fingerprint 
> out-of-band, lol).
>
> So one way to do this for the Xymon client would be to have the client 
> fetch its unique key the first time it connects to the server, and use 
> it to sign from then on. I think this much the same as the "a" through 
> "e" steps you outlined above, except not necessarily using 
> certificates - but it could be a certificate, or a private key, or 
> both client private key and server public key, or a unique per-client 
> shared key, or a one-for-all-clients shared key.
>
Right, it could even be that some tool on the client connects to the 
server and requests the certificate, and you a second interactive tool 
on the server will receive/process that request, once the admin approves 
it, then the response is sent back to the client. I mean, who doesn't 
have two ssh sessions open (one client and one server) when adding a new 
client?
> A few more comments to consider.
>
> * It's easy to do crypto badly. Whatever is done here (if anything), 
> we shouldn't be rolling our own crypto, and instead we need to use 
> standard crypto primitives provided by a well-known and 
> highly-regarded library such as OpenSSL. Even that is not sufficient; 
> one could choose the wrong type of cipher (eg a block cipher where a 
> stream cipher is appropriate), or use a hash function that requires 
> sufficient source data volume to properly initialise, to has a 
> password that could be only 8 characters. There are standardised ways 
> of doing crypto that are recognised as optimal, or at least sound, 
> such as PBKDF2 for deriving robust keys, salting hashes, and so on. If 
> it's going to be built in, it needs to be done right.
>
> * Certificates don't need to be x509. Secure shell (ssh) can support 
> certificates for authentication, but they aren't x509 certs, and from 
> what little I've read, their use seems to be further away from the 
> "full-blown PKI" end of the spectrum and closer to simple asymmetric 
> key pairs. The one thing that ssh's certs cannot do is revocation. 
> This means if I use the same cert on all boxes I login to, and my 
> private key is exposed, I need to change my key on each and every one 
> of those boxes. One solution to this is suitable key management, eg 
> storing my private key on an HSM or Yubikey; another is to use a 
> unique key for each box I want to authenticate to (essentially giving 
> myself multiple identities). This problem isn't necessarily relevant 
> to the discussion about securing client data messages, but it 
> highlights the fact that this is complex, and different requirements 
> may benefit from having quite different solutions available.
>
> * Certs give us the /potential /to use PKI, but to get all the 
> benefits of PKI one also needs ancillary services such as an OCSP or 
> CRL server and proper key management. However, many organisations 
> don't have PKI infrastructure, so the benefits of PKI such as 
> revocation lists, don't warrant the overhead, and self-signed certs or 
> other forms of key pair technology, should be available for those who 
> aren't able to afford or run a complete PKI solution. Non-PKI 
> essentially becomes mutual authentication between pairs of nodes (one 
> always being the Xymon server). If a client key is compromised, both 
> ends need remediation. If a server key is compromised, all clients 
> need remediation. This is usually good enough for most smaller 
> organisations.
>
I mostly understand, and think I agree with most of the above. At the 
end of the day, the solution needs to be:

a) backwards compatible, so you can still accept plain, unsigned, 
unencrypted connections from the coffee maker in the corner

b) simple to install and maintain, and robust. If it's too hard to 
install or maintain, then people won't want to use it.

> * I see no reason why we can't support ssh just with sufficient 
> documentation. The same goes for or stunnel, IPSec, OpenVPN, etc. 
> They're all useful to people in different circumstances. This just 
> requires documenting each one. There may be cases where a script or 
> two would benefit these different methods (an extreme case is 
> xymon-rclient.sh, which IMHO is waaaay too big to be included as a 
> simple "helper" script for one of these methods).
>
Possibly we could run the existing 1984 listener in a wrapper which 
handles all the "encryption", and then within xymon itself, there are a 
small number of additional "types" that can be sent from client, or 
returned to the client to add the additional certificate/signing stuff.

So, if we can easily add a command line flag that says enable "STARTSSL" 
command, that in itself would be a huge step forward. IMAP and POP and I 
think SMTP can all do this, so it should be a pretty well defined 
protocol, and should be well supported in common libraries.

Again, lets not try and jump and touch the moon, one step at a time...

> * If we are to do this right, we should include support for both 
> client-side and server-side authentication.
>
>     PS, as for IPv6, I am yet to see any decent level of support here, I
>     tried to start using it years ago, and it just never happened.
>     Turns my
>     routers (Cisco Meraki: www.meraki.com <http://www.meraki.com>
>     details at
>     https://documentation.meraki.com/zGeneral_Administration/Other_Topics/IPv6_Device_Compatibility)
>
>     still don't support IPv6 either, so it still doesn't seem to be as
>     important as claimed by everyone 10+ years ago when they said we
>     would
>     run out of IPv4 addresses.... It's hard to "change the world", I
>     mean,
>     we still have SMTP and DNS ;)
>
>
> Yep, this is true, overall IPv6 take-up continues to be slow. However, 
> the thing about Xymon is, it needs to be able to monitor a wide range 
> of devices and services. So if you have 1000 nodes to monitor, and 
> just one of them runs IPv6, your Xymon has to support IPv6 in order to 
> detect a service outage. This is the situation I'm in. My work-around 
> is to disable "conn" checks for such hosts, and run a script that does 
> a ping6 and creates the "conn" status messages. Also, certain sectors 
> of the industry are likely to have a relatively large IPv6 footprint, 
> such as telcos, and IoT network providers.

True, there are many Meraki customers requesting IPv6, I'm just 
acknowledging that it is a much slower process than it seemed to be 
advertised. It originally seemed like we would all be forced to IPv6 
within a couple of years.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.xymon.com/pipermail/xymon/attachments/20190314/2d908b23/attachment.html>