Setting Up Mutual TLS Authentication To Protect Your System

Every console in your system serves as an access point, and each must remain secure while connected to the system. While you might have set up other security measures such as strong passwords and transport layer security (TLS), how do you ensure some consoles are only accessible to you and a few others?

One way is by requiring mutual TLS authentication at each access point. Here’s more on mutual TLS and how to set one up.

TLS, formerly known as SSL, is an encryption protocol used all over the internet to authenticate the server in a client-server connection. It also encrypts client and server communications so outsiders cannot spy on them.

Mutual TLS (mTLS) authentication, on the other hand, ensures the parties at each end of the connection are authenticated and verify they are who they say they are. The two parties authenticate each other simultaneously in the authentication protocol; that’s why it’s also known as two-way authentication. 

A typical TLS process works as follows:

The client connects to the server

The server presents its TLS certificate

The client verifies it

They exchange information over encrypted TLS connections.

However, with mTLS, there are additional steps. After the client verifies the server’s certificate, it presents its TLS certificate to the server; the server verifies it and grants access, after which they share information.

mTLS is used as the default authentication mode for protocols such as Secure Shell Protocol (SSH), Internet Key Exchange (IKE), and others. It’s often used in the Internet of Things (IoT) since writing security schemes for such systems can be challenging, especially if they are to remain lightweight and have low computational costs.

Mutual authentication supports zero-trust networking and can help protect against various malicious attacks. Security schemes that employ mTLS use varied encryption, communication, and verification methods but share one thing: every party involved in the communication must be verified.

To set up a mutual TLS, you need a create a certificate both parties can trust. A CA is simply a public and private key with the public key containing an X.509 certificate. Anything encoded with the public key can only be decoded using the private key and vice versa. Thus, for a server to decrypt a message, it requires a key, and having it proves its authenticity.

After setting up a CA, you need to create a server certificate for the domain name system you are using. First, you will create the key by creating a certificate signing request (CSR) which the certificate authority used to create the signed server certificate.

You will repeat the same process in step 2 to create the client key and certificate.

With all the keys and certificates ready, the next step is configuring the server and client. Here the request Cert, reject Unauthorized, and ca options are employed to require the client to show a signed CA certificate so as to communicate with the server. The key and cert options allow the server to show its signed CA server certificate to the client.