Update Consul agents to securely communicate with TLS

13min
|
Consul

Securing your datacenter with TLS encryption for RPC and consensus communication is a prerequisite of the Security Model. This tutorial provides the necessary steps to update your existing datacenter for production ready TLS with zero-downtime.

You must complete the following eight-steps successfully, completing each step before starting the next, to update agents to communicate over TLS.

Warning

Failing to perform the steps in the following order may lead to unplanned cluster outages due to failures in authenticating RPC TLS Communication. Complete each step in its entirety prior to moving to the next one.

Generate server certificates. Generate a client certificate if you're using the operator method.
Distribute certificates, including the CA and private key files to every agent.
Configure the servers, allowing encrypted traffic.
Restart each server, one at a time.
Configure the clients to communicate only with TLS
Restart each client.
Reconfigure servers to only allow TLS communication.
Reload each server, one at a time.

If you want to secure service-to-service communication with TLS, review the secure service communication tutorial.

Prerequisites

Before starting this tutorial, you should have an existing Consul datacenter with running server and client agents.

You will need to generate your certificates before starting this tutorial. Follow the steps in the Secure Agent Communication with TLS Encryption tutorial to use the built-in CA or the Secure Agent Communication with Existing Certificate Authority tutorial to use OpenSSL.

Finally, you should decide which certificate distribution method works for you. There are two methods for distributing client certificates; operator or auto encryption. Both methods are described in the "Secure Agent Communication with TLS Encryption" tutorial.

The example configurations for this tutorial are in JSON. You can copy each one of the examples and add them to your existing JSON configuration file.

Distribute the certificates

Before configuring the agents, you will need to distribute the certificate and key files.

Server certificates

Distribute the following certificates and key files to each of your servers.

Note, the CA certificate may be named differently if you are not using Consul's built-in CA. The server certificate and key file should be unique for each server.

Client certificate for operator method

Distribute the client certificate and key file to all your clients. All clients should have the following certificates and key file.

Configure the servers

Now that you have created the certificates you can enable TLS on your agents, starting with the servers. For both client certificate distribution methods, operator and auto encryption, you must set the following options to false.

These options must be set to false to allow existing communication to continue uninterrupted. This will require a rolling restart on all the servers. Once the clients are configured, you can then enable verify_incoming, verify_outgoing, and verify_server_hostname on the servers. In addition to the verify options, you will set the certificates and key file the agent should use.

Below are the configurations for both methods.

Note

Consul will load all files in the configuration directory, for the following examples, you can add the configuration as new files. You can also add them to an existing configuration file, however, you should ensure that the syntax is valid before applying it to the agent. You can use consul validate to check configuration validity for both JSON and HCL files.

Operator method configuration

Consul server agent manual TLS configuration

verify_incoming = false
verify_outgoing = false
verify_server_hostname = false
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"

Auto encryption method configuration

For the auto encryption method, you will need to set auto_encrypt to true.

Consul server agent auto encrypt TLS configuration

verify_incoming = false
verify_outgoing = false
verify_server_hostname = false
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"
auto_encrypt {
  allow_tls = true
}

Setting the verify options to false on the servers allows you to update all the agents without any downtime since authenticity verification is not enforced.

Note

Leaving the verify settings as false will create an insecure configuration.

Restart the Consul servers

After configuring the servers, for either the operator or auto encryption client certificate distribution method, you can initiate a rolling restart of the servers.

Be sure to restart one server at a time, ensuring it's functioning properly before restarting the next. The leader should be restarted last, to avoid downtime.

$ systemctl restart consul

Configure the clients

Now that you have enabled TLS communication on the servers and distributed the certificates, you can configure the clients.

Operator method

For the operator method, you will configure the clients to use the operator distributed certificates and key file.

Consul client agent manual TLS configuration

verify_incoming = false
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-client-consul-0.pem"
key_file = "dc1-client-consul-0-key.pem"

Auto encryption method

For the auto encryption method, you will enable auto_encrypt.

Consul client agent auto encrypt TLS configuration

verify_incoming = false
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
auto_encrypt = {
  tls = true
}

You do not need to distribute client certificate using the auto encryption method. Consul service mesh will manage certificate distribution for you.

Restart the Consul clients

Finally, restart the clients. You will only need to restart the clients once.

$ systemctl restart consul

Warning

Restarting your Consul server agents with the consul reload command may lead to an outage since this will cause the clients to lose communication to cluster server nodes. To prevent this, adjust the TLS verification settings as prescribed in the following steps.

Reconfigure and reload the servers

Operator method

For the operator method, you will update verify_incoming, verify_outgoing, and verify_server_hostname to true.

Consul server agent manual TLS configuration

verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"

Auto encryption method

For the auto encryption method, you will update verify_incoming, verify_outgoing, and verify_server_hostname to true

Consul server agent auto encrypt configuration

verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "consul-agent-ca.pem"
cert_file = "dc1-server-consul-0.pem"
key_file = "dc1-server-consul-0-key.pem"
auto_encrypt {
  allow_tls = true
}

Reload the Consul servers

After a final Consul agent reload, your agents will only communicating with TLS.

$ consul reload

Next steps

In this tutorial, you configured your Consul agents to communicate over TLS. The other prerequisites for a secure Consul deployment are gossip encryption and ACLs with the agents configured with default deny.

Additional TLS encryption resources

For more TLS content review our other tutorials:

Generate TLS certificates with OpenSSL and secure the CLI and UI communication.

Certificate Authority (CA)

Troubleshoot