How to rotate TLS/CA certificates
This document describes the process of rotating TLS and CA certificates.
Summary
Rotation of TLS certificates
Charmed OpenSearch uses three types of TLS certificates:
app-admin: used for the administration of the OpenSearch cluster.unit-transport: used for the communication between the OpenSearch nodes.unit-http: used for the communication between OpenSearch and clients.
Two scenarios trigger the rotation of TLS certificates:
- The certificate has expired/is about to expire: In this case, the Charmed OpenSearch will automatically request a new certificate.
- You want to rotate the certificate: In this case, you can manually request a new certificate.
Manually rotate the TLS certificates
You can manually start a rotation of the TLS certificates by changing the TLS private key for the type of certificate you want to rotate.
juju run-action opensearch/leader set-tls-private-key category=<category>
Where <category> is one of app-admin, unit-transport, or unit-http.
This will automatically generate a new private key and regenerate the certificate signing request (CSR) for the specified category. The new CSR will be sent to the operator you are using to provide the certificates for signing. Once the new certificate is signed, it will be automatically applied to the OpenSearch cluster.
For more information on the different approaches to update the key please refer to the “Update keys” section of How to enable TLS encryption.
Rotation of CA certificates
The CA certificate is used to sign the TLS certificates. The CA certificate is provided to the OpenSearch cluster by the operator you are using to provide the certificates. In this section, we will describe the process of rotating the CA certificate using the self-signed-certificates operator and the manual-tls operator.
Rotate the CA certificates using the self-signed-certificates operator
Currently, the self-signed-certificates operator does not support the rotation of the CA certificate. If you need to rotate the CA certificate, you will need to start the rotation process manually.
You can manually start the rotation of the common name (CN) used for the CA certificate by changing the ca-common-name configuration option of the self-signed-certificates operator.
juju config self-signed-certificates ca-common-name=<new-ca-common-name>
The self-signed-certificates operator will automatically generate a new CA certificate with the new common name, revoke all TLS certificates that were issued with the previous CA certificate and send their invalidation to the OpenSearch cluster.
Upon receiving the information about the revoked TLS certificates, the OpenSearch cluster will automatically request new TLS certificates from the self-signed-certificates operator. After generating these certificates, they will be provided to OpenSearch. Charmed OpenSearch checks each available TLS certificate for a new CA thereby triggering a rolling restart of the cluster to apply the new CA certificate.
Until the rolling restart is complete, the OpenSearch cluster will ignore the new TLS certificates and not apply them to the nodes. This will only be done once all nodes in the cluster have updated the new CA and can communicate using the newly issued TLS certificates.
As indicated in the “Check certificates in use” section of How to enable TLS encryption, you can check the certificates in use by running the following command:
openssl s_client -showcerts -connect `leader_unit_IP:port` < /dev/null | grep issuer
Where leader_unit_IP is the IP address of the leader unit and port is the port number of the OpenSearch service. This command will show the issuer of the certificate in use which should include the new CA certificate common name.
Rotate the CA certificates using the manual-tls-certificates operator
The manual-tls-certificates operator is used to manually provide the TLS certificates to the OpenSearch cluster. To rotate the CA certificate using the manual-tls-certificates operator, you will need to manually start the TLS certificate rotation process and sign them using the new CA certificate.
To start the rotation process, remove the relation between the OpenSearch cluster and the manual-tls-certificates operator and re-add it:
juju remove-relation opensearch manual-tls-certificates
# Wait for the relation to be removed
juju integrate opensearch manual-tls-certificates
This will trigger the OpenSearch cluster to generate new CSRs for the TLS certificates. You can then sign the new CSRs using the new CA certificate and provide the new certificates to the OpenSearch cluster using the manual-tls-certificates operator:
juju run manual-tls-certificates/leader provide-certificate \
relation-id=<id> \
certificate="$(base64 -w0 certificate.pem)" \
ca-chain="$(base64 -w0 ca_chain.pem)" \
ca-certificate="$(base64 -w0 ca_certificate.pem)" \
certificate-signing-request="$(base64 -w0 csr.pem)" \
unit-name="<unit-name>"
When OpenSearch detects the new CA certificate, it will automatically issue a rolling restart to apply the new CA certificate and the new TLS certificates. This might take some time depending on the size of the cluster. During the process, the OpenSearch cluster will request new TLS certificates from the manual-tls-certificates operator so make sure to provide these certificates as well.
As indicated in the “Check certificates in use” section of How to enable TLS encryption, you can check the certificates in use by running the following command:
openssl s_client -showcerts -connect `leader_unit_IP:port` < /dev/null | grep issuer
Where leader_unit_IP is the IP address of the leader unit and port is the port number of the OpenSearch service. This command will show the issuer of the certificate in use which should include the new CA certificate common name.