Oracle Cloud Infrastructure Documentation

Encrypting Kubernetes Secrets At Rest in Etcd

The master nodes in a Kubernetes cluster store sensitive configuration data (such as authentication tokens, passwords, and SSH keys) as Kubernetes secret objects in etcd. Etcd is an open source distributed key-value store that Kubernetes uses for cluster coordination and state management. In the Kubernetes clusters created by Container Engine for Kubernetes, etcd writes and reads data to and from block storage volumes in the Oracle Cloud Infrastructure Block Volume service. Although the data in block storage volumes is encrypted, Kubernetes secrets at rest in etcd itself are not encrypted by default.

For additional security, when you create a new cluster you can specify that Kubernetes secrets at rest in etcd are to be encrypted using the Oracle Cloud Infrastructure Key Management service (see Overview of Key Management). Before you can create a cluster where Kubernetes secrets are encrypted in the etcd key-value store, you have to:

  • know the name and OCID of a suitable master encryption key in Key Management
  • create a dynamic group that includes all clusters in the compartment in which you are going to create the new cluster
  • create a policy authorizing the dynamic group to use the master encryption key

Having created the cluster and specified that you want Kubernetes secrets at rest in the etcd key-value store to be encrypted, you can optionally restrict the use of the master encryption key by modifying the dynamic group to include just that cluster.

Note the following:

  • You can only select the option to encrypt the Kubernetes secrets in the cluster's etcd key-value store when creating a new 'custom cluster'. You cannot encrypt Kubernetes secrets in the etcd key-value stores of existing 'custom clusters', or in the etcd key-value stores of 'quick clusters'.
  • You can only select the option to encrypt Kubernetes secrets in the cluster's etcd key-value store if you specify Kubernetes version 1.13.x or later as the version of Kubernetes to run on the master nodes of the cluster.
  • After you've specified a master encryption key for a new cluster and created the cluster, do not subsequently delete the master encryption key in the Key Management service. As soon as you schedule a key for deletion in Key Management, the Kubernetes secrets stored for the cluster in etcd become inaccessible. If you have already scheduled the key for deletion, it might still be in the Pending Deletion state. If that is the case, cancel the scheduled key deletion (see To cancel the deletion of a key) to restore access to the Kubernetes secrets. If you allow the scheduled key deletion operation to complete and the master encryption key to be deleted, the Kubernetes secrets stored for the cluster in etcd are permanently inaccessible. As a result, cluster upgrades will fail. In this situation, you have no choice but to delete and recreate the cluster.

Using the Console

To create a new 'custom cluster' where Kubernetes secrets are encrypted in the cluster's etcd key-value store:

  1. Log in to the Console.
  2. If you know the OCID of the master encryption key to use to encrypt Kubernetes secrets, go straight to the next step. Otherwise:
    • If a suitable master encryption key already exists in Key Management but you're not sure of its OCID, follow the instructions in To view key details and make a note of the master encryption key's OCID.
    • If a suitable master encryption key does not already exist in Key Management, follow the instructions in To create a new key to create one. Having created a new master encryption key, make a note of its OCID.
  3. Create a new dynamic group containing all the clusters in the compartment in which you intend to create the new cluster:

    1. Open the navigation menu. Under Governance and Administration, go to Identity and click Dynamic Groups.
    2. Follow the instructions in To create a dynamic group, and give the dynamic group a name (for example, acme-oke-kms-dyn-grp).
    3. Enter a rule that includes all clusters in the compartment in the format:

      ALL {resource.type = 'cluster', resource.compartment.id = '<compartment-ocid>'}

      where <compartment-ocid> is the OCID of the compartment in which you intend to create the new cluster.

      For example:

      ALL {resource.type = 'cluster', resource.compartment.id = 'ocid1.compartment.oc1..aaaaaaaa23______smwa'}
    4. Click Create Dynamic Group.

    Having created a dynamic group that includes all clusters in the compartment, you can now create a policy to give the dynamic group access to the master encryption key in Key Management.

  4. Create a new policy to enable use of the master encryption key:

    1. Open the navigation menu. Under Governance and Administration, go to Identity and click Policies.

    2. Follow the instructions in To create a policy, and give the policy a name (for example, acme-oke-kms-dyn-grp-policy).
    3. Enter a policy statement to give the dynamic group access to the master encryption key, in the format:

      Allow dynamic-group <dynamic-group-name> to use keys in compartment <compartment-name> where target.key.id = '<key-OCID>'

      where:

      • <dynamic-group-name> is the name of the dynamic group you created earlier.
      • <compartment-name> is the name of the compartment containing the master encryption key.
      • <key-OCID> is the OCID of the master encryption key in Key Management.

      For example:

      Allow dynamic-group <acme-oke-kms-dyn-grp> to use keys in compartment acme-kms-key-compartment where target.key.id = 'ocid1.key.oc1.iad.annrl______trfg'
    4. Click Create to create the new policy.
  5. Follow the instructions to create a new 'custom cluster' in Using the Console to create a 'Custom Cluster' with Explicitly Defined Settings, select the Encrypt Using Customer-Managed Keys option, and select:

    • Choose a Vault in <compartment-name>: The vault that contains the master encryption key, from the list of vaults in the specified compartment. By default, <compartment-name> is the compartment in which you are creating the cluster, but you can select a different compartment by clicking Change Compartment.
    • Choose a Key in <compartment-name>: The name of the master encryption key, from the list of keys in the specified compartment. By default, <compartment-name> is the compartment in which you are creating the cluster, but you can select a different compartment by clicking Change Compartment. Note that you cannot change the master encryption key after the cluster has been created.
  6. (Optional) Having created the cluster, for additional security:

    1. Make a note of the OCID of the new cluster you just created.
    2. Restrict the use of the master encryption key by modifying the dynamic group rule you created earlier to explicitly specify the OCID of the new cluster, rather than all clusters in the compartment. For example:

      resource.id = 'ocid1.cluster.oc1.iad.aaaaaaaaaf______yg5q'

Using the API

For information about using the API and signing requests, see REST APIs and Security Credentials. For information about SDKs, see Software Development Kits and Command Line Interface.

Use the CreateCluster operation to create a cluster.