Oracle Cloud Infrastructure Documentation

Network Resource Configuration for Cluster Creation and Deployment

Before you can use Container Engine for Kubernetes to create and deploy clusters in the regions in a tenancy:

  • The tenancy's root compartment must include a policy to allow Container Engine for Kubernetes to perform operations in the tenancy. See Create Policy for Container Engine for Kubernetes (Required).
  • Within the tenancy, there must already be a compartment to contain the necessary network resources (such as a VCN, subnets, internet gateway, route table, security lists). If such a compartment does not exist already, you will have to create it. Note that the network resources can reside in the root compartment. However, if you expect multiple teams to create clusters, best practice is to create a separate compartment for each team.
  • Within the compartment, network resources (such as a VCN, subnets, internet gateway, route table, security lists) must be appropriately configured in each region in which you want to create and deploy clusters. When creating a new cluster, you can have Container Engine for Kubernetes automatically create and configure new network resources for a new 'quick cluster'. Alternatively, you can explicitly specify the existing network resources to use for a 'custom cluster'. If you specify existing network resources, you or somebody else must have already configured those resources appropriately, as described in this topic.

This topic describes the necessary configuration for each network resource. To see details of a typical configuration, see Example Network Resource Configurations.

For an introductory tutorial, see Creating a Cluster with Oracle Cloud Infrastructure Container Engine for Kubernetes.

Root Compartment Configuration

You have to define a policy for the tenancy's root compartment to enable Container Engine for Kubernetes to perform operations on the tenancy. See Create Policy for Container Engine for Kubernetes (Required).

VCN Configuration

The VCN in which you want to create and deploy clusters must be configured as follows:

  • The VCN must have a CIDR block defined that is large enough for the number of subnets you specify for the clusters you create. For example, to create a highly available cluster will typically require five subnets to support the necessary number of hosts and load balancers. However, you can create clusters with fewer subnets. A /16 CIDR block would be large enough for almost all use cases (10.0.0.0/16 for example). The CIDR block you specify for the VCN must not overlap with the CIDR block you specify for pods and for the Kubernetes services (see CIDR Blocks and Container Engine for Kubernetes).
  • The VCN must have an internet gateway defined. See Internet Gateway Configuration.
  • If you intend to deploy worker nodes in private subnets, the VCN must have a NAT gateway defined. See NAT Gateway Configuration.
  • The VCN must have a route table defined that has a route rule specifying the internet gateway as the target for the destination CIDR block. If you intend to deploy worker nodes in private subnets, the route table must also have a route rule specifying the NAT gateway as the target for the destination CIDR block. See Route Table Configuration.
  • The VCN must have an appropriate number of subnets defined. For example, to create a typical highly available cluster will require three subnets in different availability domains in which to deploy worker nodes, and two subnets to host load balancers. However, you can create clusters with fewer worker nodes, and fewer or no load balancers, and therefore require fewer subnets. See Subnet Configuration.

  • The VCN must have security lists defined for worker node subnets and load balancer subnets (if specified). See Security List Configuration.

In addition, Oracle recommends DNS Resolution is selected for the VCN.

See VCNs and Subnets and Example Network Resource Configurations.

Internet Gateway Configuration

The VCN in which you want to create and deploy clusters must have an internet gateway. The internet gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 in a route rule in a route table.

See VCNs and Subnets and Example Network Resource Configurations.

NAT Gateway Configuration

If you intend to deploy worker nodes in private subnets, the VCN must have a NAT gateway (in addition to an internet gateway) to enable the worker nodes to initiate connections to the internet. The NAT gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 in a route rule in a separate route table.

See NAT Gateway and Example Network Resource Configurations.

Route Table Configuration

The VCN in which you want to create and deploy clusters must have a route table. The route table must have a route rule that specifies an internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy worker nodes in private subnets, a separate route table must have a route rule that specifies a NAT gateway as the target for the destination CIDR block 0.0.0.0/0 to enable the worker nodes to initiate connections to the internet. This route table is in addition to the route table specifying an internet gateway as a target.

See Internet Gateway and Example Network Resource Configurations.

DHCP Options Configuration

The VCN in which you want to create and deploy clusters must have DHCP Options configured. The default value for DNS Type of Internet and VCN Resolver is acceptable.

See DHCP Options and Example Network Resource Configurations.

Security List Configuration

The VCN in which you want to create and deploy clusters must have security lists defined for worker node subnets and load balancer subnets (if specified). The security lists for worker node subnets and load balancer subnets must be different. The security list for load balancer subnets must be unique and for their exclusive use.

Worker nodes are created with public or private IP addresses, according to whether you specify public or private subnets when defining the node pools in a cluster. In both cases, worker nodes must be able to make outbound connections to the internet. Container Engine for Kubernetes must also be able to access worker nodes.

See Security Lists and Example Network Resource Configurations.

Public Worker Node Subnet Security List configuration

When configuring a security list for public worker node subnets, the security list must have:

  • stateless ingress and egress rules that allow all traffic between the different worker node subnets
  • stateless ingress and egress rules that allow all traffic between worker node subnets and load balancer subnets (if specified)
  • an egress rule that allows all outbound traffic to the internet
  • ingress rules to allow Container Engine for Kubernetes to access worker nodes on port 22 from the following source CIDR blocks:
    • 130.35.0.0/16
    • 134.70.0.0/17
    • 138.1.0.0/16
    • 140.91.0.0/17
    • 147.154.0.0/16
    • 192.29.0.0/16

Optionally, you can include ingress rules for public worker node subnets to:

Private Worker Node Subnet Security List configuration

When configuring a security list for private worker node subnets, the security list must have:

  • stateless ingress and egress rules that allow all traffic between the different worker node subnets
  • stateless ingress and egress rules that allow all traffic between worker node subnets and load balancer subnets
  • an egress rule that allows all outbound traffic to the internet

Optionally, you can include ingress rules for private worker node subnets to:

Subnet Configuration

The characteristics of the cluster you want to create, and the number of availability domains in the region in which you want to deploy the cluster, will determine the number of subnets to configure. For example, to create a typical highly available cluster will require three subnets in different availability domains in which to deploy worker nodes, and two subnets to host load balancers.

Note

Note that Container Engine for Kubernetes does not yet support regional subnets, so you cannot currently deploy worker nodes or host load balancers in regional subnets.

The VCN in which you want to create and deploy clusters must have at least one subnet defined in which to deploy worker nodes. Worker node subnets can be either public, or private for additional security (as specified by the Subnet access property). The number of worker node subnets to create depends on the region in which you are creating the cluster:

  • If you are creating a cluster in a region with multiple availability domains, you can define multiple worker node subnets. Each worker node subnet must be in a different availability domain.
  • If you are creating a cluster in a region with a single availability domain, you can only define one worker node subnet.

You have the option to define and use load balancers in clusters you create. If you want to define and use load balancers, the VCN in which you want to create and deploy clusters must have at least one subnet defined to host the load balancers. Load balancer subnets can be public or private (as specified by the Subnet access property) and must each be in a separate availability domain. However, load balancers are optional, so you might not define load balancer subnets at all. The number of load balancer subnets to define depends on the region in which you are creating the cluster:

  • If you are creating a cluster in a region with three availability domains, you can define zero or two load balancer subnets. If you define two load balancer subnets, the two load balancer subnets must be in different availability domains.
  • If you are creating a cluster in a region with a single availability domain, you can define zero or one load balancer subnet.

In addition, all subnets must have the following properties set as shown:

  • Route Table: The name of the route table that has a route rule specifying an internet gateway (for public worker node subnets) or NAT gateway (for private worker node subnets) as the target for the destination CIDR block 0.0.0.0/0
  • DHCP options: Default

The CIDR blocks you specify for worker node and load balancer subnets must not overlap with CIDR blocks you specify for pods running in the cluster (see CIDR Blocks and Container Engine for Kubernetes).

Worker node subnets must have different security lists to load balancer subnets.

See VCNs and Subnets and Example Network Resource Configurations.