Oracle Cloud Infrastructure Documentation

Network Resource Configuration for Cluster Creation and Deployment

Before you can use Container Engine for Kubernetes to create and deploy clusters in the regions in a tenancy:

  • The tenancy's root compartment must include a policy to allow Container Engine for Kubernetes to perform operations in the tenancy. See Create Required Policy for Container Engine for Kubernetes.
  • Within the tenancy, there must already be a compartment to contain the necessary network resources (such as a VCN, subnets, internet gateway, route table, security lists). If such a compartment does not exist already, you will have to create it. Note that the network resources can reside in the root compartment. However, if you expect multiple teams to create clusters, best practice is to create a separate compartment for each team.
  • Within the compartment, network resources (such as a VCN, subnets, internet gateway, route table, security lists) must be appropriately configured in each region in which you want to create and deploy clusters. When creating a new cluster, you can have Container Engine for Kubernetes automatically create and configure new network resources for a new 'quick cluster'. Alternatively, you can explicitly specify the existing network resources to use for a 'custom cluster'. If you specify existing network resources, you or somebody else must have already configured those resources appropriately, as described in this topic.

This topic describes the necessary configuration for each network resource. To see details of a typical configuration, see Example Network Resource Configurations.

For an introductory tutorial, see Creating a Cluster with Oracle Cloud Infrastructure Container Engine for Kubernetes.

Root Compartment Configuration

You have to define a policy for the tenancy's root compartment to enable Container Engine for Kubernetes to perform operations on the tenancy. See Create Required Policy for Container Engine for Kubernetes.

VCN Configuration

The VCN in which you want to create and deploy clusters must be configured as follows:

  • The VCN must have a CIDR block defined that is large enough for the number of subnets you specify for the clusters you create. For example, to create a highly available cluster in a region with three availability domains will typically require two regional subnets (recommended) or five AD-specific subnets to support the necessary number of worker nodes and load balancers. However, you can create clusters with fewer subnets. A /16 CIDR block would be large enough for almost all use cases (10.0.0.0/16 for example). The CIDR block you specify for the VCN must not overlap with the CIDR block you specify for pods and for the Kubernetes services (see CIDR Blocks and Container Engine for Kubernetes).
  • The VCN must have an appropriate number of subnets defined. For example, to create a highly available cluster in a region with three availability domains, the VCN must include:

    • For worker nodes: a regional subnet (recommended), or three AD-specific subnets (one in each of the availability domains).
    • For load balancers: optionally (but usually) an additional regional subnet (recommended), or an additional two AD-specific subnets (each in a different availability domain).

    However, you can create clusters with fewer worker nodes, and fewer or no load balancers, and therefore require fewer subnets. Best practice is to use regional subnets to make failover across availability domains simpler to implement. See Subnet Configuration.

  • The VCN must have security lists defined for worker node subnets and load balancer subnets (if specified). See Security List Configuration.

In addition:

  • Oracle recommends DNS Resolution is selected for the VCN.
  • If you expect applications deployed on a cluster to require worker nodes to initiate connections to the internet, the VCN must have an internet gateway (if the worker nodes are in public subnets) or a NAT gateway (if the worker nodes are in private subnets). See Internet Gateway Configuration and NAT Gateway Configuration.
  • If you expect applications deployed on a cluster to pull images from Oracle Cloud Infrastructure Registry (or to use other Oracle Cloud Infrastructure resources) and you don't want to expose the data to the public internet, you can define a service gateway in the VCN. See Service Gateway Configuration.
  • If the VCN has a NAT gateway, an internet gateway, or a service gateway, it must have a route table with appropriate rules defined. See Route Table Configuration.

See VCNs and Subnets and Example Network Resource Configurations.

Internet Gateway Configuration

If you intend to deploy worker nodes in public subnets, and you expect deployed applications to require the worker nodes to initiate connections to the internet, the VCN must have an internet gateway. The internet gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 in a route rule in a worker node route table.

See VCNs and Subnets and Example Network Resource Configurations.

NAT Gateway Configuration

If you intend to deploy worker nodes in private subnets, and you expect deployed applications to require the worker nodes to initiate connections to the internet, the VCN must have a NAT gateway. The NAT gateway must be specified as the target for the destination CIDR block 0.0.0.0/0 in a route rule in a worker node route table.

See NAT Gateway and Example Network Resource Configurations.

Service Gateway Configuration

If you expect applications deployed on a cluster to pull images from Oracle Cloud Infrastructure Registry (or to use other Oracle Cloud Infrastructure resources) and you want to protect data from the public internet, you can set up a service gateway. Setting up a service gateway enables worker nodes to access other resources in the same region without exposing data to the public internet.

When setting up the service gateway, create it in the same VCN and compartment as the worker nodes, and select the All <region> Services in Oracle Services Network option.

Having created the service gateway, it must be specified as the target for All <region> Services in Oracle Services Network in a route rule in the worker node route table.

Note that if you expect deployed applications to require access to public endpoints or services not supported by a service gateway (for example, to download updates or patches), configure additional network resources (such as a NAT gateway) to access the internet.

See Access to Oracle Services: Service Gateway and Example Network Resource Configurations.

Route Table Configuration

If you intend to deploy worker nodes in public subnets, and you expect deployed applications to require the worker nodes to initiate connections to the internet, create an internet gateway. Then create a worker node route table with a route rule that specifies the internet gateway as the target for the destination CIDR block 0.0.0.0/0.

If you intend to deploy worker nodes in private subnets, and you expect deployed applications to require the worker nodes to initiate connections to the internet, create a NAT gateway. Then create a worker node route table with a route rule that specifies the NAT gateway as the target for the destination CIDR block 0.0.0.0/0.

If you expect applications deployed on a cluster to pull images from Oracle Cloud Infrastructure Registry (or to use other Oracle Cloud Infrastructure resources) and you don't want to expose the data to the public internet, create a service gateway. Then create a worker node route table with a route rule that specifies the service gateway as the target for All <region> Services in Oracle Services Network.

See Internet Gateway, NAT Gateway, Access to Oracle Services: Service Gateway, and Example Network Resource Configurations.

DHCP Options Configuration

The VCN in which you want to create and deploy clusters must have DHCP Options configured. The default value for DNS Type of Internet and VCN Resolver is acceptable.

See DHCP Options and Example Network Resource Configurations.

Security List Configuration

The VCN in which you want to create and deploy clusters must have security lists defined for worker node subnets and load balancer subnets (if specified). The security lists for worker node subnets and load balancer subnets must be different. The security list for load balancer subnets must be unique and for their exclusive use.

Worker nodes are created with public or private IP addresses, according to whether you specify public or private subnets when defining the node pools in a cluster. Container Engine for Kubernetes must be able to access worker nodes.

See Security Lists and Example Network Resource Configurations.

Public Worker Node Subnet Security List configuration

When configuring a security list for public worker node subnets, the security list must have:

  • stateless ingress and egress rules that allow all traffic between different worker node subnets
  • stateless ingress and egress rules that allow all traffic between worker node subnets and load balancer subnets (if specified)
  • ingress rules to allow Container Engine for Kubernetes to access worker nodes on port 22 from the following source CIDR blocks:
    • 130.35.0.0/16
    • 134.70.0.0/17
    • 138.1.0.0/16
    • 140.91.0.0/17
    • 147.154.0.0/16
    • 192.29.0.0/16

Optionally, you can include ingress rules for public worker node subnets to:

Optionally, you can include an egress rule that allows all outbound traffic to the internet.

Private Worker Node Subnet Security List configuration

When configuring a security list for private worker node subnets, the security list must have:

  • stateless ingress and egress rules that allow all traffic between the different worker node subnets
  • stateless ingress and egress rules that allow all traffic between worker node subnets and load balancer subnets

Optionally, you can include ingress rules for private worker node subnets to:

Optionally, you can include an egress rule that allows all outbound traffic to the internet.

Subnet Configuration

The characteristics of the cluster you want to create, and the number of availability domains in the region in which you want to deploy the cluster, will determine the number of subnets to configure. For example, to create a highly available cluster in a region with three availability domains will require:

  • For worker nodes: a regional subnet (recommended), or three AD-specific subnets (one in each of the availability domains).
  • For load balancers: optionally (but usually) an additional regional subnet (recommended), or an additional two AD-specific subnets (each in a different availability domain).

Best practice is to use regional subnets to make failover across availability domains simpler to implement.

The VCN in which you want to create and deploy clusters must have at least one subnet defined in which to deploy worker nodes. Worker node subnets can be either public, or private for additional security (as specified by the Subnet access property). The number of worker node subnets to create depends on the region in which you are creating the cluster:

  • If you are creating a cluster in a region with multiple availability domains, you can define a single regional subnet (recommended), or multiple AD-specific subnets (one in each of the availability domains).
  • If you are creating a cluster in a region with a single availability domain, you can define a single regional subnet (recommended), or a single AD-specific subnet.

You have the option to define and use load balancers in clusters you create. If you want to define and use load balancers, the VCN in which you want to create and deploy clusters must have at least one subnet defined to host the load balancers. Load balancer subnets can be public or private (as specified by the Subnet access property). However, load balancers are optional, so you might not define load balancer subnets at all. The number of load balancer subnets to define depends on the region in which you are creating the cluster:

  • If you are creating a cluster in a region with three availability domains, you can define:
    • zero or one load balancer regional subnet (recommended)
    • zero or two load balancer AD-specific subnets. If you define two load balancer AD-specific subnets, they must be in different availability domains.
  • If you are creating a cluster in a region with a single availability domain, you can define zero or one load balancer subnet:

    • zero or one load balancer regional subnet (recommended)
    • zero or one load balancer AD-specific subnet.

In addition, all subnets must have the following properties set as shown:

  • Route Table: The name of a route table, if one has been created, that has a route rule specifying an internet gateway (for public worker node subnets) or NAT gateway (for private worker node subnets) as the target for the destination CIDR block 0.0.0.0/0, and/or a route rule specifying a service gateway as the target for All <region> Services in Oracle Services Network.
  • DHCP options: Default.

The CIDR blocks you specify for worker node and load balancer subnets must not overlap with CIDR blocks you specify for pods running in the cluster (see CIDR Blocks and Container Engine for Kubernetes).

Worker node subnets must have different security lists to load balancer subnets.

See VCNs and Subnets and Example Network Resource Configurations.