Oracle Cloud Infrastructure Documentation

Example: Setting Up an Ingress Controller on a Cluster

You can set up different open source ingress controllers on clusters you have created with Container Engine for Kubernetes.

This topic explains how to set up an example ingress controller along with corresponding access control on an existing cluster. Having set up the ingress controller, this topic describes how to use the ingress controller with a hello world service, and how to verify the ingress controller is working as expected.

Ingress Controller Components

The example ingress controller comprises:

  • The default backend deployment, which handles default routes for health checks and 404 responses. This is done by using a stock image that serves the minimum required routes for a default backend.
  • The default backend service, which exposes the default backend deployment for consumption by the ingress controller deployment.
  • The ingress controller deployment, which deploys an image that contains the binary for the ingress controller and nginx. The binary manipulates and reloads the /etc/nginx/nginx.conf configuration file when an Ingress is created in Kubernetes. Nginx upstreams point to services that match specified selectors.
  • The ingress controller service, which exposes the ingress controller deployment as a LoadBalancer type service. Because Container Engine for Kubernetes uses an Oracle Cloud Infrastructure integration/cloud-provider, a load balancer will be dynamically created with the correct nodes configured as a backend set.

Setting Up the Ingress Controller

In this section, you create the access rules for ingress. You then create the example ingress controller components, and confirm they are running.

Creating the Access Rules for the Ingress Controller

  1. If you haven't already done so, download the cluster's kubeconfig configuration file. If the file does not have the expected default name and location of $HOME/.kube/config, set the KUBECONFIG environment variable to point to the file. See Downloading a kubeconfig File to Enable Cluster Access.
  2. In a terminal window, grant the Kubernetes RBAC cluster-admin clusterrole to the user by entering:

    $ kubectl create clusterrolebinding <my-cluster-admin-binding> --clusterrole=cluster-admin --user=<user_OCID>

    where:

    • <my-cluster-admin-binding> is a string of your choice to be used as the name for the binding between the user and the Kubernetes RBAC cluster-admin clusterrole. For example, jdoe_clst_adm
    • <user_OCID> is the user's OCID (obtained from the Console ). For example, ocid1.user.oc1..aaaaa...zutq (abbreviated for readability).

    For example:

    $ kubectl create clusterrolebinding jdoe_clst_adm --clusterrole=cluster-admin --user=ocid1.user.oc1..aaaaa...zutq
  3. Create and save the file ingress-controller-rbac.yaml to grant access for the ingress controller using the following code:

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: nginx-ingress-serviceaccount
      namespace: default
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
      name: nginx-ingress-clusterrole
    rules:
      - apiGroups:
          - ""
        resources:
          - configmaps
          - endpoints
          - nodes
          - pods
          - secrets
        verbs:
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - nodes
        verbs:
          - get
      - apiGroups:
          - ""
        resources:
          - services
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - "extensions"
        resources:
          - ingresses
        verbs:
          - get
          - list
          - watch
      - apiGroups:
          - ""
        resources:
          - events
        verbs:
          - create
          - patch
      - apiGroups:
          - "extensions"
        resources:
          - ingresses/status
        verbs:
          - update
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
      name: nginx-ingress-clusterrole-nisa-binding
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: nginx-ingress-clusterrole
    subjects:
      - kind: ServiceAccount
        name: nginx-ingress-serviceaccount
        namespace: default
    
  4. Using the file you just saved, grant access by executing the following command:

    $ kubectl create -f ingress-controller-rbac.yaml

Creating the Default Backend Deployment and Service

  1. Create and save the file nginx-default-backend-deployment.yaml for the default backend deployment using the following code:

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: default-http-backend
      labels:
        k8s-app: default-http-backend
      namespace: default
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            k8s-app: default-http-backend
        spec:
          terminationGracePeriodSeconds: 60
          containers:
          - name: default-http-backend
            # Any image is permissable as long as:
            # 1. It serves a 404 page at /
            # 2. It serves 200 on a /healthz endpoint
            image: gcr.io/google_containers/defaultbackend:1.0
            livenessProbe:
              httpGet:
                path: /healthz
                port: 8080
                scheme: HTTP
              initialDelaySeconds: 30
              timeoutSeconds: 5
            ports:
            - containerPort: 8080
            resources:
              limits:
                cpu: 10m
                memory: 20Mi
              requests:
                cpu: 10m
                memory: 20Mi
    
  2. Create and save the file nginx-default-backend-service.yaml for the default backend deployment using the following code:

    apiVersion: v1
    kind: Service
    metadata:
      name: default-http-backend
      namespace: default
      labels:
        k8s-app: default-http-backend
    spec:
      ports:
      - port: 80
        targetPort: 8080
      selector:
        k8s-app: default-http-backend
  3. Using the files you just saved, create the default backend deployment and service by executing these commands:

    $ kubectl create -f nginx-default-backend-deployment.yaml
    
    $ kubectl create -f nginx-default-backend-service.yaml	

Creating the Ingress Controller Nginx Deployment and Service

  1. Create and save the file nginx-ingress-controller-deployment.yaml for the Nginx ingress controller deployment using the following code:

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: nginx-ingress-controller
      labels:
        k8s-app: nginx-ingress-controller
      namespace: default
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            k8s-app: nginx-ingress-controller
        spec:
          # hostNetwork makes it possible to use ipv6 and to preserve the source 
          # IP correctly regardless of docker configuration.
          # However, it is not a hard dependency of the nginx-ingress-controller
          # itself, and it may cause issues if port 10254 already is taken on 
          # the host.
          # Since hostPort is currently broken on CNI 
          # (https://github.com/kubernetes/kubernetes/issues/31307), we have to
          # use hostNetwork where CNI is used, like with kubeadm.
          # hostNetwork: true
          terminationGracePeriodSeconds: 60
          serviceAccountName: nginx-ingress-serviceaccount
          containers:
          - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.10
            name: nginx-ingress-controller
            readinessProbe:
              httpGet:
                path: /healthz
                port: 10254
                scheme: HTTP
            livenessProbe:
              httpGet:
                path: /healthz
                port: 10254
                scheme: HTTP
              initialDelaySeconds: 10
              timeoutSeconds: 1
            ports:
            - containerPort: 80
              hostPort: 80
            - containerPort: 443
              hostPort: 443
            env:
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
            args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --update-status=false
  2. Create and save the file nginx-ingress-controller-service.yaml for the Nginx ingress controller service using the following code:

    apiVersion: v1
    kind: Service
    metadata:
      name: nginx-ingress-controller
      namespace: default
      labels:
        k8s-app: nginx-ingress-controller
    spec:
      type: LoadBalancer
      ports:
      - port: 80
        nodePort: 30021
        name: http
      - port: 443
        nodePort: 30022
        name: https
      selector:
        k8s-app: nginx-ingress-controller	
  3. Using the files you just saved, create the Nginx ingress controller deployment and service:

    $ kubectl create -f nginx-ingress-controller-deployment.yaml
    			
    $ kubectl create -f nginx-ingress-controller-service.yaml	

Verifying the Ingress Controller Services Are Running

  1. View the list of running services:

    $ kubectl get svc
    
    NAME                       CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
    default-http-backend       10.96.208.64   <none>          80/TCP                       1h
    nginx-ingress-controller   10.96.121.145  <pending>       80:30021/TCP,443:30022/TCP   1s
    

    The EXTERNAL-IP for the nginx-ingress-controller is shown as <pending> until the load balancer has been fully created in Oracle Cloud Infrastructure.

  2. Repeat the kubectl get svc command until an EXTERNAL-IP is shown for the nginx-ingress-controller:

    $ kubectl get svc
    			
    NAME                       CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
    default-http-backend       10.96.208.64   <none>          80/TCP                       48m
    nginx-ingress-controller   10.96.45.134   129.146.11.154  80:30021/TCP,443:30022/TCP   47m
    		

Using the Example Ingress Controller with a Service

In this section, you define a hello-world service and deploy it.

Creating the docker-hello-world Service Definition

  1. Create the file hello-world-ingress.yaml using the following code. This code uses a publicly available hello world image from Docker Hub. You can substitute another image of your choice that can be run in a similar manner.

    apiVersion: apps/v1beta1
    kind: Deployment
    metadata:
      name: docker-hello-world
      labels:
        app: docker-hello-world
    spec:
      replicas: 3
      template:
        metadata:
          labels:
            app: docker-hello-world
        spec:
          containers:
          - name: docker-hello-world
            image: scottsbaldwin/docker-hello-world:latest
            ports:
            - containerPort: 80
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: docker-hello-world-svc
    spec:
      selector:
        app: docker-hello-world
      ports:
        - port: 8088
          targetPort: 80
      type: ClusterIP
    

    Note the docker-hello-world service's type is ClusterIP, rather than LoadBalancer, because this service will be proxied by the ingress Nginx controller. The docker-hello-world service does not need public access directly to it. Instead, the public access will be routed from the load balancer to the ingress controller, and from the ingress controller to the upstream service.

  2. Create this new deployment and service:

    $ kubectl create -f hello-world-ingress.yaml
    

Creating the TLS Secret

A TLS secret is used for SSL termination on the ingress controller. To generate the secret for this example, a self-signed certificate is used. While this is okay for testing, for production, use a certificate signed by a Certificate Authority.

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"
			
$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt

Note

Under Windows, you may need to replace "/CN=nginxsvc/O=nginxsvc" with "//CN=nginxsvc\O=nginxsvc" . For example, this is necessary if you run the openssl command from a Git Bash shell.

Creating the Ingress Resource

  1. Create the file ingress.yaml and populate it with this code:

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: hello-world-ing
      annotations:
        kubernetes.io/ingress.class: "nginx"
    spec:
      tls:
      - secretName: tls-secret
      rules:
      - http:
          paths:
          - backend:
              serviceName: docker-hello-world-svc
              servicePort: 8088
    
  2. Create the resource:

    $ kubectl create -f ingress.yaml
    

Verifying the Ingress Controller is Working as Expected

In this section, you confirm that all of the components have been successfully created and are operating as expected. The docker-hello-world-svc service should be running as a ClusterIP service, and the nginx-ingress-controller service should be running as a LoadBalancer service.

Obtaining the External IP Address of the Load Balancer

To confirm the nginx-ingress-controller service is running as a LoadBalancer service, obtain its external IP address:

$ kubectl get svc --all-namespaces
			
NAMESPACE     NAME                       CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
default       docker-hello-world-svc     10.96.83.247    <none>           8088:31295/TCP               16s
default       kubernetes                 10.96.0.1       <none>           443/TCP                      1h
kube-system   kube-dns                   10.96.5.5       <none>           53/UDP,53/TCP                1h
default       default-http-backend       10.96.208.64    <none>           80/TCP                       1h
default       nginx-ingress-controller   10.96.121.145   129.146.11.154   80:30021/TCP,443:30022/TCP   5m

Sending cURL Requests to the Load Balancer

  1. Use the external IP address of the nginx-ingress-controller service (for example, 129.146.11.154) to curl an http request:

    $ curl -I http://129.146.11.154
    
    HTTP/1.1 301 Moved Permanently
    Via: 1.1 10.68.69.10 (McAfee Web Gateway 7.6.2.10.0.23236)
    Date: Thu, 07 Sep 2017 15:20:16 GMT
    Server: nginx/1.13.2
    Location: https://129.146.11.154/
    Content-Type: text/html
    Content-Length: 185
    Proxy-Connection: Keep-Alive
    Strict-Transport-Security: max-age=15724800; includeSubDomains;
    

    The output shows a 301 redirect and a Location header that suggest that http traffic is being redirected to https.

  2. Either cURL against the https url or add the -L option to automatically follow the location header. The -k option instructs cURL to not verify the SSL certificates.

    $ curl -ikL http://129.146.11.154
    
    HTTP/1.1 301 Moved Permanently
    Via: 1.1 10.68.69.10 (McAfee Web Gateway 7.6.2.10.0.23236)
    Date: Thu, 07 Sep 2017 15:22:29 GMT
    Server: nginx/1.13.2
    Location: https://129.146.11.154/
    Content-Type: text/html
    Content-Length: 185
    Proxy-Connection: Keep-Alive
    Strict-Transport-Security: max-age=15724800; includeSubDomains;
    
    HTTP/1.0 200 Connection established
    
    HTTP/1.1 200 OK
    Server: nginx/1.13.2
    Date: Thu, 07 Sep 2017 15:22:30 GMT
    Content-Type: text/html
    Content-Length: 71
    Connection: keep-alive
    Last-Modified: Thu, 07 Sep 2017 15:17:24 GMT
    ETag: "59b16304-47"
    Accept-Ranges: bytes
    Strict-Transport-Security: max-age=15724800; includeSubDomains;
    
    <h1>Hello webhook world from: docker-hello-world-1732906117-0ztkm</h1>
    

    The last line of the output shows the HTML that is returned from the pod whose hostname is docker-hello-world-1732906117-0ztkm.

  3. Issue the cURL request several times to see the hostname in the HTML output change, demonstrating that load balancing is occurring:

    $ curl -k https://129.146.11.154
    
    <h1>Hello webhook world from: docker-hello-world-1732906117-6115l</h1>
    
    $ curl -k https://129.146.11.154
    
    <h1>Hello webhook world from: docker-hello-world-1732906117-7r89v</h1>
    
    $ curl -k https://129.146.11.154
    
    <h1>Hello webhook world from: docker-hello-world-1732906117-0ztkm</h1>
    

Inspecting nginx.conf

The ingress controller manipulates the nginx.conf file within the pod for the nginx-ingress-controller.

  1. Find the pod name and use it with a kubectl exec command to show the contents of nginx.conf.

    $ kubectl get po
    			
    NAME                                       READY     STATUS    RESTARTS   AGE
    default-http-backend-726995137-s4lwh       1/1       Running   0          1h
    docker-hello-world-6bd6f7dfc8-2htm4        1/1       Running   0          1h
    docker-hello-world-6bd6f7dfc8-6jzhs        1/1       Running   0          1h
    docker-hello-world-6bd6f7dfc8-9pkw7        1/1       Running   0          1h
    nginx-ingress-controller-110676328-h86xg   1/1       Running   0          1h
    
    $ kubectl exec -it nginx-ingress-controller-110676328-h86xg -- cat /etc/nginx/nginx.conf
    
  2. Look for proxy_pass in the output. There will be one for the default backend and another that looks similar to:

    proxy_pass http://default-docker-hello-world-svc-8088;
    

    This shows that Nginx is proxying requests to an upstream called default-docker-hello-world-svc-8088.

  3. Locate the upstream definition in the output. It will look similar to:

    upstream default-docker-hello-world-svc-8088 {
    # Load balance algorithm; empty for round robin, which is the default
    least_conn;
    server 10.244.31.5:80 max_fails=0 fail_timeout=0;
    server 10.244.71.5:80 max_fails=0 fail_timeout=0;
    server 10.244.67.5:80 max_fails=0 fail_timeout=0;
    }	

    The upstream is proxying to three hosts that are listening on port 80.

  4. View these three hosts and the port number in the service description as well:

    $ kubectl describe svc/docker-hello-world-svc
    
    Name:                   docker-hello-world-svc
    Namespace:              default
    Labels:                 <none>
    Annotations:            <none>
    Selector:               app=docker-hello-world
    Type:                   ClusterIP
    IP:                     10.96.83.247
    Port:                   <unset> 8088/TCP
    Endpoints:              10.244.31.5:80,10.244.67.5:80,10.244.71.5:80
    Session Affinity:       None
    Events:                 <none>