Load Balance Ingress Traffic in Kubernetes Environment Using NetScaler CPX

In a Kubernetes environment, to load balance Ingress traffic for Kubernetes services you need an Ingress resource and an Ingress controller. An Ingress resource is a Kubernetes resource with which you can configure a load balancer for your Kubernetes services. The load balancer exposes the services to clients outside your Kubernetes cluster by providing externally-reachable URLs for the services, and load balances the traffic sent to those URLs. NetScaler CPX can be used an Ingress load balancer in a Kubernetes environment, to load balance the North-South traffic to your Kubernetes services by clients outside the Kubernetes cluster.

An Ingress Controller integrates the load balancer with Kubernetes. It monitors the Ingress resource through the Kubernetes API and updates the configurations of the load balancer if any of the services are changed by scaling, rolling updates, or metadata changes. The NetScaler Management and Analytics System (MAS) includes a NetScaler Ingress Controller for the Kubernetes environment. The NetScaler Ingress Controller and the NetScaler CPX instances deployed in the Kubernetes cluster enable you to handle Ingress traffic in a Kubernetes environment. For more information about NetScaler MAS, see NetScaler Management and Analytics System Product Documentation.

For more information about Ingress resources and controllers in Kubernetes, see Ingress Resources.

How the NetScaler Ingress Controller Works

After you have deployed the Kubernetes cluster, you must integrate the cluster with NetScaler MAS by providing the details of the Kubernetes environment in NetScaler MAS. NetScaler MAS monitors for changes in Kubernetes resources such as, services, pods, and Ingress rules.

When you deploy a NetScaler CPX instance as an Ingress resource in the Kubernetes cluster, it automatically registers with NetScaler MAS. As part of the registration process, NetScaler MAS learns about the NetScaler CPX instance IP address and the port on which it can reach the instance to configure NetScaler specific configuration using the NITRO REST APIs.

The Stylebook engine in NetScaler MAS processes all the information that NetScaler MAS collects from Kubernetes, such as services, pods, and Ingress rules. Using an existing provisioned Stylebook (com.citrix.adc.stylebooks/1.0/cs-lb-mon), the Stylebook engine generates NetScaler configurations, such as the virtual servers, services, and service groups required for load balancing, and applies the configurations to the NetScaler CPX Ingress Load Balancer. For more information on Stylebook, see Stylebooks.

The following diagram illustrates a Kubernetes environment that includes a NetScaler Ingress controller integrated with a NetScaler CPX Ingress resource in the Kubernetes cluster to handle the ingress traffic.

localized image

In this example, a NetScaler CPX container is deployed to load balance traffic to the Kubernetes services from outside the cluster through a virtual IP (VIP) address. The NetScaler CPX container load balances the North-South traffic by distributing the requests between the multiple Kubernetes Pods that make up services A and B.

Important

The DNS configuration for the domain, api.example.com is configured to send the traffic to the NetScaler CPX container using the NetScaler CPX host IP address. In case, if multiple NetScaler CPX containers are configured as ingress load balancer, ensure that you distribute the ingress traffic across the NetScaler CPX containers using DNS methods.

NetScaler MAS manages the NetScaler devices in the Kubernetes cluster and provides rich analytics from the devices for insight and troubleshooting. It also enables you to get visibility into application performance and security by collecting detailed traffic statistics from the NetScaler devices.

Deploying NetScaler CPX as an Ingress Load Balancer in a Kubernetes Environment

NetScaler CPX can be used as an Ingress load balancer for Kubernetes environment. You can deploy the NetScaler CPX container as a Kubernetes pod in a node within the cluster, or you can deploy it on a host outside the cluster if that host participates in the same overlay network as the other Kubernetes nodes.

Prerequisites

Before you begin, be sure to do the following:

Deploying NetScaler CPX as an Ingress Load Balancer Outside the Kubernetes Cluster

NetScaler CPX can be deployed as an Ingress load balancer outside the Kubernetes cluster. A host that is outside the cluster must participate in the same overlay network as the other Kubernetes nodes.

To deploy NetScaler CPX as an Ingress load balancer on a host outside the Kubernetes cluster: On the host, deploy the NetScaler CPX instance on the Docker container by using the following docker run command:

docker run -dt --privileged=true -p <port_number> -e NS_HTTP_PORT=<netscaler_HTTP_port> -e NS_HTTPS_PORT=<netscaler_HTTPS_port> -e EULA=yes -e NS_MGMT_SERVER=<MAS_IP_address> -e NS_MGMT_FINGER_PRINT="<MAS_finger_print>" -e NS_ROUTABLE=<True|False> -e NS_LB_ROLE=<lb_role> -e HOST=$HOSTNAME store/citrix/netscalercpx:12.0-53.6

Example:

docker run -dt --privileged=true -p 5080:80 -p 5443:443 -p 80:5080 -e NS_HTTP_PORT=5080 -p 443:5443 -e NS_HTTPS_PORT=5443 -e EULA=yes -e NS_MGMT_SERVER=10.217.212.226 -e NS_MGMT_FINGER_PRINT="74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4" -e NS_ROUTABLE=FALSE -e NS_LB_ROLE=SERVER -e HOST=$HOSTNAME store/citrix/netscalercpx:12.1.48.xx

The command deploys a NetScaler CPX docker container. The following table describes the various options and environment variables used in the docker run command:

Options and NetScaler Specific Environment Variables Descriptions
-dt Specifies that the NetScaler CPX container is run in daemon form.
–privileged=true Specifies that the NetScaler CPX container runs in privileged mode.
-p Maps the ports between the NetScaler CPX and the host. By default, the Kubernetes Ingress object assumes that the cluster is accessed used ports 80 and 443.
-p 5080:80 Binds the port 80 of the container to port 5080 of the host.
-p 5443:443 Binds the port 443 of the container to port 5443 of the host.
-p 443:5443 Binds the port 5443 of the container to port 443 of the host.
-p 80:5080 Binds the port 5080 of the container to port 80 of the host.
-e NS_HTTP_PORT or -e NS_HTTPS_PORT NetScaler CPX specific environment variable that enables you to assign custom ports for management access to NetScaler CPX. NetScaler MAS uses these ports to access the NetScaler CPX.
-e NS_MGMT_SERVER NetScaler CPX specific environment variable that allows you define the NetScaler MAS server IP address. When the NetScaler CPX is deployed, it automatically registers with the NetScaler MAS server using this IP address.
-e NS_MGMT_FINGER_PRINT NetScaler CPX specific environment variable that defines the NetScaler MAS fingerprint.
-e NS_ROUTABLE=FALSE NetScaler CPX specific environment variable specifying that the NetScaler CPX container is run in non-IP-per-container mode.
-e NS_LB_ROLE=SERVER NetScaler CPX specific environment variable specifying to NetScaler CPX and NetScaler MAS that the NetScaler CPX container is used as an Ingress resource.
-e HOST=$HOSTNAME NetScaler CPX specific environment variable specifying the host name that NetScaler MAS can use to access the NetScaler CPX container. Make sure that the host name can be resolved by NetScaler MAS, or else provide an IP address.

Once you deploy the NetScaler CPX instance on the host, it automatically registers with the NetScaler Management and Analytics System (MAS). You can view the deployed NetScaler CPX instances in the NetScaler MAS UI at: Networks > Instances > NetScaler CPX.

localized image

Deploying NetScaler CPX as an Ingress Load Balancer Within the Kubernetes Cluster

To deploy NetScaler CPX as an Ingress load balancer within a Kubernetes cluster, deploy it as a Kubernetes pod on a node in the Kubernetes cluster.

To deploy NetScaler CPX as an Ingress load balancer within the Kubernetes cluster:

  1. (Optional) If you want to deploy the NetScaler CPX as a Kubernetes pod on a particular node in the cluster, you can use a label to designate the node. To label a Kubernetes node, use the kubectl command:

    kubectl label nodes <node_IP_address> node-role=<label_name>
    

Example:

kubectl label nodes 10.217.222.224 node-role=ingress

    Once you have labeled a node, you can specify the label in the pod specification so that the pod is deployed in the node. 2. Define a pod specification for NetScaler CPX to deploy the NetScaler CPX container as a pod in the Kubernetes cluster. The pod specification is defined in a YAML file or a JSON script. The YAML file or the JSON script should contain the container type, CPX image file name, NetScaler MAS server IP address, and NetScaler MAS server fingerprint. The following is an example of a pod specification for NetScaler CPX:

    apiVersion: v1
    kind: Pod
    metadata:
    name: cpx-ingress
    annotations:
        NETSCALER_AS_APP: "True"
    spec:
        containers:
            - name: cpx-ingress
              image: "cpx:12.0-41.16"
        securityContext:
            privileged: true
        env:
            - name: "EULA"
              value: "yes"
            - name: "NS_MGMT_SERVER"
              value: "10.217.212.226"
            - name: "NS_MGMT_FINGER_PRINT"
              value: "74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4"
            - name: "NS_ROUTABLE"
              value: "FALSE"
            - name: "NS_HTTP_PORT"
              value: "5080"
            - name: "NS_HTTPS_PORT"
              value: "5443"
            - name: "NS_LB_ROLE"
              value: "SERVER"
            - name: "HOST"
              value: ""
            - name: "KUBERNETES_TASK_ID"
        valueFrom:
        fieldRef:
        fieldPath: metadata.name
            - name:"HOST"
    valueFrom:
        fieldRef:
            fieldPath: spec.nodeName
        ports:
            - containerPort: 80
              hostPort: 5080
            - containerPort: 443
              hostPort: 5443
            - containerPort: 5080
              hostPort: 80
            - containerPort: 5443
              hostPort: 443
        imagePullPolicy: Always
        nodeSelector:
            node-role: ingress

Alternatively, you can define a pod specification to deploy the NetScaler CPX as a Replication Controller, so that if NetScaler CPX goes down, Kubernetes recreates the NetScaler CPX container in the cluster. The following is a sample specification:

    apiVersion: v1
    kind: ReplicationController
    metadata:
    name: cpx-ingress
    spec:
        replicas: 1
        selector:
            app: cpx-ingress-device
        template:
        metadata:
            name: cpx-ingress
        annotations:
            NETSCALER_AS_APP: "True"
        labels:
            app: cpx-ingress-device
        spec:
            containers:
                - name: cpx-ingress
                image: "cpx:12.0-41.16"
            securityContext:
                privileged: true
            env:
                - name: "EULA"
                value: "yes"
                - name: "NS_MGMT_SERVER"
                value: "10.217.212.226"
                - name: "NS_MGMT_FINGER_PRINT"
                value: "74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4"
                - name: "NS_ROUTABLE"
                value: "FALSE"
                - name: "NS_HTTP_PORT"
                value: "5080"
                - name: "NS_HTTPS_PORT"
                value: "5443"
                - name: "NS_LB_ROLE"
                value: "SERVER"
                - name: "HOST"
                value: ""
                - name: "KUBERNETES_TASK_ID"
        valueFrom:
        fieldRef:
        fieldPath: metadata.name
            - name:"HOST"
    valueFrom:
        fieldRef:
            fieldPath: spec.nodeName
        ports:
            - containerPort: 80
            hostPort: 5080
            - containerPort: 443
            hostPort: 5443
            - containerPort: 5080
            hostPort: 80
            - containerPort: 5443
            hostPort: 443
        imagePullPolicy: Always
        nodeSelector:
            node-role: ingress

The following table describes the various sections, parameters, and environment variables used in the above example:

Section Parameter Description
containers name Name of the NetScaler CPX container.
  image Specifies the image for container creation.
securityContext privileged: true Specifies that the NetScaler CPX container runs in privileged mode.
env name: “EULA” A NetScaler CPX specific environment variable, which is required for verification that you have read and understand the End User License Agreement (EULA) available at: https://www.citrix.com/products/netscaler-adc/cpx-express.html.
  name: “NS_MGMT_SERVER” A NetScaler CPX environment variable that enables you define the NetScaler MAS server IP address. When the NetScaler CPX is deployed, it automatically registers with the NetScaler MAS server using this IP address.
  name: “NS_MGMT_FINGER_PRINT” A NetScaler CPX environment variable that enables you to define the NetScaler MAS fingerprint.
  name: “NS_ROUTABLE” A NetScaler CPX environment variable that enables you to specify that the NetScaler CPX container is run in non-IP-per-container mode. Be sure to set the value to “FALSE.”
  name: “NS_HTTP_PORT” or name: “NS_HTTPS_PORT” NetScaler CPX specific environment variables that enable you to assign custom ports for management access to NetScaler CPX. NetScaler MAS uses these ports to access the NetScaler CPX container.
  name: “NS_LB_ROLE” A NetScaler CPX environment variable that enables you to specify to NetScaler CPX and NetScaler MAS that the NetScaler CPX container is used as an Ingress resource.
  name: “HOST” The host name of the node on which the NetScaler CPX container is running. Using the host name, NetScaler MAS can access the NetScaler CPX container.
  name: “KUBERNETES_TASK_ID” Identifies the NetScaler CPX ID in the Kubernetes cluster.
  name: “HOST” The host name of the node on which the NetScaler CPX container is running. Using the host name, the NetScaler MAS can access the NetScaler CPX.
ports containerPort: or hostPort: Maps the ports between the NetScaler CPX container and the host. By default, the Kubernetes Ingress object assumes that the cluster is accessed at ports 80 and 443.
imagePullPolicy   Specifies how Kubernetes pulls the image.
nodeSelector node-role: The label of the node on which you want to deploy the pod.

3. Deploy the pod specification of the NetScaler CPX by using the following command:

kubectl create –f  (fileName | scriptName)

Example:

kubectl create –f sample.yaml

Once you deploy the NetScaler CPX instance on the host, it automatically registers with the NetScaler Management and Analytics System (MAS). You can view the deployed NetScaler CPX instances in the NetScaler MAS UI at: Networks > Instances > NetScaler CPX.

localized image

Load Balance Ingress Traffic in Kubernetes Environment Using NetScaler CPX