Load Balance Ingress Traffic in Kubernetes Environment Using Citrix ADC CPX

In a Kubernetes environment, to load balance Ingress traffic for Kubernetes services you need an Ingress resource and an Ingress controller. An Ingress resource is a Kubernetes resource with which you can configure a load balancer for your Kubernetes services. The load balancer exposes the services to clients outside your Kubernetes cluster by providing externally-reachable URLs for the services, and load balances the traffic sent to those URLs. Citrix ADC CPX can be used an Ingress load balancer in a Kubernetes environment, to load balance the North-South traffic to your Kubernetes services by clients outside the Kubernetes cluster.

An Ingress Controller integrates the load balancer with Kubernetes. It monitors the Ingress resource through the Kubernetes API and updates the configurations of the load balancer if any of the services are changed by scaling, rolling updates, or metadata changes. The NetScaler Management and Analytics System (MAS) includes a NetScaler Ingress Controller for the Kubernetes environment. The NetScaler Ingress Controller and the Citrix ADC CPX instances deployed in the Kubernetes cluster enable you to handle Ingress traffic in a Kubernetes environment. For more information about NetScaler MAS, see NetScaler Management and Analytics System Product Documentation.

For more information about Ingress resources and controllers in Kubernetes, see Ingress Resources.

How the NetScaler Ingress Controller Works

After you have deployed the Kubernetes cluster, you must integrate the cluster with NetScaler MAS by providing the details of the Kubernetes environment in NetScaler MAS. NetScaler MAS monitors for changes in Kubernetes resources such as, services, pods, and Ingress rules.

When you deploy a Citrix ADC CPX instance as an Ingress resource in the Kubernetes cluster, it automatically registers with NetScaler MAS. As part of the registration process, NetScaler MAS learns about the Citrix ADC CPX instance IP address and the port on which it can reach the instance to configure NetScaler specific configuration using the NITRO REST APIs.

The Stylebook engine in NetScaler MAS processes all the information that NetScaler MAS collects from Kubernetes, such as services, pods, and Ingress rules. Using an existing provisioned Stylebook (com.citrix.adc.stylebooks/1.0/cs-lb-mon), the Stylebook engine generates NetScaler configurations, such as the virtual servers, services, and service groups required for load balancing, and applies the configurations to the Citrix ADC CPX Ingress Load Balancer. For more information on Stylebook, see Stylebooks.

The following diagram illustrates a Kubernetes environment that includes a NetScaler Ingress controller integrated with a Citrix ADC CPX Ingress resource in the Kubernetes cluster to handle the ingress traffic.

localized image

In this example, a Citrix ADC CPX container is deployed to load balance traffic to the Kubernetes services from outside the cluster through a virtual IP (VIP) address. The Citrix ADC CPX container load balances the North-South traffic by distributing the requests between the multiple Kubernetes Pods that make up services A and B.

Important

The DNS configuration for the domain, api.example.com is configured to send the traffic to the Citrix ADC CPX container using the Citrix ADC CPX host IP address. In case, if multiple Citrix ADC CPX containers are configured as ingress load balancer, ensure that you distribute the ingress traffic across the Citrix ADC CPX containers using DNS methods.

NetScaler MAS manages the NetScaler devices in the Kubernetes cluster and provides rich analytics from the devices for insight and troubleshooting. It also enables you to get visibility into application performance and security by collecting detailed traffic statistics from the NetScaler devices.

Deploying Citrix ADC CPX as an Ingress Load Balancer in a Kubernetes Environment

Citrix ADC CPX can be used as an Ingress load balancer for Kubernetes environment. You can deploy the Citrix ADC CPX container as a Kubernetes pod in a node within the cluster, or you can deploy it on a host outside the cluster if that host participates in the same overlay network as the other Kubernetes nodes.

Prerequisites

Before you begin, be sure to do the following:

Deploying Citrix ADC CPX as an Ingress Load Balancer Outside the Kubernetes Cluster

Citrix ADC CPX can be deployed as an Ingress load balancer outside the Kubernetes cluster. A host that is outside the cluster must participate in the same overlay network as the other Kubernetes nodes.

To deploy Citrix ADC CPX as an Ingress load balancer on a host outside the Kubernetes cluster: On the host, deploy the Citrix ADC CPX instance on the Docker container by using the following docker run command:

docker run -dt --privileged=true -p <port_number> -e NS_HTTP_PORT=<netscaler_HTTP_port> -e NS_HTTPS_PORT=<netscaler_HTTPS_port> -e EULA=yes -e NS_MGMT_SERVER=<MAS_IP_address> -e NS_MGMT_FINGER_PRINT="<MAS_finger_print>" -e NS_ROUTABLE=<True|False> -e NS_LB_ROLE=<lb_role> -e HOST=$HOSTNAME store/citrix/netscalercpx:12.0-53.6

Example:

docker run -dt --privileged=true -p 5080:80 -p 5443:443 -p 80:5080 -e NS_HTTP_PORT=5080 -p 443:5443 -e NS_HTTPS_PORT=5443 -e EULA=yes -e NS_MGMT_SERVER=10.217.212.226 -e NS_MGMT_FINGER_PRINT="74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4" -e NS_ROUTABLE=FALSE -e NS_LB_ROLE=SERVER -e HOST=$HOSTNAME store/citrix/netscalercpx:12.1.48.xx

The command deploys a Citrix ADC CPX docker container. The following table describes the various options and environment variables used in the docker run command:

Options and NetScaler Specific Environment Variables Descriptions
-dt Specifies that the Citrix ADC CPX container is run in daemon form.
–privileged=true Specifies that the Citrix ADC CPX container runs in privileged mode.
-p Maps the ports between the Citrix ADC CPX and the host. By default, the Kubernetes Ingress object assumes that the cluster is accessed used ports 80 and 443.
-p 5080:80 Binds the port 80 of the container to port 5080 of the host.
-p 5443:443 Binds the port 443 of the container to port 5443 of the host.
-p 443:5443 Binds the port 5443 of the container to port 443 of the host.
-p 80:5080 Binds the port 5080 of the container to port 80 of the host.
-e NS_HTTP_PORT or -e NS_HTTPS_PORT Citrix ADC CPX specific environment variable that enables you to assign custom ports for management access to Citrix ADC CPX. NetScaler MAS uses these ports to access the Citrix ADC CPX.
-e NS_MGMT_SERVER Citrix ADC CPX specific environment variable that allows you define the NetScaler MAS server IP address. When the Citrix ADC CPX is deployed, it automatically registers with the NetScaler MAS server using this IP address.
-e NS_MGMT_FINGER_PRINT Citrix ADC CPX specific environment variable that defines the NetScaler MAS fingerprint.
-e NS_ROUTABLE=FALSE Citrix ADC CPX specific environment variable specifying that the Citrix ADC CPX container is run in non-IP-per-container mode.
-e NS_LB_ROLE=SERVER Citrix ADC CPX specific environment variable specifying to Citrix ADC CPX and NetScaler MAS that the Citrix ADC CPX container is used as an Ingress resource.
-e HOST=$HOSTNAME Citrix ADC CPX specific environment variable specifying the host name that NetScaler MAS can use to access the Citrix ADC CPX container. Make sure that the host name can be resolved by NetScaler MAS, or else provide an IP address.

Once you deploy the Citrix ADC CPX instance on the host, it automatically registers with the NetScaler Management and Analytics System (MAS). You can view the deployed Citrix ADC CPX instances in the NetScaler MAS UI at: Networks > Instances > Citrix ADC CPX.

localized image

Deploying Citrix ADC CPX as an Ingress Load Balancer Within the Kubernetes Cluster

To deploy Citrix ADC CPX as an Ingress load balancer within a Kubernetes cluster, deploy it as a Kubernetes pod on a node in the Kubernetes cluster.

To deploy Citrix ADC CPX as an Ingress load balancer within the Kubernetes cluster:

  1. (Optional) If you want to deploy the Citrix ADC CPX as a Kubernetes pod on a particular node in the cluster, you can use a label to designate the node. To label a Kubernetes node, use the kubectl command:

    kubectl label nodes <node_IP_address> node-role=<label_name>
    

Example:

kubectl label nodes 10.217.222.224 node-role=ingress

    Once you have labeled a node, you can specify the label in the pod specification so that the pod is deployed in the node. 2. Define a pod specification for Citrix ADC CPX to deploy the Citrix ADC CPX container as a pod in the Kubernetes cluster. The pod specification is defined in a YAML file or a JSON script. The YAML file or the JSON script should contain the container type, CPX image file name, NetScaler MAS server IP address, and NetScaler MAS server fingerprint. The following is an example of a pod specification for Citrix ADC CPX:

    apiVersion: v1
    kind: Pod
    metadata:
    name: cpx-ingress
    annotations:
        NETSCALER_AS_APP: "True"
    spec:
        containers:
            - name: cpx-ingress
              image: "cpx:12.0-41.16"
        securityContext:
            privileged: true
        env:
            - name: "EULA"
              value: "yes"
            - name: "NS_MGMT_SERVER"
              value: "10.217.212.226"
            - name: "NS_MGMT_FINGER_PRINT"
              value: "74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4"
            - name: "NS_ROUTABLE"
              value: "FALSE"
            - name: "NS_HTTP_PORT"
              value: "5080"
            - name: "NS_HTTPS_PORT"
              value: "5443"
            - name: "NS_LB_ROLE"
              value: "SERVER"
            - name: "HOST"
              value: ""
            - name: "KUBERNETES_TASK_ID"
        valueFrom:
        fieldRef:
        fieldPath: metadata.name
            - name:"HOST"
    valueFrom:
        fieldRef:
            fieldPath: spec.nodeName
        ports:
            - containerPort: 80
              hostPort: 5080
            - containerPort: 443
              hostPort: 5443
            - containerPort: 5080
              hostPort: 80
            - containerPort: 5443
              hostPort: 443
        imagePullPolicy: Always
        nodeSelector:
            node-role: ingress

Alternatively, you can define a pod specification to deploy the Citrix ADC CPX as a Replication Controller, so that if Citrix ADC CPX goes down, Kubernetes recreates the Citrix ADC CPX container in the cluster. The following is a sample specification:

    apiVersion: v1
    kind: ReplicationController
    metadata:
    name: cpx-ingress
    spec:
        replicas: 1
        selector:
            app: cpx-ingress-device
        template:
        metadata:
            name: cpx-ingress
        annotations:
            NETSCALER_AS_APP: "True"
        labels:
            app: cpx-ingress-device
        spec:
            containers:
                - name: cpx-ingress
                image: "cpx:12.0-41.16"
            securityContext:
                privileged: true
            env:
                - name: "EULA"
                value: "yes"
                - name: "NS_MGMT_SERVER"
                value: "10.217.212.226"
                - name: "NS_MGMT_FINGER_PRINT"
                value: "74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4"
                - name: "NS_ROUTABLE"
                value: "FALSE"
                - name: "NS_HTTP_PORT"
                value: "5080"
                - name: "NS_HTTPS_PORT"
                value: "5443"
                - name: "NS_LB_ROLE"
                value: "SERVER"
                - name: "HOST"
                value: ""
                - name: "KUBERNETES_TASK_ID"
        valueFrom:
        fieldRef:
        fieldPath: metadata.name
            - name:"HOST"
    valueFrom:
        fieldRef:
            fieldPath: spec.nodeName
        ports:
            - containerPort: 80
            hostPort: 5080
            - containerPort: 443
            hostPort: 5443
            - containerPort: 5080
            hostPort: 80
            - containerPort: 5443
            hostPort: 443
        imagePullPolicy: Always
        nodeSelector:
            node-role: ingress

The following table describes the various sections, parameters, and environment variables used in the above example:

Section Parameter Description
containers name Name of the Citrix ADC CPX container.
  image Specifies the image for container creation.
securityContext privileged: true Specifies that the Citrix ADC CPX container runs in privileged mode.
env name: “EULA” A Citrix ADC CPX specific environment variable, which is required for verification that you have read and understand the End User License Agreement (EULA) available at: https://www.citrix.com/products/netscaler-adc/cpx-express.html.
  name: “NS_MGMT_SERVER” A Citrix ADC CPX environment variable that enables you define the NetScaler MAS server IP address. When the Citrix ADC CPX is deployed, it automatically registers with the NetScaler MAS server using this IP address.
  name: “NS_MGMT_FINGER_PRINT” A Citrix ADC CPX environment variable that enables you to define the NetScaler MAS fingerprint.
  name: “NS_ROUTABLE” A Citrix ADC CPX environment variable that enables you to specify that the Citrix ADC CPX container is run in non-IP-per-container mode. Be sure to set the value to “FALSE.”
  name: “NS_HTTP_PORT” or name: “NS_HTTPS_PORT” Citrix ADC CPX specific environment variables that enable you to assign custom ports for management access to Citrix ADC CPX. NetScaler MAS uses these ports to access the Citrix ADC CPX container.
  name: “NS_LB_ROLE” A Citrix ADC CPX environment variable that enables you to specify to Citrix ADC CPX and NetScaler MAS that the Citrix ADC CPX container is used as an Ingress resource.
  name: “HOST” The host name of the node on which the Citrix ADC CPX container is running. Using the host name, NetScaler MAS can access the Citrix ADC CPX container.
  name: “KUBERNETES_TASK_ID” Identifies the Citrix ADC CPX ID in the Kubernetes cluster.
  name: “HOST” The host name of the node on which the Citrix ADC CPX container is running. Using the host name, the NetScaler MAS can access the Citrix ADC CPX.
ports containerPort: or hostPort: Maps the ports between the Citrix ADC CPX container and the host. By default, the Kubernetes Ingress object assumes that the cluster is accessed at ports 80 and 443.
imagePullPolicy   Specifies how Kubernetes pulls the image.
nodeSelector node-role: The label of the node on which you want to deploy the pod.

3. Deploy the pod specification of the Citrix ADC CPX by using the following command:

kubectl create –f  (fileName | scriptName)

Example:

kubectl create –f sample.yaml

Once you deploy the Citrix ADC CPX instance on the host, it automatically registers with the NetScaler Management and Analytics System (MAS). You can view the deployed Citrix ADC CPX instances in the NetScaler MAS UI at: Networks > Instances > NetScaler CPX.

localized image

Load Balance Ingress Traffic in Kubernetes Environment Using Citrix ADC CPX