Load Balance East-West Traffic in Kubernetes Environment Using NetScaler CPX

You can deploy NetScaler CPX in a Kubernetes cluster to load balance containerized applications in the cluster. NetScaler CPX is supported on the following Kubernetes versions:

  • 1.5.x

  • 1.6.x

For information about Kubernetes, see http://kubernetes.io/docs/.

By default, when you deploy NetScaler CPX in the Kubernetes cluster, it replaces the Kubernetes’ kube-proxy that provides basic load balancing functionality. Replacing the kube-proxy with NetScaler CPX, in addition to the load balancing functionality, you can use the NetScaler Management and Analytics System (MAS) for:

  • Visibility into the application environment in the cluster

  • Managing and monitoring the NetScaler CPX instances in the cluster

  • Using the Stylebooks feature to simplify the task of managing complex NetScaler configurations for your applications

For more information on NetScaler MAS, see NetScaler Management and Analytics System Product Documentation.

How NetScaler CPX Load Balances East-West Traffic Flow in Kubernetes Environment

After you have deployed the Kubernetes cluster, you must integrate the cluster with NetScaler MAS by providing the details of the Kubernetes environment in NetScaler MAS. NetScaler MAS monitors the changes in Kubernetes resources, such as services, endpoints, and Ingress rules.

When you deploy a NetScaler CPX instance in the Kubernetes cluster, it automatically registers with NetScaler MAS. As part of the registration process, NetScaler MAS learns about the NetScaler CPX instance IP address and the port on which it can reach the instance to configure it by using NITRO REST APIs.

The Stylebook engine in NetScaler MAS processes all the information that NetScaler MAS collects from Kubernetes, such as services, endpoints, and Ingress rules. Using an existing provisioned Stylebook (com.citrix.adc.stylebooks/1.0/cs-lb-mon), the Stylebook engine generates NetScaler-specific configurations, such as the virtual servers and service groups required for load balancing, and applies the configurations to the NetScaler CPX instances. For more information about Stylebook, see Stylebooks.

The following figure shows how NetScaler CPX load balances east-west traffic flow in a Kubernetes cluster. localized image

In this example, Node 1 and Node 2 of the Kubernetes clusters contains instances of a front-end service and a back-end service. When the NetScaler CPX instances are deployed in Node 1 and Node 2, the NetScaler CPX instances are automatically registered with NetScaler MAS. You must manually integrate the Kubernetes cluster with NetScaler MAS by configuring the Kubernetes cluster details in NetScaler MAS.

When a client requests the front-end service, the ingress resource load balances the request between the instances of the front-end service on the two nodes. When an instance of the front-end service needs information from the back-end services in the cluster, it directs the requests to the NetScaler CPX instance in its node. That NetScaler CPX instance load balances the requests between the back-end services in the cluster, thereby providing east-west traffic flow.

Deploying a NetScaler CPX Instance on a Node in Kubernetes Cluster

You can deploy NetScaler CPX instances as Kubernetes pods on the nodes in a Kubernetes cluster. A NetScaler CPX instance can be deployed as a daemon set or as a manifest.

  • Daemon set – Deploying a NetScaler CPX instance as a daemon set resource enables you to deploy a NetScaler CPX instance as a pod in the node and also ensures that a NetScaler CPX instance is deployed on new nodes that join the Kubernetes cluster. When the new node joins the cluster, the NetScaler CPX instance specified in the daemons set is installed automatically on the node.

  • Manifest – A Kubernetes manifest is a YAML or JSON formatted file containing Kubernetes object deployment and configuration instructions. You can create a Kubernetes manifest of a NetScaler CPX instance and place it in a particular directory on the nodes. A kubelet on every node monitors this directory and creates objects, that is, NetScaler CPX instances, as specified by the manifest.

Prerequisites

For this type of deployment, make sure that you:

Deploying NetScaler CPX Instances as a Daemon Set

With the daemon set approach, you can deploy a NetScaler CPX instance as a pod on a node, and it is then automatically deployed as a pod on each new node that joins the Kubernetes cluster.

To deploy a NetScaler CPX instance as a daemon set, you must write a YAML file or a JSON script. The file or script specifies the container type, CPX image file name, NetScaler MAS server IP address, and NetScaler MAS server fingerprint.

The following is a sample YAML file:

    apiVersion: extensions/v1beta1

    kind: DaemonSet

    metadata:

      name: cpx

    spec:

      template:

        metadata:

          name: cpx

          labels:

            app: cpx-daemon

          annotations:

            NETSCALER_AS_APP: "True"

        spec:

          hostNetwork: true

          containers:

            - name: cpx

              image: "<repository>/cpx:12.0-64"

              securityContext:

                 privileged: true

              env:

              - name: "EULA"

                value: "yes"

              - name: "NS_NETMODE"

                value: "HOST"

              - name: "kubernetes_url"

                value: "https://10.217.212.231:6443"

              - name: "NS_MGMT_SERVER"

                value: "10.217.212.226"

              - name: "NS_MGMT_FINGER_PRINT"

                value: "74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4"

              - name: "NS_ROUTABLE"

                value: "FALSE"

              - name: "KUBERNETES_TASK_ID"

                valueFrom:

                   fieldRef:

                      fieldPath: metadata.name

              volumeMounts:

              imagePullPolicy: Always

The following table describes the sections, parameters, and environment variables used in the sample daemon set:

Section Parameter Description
container name Name of the NetScaler CPX container.
  image Specifies the image for container creation.
SecurityContext privileged: true Specifies that the NetScaler CPX container is run in privileged mode.
  name: “EULA” A NetScaler CPX specific environment variable, which is required for verification that you have read and understand the End User License Agreement (EULA) available at: https://www.citrix.com/products/netscaler-adc/cpx-express.html.
  name: “NS_NETMODE” A NetScaler CPX specific environment variable that allows you to specify that the NetScaler CPX instance is started in host mode. After the instance starts in host mode, it configures 4 default iptable rules on the host machine for management access to the instance. It uses the following ports: 9995 for HTTP, 9996 for HTTPS, 9997 for SSH and 9998 for SNMP. Also, If you want to specify different ports, you can use the following environment variables: -e NS_HTTP_PORT, -e NS_HTTPS_PORT, -e NS_SSH_PORT, and -e NS_SNMP_PORT.
  name: “kubernetes_url” A NetScaler CPX specific environment variable that specifies the Kubernetes URL.
  name: “NS_MGMT_SERVER” A NetScaler CPX specific environment variable that describes the NetScaler MAS server IP address. When the NetScaler CPX instance is deployed, it automatically registers with the NetScaler MAS server at this IP address.
  name: “NS_MGMT_FINGER_PRINT” A NetScaler CPX specific environment variable that defines the NetScaler MAS fingerprint.
  name: “NS_ROUTABLE” A NetScaler CPX specific environment variable specifying whether the NetScaler CPX container is run in non-IP-per-container mode. Be sure to set the value to FALSE.
  name: “KUBERNETES_TASK_ID” Identifies the NetScaler CPX ID in the Kubernetes cluster.
imagePullPolicy   Specifies how Kubernetes pulls the image.

Deploying a NetScaler CPX Instance Using a Manifest

A Kubernetes manifest is a YAML or JSON formatted file containing Kubernetes object deployment and configuration instructions. You can create a Kubernetes manifest of a NetScaler CPX instance and place it in a particular directory on the nodes. A kubelet on every node monitors this directory and creates objects, that is, NetScaler CPX instances, as specified by the manifest.

The following is a sample manifest:

    apiVersion: v1

    kind: Pod

    metadata:

        name: cpx

    annotations:

        NETSCALER_AS_APP: "True"

    spec:

        hostNetwork: true

        containers:

            - name: cpx

              image: "<repository>/cpx:12.0-64"

        securityContext:

        privileged: true

        env:

            - name: "EULA"

            value: "yes”

            - name: "NS_NETMODE"

            value: "HOST"

            - name: "kubernetes_url"

            value: "https://10.217.212.231:6443"

            - name: "NS_MGMT_SERVER"

            value: "10.217.212.226"

            - name: "NS_MGMT_FINGER_PRINT"

            value: "74:EA:04:90:2C:FA:BF:7A:31:C9:52:64:D3:9C:BC:D3:O8:9F:9A:O4"

            - name: "NS_ROUTABLE"

            value: "FALSE"

            - name: "KUBERNETES_TASK_ID"

            valueFrom:

                fieldRef:

                fieldPath: metadata.name

        imagePullPolicy: Always

The following table describes the sections, parameters, and environment variables used in the sample manifest:

Section Parameter Description
container name Name of the NetScaler CPX container.
  image Specifies the image for container creation.
SecurityContext privileged: true Specifies that the NetScaler CPX container is run in privileged mode.
  name: “EULA” A NetScaler CPX specific environment variable, which is required for verification that you have read and understand the End User License Agreement (EULA) available at: https://www.citrix.com/products/netscaler-adc/cpx-express.html.
  name: “NS_NETMODE” A NetScaler CPX specific environment variable that allows you to specify that the NetScaler CPX instance is started in host mode. After the instance starts in host mode, it configures 4 default iptable rules on the host machine for management access to the instance. It uses the following ports: 9995 for HTTP, 9996 for HTTPS, 9997 for SSH and 9998 for SNMP. Also, If you want to specify different ports, you can use the following environment variables: -e NS_HTTP_PORT, -e NS_HTTPS_PORT, -e NS_SSH_PORT, and -e NS_SNMP_PORT.
  name: “kubernetes_url” A NetScaler CPX specific environment variable that specifies the Kubernetes URL.
  name: “NS_MGMT_SERVER” A NetScaler CPX specific environment variable that describes the NetScaler MAS server IP address. When the NetScaler CPX instance is deployed, it automatically registers with the NetScaler MAS server at this IP address.
  name: “NS_MGMT_FINGER_PRINT” A NetScaler CPX specific environment variable that defines the NetScaler MAS fingerprint.
  name: “NS_ROUTABLE” A NetScaler CPX specific environment variable specifying whether the NetScaler CPX container is run in non-IP-per-container mode. Be sure to set the value to FALSE.
  name: “KUBERNETES_TASK_ID” Identifies the NetScaler CPX ID in the Kubernetes cluster.
imagePullPolicy   Specifies how Kubernetes pulls the image.

Load Balance East-West Traffic in Kubernetes Environment Using NetScaler CPX