Product Documentation

Citrix ADC CPX with Kubernetes and Application Delivery Management Orchestration Validated Reference Design

Overview

Citrix ADC summary

Citrix ADC is an all-in-one application delivery controller that makes applications run up to five times better, reduces application ownership costs, optimizes the user experience, and ensures that applications are always available by using:

  • Advanced L4-7 load balancing and traffic management

  • Proven application acceleration such as HTTP compression and caching

  • An integrated application firewall for application security

  • Server offloading to significantly reduce costs and consolidate servers

As an undisputed leader of service and application delivery, Citrix ADC is deployed in thousands of networks around the world to optimize, secure, and control the delivery of all enterprise and cloud services. Deployed directly in front of web and database servers, Citrix ADC combines high-speed load balancing and content switching, http compression, content caching, SSL acceleration, application flow visibility, and a powerful application firewall into an integrated, easy-to-use platform. Meeting SLAs is greatly simplified with end-to-end monitoring that transforms network data into actionable business intelligence. Citrix ADC allows policies to be defined and managed using a simple declarative policy engine with no programming expertise required.

Citrix ADC CPX overview

Users and customers prefer their applications to be available from anywhere and on any device. As a result, enterprises are increasingly developing cloud native applications using the microservice architecture.

A microservice architecture provides benefits that make it a perfect fit for cloud native applications:

  1. Deploy applications in shorter period of time.

  2. Scale Up/Down in seamless fashion.

  3. Continuous delivery – Push the new features/fix to production without impacting end user traffic.

  4. Allow each microservice to be under the control of a specific development team.

  5. Provide with the ability to use the best programming language for a particular microservice.

There are tools available to manage the microservices deployment, but still some challenges remain:

  1. Lack of visibility into the North-South and inter services traffic (East – West) which makes it hard to troubleshoot application performance issues.

  2. Lack of Security.

  3. Failure handling.

This document focuses on how Citrix ADC solves these challenges and how Citrix ADC can be inserted into the miroservices architecture. This document includes:

  1. Introduction to microservices architecture and management tools.

  2. Design/Deployment guidelines on using Citrix ADC in a Kubernetes container cluster environment.


Microservices - Enabling Cloud native applications

Introduction to Microservices

Microservices refer to decomposing a monolithic application into a number of different small lightweight applications. Containers are lightweight and they fit perfectly for this type of requirement which is the predominant form factor in the microservices world.

Microservices architecture should provide the following set of services:

  1. Ability to spin up new services

  2. Ability to scale up/down the new services based on traffic

  3. New services should be published so that they are discovered automatically

Container cluster management tools

Increasingly the listed above services provide the container cluster management tools. Some of the popular tools are:

  1. Kubernetes

  2. Apache Mesos and Marathon

  3. Amazon Container service

  4. Google Container Engine (GKE)

  5. Azure Container Service

  6. Docker Swarm and Datacenter

Kubernetes and Apache Mesos can work in multi/hybrid cloud environment. Kubernetes is increasingly becoming a popular choice as container cluster management platform. This document focuses on the integration of Citrix ADC with Kubernetes.


Kubernetes

Kubernetes architecture

image-cpx-with-kubernetes-01

Kubernetes manages a cluster of nodes. One of the nodes is the master, and the rest of the nodes host the applications.

Components of a Kubernetes master

  1. Etcd - A persistent lightweight distributed key-value data store that stores the state of the cluster and it nodes.

  2. API server – Clients can use the API server to access the etcd key-value store and it enables clients to configure new services in the cluster

  3. Scheduler – Selects the nodes on which a new pod/application should be launched based on the current CPU/ Memory/Disk Usage

  4. Controller manager – Updates the API server on new nodes and created service endpoints

Components of a Kubernetes node

  1. Kubelet – Starts the services on a particular node based on the requests from the API server

  2. Kube-proxy – Routes traffic to the appropriate container based on the IP address and port number, and provides a basic load balancing capability using round robin load balancing algorithm

  3. cAdvisor – Gathers CPU, memory usage, and passes the data to the Kubernetes Master

Kubernetes design

Kubernetes provides a set of blocks which can be used to deploy and scale applications.

The basic scheduling unit is called a pod. A pod consists of one or more container applications that are guaranteed to be co-located on the same node. Each pod is assigned a unique IP address.

A service is a set of pods that work together. A set of pods that constitutes a service is defined by label selector and has a reachable IP address called endpoints.

image-cpx-with-kubernetes

Kubernetes commands

Below is a sample output of kubectl commands that list the services and pods:

root@ubuntu# kubectl get services

NAME            CLUSTER-IP      EXTERNAL-IP     PORT(S)     AGE

frontend        10.108.25.128   10.105.158.147  80/TCP      13D
kubernetes      10.96.0.1       <none>          443/TCP     13D
redis-master    10.101.42.129   <none>          6379/TCP    13D
redis-slave     10.107.235.135  <none>          6379/TCP    13D

root@ubuntu# kubectl get pods

NAME                       READY           STATUS          RESTARTS    AGE

cpx-1                      1/1             Running         0           11D
cpx-3                      1/1             Running         0           11D
cpx-4                      1/1             Running         0           11D
redis-slave                1/1             Running         0           13D
frontend-1768566195-4qdv6      1/1             Running         0           13D
frontend-1768566195-fjx61      1/1             Running         0           13D
frontend-1768566195-j1prp      1/1             Running         0           13D
redis-master-106238132-533tw   1/1             Running         0           13D
redis-slave-3837281623-4b8l3   1/1             Running         0           13D
redis-slave-3837281623-n80fb   1/1             Running         0           13D

root@cpxkube-4:~#

Ingress controller - Why it is required

Usually services and pods have an IP address that is routable only inside the cluster network.

An ingress controller is the collection of rules which allow the end client/user connections to reach a respective application/service hosted in the kubernetes cluster.

The ingress controller can be configured to give the following set of services:

  1. Publish externally reachable url for applications hosted on kubernetes

  2. Load balance traffic

  3. SSL offload

Use cases of Citrix ADC in Kubernetes

The use cases are:

  1. Citrix ADC acts as the ingress-controller for North-South and East-West traffic for the front-end apps.

  2. Citrix ADC acts as the replacement for kube-proxy for East-West traffic within Kuberntes.

For both of the above use cases, Citrix ADM is an integral part of the deployment. It provides the following functionalities:

Service discovery

Listens to the kube-api server and based on events makes the necessary configuration changes in the Citrix ADC using the Stylebook/NITRO APIs.

Visibility

Provides application performance analytics and helps the administrator in the following areas using Application performance analytics and app scoring:

  1. Whether the new changes to application have the desired effect

  2. Blue/Green Deployments

  3. Troubleshoot application performance issue.


Citrix Application Delivery Management as Kubernetes Ingress controller

Ingress Traffic Management consists of two parts:

Ingress controller:

Ingress controller listens to the kube-api for ingress related updates and makes necessary configuration changes to make it effective on the Ingress device.

Ingress device:

Ingress device handles the incoming traffic and routes the traffic to the appropriate service based on the configured ingress rules, and provides reverse proxy functionality.

Since Ingress controller acts as an entry point and it is in the path of N-S traffic, it adds a great value by providing the following set of capabilities:

  1. Rewrite

  2. SSL Offload

  3. L4/L7 Rate Limit

  4. L4/L7 DDoS Protection

  5. Application performance and troubleshooting

Citrix ADC + Citrix ADM provides the above set of capabilities. Citrix ADM using application health score can enable administrators to troubleshoot the issues.

Citrix ADM + Citrix ADC acts as the Ingress controller. Citrix ADM listens to the kube-api server and uses the stylebooks to configure the Citrix ADC which does the actual routing.

Background of Stylebooks: Stylebook is a feature in Citrix ADM. It is a template that can be used to create and manage complex Citrix ADC configurations.

In this case, a customer has an option to provide the LB methods, persistency, etc. in the stylebook, and the stylebook configures Citrix ADC using the REST APIs.

Below are the benefits of using Citrix ADM as Ingress controller:

  1. Security (Ratelimit, DDoS Protection)

  2. Analytics (Application performance troubleshooting)

  3. Service discovery and automatic configuration using Citrix ADM + Stylebooks

Flow diagram with Citrix ADC as ingress controller

image-cpx-with-kubernetes-03

How to make the choice of CPX vs MPX/VPX

A customer can choose either CPX or VPX/MPX/SDX as the ingress controller.

For many customers, the use of a hardware-based approach with MPX or SDX versus a software-based approach with VPX and CPX depends on how far they have moved towards the microservices architecture.

Customers who have started the journey may leverage an available MPX or SDX in front of Kubernetes, especially if they need to go into production for large clusters. For smaller clusters, the CPX would be suitable.

There are pros and cons for each of the form factors:

CPX form factor:

Pros:

  1. No additional integration required for participating in the overlay network

  2. CPX can be spun up or down through Kubernetes automatically

  3. Kubernetes replication controller can be used for maintaining HA

Cons:

  1. Limited throughput to < 10 Gbps

  2. Web Application Firewall is not supported in CPX

VPX/MPX/SDX form factor:

Pros:

  1. All Citrix ADC features are available

  2. High scale throughput and SSL performance

  3. Availability of clustering

Cons:

  1. Need additional integration in participating in the overlay network

The customer’s choice on VPX/MPX vs CPX depends on the performance requirement and the features required.

CPX with multi core can scale up to 7000 SSL TPS and 10 Gbps of throughput. But if the customer’s requirement is beyond the above numbers, then VPX/MPX should be chosen as Ingress device.


Configuration of Kubernetes cluster for Citrix ADC CPX

Below is the command to add the CPX as Ingress controller:

docker run -dt --privileged=true -p 5080:80 -p 5443:443 -p 80:5080 -e NS_HTTP_PORT=5080 -p 443:5443 -e NS_HTTPS_PORT=5443 -e EULA=yes -e NS_MGMT_SERVER=<MAS-IP> -e NS_MGMT_FINGER_PRINT="9C:2C:E7:64:38:C9:97:F1:0A:55:47:16:70:07:5B:70:B-

B:25:32:A3" -e NS_ROUTABLE=FALSE -e NS_LB_ROLE=SERVER -e HOST=$HOSTNAME 10.217.6.101:5000/cpx:latest

Run the above command on the Kubernetes Master for registering CPX to the Citrix ADM and act as Ingress controller.

Extract fingerprint from Citrix ADM

Get the fingerprint from the MAS using the following steps: bash-2.05b# more fingerprint.sh

#!/bin/sh

CHOST=${1:-localhost}

echo | openssl s_client -connect $CHOST:443 | openssl x509 -fingerprint -noout | cut -d'=' - f2

bash-2.05b# pwd

/root

bash-2.05b# ls

.bash_history .ssh

bash-2.05b# sh fingerprint.sh

depth=0 C = US, ST = California, L = San Jose, O = Citrix ADC, OU = Internal, CN = Test Only Cert

verify error:num=18:self signed certificate

verify return:1

depth=0 C = US, ST = California, L = San Jose, O = Citrix ADC, OU = Internal, CN = Test Only Cert

verify return:1

DONE

5F:97:C3:14:72:66:5E:C4:EB:E2:5B:9E:FA:D2:74:7F:AC:9D:59:F8

bash-2.05b#

Configuration of Citrix ADM for Container Management

Go to Menu > Orchestration > Container Orchestration.

Provide the kube-api url and the certificate/key for authentication purpose. Below is the Citrix ADM snapshot. The cert/ key data is available in Kubernetes master node in the file /etc/kubernetes/admin.conf, and the same should be used while adding the kube-api server url in the Citrix ADM.


Citrix ADC CPX as kube-proxy replacement

Kube Proxy

Kube-proxy listens to the kube-api server and adds/removes iptables rules based on the addition/removal of services, so that the new services are accessible by the clients.

CPX can replace kube-proxy. It provides the following benefits:

  1. Application health score.
    1. CPX is integrated with Citrix ADM. CPX provides the telemetry data to the MAS, and Citrix ADM applications can be defined and application performance analytics can be done on the microservices. Some of the use cases are listed below.

    2. Health of the backend services can be monitored

    3. Administrator can isolate the issue to particular node using the data from Citrix ADM

  2. SSL Offload

  3. Rate Limit

image-cpx-with-kubernetes-04

Steps to register CPX with ADM and act as kube-proxy replacement

  1. Extract the fingerprint from Citrix ADM.

    Get the fingerprint from the MAS using the following steps:

    bash-2.05b# more fingerprint.sh
    
    #!/bin/sh
    
    CHOST=${1:-localhost}
    
    echo | openssl s_client -connect $CHOST:443 | openssl x509 -fingerprint -noout | cut -d'=' -f2
    
    bash-2.05b# pwd
    
    /root
    
    bash-2.05b# ls
    
    .bash_history .ssh  fingerprint.sh
    
    bash-2.05b# sh fingerprint.sh
    
    depth=0 C = US, ST = California, L = San Jose, O = Citrix ADC, OU = Internal, CN = Test Only Cert
    
    verify error:num=18:self signed certificate
    
    verify return:1
    
    depth=0 C = US, ST = California, L = San Jose, O = Citrix ADC, OU = Internal, CN = Test Only Cert
    
    verify return:1
    
    DONE
    
    5F:97:C3:14:72:66:5E:C4:EB:E2:5B:9E:FA:D2:74:7F:AC:9D:59:F8
    
    bash-2.05b#
    

    Last line in the above output (5F:97:C3:14:72:66:5E:C4:EB:E2:5B:9E:FA:D2:74:7F:AC:9D:59:F8) is the fingerprint and this value should be used by the CPX in registering with the Citrix ADM.

  2. Create YAML File(s)

    Sample YAML file format:

    image-cpx-with-kubernetes-05

    The following table defines values for some of the most common key fields in the yaml file:

    Field Definition
    ImagePullPolicy This field indicates whether the docker image should be downloaded from the docker repository every time the cpx is started.
    nodeSelector: Kube_no The label on which the cpx should be started. Label and kubernetes node are key value pairs and created using the command kubectl label nodes cpxkube-1 Kube_no=minion_
    NS_MGMT_SERVER Citrix ADM IP
    NS_MGMT_FINGER Finger print of ADM and used in registration
  3. Label the CPX Nodes

    Label the nodes using the below command. In the below command cpxkube-1 is one of the nodes in kuber netes cluster.

    kubectl label nodes cpxkube-1 Kube_no=minion_1

  4. Create the CPX Instances

Start using kubectl –f create cpx_1.yaml

The last step of kubectl –f create needs to be repeated for each of the nodes in the cluster.

After the CPX is launched, the CPX registers itself with Citrix ADM.

Citrix ADM using Stylebook/NITRO APIs configures the CPX based on the events received from kube-api.


Daemonset

Another approach to launch CPX as kube-proxy on the all nodes is daemonset. A key advantage of using daemonset is that the administrator doesn’t need to launch cpx from each master node, because daemonset configuration takes care of the launching CPX on all of the cluster nodes.

Below is the yaml file with the daemonset configuration:

image-cpx-with-kubernetes-06


Conclusion

Citrix ADC along with Citrix ADM solves the following challenges faced in the deployment of microservice environment.

  1. Reliable/secure delivery of requests

  2. Failure handling of services

  3. Visibility into application traffic and ability to troubleshoot application performance issues using application health score


Additional references

Citrix CPX datasheet

Cluster management tools

Kubernetes reference material

Lyft engineering reference