Advanced Concepts

Service Migration to Citrix ADC using Routes in OpenShift Validated Reference Design

Static and Auto Routes in an OpenShift cluster

Static Routes (Default) - maps the OpenShift HostSubnet to the External ADC via a Static route

Static routes are common in legacy OpenShift deployments using HAProxy. The static routes can be used in parallel with Citrix node controller (CNC), Citrix ingress controller (CIC), and CPX when migrating services from one Service Proxy to another without disrupting the deployed namespaces in a functioning cluster.

Example static route configuration for Citrix ADC:

    oc get hostsubnet (Openshift Cluster) snippet
        oc311-master.example.com 10.x.x.x 10.128.0.0/23
        oc311-node1.example.com 10.x.x.x 10.130.0.0/23
        oc311-node2.example.com 10.x.x.x 10.129.0.0/23

    show route (external Citrix VPX) snippet
        10.128.0.0 255.255.254.0 10.x.x.x STATIC
        10.129.0.0 255.255.254.0 10.x.x.x STATIC
        10.130.0.0 255.255.254.0 10.x.x.x STATIC

Auto Routes - uses the CNC (Citrix Node Controller) to automate the external routes to the defined route shards

You can integrate Citrix ADC with OpenShift in two ways, both of which support OpenShift router sharding.

Route Types

  • unsecured - external load balancer to CIC router, HTTP traffic is not encrypted.
  • secured-edge - external load balancer to CIC router terminating TLS.
  • secured-passthrough - external load balancer to destination terminating TLS
  • secured reencrypt - external load balancer to CIC router terminating TLS. CIC router encrypting to destination using TLS.

See more about the different route types in Citrix ingress controller deployment solutions.

Deploy the Citrix Ingress Controller with OpenShift router sharding support

The Citrix Ingress Controller (CIC) acts as a Router and redirects the traffic to various pods to distribute incoming traffic among various available pods.

This migration process can also be part of a cluster upgrade process from legacy OpenShift topologies to automated deployments with Citrix CNC, CIC, and CPX components for cluster migration and upgrade procedures.

This solution can be achieved by two methods:

  • CIC Router Plug-In (Pod)
  • CPX Router inside OpenShift (Sidecar)

Both of the methods are described below along with migration examples.

OpenShift router sharding allows distributing a set of routes among multiple OpenShift routers. By default, an OpenShift router selects all routes from all namespaces. In router sharding, labels are added to routes or namespaces and label selectors to routers for filtering routes. Each router shard selects only routes with specific labels that match its label selection parameters.

To configure router sharding for a Citrix ADC deployment on OpenShift, a Citrix ingress controller instance is required per shard. The Citrix ingress controller instance is deployed with route or namespace labels or both as environment variables depending on the criteria required for sharding. When the Citrix ingress controller processes a route, it compares the route’s labels or route’s namespace labels with the selection criteria configured on it. If the route satisfies the criteria, the appropriate configuration is applied to Citrix ADC, otherwise it does not apply the configuration.

In router sharding, selecting a subset of routes from the entire pool of routes is based on selection expressions. Selection expressions are a combination of multiple values and operations. See more about the expressions, values, and operations in this Citrix blog.

Bookinfo deployment

The architecture for the Bookinfo application is seen in the figure below. A CIC is deployed as an OpenShift router plug-in in the first tier, configuring the Citrix ADC VPX to route North-South traffic to the Product Page. In the second tier, a Citrix ADC CPX is deployed as an OpenShift router, routing East-West traffic between the Details and Product Page microservice, whereas East-West traffic between the Product Page, Reviews, and Ratings microservices are being routed through the default HAProxy router.

Route sharding diagram of the Bookinfo app deployed as a service mesh lite architecture

Citrix Components

  • VPX - The Ingress ADC that is presenting the cluster services to DNS.
  • CIC - provides the ROUTE_LABELS and NAMESPACE_LABELS to the external Citrix ADC via the CNC route.
  • CPX - provides OpenShift routing within the OpenShift cluster.

Deployment Steps

  1. Create a namespace for the deployment. oc create ns sml
  2. Deploy the Bookinfo application into the namespace. oc apply -f bookinfo.yaml

    ##################################################################################################
    # Details service
    ##################################################################################################
    apiVersion: v1
    kind: Service
    metadata:
      name: details
      labels:
        app: details
        service: details
    spec:
      ports:
      - port: 9080
        name: http
      selector:
        app: details
    ---
    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: details-v1
      labels:
        app: details
        version: v1
    spec:
      replicas: 1
      template:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
          labels:
            app: details
            version: v1
        spec:
          containers:
    "bookinfo.yaml" 224L, 5120C
    
    
  3. Deploy the route file that maps to our product page service. Specify frontend-ip (This is the content switching vip on the Tier 1 ADC) oc apply -f routes-productpage.yaml

    apiVersion: v1
    kind: Route
    metadata:
       name: productpage-route
       namespace: sml
       annotations:
        ingress.citrix.com/frontend-ip: "X.X.X.X"
       labels:
        name: productpage
    spec:
       host: bookinfo.com
       path: /
       port:
         targetPort: 80
       to:
         kind: Service
         name: productpage-service
    
  4. Deploy the RBAC file for the sml namespace that gives the CIC the necessary permissions to run. The RBAC file is already namespaced. oc apply -f rbac.yaml

     kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: cpx
    rules:
      - apiGroups: [""]
        resources: ["endpoints", "ingresses", "services", "pods", "secrets", "nodes", "routes", "namespaces","tokenreviews","subjectaccessreview"]
        verbs: ["get", "list", "watch"]
      # services/status is needed to update the loadbalancer IP in service status for integrating
      # service of type LoadBalancer with external-dns
      - apiGroups: [""]
        resources: ["services/status"]
        verbs: ["patch"]
      - apiGroups: ["extensions"]
        resources: ["ingresses", "ingresses/status"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["apiextensions.k8s.io"]
        resources: ["customresourcedefinitions"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["apps"]
        resources: ["deployments"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["citrix.com"]
        resources: ["rewritepolicies", "canarycrds", "authpolicies", "ratelimits"]
        verbs: ["get", "list", "watch"]
      - apiGroups: ["citrix.com"]
        resources: ["vips"]
        verbs: ["get", "list", "watch", "create", "delete"]
      - apiGroups: ["route.openshift.io"]
        resources: ["routes"]
        verbs: ["get", "list", "watch"]
    
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1beta1
    metadata:
      name: cpx
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cpx
    subjects:
    - kind: ServiceAccount
      name: cpx
      namespace: sml
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
    "rbac.yaml" 51L, 1513C
    
  5. Deploy your CIC to push the route configs to your VPX. Match the parameter ROUTE_LABELS to the label specified in the route-productpage.yaml. For more information on syntax for ROUTE_LABELS, please see this blog. oc apply -f cic-productpage-v2.yaml

    apiVersion: v1
    kind: Pod
    metadata:
      name: cic
      labels:
        app: cic
    spec:
      serviceAccount: cpx
      containers:
      - name: cic
        image: "quay.io/citrix/citrix-k8s-ingress-controller:1.7.6"
        securityContext:
           privileged: true
        env:
        - name: "EULA"
          value: "yes"
        # Set NetScaler NSIP/SNIP, SNIP in case of HA (mgmt has to be enabled)
        - name: "NS_IP"
          value: "X.X.X.X"
        # Set NetScaler VIP that receives the traffic
    #    - name: "NS_VIP"
    #      value: "X.X.X.X"
        - name: "NS_USER"
          value: "nsroot"
        - name: "NS_PASSWORD"
          value: "nsroot"
        - name: "NS_APPS_NAME_PREFIX"
          value: "BOOK"
        - name: "ROUTE_LABELS"
          value: "name in (productpage)"
    #    - name: "NAMESPACE_LABELS"
    #      value: "app=hellogalaxy"
        # Set username for Nitro
    #    - name: "NS_USER"
    #      valueFrom:
    #       secretKeyRef:
    #        name: nslogin
    #        key: nsroot
    #    # Set user password for Nitro
    #    - name: "NS_PASSWORD"
    #      valueFrom:
    #       secretKeyRef:
    #        name: nslogin
    #        key: nsroot
        args:
    #    - --default-ssl-certificate
    #      default/default-cert
        imagePullPolicy: Always
    ~
    "cic-productpage-v2.yaml" 48L, 1165C
    
  6. Now we must create a headless service that points users looking for details to our CPX using the DNS pods in our cluster. oc apply -f detailsheadless.yaml

    ##################################################################################################
    # Details service
    ##################################################################################################
    apiVersion: v1
    kind: Service
    metadata:
      name: details
    spec:
      ports:
      - port: 9080
        name: http
      selector:
        app: cpx
    
  7. Deploy a new service to expose the details container. oc apply -f detailsservice.yaml

    ##################################################################################################
    # Details service
    ##################################################################################################
    apiVersion: v1
    kind: Service
    metadata:
      name: details-service
      labels:
        app: details-service
        service: details-service
    spec:
      clusterIP: None
      ports:
      - port: 9080
        name: http
      selector:
        app: details
    
  8. Deploy a new route definition that sits in front of the details service that we created. Notice the label is “name: details”. oc apply -f detailsroutes.yaml

    apiVersion: v1
    kind: Route
    metadata:
       name: details-route
       namespace: sml
       annotations:
        ingress.citrix.com/insecure-port: "9080"
       labels:
        name: details
    spec:
       host: details
       path: /
       port:
          targetPort: 9080
       to:
         kind: Service
         name: details-service
    
  9. Deploy CPX for E/W traffic. A CIC is deployed as a sidecar and is configured with a ROUTE_LABEL parameter to match the label in detailsroutes.yaml. oc apply -f cpx.yaml

    apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: cpx
      labels:
        app: cpx
        service: cpx
    spec:
      replicas: 1
      template:
        metadata:
          name: cpx
          labels:
            app: cpx
            service: cpx
          annotations:
            NETSCALER_AS_APP: "True"
        spec:
          serviceAccountName: cpx
          containers:
            - name: cpx
              image: "quay.io/citrix/citrix-k8s-cpx-ingress:13.0-36.28"
              securityContext:
                 privileged: true
              env:
              - name: "EULA"
                value: "yes"
              - name: "KUBERNETES_TASK_ID"
                value: ""
              - name: "MGMT_HTTP_PORT"
                value: "9081"
              ports:
              - name: http
                containerPort: 9080
              - name: https
                containerPort: 443
              - name: nitro-http
                containerPort: 9081
              - name: nitro-https
                containerPort: 9443
    # readiness probe?
              imagePullPolicy: Always
            # Add cic as a sidecar
            - name: cic
              image: "quay.io/citrix/citrix-k8s-ingress-controller:1.7.6"
              env:
              - name: "EULA"
                value: "yes"
              - name: "NS_IP"
    "cpx.yaml" 75L, 1939C
    

Continuous Delivery choices in a microservices environment

Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day.

Continuous Delivery (CD) is the natural extension of Continuous Integration: an approach in which teams ensure that every change to the system is releasable, and that we can release any version at the push of a button.

The different CD choices, along with their pros and cons, are:

  • Recreate - Version 1 (V1) is terminated and then Version 2 (V2) is rolled out.
    • Pros
      • Easy to set up.
      • Application state is entirely renewed.
    • Cons
      • High impact on the user. Expect downtime that depends on both shutdown and boot duration.
  • Ramped/Rolling Update - V2 is slowly rolled out and replaces V1.
    • Pros
      • Easy to set up.
      • Version is slowly released across instances.
      • Convenient for stateful applications that can handle rebalancing of the data.
    • Cons
      • Rollout/rollback can take time.
      • Supporting multiple APIs is hard.
      • Little control of traffic.
  • Blue Green - V2 is released alongside V1, then the traffic is switched to V2.
    • Pros
      • Instant rollout/rollback.
      • Avoid version issue since the entire application is changed at once.
    • Cons
      • Expensive as it requires double the resources.
      • Proper test of the entire platform should be done before releasing to production.
      • Handling multiple stateful applications can be hard.
  • Canary - V2 is released to a subset of users and then proceeds to a full rollout.
    • Pros
      • Version released for a subset of users.
      • Convenient for error rate and performance monitoring.
      • Fast rollback.
    • Cons
      • Slow rollout.
      • Handling multiple stateful applications can be hard.
  • A/B Testing - V2 is released to a subset of users under specific conditions.
    • Pros
      • Several versions run in parallel.
      • Full control over the traffic distribution.
    • Cons
      • Requires intelligent load balancer.
      • Hard to troubleshoot errors for a given session, so distributed tracing becomes mandatory.
  • Shadow - V2 receives real-world traffic alongside V11 and does not impact the response.
    • Pros
      • Performance testing of the application with production traffic.
      • No impact on the user.
      • No rollout until the stability and performance of the application meet the requirements.
    • Cons
      • Expensive as it requires double the resources.
      • Not a true user test and can be misleading.
      • Complex to set up.
      • Requires mocking service for certain cases.

Reference Materials

Citrix GitHub: “OpenShift routes and ingress”

Citrix Developer Docs: “Deployment solutions”

Citrix Blog: “Enable OpenShift router sharding support with Citrix ADC”

OpenShift Route Documentation:

Service Migration to Citrix ADC using Routes in OpenShift Validated Reference Design