NetScaler ingress controller

Deploy the NetScaler Ingress Controller as an OpenShift router plug-in

In an OpenShift cluster, external clients need a way to access the services provided by pods. OpenShift provides two resources for communicating with services running in the cluster: routes and Ingress.

In an OpenShift cluster, a route exposes a service on a given domain name or associates a domain name with a service. OpenShift routers route external requests to services inside the OpenShift cluster according to the rules specified in routes. When you use the OpenShift router, you must also configure the external DNS to make sure that the traffic is landing on the router.

The NetScaler Ingress Controller can be deployed as a router plug-in in the OpenShift cluster to integrate with NetScalers deployed in your environment. The NetScaler Ingress Controller enables you to use the advanced load balancing and traffic management capabilities of NetScaler with your OpenShift cluster.

OpenShift routes can be secured or unsecured. Secured routes specify the TLS termination of the route.

The NetScaler Ingress Controller supports the following OpenShift routes:

  • Unsecured Routes: For Unsecured routes, HTTP traffic is not encrypted.

  • Edge Termination: For edge termination, TLS is terminated at the router. Traffic from the router to the endpoints over the internal network is not encrypted.

  • Passthrough Termination: With passthrough termination, the router is not involved in TLS offloading and encrypted traffic is sent straight to the destination.

  • Re-encryption Termination: In re-encryption termination, the router terminates the TLS connection but then establishes another TLS connection to the endpoint.

For detailed information on routes, see the OpenShift documentation.

You can either deploy a NetScaler MPX or VPX appliance outside the OpenShift cluster or deploy NetScaler CPXs as pods inside the cluster. The NetScaler Ingress Controller integrates NetScalers with the OpenShift cluster and automatically configures NetScalers based on rules specified in routes.

Based on how you want to use NetScaler, there are two ways to deploy the NetScaler Ingress Controller as a router plug-in in the OpenShift cluster:

  • As a sidecar container alongside NetScaler CPX in the same pod: In this mode, the NetScaler Ingress Controller configures the NetScaler CPX.

  • As a standalone pod in the OpenShift cluster: In this mode, you can control the NetScaler MPX or VPX appliance deployed outside the cluster.

For information on deploying the NetScaler Ingress Controller to control the OpenShift ingress, see the NetScaler Ingress Controller for Kubernetes.

You can use NetScaler for load balancing OpenShift control plane (master nodes). NetScaler provides a solution to automate the configuration of NetScaler using Terraform instead of manually configuring the NetScaler. For more information, see NetScaler as a load balancer for the OpenShift control plane.

Alternate Backend Support

OpenShift Alternate backends is now supported by NetScaler Ingress Controller.

NetScaler is configured according to the weights provided in the routes definition and traffic is distributed among the service pods based on those weights.

The following is an example of a route manifest with alternate backend:

 kind: Route
 apiVersion: route.openshift.io/v1
 metadata:
   name: r1
   labels:
     name: apache
   annotations:
     ingress.citrix.com/frontend-ip: "<Frontend-ip>"
 spec:
   host: some.alternate-backends.com
   to:
     kind: Service
     name: apache-1
     weight: 30
   alternateBackends:
     - kind: Service
       name: apache-2
       weight: 20
     - kind: Service
       name: apache-3
       weight: 50
   port:
     targetPort: 80
   wildcardPolicy: None
<!--NeedCopy-->

For this route, 30 percent of the traffic is sent to the service apache-1 and 20 percent is sent to the service apache2 and 50 percent to the service apache-3 based on weights provided in the route manifest

Supported Citrix components on OpenShift

Citrix components Versions
NetScaler Ingress Controller Latest
NetScaler VPX 12.1 50.x and later
NetScaler CPX 13.0–36.28

Note:

CRDs provided for the NetScaler Ingress Controller is not supported for OpenShift routes. You can use OpenShift ingress to use CRDs.

Deploy NetScaler CPX as a router within the OpenShift cluster

In this deployment, you can use the NetScaler CPX instance for load balancing the North-South traffic to microservices in your OpenShift cluster. The NetScaler Ingress Controller is deployed as a sidecar alongside the NetScaler CPX container in the same pod using the cpx_cic_side_car.yaml file.

Before you begin: When you deploy NetScaler CPX as a router, port conflicts can arise with the default router in OpenShift. You should remove the default router in OpenShift before deploying NetScaler CPX as a router. To remove the default router in OpenShift, perform the following steps:

  1. Back up the default router configuration using the following command.

    oc get -o yaml dc/router clusterrolebinding/router-router-role serviceaccount/router > default-router-backup.yaml
    
  2. Delete the default router using the following command.

    oc delete -f default-router-backup.yaml
    

Perform the following steps to deploy NetScaler CPX as a router with the NetScaler Ingress Controller as a sidecar.

  1. Download the cpx_cic_side_car.yaml file using the following command:

    wget https://raw.githubusercontent.com/citrix/citrix-k8s-ingress-controller/master/deployment/openshift/manifest/cpx_cic_side_car.yaml
    
  2. Add the service account to privileged security context constraints (SCC) of OpenShift.

    oc adm policy add-scc-to-user privileged system:serviceaccount:default:citrix
    
  3. Deploy the NetScaler Ingress Controller using the following command:

    oc create -f cpx_cic_side_car.yaml
    
  4. Verify if the NetScaler Ingress Controller is deployed successfully using the following command:

    oc get pods --all-namespaces
    

Deploy NetScaler MPX/VPX as a router outside the OpenShift cluster

In this deployment, the NetScaler Ingress Controller which runs as a stand-alone pod allows you to control the NetScaler MPX, or VPX appliance from the OpenShift cluster. You can use the cic.yaml file for this deployment.

Note: The NetScaler MPX or VPX can be deployed in standalone, high-availability, or clustered modes.

Prerequisites

  • Determine the IP address needed by the NetScaler Ingress Controller to communicate with the NetScaler appliance. The IP address might be any one of the following depending on the type of NetScaler deployment:
    • NSIP (for standalone appliances): The management IP address of a standalone NetScaler appliance. For more information, see IP Addressing in NetScaler.
    • SNIP (for appliances in High Availability mode): The subnet IP address. For more information, see IP Addressing in NetScaler.
    • CLIP (for appliances in clustered mode): The cluster management IP (CLIP) address for a clustered NetScaler deployment. For more information, see IP addressing for a cluster.
  • The user name and password of the NetScaler VPX or MPX appliance used as the Ingress device. If you are not using the default credentials, the NetScaler appliance must have a system user account with certain privileges so that the NetScaler Ingress Controller can configure the NetScaler MPX, or VPX appliance. To create a system user account on NetScaler, see Create a system user account for the NetScaler Ingress Controller in NetScaler.

    You can directly pass the user name and password as environment variables to the NetScaler Ingress Controller or use OpenShift secrets (recommended). If you want to use OpenShift secrets, create a secret for the user name and password using the following command:

     oc create secret generic nslogin --from-literal=username=<username> --from-literal=password=<password>
    

Create a system user account for the NetScaler Ingress Controller in NetScaler

The NetScaler Ingress Controller configures a NetScaler appliance (MPX or VPX) using a system user account of the NetScaler appliance. The system user account must have the permissions to configure the following tasks on the NetScaler:

  • Add, Delete, or View Content Switching (CS) virtual server
  • Configure CS policies and actions
  • Configure Load Balancing (LB) virtual server
  • Configure Service groups
  • Cofigure SSL certkeys
  • Configure routes
  • Configure user monitors
  • Add system file (for uploading SSL testkeys from OpenShift)
  • Configure Virtual IP address (VIP)
  • Check the status of the NetScaler appliance
  • Configure SSL actions and policies
  • Configure SSL vServer
  • Configure responder actions and policies

To create the system user account, perform the following:

  1. Log on to the NetScaler appliance using the following steps:

    1. Use an SSH client, such as PuTTy, to open an SSH connection to the NetScaler appliance.

    2. Log on to the appliance by using the administrator credentials.

  2. Create the system user account using the following command:

    add system user <username> <password>
    

    For example:

    add system user cic mypassword
    
  3. Create a policy to provide required permissions to the system user account. Use the following command:

    add cmdpolicy cic-policy ALLOW '^\(\?!shell)\(\?!sftp)\(\?!scp)\(\?!batch)\(\?!source)\(\?!.\*superuser)\(\?!.\*nsroot)\(\?!install)\(\?!show\s+system\s+\(user|cmdPolicy|file))\(\?!\(set|add|rm|create|export|kill)\s+system)\(\?!\(unbind|bind)\s+system\s+\(user|group))\(\?!diff\s+ns\s+config)\(\?!\(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).\*|\(^install\s\*\(wi|wf))|\(^\S+\s+system\s+file)^\(\?!shell)\(\?!sftp)\(\?!scp)\(\?!batch)\(\?!source)\(\?!.\*superuser)\(\?!.\*nsroot)\(\?!install)\(\?!show\s+system\s+\(user|cmdPolicy|file))\(\?!\(set|add|rm|create|export|kill)\s+system)\(\?!\(unbind|bind)\s+system\s+\(user|group))\(\?!diff\s+ns\s+config)\(\?!\(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).\*|\(^install\s\*\(wi|wf))|\(^\S+\s+system\s+file)'
    

    Note: The system user account would have privileges based on the command policy that you define.

    The command policy mentioned in step 3 is similar to the built-in sysAdmin command policy with another permission to upload files.

    In the command policy specification provided, special characters which need to be escaped are already omitted to easily copy-paste into the NetScaler command line.

    For configuring the command policy from the NetScaler configuration wizard (GUI), use the following command policy specification.

    ^\(?!shell)\(?!sftp)\(?!scp)\(?!batch)\(?!source)\(?!.\*superuser)\(?!.\*nsroot)\(?!install)\(?!show\s+system\s+\(user|cmdPolicy|file))\(?!\(set|add|rm|create|export|kill)\s+system)\(?!\(unbind|bind)\s+system\s+\(user|group))\(?!diff\s+ns\s+config)\(?!\(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).\*|\(^install\s\*\(wi|wf))|\(^\S+\s+system\s+file)^\(?!shell)\(?!sftp)\(?!scp)\(?!batch)\(?!source)\(?!.\*superuser)\(?!.\*nsroot)\(?!install)\(?!show\s+system\s+\(user|cmdPolicy|file))\(?!\(set|add|rm|create|export|kill)\s+system)\(?!\(unbind|bind)\s+system\s+\(user|group))\(?!diff\s+ns\s+config)\(?!\(set|unset|add|rm|bind|unbind|switch)\s+ns\s+partition).\*|\(^install\s\*\(wi|wf))|\(^\S+\s+system\s+file)
    
  4. Bind the policy to the system user account using the following command:

    bind system user cic cic-policy 0
    

Deploy the NetScaler Ingress Controller as a pod in an OpenShift cluster

Perform the following steps to deploy the NetScaler Ingress Controller as a pod:

  1. Download the cic.yaml file using the following command:

    wget https://raw.githubusercontent.com/citrix/citrix-k8s-ingress-controller/master/deployment/openshift/manifest/cic.yaml
    
  2. Edit the cic.yaml file and enter the values for the following environmental variables:

    Environment Variable Mandatory or Optional Description
    NS_IP Mandatory The IP address of the NetScaler appliance. For more details, see Prerequisites.
    NS_USER and NS_PASSWORD Mandatory The user name and password of the NetScaler VPX or MPX appliance used as the Ingress device. For more details, see Prerequisites.
    EULA Mandatory The End User License Agreement. Specify the value as Yes.
    NS_VIP Optional NetScaler Ingress Controller uses the IP address provided in this environment variable to configure a virtual IP address to the NetScaler that receives Ingress traffic. Note: NS_VIP acts as a fallback when the frontend-ip annotation is not provided in Ingress or Route yaml. Not supported for Type Loadbalancer service.
  3. Add the service account to privileged security context constraints (SCC) of OpenShift.

    oc adm policy add-scc-to-user privileged system:serviceaccount:default:citrix
    
  4. Save the edited cic.yaml file and deploy it using the following command:

     oc create -f cic.yaml
    
  5. Verify if the NetScaler Ingress Controller is deployed successfully using the following command:

    oc create get pods --all-namespaces
    
  6. Configure static routes on NetScaler VPX or MPX to reach the pods inside the OpenShift cluster.

    1. Use the following command to get the information about host names, host IP addresses, and subnets for static route configuration.

      oc get hostsubnet
      
    2. Log on to the NetScaler instance.

    3. Add the route on the NetScaler instance using the following command.

      add route <pod_network> <netmask> <gateway>
      
    <B>Example:</b>
    
          oc get hostsubnet
      
          NAME            HOST           HOST IP        SUBNET
          os.example.com  os.example.com 192.168.122.46 192.1.1.0/24
    
     From the output of the `oc get hostsubnet` command:
    
          <pod_network> = 192.1.1.0
          Value for subnet = 192.1.1.0/x where x = 24 that means <netmask>= 255.255.255.0
          <gateway> = 192.168.122.46
    
     The required static route to add on NetScaler is as follows:
    
          add route 10.1.1.0 255.255.255.0 192.168.122.46
    

Example: Deploy the NetScaler Ingress Controller as a router plug-in in an OpenShift cluster

In this example, the NetScaler Ingress Controller is deployed as a router plug-in in the OpenShift cluster to load balance an application.

  1. Deploy a sample application (apache.yaml) in your OpenShift cluster and expose it as a service in your cluster using the following command.

    oc create -f https://raw.githubusercontent.com/citrix/citrix-k8s-ingress-controller/master/deployment/openshift/manifest/apache.yaml
    

    Note: When you deploy a normal Apache pod in OpenShift, it may fail as Apache pod always runs as a root pod. OpenShift has strict security checks which block running a pod as root or binding to port 80. As a workaround, you can add the default service account of the pod to the privileged security context of OpenShift by using the following commands:

        oc adm policy add-scc-to-user privileged system:serviceaccount:default:default
        oc adm policy add-scc-to-group anyuid system:authenticated
    

    The content of the apache.yaml file is given as follows.

    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        name: apache-only-http
      name: apache-only-http
    spec:
      replicas: 4
      selector:
        matchLabels:
          app: apache-only-http
      template:
        metadata:
          labels:
            app: apache-only-http
        spec:
          containers:
          - image: raghulc/apache-multiport-http:1.0.0
            imagePullPolicy: IfNotPresent
            name: apache-only-http
            ports:
            - containerPort: 80
              protocol: TCP
            - containerPort: 5080
              protocol: TCP
            - containerPort: 5081
              protocol: TCP
            - containerPort: 5082
              protocol: TCP
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        name: apache-only-ssl
      name: apache-only-ssl
    spec:
      replicas: 4
      selector:
        matchLabels:
          app: apache-only-ssl
      template:
        metadata:
          labels:
            app: apache-only-ssl
        spec:
          containers:
          - image: raghulc/apache-multiport-ssl:1.0.0
            imagePullPolicy: IfNotPresent
            name: apache-only-ssl
            ports:
            - containerPort: 443
              protocol: TCP
            - containerPort: 5443
              protocol: TCP
            - containerPort: 5444
              protocol: TCP
            - containerPort: 5445
              protocol: TCP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: svc-apache-multi-http
    spec:
      ports:
      - name: apache-http-6080
        port: 6080
        targetPort: 5080
      - name: apache-http-6081
        port: 6081
        targetPort: 5081
      - name: apache-http-6082
        port: 6082
        targetPort: 5082
      selector:
        app: apache-only-http
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: svc-apache-multi-ssl
    spec:
      ports:
      - name: apache-ssl-6443
        port: 6443
        targetPort: 5443
      - name: apache-ssl-6444
        port: 6444
        targetPort: 5444
      - name: apache-ssl-6445
        port: 6445
        targetPort: 5445
      selector:
        app: apache-only-ssl
    ---
    
  2. Deploy the NetScaler Ingress Controller for NetScaler VPX as a stand-alone pod in the OpenShift cluster using the steps in Deploy the NetScaler Ingress Controller as a pod.

    oc create -f cic.yaml
    

    Note: To deploy the NetScaler Ingress Controller with NetScaler CPX in the OpenShift cluster, perform the steps in Deploy the NetScaler Ingress Controller as a sidecar with NetScaler CPX.

  3. Create an OpenShift route for exposing the application.

    • For creating an unsecured OpenShift route (unsecured-route.yaml), use the following command:

       oc create -f unsecured-route.yaml
      
    • For creating a secured OpenShift route with edge termination (secured-edge-route.yaml), use the following command.

       oc create -f secured-route-edge.yaml
      
    • For creating a secured OpenShift route with passthrough termination (secured-passthrough-route.yaml), use the following command.

       oc create -f secured-passthrough-route.yaml
      
    • For creating a secured OpenShift route with re-encryption termination (secured-reencrypt-route.yaml), use the following command.

       oc create -f secured-reencrypt-route.yaml
      

    To see the contents of the YAML files for OpenShift routes in this example, see YAML files for routes.

    Note:

    For a secured OpenShift route with passthrough termination, you must include the default certificate.

  4. Access the application using the following command.

      curl http://<VIP of NetScaler VPX>/ -H 'Host: < host-name-as-per-the-host-configuration-in-route >'
      <!--NeedCopy-->
    

YAML files for routes

This section contains YAML files for unsecured and secured routes specified in the example.

Note: Keys used in this example are for testing purpose only. You must create your own keys for the actual deployment.

The contents of the unsecured-route.yaml file is given as follows:

apiVersion: v1
kind: Route
metadata:
  name: unsecured-route
spec:
  host: unsecured-route.openshift.citrix-cic.com
  path: "/"
  to:
    kind: Service
    name: svc-apache-multi-http
<!--NeedCopy-->

See, secured-edge-route for the contents of the secured-edge-route.yaml file.

The contents of the secured-passthrough-route is given as follows:

apiVersion: v1
kind: Route
metadata:
  name: secured-passthrough-route
spec:
  host: secured-passthrough-route.openshift.citrix-cic.com
  to:
    kind: Service
    name: svc-apache-multi-ssl
  tls:
    termination: passthrough
<!--NeedCopy-->

See, secured-reencrypt-route for the contents of the secured-reencrypt-route.yaml file.