Citrix ADC CPX troubleshooting

This document explains how to troubleshoot issues that you may encounter while using Citrix ADC CPX. Using this document, you can collect logs to determine the causes and apply workarounds for some of the common issues related to the installation and configuration of Citrix ADC CPX.

  • How can I view Citrix ADC CPX logs?

    You can view Citrix ADC CPX logs using the kubectl logs command if Citrix ADC CPX is deployed with the tty:true option. You can run the following command to display the logs:

     kubectl logs <pod-name> [-c <container-name>] [-n <namespace-name>]
    

    Example,

     kubectl logs cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc
    

    The following is an example of the Citrix ADC CPX pod deployment with the tty:true option:

       containers:
         - name: cpx-ingress
           image: "quay.io/citrix/citrix-k8s-cpx-ingress:13.0-58.30"
           tty: true
           securityContext:
              privileged: true
           env:
    
     <!--NeedCopy-->
    

    You can find more boot logs in the /cpx/log/boot.log file of Citrix ADC CPX file system.

    Note: To get the pod name, run the kubectl get pods -o wide command.

  • How can I collect the technical support bundle from Citrix ADC CPX?

    You can run the following command on the shell interface of the Kubernetes master node to collect the Citrix ADC CPX technical support bundle:

     kubectl exec <cpx-pod-name> [-c <cpx-container-name>] [-n <namespace-name>] /var/netscaler/bins/cli_script.sh "show techsupport"
    

    You can view the technical support bundle in the /var/tmp/support directory of the Citrix ADC CPX’s file system. Use scp or kubectl cp to copy the technical support bundle from Citrix ADC CPX to the desired destination.

    Example:

     root@localhost# kubectl exec cpx-ingress1-55b9b6fc75-t5kc6 -c cpx -n citrix-adc /var/netscaler/bins/cli_script.sh "show techsupport"
     exec: show techsupport
       Scope:  NODE
     Done
     root@localhost# kubectl cp cpx-ingress1-55b9b6fc75-t5kc6:var/tmp/support/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz -c cpx
     root@localhost# ll /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz
     -rw-r--r-- 1 root root 1648109 Aug 31 13:23 /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz
    
  • Why is Citrix ADC CPX pod stuck while booting?

    You can check the pod status using the kubectl describe pods command. Run the following command to know the pod status:

     kubectl describe pods <pod-name> [-c <container-name>] [-n <namespace-name>]
    

    Example:

     kubectl describe pods cpx-ingress1-69b9b8c648-t8bgn
    

    If the pod events show that container is started, then you must check the pod logs.

  • How do I copy files between the Citrix ADC CPX pod and the Kubernetes master node?

    It is recommended to use the volume mount feature of docker to mount the /cpx directory to the file system of the host. If a Citrix ADC CPX container exits core-dumps, logs and other important data are available on the mount point.

    You can use any one of the following commands to copy files between the Citrix ADC CPX pod and the Kubernetes master node:

    kubectl cp: You can run the following command to copy files from pod to node:

     kubectl cp <pod-name>:<absolute-src-path> <dst-path> [-c <container-name>] [-n <namespace-name>]
    

    Example:

     root@localhost:~# kubectl cp cpx-ingress-596d56bb6-zbx6h:cpx/log/boot.log /tmp/cpx-boot.log -c cpx-ingress
     root@localhost:~# ll /tmp/cpx-boot.log
     -rw-r--r-- 1 root root 7880 Sep 11 00:07 /tmp/cpx-boot.log
    

    scp: You can use the command to copy files between the Citrix ADC CPX pod and the Kubernetes node. Run the following command to copy files from pod to node. When it prompts for the password, provide the password for the SSH user:

     scp <user>@<pod-ip>:<absolute-src-path> <dst-path>
    

    Example:

     root@localhost:~# scp nsroot@192.168.29.198:/cpx/log/boot.log /tmp/cpx-boot.log
     nsroot@192.168.29.198's password:
     boot.log
     100% 7880     5.1MB/s   00:00
     root@localhost:~#
    
  • How do I capture packets on Citrix ADC CPX?

    To capture packets on Citrix ADC CPX, launch the shell interface of Citrix ADC CPX using the kubectl exec command. Run the following command to launch the shell interface of the Citrix ADC CPX pod:

         kubectl exec -it pod-name [-c container-name] [-n namespace-name] bash
    

    Example:

         kubectl exec -it cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc bash
    

    And, run the following command to begin packet capture:

         cli_script.sh “start nstrace -size 0”
    

    If you want to stop the ongoing packet capture, run the following command:

         cli_script.sh “stop nstrace”
    

    You can view the packets captured in a .cap file in the /cpx/nstrace/time-stamp directory on the Citrix ADC CPX file system.

  • Why is the license server not configured even when Citrix ADC CPX is deployed with the LS_IP=<ADM-IP> environment variable?

    Ensure that the license server is accessible from the node on which Citrix ADC CPX is deployed. You can use the ping <ADM-IP> command to verify the connectivity from the Citrix ADC CPX node to Citrix ADM.

    If Citrix ADM is accessible from the node, then you must verify the license server configuration logs in the /cpx/log/boot.log file. You can also check for license server configuration using the following command on the shell interface of the Citrix ADC CPX pod:

     cli_script.sh "show licenseserver"
    

    Example:

     root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserver"
     exec: show licenseserver
     ServerName:  10.106.102.199Port:  27000 Status:  1   Grace:  0  Gptimeleft:  720
     Done
    
  • Why is pooled license not configured on Citrix ADC CPX even after a successful license server configuration on Citrix ADC CPX?

    Verify the license configuration logs in the /cpx/log/boot.log file. You can also verify the configured pooled license on Citrix ADC CPX using the following command on the shell interface of the Citrix ADC CPX pod:

     cli_script.sh “show capacity”
    

    Example,

     root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show capacity"
     exec: show capacity
     Actualbandwidth:  1000  MaxVcpuCount:  2  Edition:  Platinum      Unit:  Mbps   Bandwidth:  0`  `Maxbandwidth:  40000        Minbandwidth:  20       Instancecount:  1
     Done
    

    Also, ensure that the required license files are uploaded in the license server. You can also verify the available licenses on the license server once it is successfully configured on Citrix ADC CPX by using the following command. Run the command on the shell interface of Citrix ADC CPX pod:

     cli_script.sh “sh licenseserverpool”
    

    Example:

     root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserverpool"
     exec: show licenseserverpool
         Instance Total                     : 5
         Instance Available                 : 4
         Standard Bandwidth Total           : 0 Mbps
         Standard Bandwidth Availabe        : 0 Mbps
         Enterprise Bandwidth Total         : 0 Mbps
         Enterprise Bandwidth Available     : 0 Mbps
         Platinum Bandwidth Total           : 10.00 Gbps
         Platinum Bandwidth Available       : 9.99 Gbps
         CP1000 Instance Total              : 100
         CP1000 Instance Available          : 100
       Done
     <!--NeedCopy-->
    
  • Why do NITRO API calls get Connection Refused response from Citrix ADC CPX?

    The default port for NITRO APIs is 9080 (unsecure) and 9443 (secure) from the Citrix ADC CPX release 12.1 onwards. Ensure that the NITRO port of Citrix ADC CPX you try to access is exposed on the pod. You can run the kubectl describe command to view the exposed and mapped port of the Citrix ADC CPX container in the Citrix ADC CPX container section:

     kubectl describe pods <pod-name> | grep -i port
    

    Example:

          ng472 | grep -i port
             Ports:          80/TCP, 443/TCP, 9080/TCP, 9443/TCP
             Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
               NS_HTTP_PORT:            9080
               NS_HTTPS_PORT:           9443
             Port:          <none>
             Host Port:     <none>
               NS_PORT:                 80
     <!--NeedCopy-->
    
  • Why does the NSPPE process in Citrix ADC CPX consume most of the CPU usage even when there is no or little traffic?

    If Citrix ADC CPX is deployed with the CPX_CONFIG=’{“YIELD”:”NO”}' environment variable, the NSPPE process consumes 100 percent CPU usage even when there is no or little traffic. If you want the NSPPE process not to consume the CPU usage, you must deploy Citrix ADC CPX without the CPX_CONFIG=’{“YIELD”:”NO”} environment variable. By default, the NSPPE process in CPX is configured not to hog or consume the CPU usage.

  • Why is Citrix ADC CPX not listed in Citrix ADM even when it was deployed with the required environment variables for registration with Citrix ADM?

    You can find the logs for Citrix ADC CPX registration with Citrix ADM in the /cpx/log/boot.log file on the Citrix ADC CPX file system.

    You can verify the accessibility of the Citrix ADM IP address from the Citrix ADC CPX pod using the ping command. Also, ensure that all the required environment variables for Citrix ADM registration are configured for the Citrix ADC CPX container.

    • NS_MGMT_SERVER: Specifies the ADM-IP address or FQDN.
    • HOST: Specifies the node IP address.
    • NS_HTTP_PORT: Specifies the mapped HTTP- port on node.
    • NS_HTTPS_PORT: Specifies the mapped HTTPS port on node.
    • NS_SSH_PORT: Specifies the mapped SSH port on node.
    • NS_SNMP_PORT: Specifies the mapped SNMP port on node.
    • NS_ROUTABLE: Citrix ADC CPX pod IP address is not routable from outside.
    • NS_MGMT_USER: Specifies the ADM username.
    • NS_MGMT_PASS: Specifies the ADM password.
  • Why does cli_script.sh show Invalid user name or password error message after changing the password for nsroot user?

    The command cli_script.sh is a wrapper utility for NSCLI on Citrix ADC CPX. It runs the first argument as command string or file path and the second argument is optional which is credentials. If the password for the nsroot user is changed, you need to provide credentials as the second argument to cli_script.sh. You can run the following command to run NSCLI with credentials:

     cli_script.sh “<command>” “:<username>:<password>”
    

    Example:

     root@087a1e34642d:/# cli_script.sh "show ns ip"
     exec: show ns ip
    
     ERROR: Invalid username or password
    
     root@087a1e34642d:/# cli_script.sh "show ns ip" ":nsroot:nsroot123"
    
     exec: show ns ip
    
     Ipaddress        Traffic Domain         Type             Mode     Arp     Icmp     Vserver  State        
     --------------   ----                   ----             ---     ----     -------  ------   -----
     172.17.0.3       0                   NetScaler IP     Active   Enabled  Enabled  NA       Enabled
     192.0.0.1        0                   SNIP             Active   Enabled  Enabled  NA       Enabled
     Done
    
  • Why does SSH to Citrix ADC CPX fail with root and nsroot user?

    From 13.0-64.35 release onwards, Citrix ADC CPX generates a default password and updates it for SSH users - root and nsroot. If you have not changed the password manually, password for SSH users can be found in /var/deviceinfo/random_id on Citrix ADC CPX’s file-system.

Citrix ADC CPX troubleshooting

In this article