Citrix ADC CPX

Citrix ADC CPX troubleshooting

October 1, 2020

Contributed by:

This document explains how to troubleshoot issues that you may encounter while using Citrix ADC CPX. Using this document, you can collect logs to determine the causes and apply workarounds for some of the common issues related to the installation and configuration of Citrix ADC CPX.

How can I view Citrix ADC CPX logs?

You can view Citrix ADC CPX logs using the kubectl logs command if Citrix ADC CPX is deployed with the tty:true option. You can run the following command to display the logs:
```
 kubectl logs <pod-name> [-c <container-name>] [-n <namespace-name>]
```
Example,
```
 kubectl logs cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc
```
The following is an example of the Citrix ADC CPX pod deployment with the tty:true option:

      containers:
        - name: cpx-ingress
          image: "quay.io/citrix/citrix-k8s-cpx-ingress:13.0-58.30"
          tty: true
          securityContext:
             privileged: true
          env:

You can find more boot logs in the */cpx/log/boot.log* file of Citrix ADC CPX file system.

**Note**: To get the pod name, run the `kubectl get pods -o wide` command.

How can I collect the technical support bundle from Citrix ADC CPX?

You can run the following command on the shell interface of the Kubernetes master node to collect the Citrix ADC CPX technical support bundle:

 kubectl exec <cpx-pod-name> [-c <cpx-container-name>] [-n <namespace-name>] /var/netscaler/bins/cli_script.sh "show techsupport"

You can view the technical support bundle in the /var/tmp/support directory of the Citrix ADC CPX’s file system. Use scp or kubectl cp to copy the technical support bundle from Citrix ADC CPX to the desired destination.

Example:

 root@localhost# kubectl exec cpx-ingress1-55b9b6fc75-t5kc6 -c cpx -n citrix-adc /var/netscaler/bins/cli_script.sh "show techsupport"
 exec: show techsupport
   Scope:  NODE
 Done
 root@localhost# kubectl cp cpx-ingress1-55b9b6fc75-t5kc6:var/tmp/support/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz -c cpx
 root@localhost# ll /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz
 -rw-r--r-- 1 root root 1648109 Aug 31 13:23 /tmp/collector_P_192.168.29.232_31Aug2020_07_30.tar.gz

Why is Citrix ADC CPX pod stuck while booting?

You can check the pod status using the kubectl describe pods command. Run the following command to know the pod status:
```
 kubectl describe pods <pod-name> [-c <container-name>] [-n <namespace-name>]
```
Example:
```
 kubectl describe pods cpx-ingress1-69b9b8c648-t8bgn
```
If the pod events show that container is started, then you must check the pod logs.
How do I copy files between the Citrix ADC CPX pod and the Kubernetes master node?

It is recommended to use the volume mount feature of docker to mount the /cpx directory to the file system of the host. If a Citrix ADC CPX container exits core-dumps, logs and other important data are available on the mount point.

You can use any one of the following commands to copy files between the Citrix ADC CPX pod and the Kubernetes master node:

kubectl cp: You can run the following command to copy files from pod to node:
```
 kubectl cp <pod-name>:<absolute-src-path> <dst-path> [-c <container-name>] [-n <namespace-name>]
```
Example:
```
 root@localhost:~# kubectl cp cpx-ingress-596d56bb6-zbx6h:cpx/log/boot.log /tmp/cpx-boot.log -c cpx-ingress
 root@localhost:~# ll /tmp/cpx-boot.log
 -rw-r--r-- 1 root root 7880 Sep 11 00:07 /tmp/cpx-boot.log
```
scp: You can use the command to copy files between the Citrix ADC CPX pod and the Kubernetes node. Run the following command to copy files from pod to node. When it prompts for the password, provide the password for the SSH user:
```
 scp <user>@<pod-ip>:<absolute-src-path> <dst-path>
```
Example:
```
 root@localhost:~# scp nsroot@192.168.29.198:/cpx/log/boot.log /tmp/cpx-boot.log
 nsroot@192.168.29.198's password:
 boot.log
 100% 7880     5.1MB/s   00:00
 root@localhost:~#
```
How do I capture packets on Citrix ADC CPX?

To capture packets on Citrix ADC CPX, launch the shell interface of Citrix ADC CPX using the kubectl exec command. Run the following command to launch the shell interface of the Citrix ADC CPX pod:
```
     kubectl exec -it pod-name [-c container-name] [-n namespace-name] bash
```
Example:
```
     kubectl exec -it cpx-ingress1-69b9b8c648-t8bgn -c cpx -n citrix-adc bash
```
And, run the following command to begin packet capture:
```
     cli_script.sh “start nstrace -size 0”
```
If you want to stop the ongoing packet capture, run the following command:
```
     cli_script.sh “stop nstrace”
```
You can view the packets captured in a .cap file in the /cpx/nstrace/time-stamp directory on the Citrix ADC CPX file system.
Why is the license server not configured even when Citrix ADC CPX is deployed with the LS_IP=<ADM-IP> environment variable?

Ensure that the license server is accessible from the node on which Citrix ADC CPX is deployed. You can use the ping <ADM-IP> command to verify the connectivity from the Citrix ADC CPX node to Citrix ADM.

If Citrix ADM is accessible from the node, then you must verify the license server configuration logs in the /cpx/log/boot.log file. You can also check for license server configuration using the following command on the shell interface of the Citrix ADC CPX pod:
```
 cli_script.sh "show licenseserver"
```
Example:
```
 root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserver"
 exec: show licenseserver
 ServerName:  10.106.102.199Port:  27000 Status:  1   Grace:  0  Gptimeleft:  720
 Done
```
Why is pooled license not configured on Citrix ADC CPX even after a successful license server configuration on Citrix ADC CPX?

Verify the license configuration logs in the /cpx/log/boot.log file. You can also verify the configured pooled license on Citrix ADC CPX using the following command on the shell interface of the Citrix ADC CPX pod:
```
 cli_script.sh “show capacity”
```
Example,
```
 root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show capacity"
 exec: show capacity
 Actualbandwidth:  1000  MaxVcpuCount:  2  Edition:  Platinum      Unit:  Mbps   Bandwidth:  0`  `Maxbandwidth:  40000        Minbandwidth:  20       Instancecount:  1
 Done
```
Also, ensure that the required license files are uploaded in the license server. You can also verify the available licenses on the license server once it is successfully configured on Citrix ADC CPX by using the following command. Run the command on the shell interface of Citrix ADC CPX pod:
```
 cli_script.sh “sh licenseserverpool”
```
Example:

    root@cpx-ingress-596d56bb6-zbx6h:/cpx/log# cli_script.sh "show licenseserverpool"
exec: show licenseserverpool
        Instance Total                     : 5
        Instance Available                 : 4
        Standard Bandwidth Total           : 0 Mbps
        Standard Bandwidth Availabe        : 0 Mbps
        Enterprise Bandwidth Total         : 0 Mbps
        Enterprise Bandwidth Available     : 0 Mbps
        Platinum Bandwidth Total           : 10.00 Gbps
        Platinum Bandwidth Available       : 9.99 Gbps
        CP1000 Instance Total              : 100
        CP1000 Instance Available          : 100
      Done

Why do NITRO API calls get Connection Refused response from Citrix ADC CPX?

The default port for NITRO APIs is 9080 (unsecure) and 9443 (secure) from the Citrix ADC CPX release 12.1 onwards. Ensure that the NITRO port of Citrix ADC CPX you try to access is exposed on the pod. You can run the kubectl describe command to view the exposed and mapped port of the Citrix ADC CPX container in the Citrix ADC CPX container section:
```
 kubectl describe pods <pod-name> | grep -i port
```
Example:

         ng472 | grep -i port
            Ports:          80/TCP, 443/TCP, 9080/TCP, 9443/TCP
            Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
              NS_HTTP_PORT:            9080
              NS_HTTPS_PORT:           9443
            Port:          <none>
            Host Port:     <none>
              NS_PORT:                 80

Why does the NSPPE process in Citrix ADC CPX consume most of the CPU usage even when there is no or little traffic?

If Citrix ADC CPX is deployed with the CPX_CONFIG=’{“YIELD”:”NO”}' environment variable, the NSPPE process consumes 100 percent CPU usage even when there is no or little traffic. If you want the NSPPE process not to consume the CPU usage, you must deploy Citrix ADC CPX without the CPX_CONFIG=’{“YIELD”:”NO”} environment variable. By default, the NSPPE process in CPX is configured not to hog or consume the CPU usage.
Why is Citrix ADC CPX not listed in Citrix ADM even when it was deployed with the required environment variables for registration with Citrix ADM?

You can find the logs for Citrix ADC CPX registration with Citrix ADM in the /cpx/log/boot.log file on the Citrix ADC CPX file system.

You can verify the accessibility of the Citrix ADM IP address from the Citrix ADC CPX pod using the ping command. Also, ensure that all the required environment variables for Citrix ADM registration are configured for the Citrix ADC CPX container.
- NS_MGMT_SERVER=<ADM-IP>
- NS_MGMT_FINGER_PRINT=<ADM-Fingerprint>
- HOST=<Node-IP>
- NS_HTTP_PORT=<Mapped-HTTP-Port-on-Node>
- NS_HTTPS_PORT=<Mapped-HTTPS-Port-on-Node>
- NS_SSH_PORT=<Mapped-SSH-Port-on-Node>
- NS_SNMP_PORT=<Mapped-SNMP-Port-on-Node>
- NS_ROUTABLE=0 (Citrix ADC CPX pod IP address is not routable from outside.)
Why does cli_script.sh show Invalid user name or password error message after changing the password for nsroot user?

The command cli_script.sh is a wrapper utility for NSCLI on Citrix ADC CPX. It runs the first argument as command string or file path and the second argument is optional which is credentials. If the password for the nsroot user is changed, you need to provide credentials as the second argument to cli_script.sh. You can run the following command to run NSCLI with credentials:
```
 cli_script.sh “<command>” “:<username>:<password>”
```

Example:

    root@087a1e34642d:/# cli_script.sh "show ns ip"
    exec: show ns ip

    ERROR: Invalid username or password

    root@087a1e34642d:/# cli_script.sh "show ns ip" ":nsroot:nsroot123"

    exec: show ns ip

    Ipaddress        Traffic Domain         Type             Mode     Arp     Icmp     Vserver  State        
    --------------   ----                   ----             ---     ----     -------  ------   -----
    172.17.0.3       0                   NetScaler IP     Active   Enabled  Enabled  NA       Enabled
    192.0.0.1        0                   SNIP             Active   Enabled  Enabled  NA       Enabled
    Done

The official version of this content is in English. Some of the Citrix documentation content is machine translated for your convenience only. Citrix has no control over machine-translated content, which may contain errors, inaccuracies or unsuitable language. No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, suitability, or correctness of any translations made from the English original into any other language, or that your Citrix product or service conforms to any machine translated content, and any warranty provided under the applicable end user license agreement or terms of service, or any other agreement with Citrix, that the product or service conforms with any documentation shall not apply to the extent that such documentation has been machine translated. Citrix will not be held responsible for any damage or issues that may arise from using machine-translated content.

Citrix ADC CPX troubleshooting

October 1, 2020

Contributed by:

October 1, 2020

Contributed by:

Citrix ADC CPX troubleshooting

In this article