Citrix ADC

How to record a packet trace on Citrix ADC

This troubleshooting article explains how an administrator can record a network packet trace using the Citrix ADC GUI.

Points to remember

  • Citrix recommends you to use the recent Wireshark version from the “automated build section” available in the following webpage: http://www.wireshark.org/download/automated.

  • In Citrix ADC version 10.5 or later, to decrypt the capture and ensure ECC (Elliptic Curve Cryptography), Session Reuse and DH parameters are disabled from the virtual server before the trace is captured.

Record packet trace on NetScaler version 11.1

  1. Navigate to System > Diagnostics page.
  2. click the Start new trace link in the Diagnostic page, as shown in the following screenshot.

    Accessing Diagnostic page

  3. Update the packet size to 0 in the Packet size field.

    Packet size

  4. Click Start to start recording the network packet trace.
  5. Click Stop and Download to stop recording the network packet trace after the test is complete.

    Stop and Download trace

  6. Select the required file and click Select and click Download.

    Download packet trace

  7. Open the network packet trace file with the Wireshark utility to display the content of the file.

Record packet trace on NetScaler 10.5 appliance

  1. Navigate to System > Diagnostics page.

    Access diagnostic page

  2. Click the Start new trace link under Technical Support Tools as shown in the following screenshot.
  3. Update the packet size to 0 in the Packet Size field.

    Packet size

    Note: If appliance headers are not required then select Capture trace in .pcap format.

  4. Click Start to start recording the network packet trace.
  5. Click OK to stop recording the network packet trace after the test is complete.

    Stop trace recording

    An nstrace.cap file is generated, which contains the network packet trace.

  6. Highlight the required file and click Download.

    Download the file

  7. Specify a destination and save the packet trace.
  8. Open the network packet trace file with the Wireshark utility to display the content of the file.

    Note: Select Decrypted SSL packets (SSLPLAIN) to decrypt the packet trace without the private key.

    Decrypted SSL packets

Capture SSL master keys

In the 11.0, 11.1 version and above there is an option to capture the session keys which is valid for only for that particular session/nstrace and this option can be used if you do not want to share the private key or use SSLPLAIN mode. For more information, see https://support.citrix.com/article/CTX135889.

Export Session Keys without sharing Private key

In most of the scenarios the private key is not available or shared. In such scenarios we can suggest exporting the SSL session keys instead of the private key. Read, [How to Export and Use SSL Session Keys to Decrypt SSL Traces Without Sharing the SSL Private Key, see https://support.citrix.com/article/CTX135889.

Filters

Also, it is always recommended to add IP based filters while taking traces. The process ensures that you capture only interested traffic which eases your troubleshooting. Adding filters will also decrease the load on the appliance while taking traces.

Filter section

Simple IP-based filters are enough to get the right captures. For more information about nstrace filters and examples, see Citrix Documentation page.

Use case to capture a packet trace with virtual server IP filter (both front-end and back end)

Using a filter of the virtual server IP address and enabling the option “–link” in CLI or select the option “Trace filtered connection peer traffic” in GUI (available 10.1 and above), you can capture both the front end and back-end traffic for that particular IP address. With this option it is not recommended to mention a source IP or destination IP filter.

start nstrace -size 0 -filter "CONNECTION.IP.EQ(1.1.1.1)" -link ENABLED

show nstrace
        State:  RUNNING          Scope:  LOCAL            TraceLocation:  "/var/nstrace/24Mar2017_16_00_19/..." Nf:  24                  Time:  3600              Size:  0                 Mode:  TXB NEW_RX
        Traceformat:  NSCAP      PerNIC:  DISABLED        FileName:  24Mar2017_16_00_19 Filter:  "CONNECTION.IP.EQ(1.1.1.1)" Link:  ENABLED           Merge:  ONSTOP           Doruntimecleanup:  ENABLED
        TraceBuffers:  5000      SkipRPC:  DISABLED       Capsslkeys:  DISABLED    InMemoryTrace:  DISABLED

Merge

Capturing cyclic traces

It is always challenging to troubleshoot an intermittent issue. Cyclic tracing is best suited for issues which are intermittent. The traces can be run over a span of few hours or days before the issue occurs. Also, you can use a specific filter and evaluate the size of the trace files that are generated before you run it for a longer time.

Run the following command from the CLI:

start nstrace -nf 60 -time 30 -size 0
This particular trace will create 60 files each of them for 30 sec. This means the files will start getting overwritten after 60 trace files or 30 mins
Show nstrace à To check the status of the nstrace
Stop nstrace à To stop the nstrace.

Best Practices

On a unit handling GB of traffic per second, capturing traffic is a very resource intensive process. The impact to resources is mainly in terms of the CPU and the disk space. Disk space impact can be reduced by using filtering expressions. However, the impact on the CPU remains and sometimes causes a slight increase as the appliance now needs to process packets according to the filter before capturing them.

The best practice with regards to tracing is:

  1. The duration for which the trace is run must be as limited as possible when you still ensure the packets of interest are captured.
  2. Schedule the tracing activity to happen at a time when the number of users (and hence the traffic) is greatly reduced, such as during off hours.

More resources

Disable session reuse on virtual server from the GUI

Session reuse is disabled when you capture a trace to complete an SSL handshake in the trace. When it is enabled, you can capture a partial handshake in the trace. Ensure you enable the option after the trace collection. Do not disable an SSL session reuse when the persistence method is sslsession, as it breaks the persistence for existing connections. For more information refer to https://support.citrix.com/article/CTX121925.

  1. Open the virtual server and navigate to SSL Parameters.
  2. Disable Enable Session Reuse if enabled.

    Enable session reuse

Disable session reuse on virtual server from the CLI

  1. SSH to the appliance console.
  2. Run the following command to disable DH Param from the virtual server:

    set ssl vserver "vServer_Name" -sessReuse DISABLED

Disable DH parameter on virtual server from the GUI

Refer to https://support.citrix.com/article/CTX213335 To understand about DH Parameter.

  1. Open the virtual server and navigate to SSL Parameters.
  2. Disable DH Param if enabled.

    SSL Parameters

Disable DH parameter on virtual server from the CLI

  1. SSH to the appliance console.
  2. Run the following command to disable DH Param from the virtual server:

    set ssl vserver "vServer_Name" -dh DISABLED

Disable ECC curve on virtual server from the GUI

ECC curve is disabled to decrypt the captured SSL trace with private key. You must not disable the keys if the related SSL ciphers are used. For more information about ECC curve, see https://support.citrix.com/article/CTX205289

  1. Open the virtual server and navigate to ECC Curve.

    ECC Curve

  2. If there is no ECC Curve bound to the virtual server then no other action is required.

    No CC Curve

  3. If any ECC Curve is bound to the virtual server then click the ECC Curve and Unbind it from the virtual server.

Disable ECC curve on virtual server from the CLI

  1. SSH to the appliance console.
  2. Run the following command for each ECC Curve bound to the virtual server:

    unbind ssl vserver "vServer_Name" -eccCurveName "ECC_Curve_Name"

How to record a packet trace on Citrix ADC