Product Documentation

How to record a packet trace on Citrix ADC

This troubleshooting article explains how an administrator can record a network packet trace using the Citrix ADC GUI.

Points to remember

  • Citrix recommends you to use the recent Wireshark version from the “automated build section” available in the following web page: http://www.wireshark.org/download/automated.

  • In Citrix ADC software release 10.5 and later, to decrypt the capture, ensure that ECC (Elliptic Curve Cryptography), Session Reuse and DH Param are disabled/removed from the virtual server before the trace is captured. Refer to the Additional Resources section of this article before making these changes

Record packet trace on NetScaler 11.1 appliance

To record a packet trace on a NetScaler 11.1 appliance, you must complete the following procedure.

  1. Sign into NetScaler GUI and navigate to System > Diagnostics page.
  2. click the Start new trace link in the Diagnostic page, as shown in the following screenshot.

    Accessing Diagnostic page

  3. Update the packet size to 0 in the Packet size field.

    Packet size

  4. Click Start to start recording the network packet trace.
  5. Click Stop and Download to stop recording the network packet trace after the test is complete.

    Stop and Download trace

  6. Select the required file and click Select and click Download.

    Download packet trace

  7. Open the network packet trace file with the Wireshark utility to display the content of the file.

Record packet trace on NetScaler 10.5 appliance

To record a packet trace on a NetScaler 10.5 appliance, you must complete the following procedure:

  1. Sign into NetScaler GUI and navigate to System > Diagnostics page.

    Access Diagnostic page

  2. Click the Start new trace link under Technical Support Tools as shown in the following screen shot.
  3. Update the packet size to 0 in the Packet Size field.

    Packet size

    Note: If NetScaler headers are not required then select Capture trace in .pcap format.

  4. Click Start to start recording the network packet trace.
  5. Click OK to stop recording the network packet trace after the test is complete.

    Stop trace recording

    An nstrace.cap file is generated, which contains the network packet trace.

  6. Highlight the required file and click Download.

    Download the file

  7. Specify a destination and save the packet trace.
  8. Open the network packet trace file with the Wireshark utility to display the content of the file.

    Note: Select Decrypted SSL packets (SSLPLAIN) to decrypt the packet trace without the private key.

    Decrypted SSL packets

Capture SSL master keys

In the latest 11.0 and 11.1 version and above there is an option to capture the session keys which will be valid for only for that particular session/nstrace and this option can be used if you donot want to share the private key or use SSLPLAIN mode. For more information please refer to How to use the new “capsslkeys” option when trying to capture nstrace on Netscaler.

Export Session Keys without sharing Private key

In most of the scenarios the private key is not available or shared. In such scenarios we can suggest to export the SSL session keys instead of the private key. Please refer to the below article for more information. How to Export and Use SSL Session Keys to Decrypt SSL Traces Without Sharing the SSL Private Key.

Filters

Additionally, it is always recommended to add IP based filters while taking traces. This will ensure that you will capture only interested traffic which will further ease your troubleshooting. Adding filters will also decrease the load on NetScaler while taking traces.

You will find the option to configure filters on the same page:

screenshot

Simple IP based filters are enough to get the right captures. For a detailed list of filters and examples, refer to Citrix Documentation - nstrace.

Use case to capture a packet trace with virtual server IP filter (both frontend and backend)

With a filter of the virtual server IP address and enabling the option “–link” in CLI or select the option “Trace filtered connection peer traffic” in GUI (available 10.1 and above), you will be able to capture both the front end and backend traffic for that particular IP address. With this option it is not recommended to mention a source IP or destination IP filter.

start nstrace -size 0 -filter "CONNECTION.IP.EQ(1.1.1.1)" -link ENABLED

show nstrace
        State:  RUNNING          Scope:  LOCAL            TraceLocation:  "/var/nstrace/24Mar2017_16_00_19/..." Nf:  24                  Time:  3600              Size:  0                 Mode:  TXB NEW_RX
        Traceformat:  NSCAP      PerNIC:  DISABLED        FileName:  24Mar2017_16_00_19 Filter:  "CONNECTION.IP.EQ(1.1.1.1)" Link:  ENABLED           Merge:  ONSTOP           Doruntimecleanup:  ENABLED
        TraceBuffers:  5000      SkipRPC:  DISABLED       Capsslkeys:  DISABLED    InMemoryTrace:  DISABLED

Merge

Capturing cyclic traces

It is always challenging to troubleshoot an intermittent issue. Cyclic tracing is best suited for issues which are intermittent. These traces can be run over a span of few hours or days based on the occurance of the issue. Also a specific filter can be used as mentioned above. Please evaluate the size of the trace files that are being generated before running it for a longer time (as this can fill up the var space)

Run the following command from CLI:

start nstrace -nf 60 -time 30 -size 0 
This particular trace will create 60 files each of them for 30 sec. This means the files will start getting overwritten after 60 trace files or 30 mins
Show nstrace à To check the status of the nstrace
Stop nstrace à To stop the nstrace.

Best Practices

On a unit handling gigabytes of traffic per second, capturing traffic is a very resource intensive process. The impact to resources is mainly in terms of CPU and Disk Space. Disk Space impact can be reduced by using filtering expressions (capturing traffic only related to a particular IP). However the impact on CPU remains despite using expressions and in some cases might cause a slight further increase as NetScaler now needs to process packets according to the filter before capturing them.

The best practice with regards to tracing are:

  1. The duration for which the trace is run should be as limited as possible while still ensuring the packets of interest are captured.
  2. Schedule the tracing activity to happen at a time when the number of users (and hence the traffic) is greatly reduced, such as during off hours.

Additional resources

Disable session reuse on virtual server from NetScaler GUI

Session reuse is disabled while capturing a trace in order to capture a complete SSL handshake in the trace. When it is enabled it is likely that you will only capture a partial handshake in the trace. Ensure to enable the option after trace collection. Do not disable SSL session reuse when the persistence method is sslsession, as it will break the persistence for existing connections. For more information refer to CTX121925- SSL Renegotiation Process and Session Reuse on NetScaler appliance.

  1. Open the virtual server and navigate to SSL Parameters.
  2. Disable Enable Session Reuse if enabled.

    Enable session reuse

Disable session reuse on virtual server from NetScaler CLI

  1. SSH to the NetScaler.
  2. Run the following command to disable DH Param from the virtual server:

    set ssl vserver "vServer_Name" -sessReuse DISABLED

Disable DH parameter on virtual server from NetScaler GUI

Refer to CTX213335 - How do I setup a Diffie-Hellman key on NetScaler? to understand about DH Parameter.

  1. Open the virtual server and navigate to SSL Parameters.
  2. Disable DH Param if enabled.

    SSL Parameters

Disable DH parameter on virtual server from NetScaler CLI

  1. SSH to the NetScaler.
  2. Run the following command to disable DH Param from the virtual server:

    set ssl vserver "vServer_Name" -dh DISABLED

Disable ECC curve on virtual server from NetScaler GUI

ECC curve are disabled in order to decrypt the captured SSL trace with private key. These keys should not be disabled if the related SSL ciphers are used. For more information on ECC curve refer to CTX20 - How Do I Setup ECC on NetScaler?

  1. Open the virtual server and navigate to ECC Curve.

    ECC Curve

  2. If no ECC Curve is bound to the virtual server then no other action is required.

    No CC Curve

  3. If any ECC Curve is bound to the virtual server then click the ECC Curve and Unbind it from the virtual server.

Disable ECC curve on virtual server from NetScaler CLI

  1. SSH to the NetScaler.
  2. Run the following command for each ECC Curve bound to the virtual server:

    unbind ssl vserver "vServer_Name" -eccCurveName "ECC_Curve_Name"