Application Delivery Management

Troubleshoot HDX Insight issues

If the HDX Insight solution is not functioning as expected, the issue might be with one of the following. Refer to the checklists in the respective sections for troubleshooting.

  • HDX Insight configuration.

  • Connectivity between NetScaler and NetScaler Console.

  • Record generation for HDX/ICA traffic in NetScaler.

  • Population of records in NetScaler Console.

HDX Insight configuration checklist

  • Make sure that the AppFlow feature is enabled in NetScaler. For details, see Enabling AppFlow.

  • Check HDX Insight configuration in the NetScaler running configuration.

    Run the show running | grep -i <appflow_policy> command to check the HDX Insight configuration. Make sure that the bind type is ICA REQUEST. For example;

    bind vpn vserver afsanity -policy afp -priority 100 -type ICA_REQUEST

    For transparent mode, the bind type must be ICA_REQ_DEFAULT. For example;

    bind appflow global afp 100 END -type ICA_REQ_DEFAULT

  • For single-hop/Access Gateway or double-hop deployment, make sure that HDX Insight AppFlow policy is bound to the VPN virtual server, where HDX/ICA traffic is flowing.

  • For Transparent mode or LAN user mode make sure the ICA ports 1494 and 2598 are set.

  • Check appflowlog parameter in NetScaler Gateway or VPN virtual server is enabled for Access Gateway or double-hop deployment. For details, see Enabling AppFlow for Virtual Servers.

  • Check “Connection Chaining” is enabled in double-hop NetScaler. For details see, Configuring NetScaler Gateway appliances to export data.

  • After HA Failover if the HDX Insight details are Skip parsed, check ICA param “enableSRonHAFailover” is enabled. For details, see Session Reliability on NetScaler High Availability Pair.

Connectivity between NetScaler and NetScaler Console checklist

  • Check AppFlow collector status in NetScaler. For details, see How to check the status of connectivity between NetScaler and AppFlow Collector.

  • Check HDX Insight AppFlow policy hits.

    Run the command show appflow policy <policy_name> to check the AppFlow policy hits.

    You can also navigate to Settings > AppFlow > Policies in the GUI to check the AppFlow policy hits.

  • Validate any firewall blocking AppFlow ports 4739 or 5557.

Record generation for HDX/ICA traffic in NetScaler checklist

Run the command tail -f /var/log/ns.log | grep -i "default ICA Message" for log validation. Based on the logs that are generated, you can use this information for troubleshooting.

  • Log: Skipped parsing ICA connection - HDX Insight not supported for this host

    Cause: Unsupported Citrix Virtual Apps and Desktops versions

    Workaround: Upgrade the Citrix Virtual Apps and Desktops servers to a supported version.

  • Log: Client type received 0x53, NOT SUPPORTED

    Cause: Unsupported version of Citrix Workspace

    Solution: Upgrade Citrix Workspace to a supported version. For details, see Citrix Workspace app.

  • Log: Error from Expand Packet - Skipping all hdx processing for this flow

    Cause: Issue with uncompressing ICA traffic

    Solution: No reports are available for this ICA session until a new session is established.

  • Log: Invalid transition: NS_ICA_ST_FLOW_INIT/NS_ICA_EVT_INVALID -> NS_ICA_ST_UNINIT”

    Cause: Issue with parsing the ICA handshake

    Solution: No reports are available for this particular ICA session until a new session is established.

  • Log: Missing EUEM ICA RTT

    Cause: Unable to parse End-User Experience Monitoring channel data

    Solution: Make sure End-User Experience Monitoring service in started on the Citrix Virtual Apps and Desktops servers. Make sure you are using the supported versions of Citrix Workspace App.

  • Log: Invalid Channel Header

    Cause: Unable to identify channel header

    Solution: No reports are available for this particular ICA session until a new session is established.

  • Log: Skip code

    If you see any of the following values for skip code, then the Insight details are skip parsed.

    Skip code 0 indicates that the record is successfully exported from NetScaler.

Skip Code Error message Cause of error
100 NS_ICA_ERR_NULL_FRAG Error handling ICA fragments, likely due to memory conditions
101 NS_ICA_ERR_INVALID_HS_CMD Invalid handshake command received
102 NS_ICA_ERR_REDUC_PARAM_CNT Invalid parameter specified for V3 expander initialization
103 NS_ICA_ERR_REDUC_INIT Unable to initialize the V3 expander correctly
104 NS_ICA_ERR_REDUC_PARAM_BYTES Insufficient bytes to assign a coder to a channel
105 NS_ICA_ERR_INVALID_CHANNEL Invalid ICA channel number
106 NS_ICA_ERR_INVALID_DECODER Invalid decoder specified for a channel
107 NS_ICA_ERR_INVALID_TW_PARAM Invalid parameter count specified on Thinwire channel
108 NS_ICA_ERR_INVALID_TW_DECODER Invalid decoder for Thinwire channel
109 NS_ICA_ERR_REDUC_NO_DECODER No decoder defined for channel
110 NS_ICA_ERR_REDUC_V3_EXPANDER Failed to expand channel data
111 NS_ICA_ERR_REDUC_BYTES_V3_OOR Expander error: Bytes consumed more than bytes available
112 NS_ICA_ERR_REDUC_BYTES_OOR Error: Uncompressed data overrun
113 NS_ICA_ERR_REDUC_INVALID_CMD Undefined Expander command
114 NS_ICA_ERR_CGP_FILL_HOLE Error while handling split CGP frames
115 NS_ICA_ERR_MEM_NSB_ALLOC NSB allocation error – due to low memory conditions
116 NS_ICA_ERR_MEM_REDUC_CTX_ALLOC Memory allocation error for expander context
117 NS_ICA_ERR_ICA_OLD_SERVER Old server, capability blocks not supported
118 NS_ICA_ERR_PIR_MANY_FRAG Packet Init request is fragmented, unable to process
119 NS_ICA_ERR_INIT_ICA_CAPS ICA capability initialization error
120 NS_ICA_ERR_NO_MSI_SUPPORT Host does not support MSI feature. Indicates for XenApp version lower than 6.5 or XenDesktop versions lower than 5.0
121 NS_ICA_ERR_CGP_INVALID_CMD Invalid CGP command encountered
122 NS_ICA_ERR_INSUFFICENT_CHANNEL_BYTES Insufficient bytes over channel
123 NS_ICA_ERR_CHANNEL_DATA Incorrect data on EUEM, CONTROL, or SEAMLESS channel
124 NS_ICA_ERR_INVALID_PURE_CMD Invalid command received while processing pure ICA channel data
125 NS_ICA_ERR_INVALID_PURE_LEN0 Invalid length encountered while processing pure ICA channel data
126 NS_ICA_ERR_INVALID_PURE_LEN Invalid length encountered while processing PURE ICA channel data
127 NS_ICA_ERR_INVALID_CLNT_DATA Invalid data length received from client
128 NS_ICA_ERR_MSI_GUID_SZ Error in MSI GUID size
129 NS_ICA_ERR_INVALID_CHANNEL_HEADER Detected invalid channel header
130 NS_ICA_ERR_CGP_PARSE_RECONNECT_ID Retrieval of reconnected session failed
131 NS_ICA_ERR_DISABLE_SR_NON_NS_RECONNECT Error in disabling SR
132 NS_ICA_ERR_REDUC_NOT_V3 Unsupported ICA Reducer version
133 NS_ICA_ERR_HS_COMPRESSION_DISABLED Compression disabled, not honored by host
134 NS_ICA_ERR_IDENT_PROTO Unable to identify ICA or CGP protocol, seen with incorrect workspaces
135 NS_ICA_ERR_INVALID_SIGNATURE Incorrect ICA signature or magic string
136 NS_ICA_ERR_PARSE_RAW Error while parsing the ICA handshake packet
137 NS_ICA_ERR_INCOMPLETE_PKT Incomplete packet received in handshake
138 NS_ICA_ERR_ICAFRAME_TOO_LARGE ICA frame is too large, exceeds 1460 bytes
139 NS_ICA_ERR_FORWARD Error while forwarding the ICA data
140 NS_ICA_ERR_MAX_HOLES Unable to process CGP command as it is split beyond supported limit
141 NS_ICA_ERR_ASSEMBLE_FRAME Unable to reassemble ICA frame correctly
142 NS_ICA_ERR_UNSUPPORTED_RECEIVER_VERSION Skipped ICA parsing for this workspace (client) as it is not in the allow list
143 NS_ICA_ERR_LOOKUP_RECONNECT_ID Unable to detect parsing state for client reconnect cookie
144 NS_ICA_ERR_SYNCUP_RECONNECT_ID Invalid reconnect cookie length detected post client reconnect
145 NS_ICA_ERR_INVALID_RECONNECT_ID Client reconnects cookie missed the needed constraint
146 NS_ICA_ERR_INVALID_CLIENT_VERSION Invalid workspace version string received from client
147 NS_ICA_ERR_UNKNOWN_CLIENT_PRODUCT_ID Invalid product ID received from client
148 NS_ICA_ERR_V3_HDR_CORRUPT_LEN Invalid channel length post expansion
149 NS_ICA_ERR_SPECIAL_THINWIRE Decompression error
150 NS_ICA_ERR_SEAMLESS_INSUFFBYTE Encountered insufficient bytes for seamless command
151 NS_ICA_ERR_EUEM_INSUFFBYTE Encountered insufficient bytes for EUEM command
152 NS_ICA_ERR_SEAMLESS_INVALID_EVENT Invalid event for seamless channel parsing
153 NS_ICA_ERR_CTRL_INVALID_EVENT Invalid event for CTRL channel parsing
154 NS_ICA_ERR_EUEM_INVALID_EVENT Invalid event for EUEM channel parsing
155 NS_ICA_ERR_USB_INVALID_EVENT Invalid event for USB channel parsing
156 NS_ICA_ERR_PURE_INVALID_EVENT Invalid event for pure channel parsing
157 NS_ICA_ERR_VCP_INVALID_EVENT Invalid event for virtual channel parsing
158 NS_ICA_ERR_ICAP_INVALID_EVENT Invalid event for ICA data parsing
159 NS_ICA_ERR_CGPP_INVALID_EVENT Invalid event for CGP data parsing
160 NS_ICA_ERR_BASICCRYPT_INVALIDSTATE Invalid state for a crypt command in basic encryption
161 NS_ICA_ERR_BASICCRYPT_INVALIDCRYPTCMD Invalid crypt command in basic encryption
162 NS_ICA_ERR_ADVCRYPT_INVALIDSTATE Invalid state for a crypt command in RC5 encryption
163 NS_ICA_ERR_ADVCRYPT_INVALIDCRYPTCMD Invalid crypt command in RC5 encryption
164 NS_ICA_ERR_ADVCRYPT_ENC Error in RC5 encryption/decryption
165 NS_ICA_ERR_ADVCRYPT_DEC Error in RC5 encryption/decryption
166 NS_ICA_ERR_SERVER_NOT_REDUCER_V3 VDA does not support Reducer Version 3
167 NS_ICA_ERR_CLIENT_NOT_REDUCER_V3 Workspace does not support Reducer Version 3
168 NS_ICA_ERR_ICAP_INSUFFBYTE Unexpected number of bytes in ICA handshake
169 NS_ICA_ERR_HIGHER_RECONSEQ Higher CGP resumption sequence number from peer post reconnects
170 NS_ICA_ERR_DESCSRINFO_ABSENT Unable to restore ICA parsing state post reconnect
171 NS_ICA_ERR_NSAP_PARSING Error while parsing Insight channel data
172 NS_ICA_ERR_NSAP_APP Error while parsing app details from Insight channel data
173 NS_ICA_ERR_NSAP_ACR Error while parsing ACR details from Insight channel data
174 NS_ICA_ERR_NSAP_SESSION_END Error while parsing session end details from Insight channel data
175 NS_ICA_ERR_NON_NSAP_SN Skipped ICA parsing on service node due to the absence of Insight channel support
176 NS_ICA_ERR_NON_NSAP_CLIENT NSAP is not supported by client
177 NS_ICA_ERR_NON_NSAP_SERVER NSAP is not supported by VDA
178 NS_ICA_ERR_NSAP_NEG_FAIL Error while NSAP data negotiation
179 NS_ICA_ERR_SN_RECONNECT_TKT_FETCH Error in fetching service reconnects ticket in service node
180 NS_ICA_ERR_SN_HIGHER_RECONSEQ Error when receiving higher reconnect sequence number in service node
181 NS_ICA_ERR_DISABLE_HDXINSIGHT_NONNSAP Error while disabling HDX Insight for non-NSAP connections

Sample logs:

Jan 9 22:57:02 <local0.notice> 10.106.40.223 01/09/2020:22:57:02 GMT ns-223 0-PPE-2 : default ICA Message 1234 0 : "Session setup data send: Session GUID [57af35043e624abab409f5e6af7fd22c], Client IP/Port [10.105.232.40/52314], Server IP/Port [10.106.40.215/2598], MSI Client Cookie [Non-MSI], Session setup time [01/09/2020:22:56:49 GMT], Client Type [0x0052], Receiver Version [19.12.0.23], User [user1], Client [10.105.232.40], Server [WIN2K12-215], Ctx Flags [0x8820220228], Track Flags [0x1775010c3fc], Skip Code [0]"

Jan 9 22:55:41 <local0.notice> 10.106.40.223 01/09/2020:22:55:41 GMT ns-223 0-PPE-0 : default ICA Message 156 0 : "Skipping ICA flow: Session GUID [4e3a91175ebcbe686baf175eec7e0200], Client IP/Port [10.105.232.40/60059], Server IP/Port [10.106.40.219/2598], MSI Client Cookie [Non-MSI], Session setup time [01/09/2020:22:55:39 GMT], Client Type [0x0052], Receiver Version [19.12.0.23], User [user1], Client [10.105.232.40], Server [10.106.40.219], Ctx Flags [0x8820220008], Track Flags [0x1600010c040], Skip Code [171]"

Error counters

Various counters are captured ICA parsing. The following table lists the various counters for ICA parsing. Run the command nsconmsg –g hdx –d statswt0for viewing the counter details.

HDX counter name Purpose Category(Stats/Error/Diagnostics)
hdx_tot_ica_conn Indicates total number of Pure ICA connections detected by NS. Incremented whenever an ICA connection based on the ICA signature on a client PCB is detected. Stats
hdx_tot_cgp_conn Indicates total number of CGP connections detected by NS (Session Reliability ON). Incremented whenever a CGP connection based on the CGP signature on a client PCB is detected. Stats
hdx_dbg_tot_udt_conn Indicates total number of UDP ICA connections detected by NS Stats
hdx_dbg_tot_nsap_conn Indicates total number of NSAP supported connections detected by NS Stats
hdx_tot_skip_conn Indicates how many ICA connections were skipped by parser due to invalid ICA or CGP signature. Stats
hdx_dbg_active_conn Total Active EDT/CGP/ICA connections at that instant. Stats
hdx_dbg_active_nsap_conn Total Active EDT/CGP/ICA NSAP connections at that instant. Stats
hdx_dbg_skip_appflow_disabled Total number of instances where AppFlow was detached from a session because of disabling AppFlow Stats/Diagnostics
hdx_dbg_transparent_user Total number of transparent user access Stats/Diagnostics
hdx_dbg_ag_user Total number of Access Gateway user access Stats/Diagnostics
hdx_dbg_lan_user Total number of LAN user mode access Stats/Diagnostics
hdx_basic_enc Indicates the number of ICA connections using basic encryption Stats/Diagnostics
hdx_advanced_enc Indicates the number of ICA connections using advanced RC5 based encryption Stats/Diagnostics
hdx_dbg_reconnected_session Total number of reconnect requests from client without any NetScaler error Stats/Diagnostics
hdx_dbg_host_rejected_ns_reconnect Total number of hosts rejected reconnects requests by client Stats/Diagnostics
hdx_euem_available Indicates the number of connections having the End User Experience Monitoring channel available. End User Experience Monitoring channel is required to collect statistics such as ICA RTT. Stats/Diagnostics
hdx_err_disabled_sr Session Reliability is disabled using nsapimgr knob. Session does not work for this session. Error
hdx_err_skip_no_msi XA/XD server is Missing MSI capability. This indicates an older server version and HDX Insight skips this connection. Error
hdx_err_skip_old_server Old unsupported server version Error
hdx_err_clnt_not_whitelist Client workspace not in allow list, HDX Insight skips this connection Error
hdx_sm_ica_cam_channel_disabled Total number of NS_ICA_CAM_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_usb_channel_disabled Total number of NS_ICA_USB_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_clip_channel_disabled Total number of NS_ICA_CLIP_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_ccm_channel_disabled Total number of NS_ICA_CCM_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_cdm_channel_disabled Total number of NS_ICA_CDM_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_com1_channel_disabled Total number of NS_ICA_COM1_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_com2_channel_disabled Total number of NS_ICA_COM2_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_cpm_channel_disabled Total number of NS_ICA_CPM_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_lpt1_channel_disabled Total number of NS_ICA_LPT1_CHANNEL disabled via SmartAccess policy Diagnostics
hdx_sm_ica_lpt2_channel_disabled Total number of NS_ICA_LPT2_CHANNEL disabled via SmartAccess policy Diagnostics
dx_dbg_sm_ica_msi_disabled Total number of cases where MSI is disabled via SmartAccess policy Diagnostics
hdx_sm_ica_file_channel_disabled Total number of NS_ICA_FILE_CHANNEL is disabled via SmartAccess policy Diagnostics
hdx_dbg_usb_accept_device Total number of USB devices accepted Diagnostics
hdx_dbg_usb_reject_device Total number of USB devices rejected Diagnostics
hdx_dbg_usb_reset_endpoint Total number of USB endpoints reset Diagnostics
hdx_dbg_usb_reset_device Total number of USB devices reset Diagnostics
hdx_dbg_usb_stop_device Total number of USB devices stopped Diagnostics
hdx_dbg_usb_stop_device_response Total number of responses from stopped USB devices Diagnostics
hdx_dbg_usb_device_gone Total number of USB devices gone Diagnostics
hdx_dbg_usb_device_stopped Total number of USB devices stopped Diagnostics

nstrace validation

Check for CFLOW protocol to see all AppFlow records going out of NetScaler.

Population of records in NetScaler Console checklist

  • Run the command tail -f /var/mps/log/mps_afdecoder.log | grep -i "Data Record: ica_"and check logs to confirm NetScaler Console is receiving AppFlow records.

  • Confirm NetScaler instance is added to NetScaler Console.
  • Validate NetScaler Gateway/VPN virtual server is licensed in NetScaler Console.
  • Make sure multi-hop parameter setting is enabled for double-hop.
  • Make sure NetScaler Gateway is cleared for second-hop in double-hop deployment.

Before contacting Citrix technical support

For a speedy resolution, make sure that you have the following information before contacting Citrix technical support:

  • Details of the deployment and network topology.

  • NetScaler and NetScaler Console versions.
  • Citrix Virtual Apps and Desktops server versions.
  • Client Workspace versions.
  • Number of Active ICA sessions when the issue occurred.
  • Tech support bundle captured by running the show techsupport command at the NetScaler command prompt.
  • Tech support bundle captured for NetScaler Console.
  • Packet traces captured on all NetScaler. To start a packet trace, type, start nstrace -size 0' To stop a packet trace, type, stop nstrace
  • Collect entries in the system’s ARP table by running the show arp command.

Known Issues

Refer NetScaler release notes for known issues on HDX Insight.

Troubleshoot HDX Insight issues