Product Documentation

Troubleshooting Tips

Dec 30, 2015

Following are some reported problems and their resolutions.

I cannot see any records on the NetScaler Insight Center dashboard.
If no reports appear on the dashboard after you have enabled AppFlow for at least one virtual server on a NetScaler ADC or NetScaler Gateway that has been added to the inventory, check the following:
  • Is the NetScaler Insight Center version the same as or higher than the NetScaler ADC version?
  • Does the NetScaler ADC have the required license for NetScaler Insight Center to collect data? For details, see Licensing Information.
  • Are all the configurations implemented from NetScaler Insight center, and not from the NetScaler ADC?
  • Did you wait for 5 minutes for the data to appear on the dashboard? If not, wait for at least 5 minutes for NetScaler Insight Center to display the data.
  • Is the configuration utility displaying any repeating error messages? If so, check to see if there are any core dumps in the /var/core directory.
  • Is AppFlow enabled on the NetScaler appliance?
  • Is AppFlow logging enabled on the virtual servers (load balancing, content switching, or VPN virtual servers) of the NetScaler appliance?
  • Is AppFlow enabled for the services and service groups that are bound to the load balancing virtual servers?
  • Does the AppFlow policy for the virtual server have the highest priority for which AppFlow is most recently enabled? (If AppFlow is enabled for a virtual server on more than one NetScaler Insight Center virtual appliance, the virtual appliance on which AppFlow was most recently enabled for the NetScaler appliance collects the information). Follow the below procedure to check the priority:
    1. On the NetScaler appliance, navigate to Traffic Management.
    2. Expand Load Balancing, or expand Content Switching and then click Virtual servers.
    3. Double-click the virtual server for which you want to see if AppFlow is enabled.
    4. In the Configure Virtual Server (Load Balancing) or Configure Virtual Server (Content Switching) dialog box, on the Policies tab, click the arrow in the right corner of the dialog box.
    5. Select AppFlow from the drop-down list.
    6. Check if the AppFlow policy name has the highest priority.

    Also, from the command line interface, run the sh appflow global command to make sure that the global AppFlow policies do not override the virtual server policies.

  • Are the NetScaler Insight Center virtual appliances and port values set correctly on the NetScaler appliance? To check:
    1. On the Configuration tab of NetScaler appliance, navigate to System > AppFlow.
    2. Under Policy manager select AppFlow policy Manager.
    3. Select a virtual server (LB virtual server or CS Virtual server).
    4. Double click Action Name and verify that if the NetScaler Insight Center IP address and port are correct.
  • On the NetScaler appliance, is traffic flowing through the virtual server for which data collection was enabled in NetScaler Insight Center? Also verify that the hits counter is increasing on the AppFlow policy, by running the following command from the NetScaler command line interface:

    sh appflow policy <policyname>

  • Is UDP port 4739 on the NetScaler Insight Center virtual appliance reachable from NetScaler appliance?
  • To see Web Insight records, run the following command on the NetScaler command line interface to check if Log only client-side traffic is set to NO.

    show appflow param

    Example
    > show appflow param 
            AppFlow parameters 
     
            IPFIX template refresh interval: 3600 seconds 
            Appname refresh interval: 60 seconds 
            IPFIX flow record export interval: 60 seconds 
            IPFIX UDP Path MTU: 1472 bytes 
            HTTP URL logging: ENABLED 
            AAA username logging: ENABLED 
            HTTP cookie logging: ENABLED 
            HTTP referer logging: ENABLED 
            HTTP method logging: ENABLED 
            HTTP host logging: ENABLED 
            HTTP user-agent logging: ENABLED 
            HTTP Content-Type header logging: ENABLED 
            HTTP Authorization header logging: ENABLED 
            HTTP Via header logging: ENABLED 
            HTTP X-Forwarded-For header logging: ENABLED 
            HTTP Location header logging: ENABLED 
            HTTP Setcookie header logging: ENABLED 
            HTTP Setcookie2 header logging: ENABLED 
            Log only client-side traffic: NO 
            Connection Chaining: DISABLED 
    
    If Log only client-side traffic is set to YES, then run the following command to change the setting to NO:

    set appflow param -clientTrafficOnly (YES | NO)

    Example:
    set appflow param -clientTrafficOnly NO
    
  • Is NetScaler Insight Center receiving data records from the device?
    To verify that NetScaler Insight Center is receiving data records,
    • On the NetScaler Insight Center virtual machine, open the mps_afdecoder.log file located in /var/mps/log/ and check to see if the file displays text similar to the following once every second:

      Monday, 22 Dec 14 20:12:03 +0000 [Debug] For AFProto Thread id :: 34473829696, Elapsed time is: 1010014 micro sec, Packet pick up is: 19, Packet processed is: 19, Decode rate is: 18/sec, Mean decode rate is: 13/sec, Data record processed is: 29, Data record routed is: 15

      Following are descriptions of the parameters in the above text:
      • Elapsed time is: Time consumed, in microseconds, since NetScaler Insight Center last displayed this text.
      • Packet pick up is: Number of UDP packets that NetScaler Insight Center has received in the elapsed time.
      • Packet processed is: Number of UDP packets that NetScaler Insight Center has processed in the elapsed time.
      • Decode rate is and Mean decode rate is: Rate at which NetScaler Insight Center has received UDP packets.
      • Data record processed is: Number of data records present in the UDP packets that were successfully decoded. (One UDP packet can contain more than one data record).
      • Data record routed is: Number of data records that were routed properly to a proper handler (such as Web Insight or HDX Insight).
      If the Data record routed is parameter's value is consistently zero, the NetScaler Insight Center virtual machine is not receiving any valid data records. Check the device's AppFlow configuration, or contact the technical support team to debug the issue.
    • On the NetScaler Insight Center virtual machine, in the mps_afdecoder.log file located at /var/mps/log/, check to see if NetScaler Insight Center displays the data records, such as :
      • ica_session_setup, ica_session_network_update, and ica_session_update, every second for HDX Insight.
      • l7_clt_to_ns, ns_es4ns_client_load and ns_es4ns_client_render records every second for Web Insight.
      If it displays these records, the data records are being sent to NetScaler Insight Center. Otherwise, check the device's AppFlow configuration.
  • Are the data record logs enabled for Web Insight and/or HDX Insight?
    To check, on the Configuration tab, click System, and in the right pane, select Change Data Record Log Settings and verify that the HDX Insight Logs or/and Web Insight logs option is selected.
    Note: Web Insight logs are disabled by default.

    Enable Web Insight Logs only if the value of Data record routed is parameter in the var/mps/log/mps_afdecoder.log file is in the range of 1 to 20. If the value is more than 20, disable AppFlow on the virtual server that has high traffic. If you enable Web Insight logs for a virtual server that has high traffic, the load on the mps_afdecoder.log file can become heavy and the appliance can become unresponsive.

NetScaler Insight Center does not display reports for a particular user name. What should I do?
If you suspect that a particular user report is not being displayed, on the NetScaler Insight Center virtual machine, open the mps_afdecoder.log files located in /var/mps/log/ file and run the following command:

grep <username> < mps_afdecoder.log>

If the output includes the user name, the user record is displayed in the reports. If the output does not display the user name, verify that the Citrix Receiver version being used is supported by NetScaler Insight Center. For details, see Supported Software.

Why does the WAN latency and DC latency metrics display the values zero?
If the latency is less than 1 millisecond, the WAN latency and DC latency values are displayed as zero.
I am not able to add a NetScaler appliance. What are the possible reasons?
  • Make sure that NetScaler appliance you add is UP and reachable when you add it to the Inventory. If the appliance is DOWN, or OUT-OF-SERVICE, you cannot add it to the Inventory.
  • Make sure that you have not added a standalone Access Gateway Standard appliance.
The session up time value is incorrect.
Make sure that the times set for the NetScaler appliance and NetScaler Insight Center are correct and in sync. For more information, see Configuring Clock Synchronization.
The graphs are not clearly displayed or they look scrambled.
Clear the browser cache and retry. Also make sure the NetScaler Insight Center is supported on the browser. For details, see Accessing NetScaler Insight Center.
I am unable to log on to NetScaler Insight Center through the Internet Explorer browser.

The browser might be set in compatibility mode. To disable compatibility mode, go to Tools >Compatibility View Settings and clear the Display all websites in Compatibility View check box.

If you still cannot access NetScaler Insight Center after you disable the compatibility mode in Internet Explorer version 8 or 9, make sure that the browser mode and document mode in the browser are set to the same version. To view the configuration, press F12. Set the values to either Internet Explorer 8 or Internet Explorer 9.

Why does the RTT metric display an incorrect value? Or, The ICA RTT is shown as N/A on the NetScaler Insight Center reports. What should I do?
To debug this issue, check to see if the EUEM value on XenApp or XenDesktop is enabled or not. If EUEM is disabled, then NetScaler Insight Center does not display the ICA RTT values. If EUEM is enabled, and ICA RTT is shown as N/A, then perform each of the following diagnostic operations sequentially till the issue is resolved.
  • Check if the Citrix End User Experience Monitoring (EUEM) configurations are enabled on the XenApp server:
    1. From the Citrix AppCenter, navigate to Citrix Resources > XenApp > Farm > Policies.
    2. On the Settings tab, go to ICA > End User Monitoring and make sure that the following configurations are enabled:
      • ICA Round Trip Calculation
      • ICA Round Trip Calculation Interval
      • ICA Round Trip Calculation for Idle Connections
    3. Verify the configurations.
      1. Check for ICA RTT policy entries in the registry.
        1. Open the command prompt from your Windows machine.
        2. Type regedit and press Enter.
        3. Navigate to HKEY_LOCAL_MACHINE > Software > Policies > Citrix > EndUserMonitoring.
        4. Make sure that the following configurations are available:
          • ICA Round Trip Calculation
          • ICA Round Trip Calculation Interval
          • ICA Round Trip Calculation for Idle Connections
      2. Make sure that the services for these configurations are set to Automatic.
        1. Open the command prompt from your Windows machine.
        2. Type services.msc and press Enter.
        3. In the Services dialog box, make sure that Citrix End User Experience Monitoring is listed and the Startup Type is set to Automatic.
  • If you are using a virtual desktop application, make sure that the Startup Type is set to Automatic. This has to be done on the master image.
    1. Open the command prompt from your Windows machine.
    2. Type services.msc and press Enter.
    3. In the Services dialog box, make sure that Citrix End User Experience Monitoring is listed and the Startup Type is set to Automatic.
  • Make sure that the Citrix EUEM Hotfix is available.
    1. Open the command prompt from your Windows machine.
    2. Type appwiz.cpl and press Enter.
    3. On the Uninstall or Change a Program dialog box, make sure that the Citrix HotFix XA650R01W2K8R2X64064 for the XenApp 6.5 with R01 server is installed.
  • Make sure that the value of flowRecordInterval parameter on the NetScaler ADC is set to 60 seconds. To verify, run the following command on the NetScaler appliance:

    show appflow param

    Confirm by checking if the following output parameter is set to 60:
    IPFIX flow record export interval: 60 seconds
    
    If the value is not 60, then set the value by running the following command:

    set appflow param -flowRecordInterval 60

  • After you start the XenApp or XenDesktop traffic, wait for two or three minutes and check to see the ICA RTT value in NetScaler Insight Center reports.
  • Check if NetScaler Insight Center dashboard displays session ID as NON-EUEM as shown in the following image:
    Figure 1. NetScaler Insight Center Report
  • Make sure that the ICA connections actively send data.
  • Verify the XenApp or XenDesktop versions.
    Table 1. XenApp/XenDesktop Versions and builds

    Product

    HDX Insight

    XenApp

    6.5, build 6682 with HRP01

    XenDesktop

    5.6, build 56060

    7.0 and higher versions

    Table 2. Operating systems and receiver details

    Operating system

    Receiver version

    Windows 7

    3.4 Enterprise Edition

    4.0 Standard Edition

    Windows 8

    3.4 Enterprise Edition

    4.0 Standard Edition

    Mac

    11.8, build 238301 and above

    Note: Mac client does not support ICA RTT reports for CloudBridge version 7.3.0.

    Linux

    13 and above

  • Make sure that the ICA RTT calculation for idle connections is enabled on the XenApp server.
Note: If none of the debugging options work, contact technical support to install the Sems Com Plugin test tool on Xenapp Server to see if you receive ICA RTT events.
 
 
An error message is displayed when I add the NetScaler appliance to the NetScaler Insight Center inventory.
Make sure that the GUI is accessible, by verifying that port is open for communication. To do so, run the following command from the command line interface:

show ns ip <ipaddress>

Example: show ns ip 10.102.60.31
IP: 10.102.60.31 
            Netmask: 255.255.255.128 
            Type: NetScaler IP 
            ... 
            ... 
            ...									 
            gui: Enabled 
            ... 
            ... 
            ... 

If GUI is set to secureonly, then make sure that all communication between NetScaler Insight Center and a NetScaler appliance is over a secure channel. For more information, see Modifying System Security Settings.

Also, make sure that proper licenses are available for the NetScaler appliances that are added in NetScaler Insight Center inventory. For more information, see Licensing Information. Also check if the required ports are open for communication. For more details, see Ports.

I am not able to clear all the AppFlow related configurations for a selected virtual server by using the NetScaler Insight Center graphical user interface.
Log on to the NetScaler appliance by using the graphical user interface and delete the required collector and action. To delete the action name, perform the following procedure:
  1. Navigate to System > AppFlow > Actions.
  2. Select the AppFlow action name that you want to delete.
  3. Click Remove.
To delete the collector, perform the following procedure:
  1. Navigate to System > AppFlow > Collectors.
  2. Select the Collector name that you want to delete.
  3. Click Remove.
The following error is displayed when I upgrade the NetScaler Insight Center appliance: "Backup/ Restore operation in progress. Try after some time."
Reboot the NetScaler Insight Center appliance or run the following commands at the NetScaler Insight Center shell prompt:
/etc/rc.d/analyticsd stop 
/etc/rc.d/analyticsd start
The HDX Insight node does not appear on the NetScaler Insight Center dashboard.
Make sure that proper licenses are available for the NetScaler appliances that are added in NetScaler Insight Center inventory. For more information, see Licensing Information. Also check if the required ports are open for communication. For more details, see Ports.
An error message is displayed when I access NetScaler Insight Center using Internet Explorer 8.
Some features of NetScaler Insight Center are not support on Internet Explorer 8. You can access the appliance using Internet Explorer 9. For more details on browser support, see Accessing NetScaler Insight Center.
I cannot see the waterfall chart or the Page Analysis button on the NetScaler Insight Center dashboard.
Check the following possible causes:
  1. Is the HTML Injection check box selected?
  2. Is the URL response content type of the web page in text or HTML format? NetScaler Insight Center does not display the waterfall charts if the response type is anything other than text or HTML.
  3. Was the transaction for the web page successful? Make sure the response header indicates 200 OK.
  4. Were the page rendering and loading successful? If either fails, the timing information is not received from the client and the waterfall charts are not be displayed.
  5. Check to see if the transactions are served by the NetScaler appliance (either NetScaler generated or served from NetScaler cache). If so, the waterfall charts are not displayed.
  6. Is the appropriate license installed on the NetScaler appliance?
NetScaler Insight Center does not display the HDX Insight reports for CloudBridge appliances. What should I do?
  • Check the firewall configuration and make sure that the CloudBridge appliance and NetScaler Insight Center communicate over port 4739.
  • Wait for 2 to 3 minutes after the traffic is generated. NetScaler Insight Center usually takes two to three minutes to display the HDX Insight reports for a user.
  • Check the CloudBridge discovery status on the NetScaler Insight Center Inventory.
  • Make sure that CloudBridge ICA connections are accelerated with Disk Based Compression (DBC) policy. To verify, on the Monitoring tab, navigate to Optimization > Connections, and in the right pane check to see if the Compression Type is Disk for ICA service class.

    Also, on the Configuration tab, navigate to Optimization Rules > Service Classes and in the right pane, expand ICA and click Edit. The Acceleration Policy selected must be Disk.

  • On the CloudBridge appliance, make sure that the configuration for Appflow HDX Data set is enabled. Also, make sure that the update Interval is set to one minute and NetScaler Insight Center collector IP or port should not be deleted. To verify, on the Configuration tab, navigate to Appliance Settings > AppFlow and verify the values on the right pane.
The HDX Insight reports display the Uptime as 'Negative.' What should I do?

Make sure that the NTP server is configured on both CloudBridge appliance and NetScaler Insight Center. To add an NTP server on NetScaler Insight Center, see Configuring Clock Synchronization.

Note: If you have configured the NTP server at a later point of time, then, run the drop_table command from the NetScaler Insight Center Command Line Interface (CLI). Then discover the appliance again and establish a new ICA connection.
NetScaler Insight Center does not display the geo maps. What should I do?
If NetScaler Insight Center does not display the geo maps, then make sure to follow the below steps:
  1. Make sure to upload the geo database file:
    1. On the Configuration tab, expand NetScaler Insight Center, and then click Geo Database Files.
    2. From the Action drop-down menu, select Upload.
    3. In the Upload Geo Database File window, click Browse.
    4. Navigate to the location of the geo database file, GeoLiteCity.dat, and then click Upload.
  2. Enable geo data collection.
    1. On the Configuration tab, click Inventory.
    2. From the inventory list, select the IP address of the NetScaler appliance for which you want to enable geo data collection.
    3. In the NetScaler Insight Center Inventory Setup pane, select the Enable geo data collection check box.
  3. If the geo maps are not displayed even after uploading the geo database file and enabling geo data collection, remove (drop) the tables in NetScaler Insight Center.
    1. Run the following commands from the NetScaler insight Center command line interface:
      /etc/rc.d/analyticsd stop 
      cd /mps 
      ./drop_table 
      
    2. Restart NetScaler Insight Center.