Citrix Application Delivery Management

Troubleshoot Gateway Insight issues

If the Gateway Insight solution is not functioning as expected, the issue might be with one of the following. Refer to the checklists in the respective sections for troubleshooting.

  • Gateway Insight configuration.
  • Connectivity issue between Citrix ADC and Citrix ADM.
  • Record generation in Citrix ADC.
  • Validations in Citrix ADM.

Gateway Insight configuration checklist

  • Make sure that the AppFlow feature is enabled in the Citrix ADC appliance. For details, see Enabling AppFlow.

  • Check the Gateway Insight configuration in the Citrix ADC running configuration.

    Run the show running | grep -i <appflow_policy> command to check the Gateway Insight configuration. Make sure that the bind type is REQUEST. For example;

     bind vpn vserver afsanity -policy afp -priority 100 -type REQUEST
     <!--NeedCopy-->
    

    Bind type OTHERTCP_REQUEST is also required for Gateway Insight.

     bind vpn vserver afsanity -policy afp -priority 100 -type OTHERTCP_REQUEST
     <!--NeedCopy-->
    
  • For single-hop, Access Gateway, or Unified Gateway deployment, make sure that Gateway Insight AppFlow policy is bound to the VPN virtual server, where VPN traffic is flowing. For details, see Enabling HDX Insight data collection.
  • For double-hop, Gateway Insight must be configured on both the hops.
  • Check appflowlog parameter in Citrix Gateway/VPN virtual server. For details, see Enabling AppFlow for Virtual Servers.

Connectivity between Citrix ADC and Citrix ADM checklist

  • Check AppFlow collector status in Citrix ADC. For details, see How to check the status of connectivity between Citrix ADC and AppFlow Collector.

  • Check Gateway Insight AppFlow policy hits.

    Run the command show appflow policy <policy_name> to check the AppFlow policy hits.

    You can also navigate to System > AppFlow > Policies in the GUI to check the AppFlow policy hits.

  • Validate any firewall blocking AppFlow ports 4739 or 5557.

Record generation in Citrix ADC checklist

  • Run the nsconmsg -d stats -g ai_tot command and check for the stats increments in Citrix ADC.
  • Capture nstrace logs and check for CFLOW packets to confirm Citrix ADC exports AppFlow records.

    Note:

    The nstrace logs are required only for IPFIX. For Logstream, nstrace logs do not confirm if the ADC appliance exported the AppFlow records.

Validation of records in Citrix ADM

  • Run the tail -f /var/mps/log/mps_afdecoder.log | grep -i "Data Record: vpn_" command to check the logs to confirm Citrix ADM is receiving AppFlow records.
  • Make sure that the Citrix ADC instance is added to the Citrix ADM.
  • Make sure that the Citrix Gateway/VPN virtual server is licensed in Citrix ADM.

Validation of Logstream logs in Citrix ADM

Validation of Logstream data received by Citrix ADM can be done using the following methods:

  • Enabling data record logging in Citrix ADM

    Once enabled, the logs can be seen in the /var/mps/log/mps_afdecoder.log

  • Enabling ULFD library logging

    Run the command /mps/decoder_enable_debug

    The logs are captured in/var/ulflog/libulfd.log

    You can disable logging by using the command /mps/decoder_disable_debug

Gateway Insight counters

The following Gateway Insight counters are available.

  • ai_tot_preauth_epa_export
  • ai_tot_auth_export
  • ai_tot_auth_session_id_update_export
  • ai_tot_postauth_epa_export
  • ai_tot_vpn_update_export
  • ai_tot_ica_fileinfo_export
  • ai_tot_app_launch_failure
  • ai_tot_logout_export
  • ai_tot_skip_appflow_export
  • ai_tot_sso_appflow_export
  • ai_tot_authz_appflow_export
  • ai_tot_appflow_pol_eval_failure
  • ai_tot_vpn_export_state_mismatch
  • ai_tot_appflow_disabled
  • ai_tot_appflow_pol_eval_in_gwinsight
  • ai_tot_app_launch_success

AppFlow records in Citrix ADC log

Starting from release 13.0 build 71.x, you can check the Citrix ADC logs to confirm if the AppFlow records are exported. The default log level of syslogparams captures all the error and information logs. In case you do not find a clue about the errors, enable all log levels including DEBUG in syslogparams to capture even the DEBUG logs.

Sample logs

<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 147 0 : "GwInsight: Sent auth record Func=ns_sslvpn_export_auth_data Username=<name> Clientip=<ip>:<port> Destip=0:80 SessSeq=0 Sessid=<sessid> Gwip=<ip>:443 StatusCode=0 CSappid=0 CSAppname=(null) VPNfqdn=<vpnfqdn> Authtype=3 EPAid=(null) AuthStage=1 AuthDuration=309 AuthAgent=<auth_server_ip> Groupname= Policyname=<name> CurfactorPolname=<name> NextfactorPolname= CSecExpr= Devicetype=16777219 Deviceid=0 email="
<local0.err> … GMT 0-PPE-0 : default SSLVPN Message 143 0 : "GwInsight: Func=ns_aaa_copy_email_id_to_vpn_record input hash_attrs_len is zero"
<local0.err> … GMT 0-PPE-0 : default SSLVPN Message 148 0 : "GwInsight: Func=update_session_appflow_collector pcb or session is NULL"
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 165 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=1 Sessid=<sessid> Gwip=<ip>:443 StatusCode=0 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=0 SessState=2 SessMode=2 IIP=0 AppByteCount=0 ReqURL=/Citrix/Store
Web BackendServername= SSOurl= email="
SSO logs:
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 463 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=2 Sessid=<sessid> Gwip=<ip>:443 StatusCode=150 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=1 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 582 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=2 Sessid=<sessid> Gwip=<ip>:443 StatusCode=150 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=3 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 513 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:80 SessSeq=2 Sessid=<sessid> Gwip=<ip>:443 StatusCode=150 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=2 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->
<local0.info> … GMT 0-PPE-0 : default SSLVPN Message 29796 0 : "GwInsight: Sent session update record Func=ns_sslvpn_send_update_record Username=<name> Clientip=<ip>:<port> Destip=<ip>:443 SessSeq=c Sessid=<sessid> Gwip=<ip>:443 StatusCode=155 CSappid=0 CSAppname=(null) VPNfqdn=<fqdn> SSOAuthMethod=6 SessState=4 SessMode=3 IIP=0 AppByteCount=0 ReqURL= BackendServername=<> SSOurl= email="
<!--NeedCopy-->

Contact Citrix technical support

For a speedy resolution, make sure that you have the following information before contacting Citrix technical support:

  • Details of the deployment and network topology.
  • Citrix ADC and Citrix ADM versions.
  • Tech support bundle for Citrix ADC and Citrix ADM.
  • nstrace capture during the issue.

Known Issues

Refer Citrix ADC release notes for known issues on Gateway Insight.