Citrix SD-WAN

AppFlow and IPFIX

AppFlow and IPFIX are flow export standards used to identify and collect application and transaction data in the network infrastructure. This data gives better visibility into application traffic utilization and performance.

The collected data, called flow records are transmitted to one or more IPv4 or IPv6 collectors. The collectors aggregate the flow records and generate real-time or historical reports.

AppFlow

AppFlow exports flow level data for HDX / ICA connections only. You can enable either the TCP only for HDX dataset template or the HDX dataset template. The TCP only for HDX dataset provides multi-hop data. The HDX dataset provides HDX insight data.

Note

HDX template is available for Citrix SD-WAN PE edition and Two-box appliances only. It should be enabled on the Data Center appliance.

AppFlow Collectors like Splunk and Citrix ADM have dashboards to interpret and present these templates.

IPFIX

IPFIX is a collector export protocol used for exporting flow level data for all connections. For any connection, you can view information such as packet count, byte count, type of service, flow direction, routing domain, application name and so on. IPFIX flows are transmitted through the management interface. Most collectors can receive IPFIX flow records, but may need to build a custom dashboard to interpret IPFIX template.

The IPFIX template defines the order in which the data stream is to be interpreted. The collector receives a template record, followed by the data records. Citrix SD-WAN uses templates 611 and 613 to export IPv4 IPFIX flow data, 615 and 616 to export IPv6 IPFIX flow data along with Options template 612.

Application Flow Info (IPFIX) exports data sets as per templates 611 for IPv4 flows, 615 for IPv6 flows and 612 options Template with Application info.

Basic Properties (IPFIX) exports data sets as per templates 613 for IPv4 flows and 616 for IPv6 flows.

The following tables provide the detailed list of flow data associated with each IPFIX template.

Application Flow Info (IPFIX) - V10 templates

Template ID - 611

Info Element (IE) IE name & ID Type and len Description
Observation point ID observationPointId, 138 Unsigned32, 4  
Export process ID exportingProcessId, 144 Unsigned32, 4  
Flow ID flowId, 148 Unsigned64, 8  
Ipv4 SRC IP sourceIPv4Address, 8 Ipv4address, 4  
Ipv4 DST IP destinationIpv4Addres, 12 Ipv4address, 4  
Ipversion ipVersion, 60 Unsigned8, 1 Set to 4.
IP protocol number protocoldentifier,4 Unsigned8, 1  
Padding N/A Unsigned16, 2  
SRC Port sourceTransportPort, 7 Unsigned16, 2  
DST Port destinationTransportPort,11 Unsigned16, 2  
Pkt Count packetDeltaCount, 2 Unsigned64, 8  
Byte Count octetDeltaCount, 1 Unsigned64, 8  
Time for first pkt in microseconds flowStartMicroseconds, 154 dateTimeMicroseconds, 8  
Time for lastpkt in microseconds flowEndMicroseconds, 155 dateTimeMicroseconds, 8  
IP ToS ipClassOfService, 5 Unsigned8, 1  
Flow Flags tcpControlBits, 6 Unsigned8, 2 Currently set to 0.
Flow Direction flowDirection, 61 Unsigned8, 1 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN
Input Interface ingressInterface, 10 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.
Output Interface egressInterface, 14 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.
Input Vlan ID vlanId, 58 Unsigned16, 2  
Output Vlan ID postVlanId, 59 Unsigned16, 2  
VRF ID ingressVRFID, 234 Unsigned32, 4  
Flow Key Indicator flowKeyIndicator, 173 Unsigned64, 8 Set to 0x1E037F.
Application ID applicationId, 95 octetArray, variable The Application ID is same as the ID of the applications classified by the DPI engine. The application IDs remain constant. The application IDs for Custom domain name based applications change with every configuration update.

Template ID – 615 (IPv6 flows)

Info Element (IE) IE name & ID Type and len Comment  
Observation point ID observationPointId, 138 Unsigned32, 4  
Export process ID exportingProcessId, 144 Unsigned32, 4    
Flow ID flowId, 148 Unsigned64, 8    
Ipv6 SRC IP sourceIPv6Address, 27 Ipv6address, 16    
Ipv6 DST IP destinationIpv6Addres, 28 Ipv6address, 16    
Ipversion ipVersion, 60 Unsigned8, 1 Set to 6  
IP protocol number protocoldentifier, 4 Unsigned8, 1    
Padding N/A Unsigned16, 2    
SRC Port sourceTransportPort, 7 Unsigned16, 2    
DST Port destinationTransportPort, 11 Unsigned16, 2    
Pkt Count packetDeltaCount, 2 Unsigned64, 8    
Byte Count octetDeltaCount, 1 Unsigned64, 8    
Time for first pkt in microseconds flowStartMicroseconds, 154 dateTimeMicroseconds, 8    
Time for lastpkt in microseconds flowEndMicroseconds, 155 dateTimeMicroseconds, 8    
IP ToS ipClassOfService, 5 Unsigned8, 1    
Flow Flags tcpControlBits, 6 Unsigned8, 2 Currently set to 0.  
Flow Direction flowDirection, 61 Unsigned8, 1 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN  
Input Interface ingressInterface, 10 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.  
Output Interface egressInterface, 14 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.  
Input Vlan ID vlanId, 58 Unsigned16, 2    
Output Vlan ID postVlanId, 59 Unsigned16, 2    
VRF ID ingressVRFID, 234 Unsigned32, 4    
Flow Key Indicator flowKeyIndicator, 173 Unsigned64, 8 Set to 0x1E037F.  
Application ID applicationId, 95 octetArray, variable The Application ID is same as the ID of the applications classified by the DPI engine. The application IDs remain constant. The application IDs for Custom domain name based applications change with every configuration update.  

Template 612 (Options Template)

Info Element (IE) IE name & ID Type Comment
Application ID applicationId, 95 octetArray The Application ID is same as the ID of the applications classified by the DPI engine. The application IDs remain constant. The application IDs for Custom domain name based applications change with every configuration update.
Application Name applicationName, 96 string Specifies the name of the Citrix SDWAN specific proprietary application.
Application Description applicationDescription, 94 string Specifies the description of the application.

Basic Properties (IPFIX) – V9 compliant template - Template 613 (IPv4 flows)

Info Element (IE) IE name & ID Type and len Comment
Ipv4 SRC IP sourceIPv4Address, 8 Ipv4address, 4  
Ipv4 DST IP destinationIpv4Addres, 12 Ipv4address, 4  
Ipversion ipVersion, 60 Unsigned8, 1  
IP protocol number protocoldentifier, 4 Unsigned8, 1  
IP ToS ipClassOfService, 5 Unsigned8, 1  
Flow Direction flowDirection, 61 Unsigned8, 1 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN
SRC Port sourceTransportPort, 7 Unsigned16, 2  
DST Port destinationTransportPort, 11 Unsigned16, 2  
Pkt Count packetDeltaCount, 2 Unsigned64, 8  
Byte Count octetDeltaCount, 1 Unsigned64, 8  
Input Interface ingressInterface, 10 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.
Output Interface egressInterface, 14 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.
Input Vlan ID vlanId, 58 Unsigned16, 2  
Output Vlan ID postVlanId, 59 Unsigned16, 2  

Template ID – 616 (IPv6 flows)

Info Element (IE) IE name & ID Type and len Comment  
Ipv6 SRC IP sourceIPv6Address, 27 Ipv6address, 16    
Ipv6 DST IP destinationIpv6Addres, 28 Ipv6address, 16    
Ipversion ipVersion, 60 Unsigned8, 1 Set to 6  
IP protocol number protocoldentifier,4 Unsigned8, 1    
IP ToS ipClassOfService, 5 Unsigned8, 1    
Flow Direction flowDirection, 61 Unsigned8, 1 0x00: ingress flow0x01: egress flowWAN-WAN and LAN-LAN flows are a possibility in SDWAN  
SRC Port sourceTransportPort, 7 Unsigned16, 2    
DST Port destinationTransportPort, 11 Unsigned16, 2    
Pkt Count packetDeltaCount, 2 Unsigned64, 8    
Byte Count octetDeltaCount, 1 Unsigned64, 8    
Input Interface ingressInterface, 10 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.  
Output Interface egressInterface, 14 Unsigned32, 4 Citrix SD-WAN load balances data flows through multiple member paths, hence a single data flow can have multiple input/output interface combinations.  
Input Vlan ID vlanId, 58 Unsigned16, 2    
Output Vlan ID postVlanId, 59 Unsigned16, 2    

Limitations

  • AppFlow does not support IPv6 collector and flow records.
  • The export interval for Net Flow is increased from 15 seconds to 60 seconds.
  • AppFlow/IPFIX flows are transmitted over UDP, on connection loss not all data is retransmitted. If the export interval is set to X minutes, the appliance stores X minutes of data only. Which is retransmitted after X minutes of connection loss.
  • In Citrix SD-WAN, release 10 version 2 the AppFlow settings are made local to every appliance, while in the previous releases it was a global setting. If the SD-WAN software release is downgraded to any of the previous releases and if AppFlow is configured on any one of the appliances, it will be applied globally to all alliances.

Configuring AppFlow/IPFIX

You can configure AppFlow / IPFIX on individual SD-WAN appliances or configure it on SD-WAN Center and push the configuration to a group of appliances.

To configure AppFlow / IPFIX on SD-WAN appliances:

  1. In Citrix SD-WAN SE/PE web interface, navigate to Configuration > AppFlow/IPFIX.
  2. Click Enable.

    AppFlow/IPFIX configuration

  3. In the Data Update Interval field, specify the time interval, in minutes, at which the flow reports are exported to AppFlow/IPFIX collector. The maximum interval is 10 minutes.

  4. Select the AppFlow dataset template, you can choose either one of the following dataset templates:
    • TCP only for HDX (AppFlow): The AppFlow dataset template to collect and send multi-hop data of ICA connections to the AppFlow collector.
    • HDX (AppFlow): The AppFlow dataset template to collect and send HDX insight data of ICA connections to AppFlow collector.

    Note

    HDX template is available for Citrix SD-WAN PE and Two Box appliances only.

  5. You can configure up to four AppFlow / IPFIX collectors. For each collector specify the following parameters:

    • IP Address: The IP Address of the external AppFlow / IPFIX collector system.
    • Port: The port number on which the external AppFlow / IPFIX collector system listens. The default value is 4739. You can change the port number depending on the collector used.

    • Application Flow Info (IPFIX): Sends flow records, as per IPFIX templates 611, 615, and 612 to IPFIX collectors.

    • Basic Properties (IPFIX): Sends flow records, as per IPFIX template 613 and 616 to IPFIX collectors.

    • Citrix ADM: Select this to use Citrix ADM as the AppFlow collector.

    Note

    • Citrix ADM currently does not support IPFIX collection.
    • Citrix ADM does not support IPv6 addresses for AppFlow and IPFIX.
    • Citrix ADM User: User name of the Citrix ADM collector

    • Password: Citrix ADM collector password.

    The user name and password are used to seamlessly log in into Citrix ADM and store flow data.

  6. Click Apply Settings.

To configure AppFlow / IPFIX collector using Citrix SD-WAN Center:

  1. In Citrix SD-WAN Center management UI, navigate to Configuration > Appliance Settings.
  2. Navigate to the AppFlow / IPFIX section and choose Include in File.
  3. Select Enable IPFIX / AppFlow Collection.

    Enable ApplFlow/IPFIX collection

  4. In the Data Update Interval field, specify the time interval, in minutes, at which the AppFlow reports are exported to the AppFlow / IPFIX collector.
  5. Select the AppFlow dataset template, you can choose either one of the following dataset templates:
    • TCP only for HDX: The AppFlow dataset template to collect and send multi-hop data of ICA connections to the AppFlow collector.
    • HDX: The AppFlow dataset template to collect and send HDX insight data of ICA connections to AppFlow collector.

    Note

    HDX template is available for Citrix SD-WAN PE and Two Box appliances only.

  6. You can configure up to four AppFlow / IPFIX collectors. For each collector specify the following parameters:

    • IPFIX / AppFlow Collector: The IP Address of the external AppFlow / IPFIX collector system.
    • Port: The port number on which the external AppFlow / IPFIX collector system listens. The default value is 4739. You can change the port number depending on the collector used.
    • Application Flow Info: Sends flow records, as per IPFIX templates 611, 615, and 612 to IPFIX collectors.
    • Basic Properties (IPFIX): Sends flow records, as per IPFIX template 613 and 616 to IPFIX collectors.
    • Citrix ADM: Select this to use Citrix ADM as the AppFlow collector.

      Note

      Citrix ADM currently does not support IPFIX collection.

    • Citrix ADM User: User name of the Citrix ADM collector.
    • Password: Citrix ADM collector password.

      The user name and password are used to seamlessly log in into Citrix ADM and store flow data.

  7. Save and Export the configuration to the managed appliances.

Note

If SD-WAN Center version is lower than 10.2 and SD-WAN appliances version is 10.2 and above then you can observe the following conditions.

  • If local collectors are enabled on the appliances, the AppFlow / IPFIX configuration pushed from SD-WAN center does not affect the existing configuration.
  • If local collectors are not enabled on the appliances, the AppFlow/IPFIX configuration pushed from SD-WAN center will be applied to the appliance.
  • If the global AppFlow/IPFIX configuration is enabled in SD-WAN Center configuration, all the local collectors are enabled on the appliances.

Log files

For troubleshooting issues related to AppFlow / IPFIX export protocols, you can view and download the SDWAN_export.log files. Navigate to Configuration > Logging / Monitoring and select the SDWAN_export.log files.

Logging and monitoring

AppFlow and IPFIX