Citrix Application Delivery Management service

Troubleshoot pooled capacity license issues

This section describes how to analyze and troubleshoot common pooled capacity issues.

Check license status

The ADM acts as a licensing server for your ADC pooled capacity license. You can use the ADM GUI to check the status of the license. Navigate to Networks > Licenses > Pooled Capacity > License Usage.

Understand license status

The following table lists the types of license status and what they mean

Status What it means
Allocated The license state is fine.
Allocated: not applied on ADC Citrix ADC might require reboot if license is checked-out or checked-in from ADC, but Citrix ADC hasn’t rebooted yet.
Not allocated License is not allocated in the ADC instance.
Grace Citrix ADC instance is in the license grace period for 30 days
Sync in progress Citrix ADM fetches information from Citrix ADC in a 2-minute intervals. Synchronizing licenses between Citrix ADM and Citrix ADC might take as long as 15 minutes. Citrix ADM might have rebooted or ADM HAS failover is triggered.
Partially allocated Citrix ADC cannot accept the capacity allocated because it might be running at its maximum allocation. For example, Citrix ADC is running with 10 Gbps license pool capacity. When ADC reboots, the 10 Gbps is checked-in back to ADM license server. When Citrix ADC comes back online, it tries to check out the earlier allocated 10 Gbps automatically. Meanwhile, other ADC instances might have checked out that bandwidth. Partially Allocated appears if the license pool does not have enough capacity to allocate complete 10 Gbps or even partial capacity to this ADC.
Not managed Citrix ADC is not added to ADM for manageability. This does not have impact on Citrix ADC licensing, but it can impact license monitoring from ADM.
Not managed Citrix ADC is not added to ADM for manageability. This does not have impact on Citrix ADC licensing, but it can impact license monitoring from ADM.
Connection lost Citrix ADC is not reachable from ADM for manageability. For example, there are network connectivity issues, NITRO is not working, or Citrix ADC password mismatches. If NITRO is not working or Citrix ADC password mismatches, this does not have an impact on Citrix ADC licensing. However, it can impact license monitoring from ADM.

Check server status

This section describes the common server status issues and possible reasons and fixes.

Issue: ADC displays license server as unreachable and and license status changes to grace.

  • Connection to license server (ADM or ADM Service Agent) has severed for more than 15 minutes. Verify if the license server is up and reachable.

  • ADC is in grace mode.

Issue: ADC displays license server status as reachable but user attempt to change allocation has no effect. Clicking Change Allocation returns 0 0. This value might make it appear the configured capacity has been lost.

  • Connection to license server has recently gone down but the ADC still hasn’t missed the second heartbeat. Therefore, it is not in Grace (yet). Verify if the license server is up and reachable.

Issue: ADC displays capacity and instance counts but the license server is Reachable/Unreachable. Clicking Change Allocation returns some numbers but does not account for configured capacity.

  • Connection to license server got restored but the ADC is still to miss the second heartbeat or send the reconnect probe.

Issue: ADC says Cannot connect to license server when configuring Pooled licensing with ADM service

  • Check firewall rules to ensure that port 27000 and 7279 are open.
  • Agent is not registered. For more information, see Getting Started.
  • ADM service does not have license files uploaded. For more information, see Configure Citrix ADC pooled capacity
  • ADM has the wrong license file files.

Check usage report of license

Under Networks > Licenses > Pooled Capacity > License Usage in the ADM GUI, you can see the monthly peak of your license usage. You can use this report to increase your license usage or plan purchase of extra license.

The following are some details how the report is generated and can be used.

Polling: License data is polled from the ADC instances every 15 minutes.

Maintaining peaks per hour: ADM maintains only maximum license usage in an hour, per device.

Reporting: You can generate GUI report for each instance, for a specific time range.

Exporting: You can export reports either as in CSV format or XLS format.

Purging: ADM purges data on the first of every month at 12:10 a.m. The purge period is configurable (the default period is two months).

Counters and statistics for pooled capacity licensing

The following counters, logs, and commands expose the Citrix ADC pooled licensing metrics that indicate the behavior of both ADM and ADC instances in pooled licensing mode.

  • SNMP traps: available from ADC version 13.xx.
  • NSCONMSG counters for rate limiting: available from ADC version 12.1 57.xx.
  • ADM counters ADM Command Actions are available in Citrix ADC Cloud Service.

SNMP traps

You can configure the following SNMP traps v.13 pooled license alarms

  • POOLED-LICENSE-CHECKOUT-FAILURE
  • POOLED-LICENSE-ONGRACE
  • Configure POOLED-LICENSE-PARTIAL

For more information about these alarms, see Citrix ADC SNMP OID Reference.

NSCONMSG Counter

Check the following NCCONMSG counters and what they mean:

  • allnic_err_rl_cpu_pkt_drops: aggregate (all NICs) packet drops after CPU limit was reached
  • allnic_err_rl_pps_pkt_drops: aggregate packet drops system wide after pps limit
  • allnic_err_rl_rate_pkt_drops: aggregate rate drops system wide
  • allnic_err_rl_pkt_drops: cumulative rate limiting drops due to rate, pps, and CPU
  • rl_tot_ssl_rl_enforced: number of times SSL RL was applied (on new SSL connections)
  • rl_tot_ssl_rl_data_limited: number of times SSL throughput limit was reached
  • rl_tot_ssl_rl_sess_limited: number of times SSL TPS limit was reached

ADM counters

When you choose the Run Command Action event action, you can create a command or a script that can be run on Citrix ADM for events matching a particular filter criterion. You can also set the following parameters for the Run Command Action script:

Parameter Description
$source This parameter corresponds to the source IP address of the received event.
$category This parameter corresponds to the type of traps defined under the category of the filter.
$entity This parameter corresponds to the entity instances or counters for which an event has been generated. It can include the counter names for all threshold-related events, entity names for all entity-related events, and certificate names for all certificate-related events.
$severity This parameter corresponds to the severity of the event.
$failureobj The failure object affects the way an event is processed and ensures that the failure object reflects the exact problem as notified. This can be used to track down problems quickly and to identify the reason for failure, instead of simply reporting raw events.

Note

During command execution, these parameters are replaced with actual values.

Troubleshoot pooled capacity license issues