Citrix ADC

Bot Detection

April 27, 2021

Contributed by:

The Citrix ADC bot management system uses six different techniques to detect the incoming bot traffic. The techniques are used as detection rules to detect the bot type. The techniques are bot allow list, bot block list, IP reputation, device fingerprinting, rate limiting, bot trap, TPS, and CAPTCHA.

Note:

Bot management supports a maximum of 32 configuration entities for block list, allow list, and rate limiting techniques.

Bot white list. A customized list of IP addresses, subnets, and policy expressions that can be bypassed as an allowed list.

Bot black list. A customized list of IP addresses, subnets, and policy expressions that has to be blocked from accessing your web applications.

IP reputation. This rule detects if the incoming bot traffic is from a malicious IP address.

Device fingerprint. This rule detects if the incoming bot traffic has the device fingerprint ID in the incoming request header and browser attributes of an incoming client bot traffic.

Limitation:

Java Script must be enabled in the client browser.

Does not support in headless browsers.

Does not work for XML responses.

Bot log expression. The detection technique enables you to capture additional information as log messages. The data can be the name of the user who requested the URL, the source IP address, and the source port from which the user sent the request or data generated from an expression.

Rate limit. This rule rate limits multiple requests coming from the same client.

Bot trap. Detects and blocks automated bots by advertising a trap URL in the client response. The URL appears invisible and not accessible if the client is a human user. The detection technique is effective in blocking attacks from automated bots.

TPS. Detects incoming traffic as bots if the maximum number of requests and percentage increase in requests exceeds the configured time interval.

CAPTCHA. This rule uses a CAPTCHA for mitigating bot attacks. A CAPTCHA is a challenge-response validation to determine if the incoming traffic is from a human user or an automated bot. The validation helps block automated bots that cause security violations to web applications. You can configure CAPTCHA as a bot action in IP reputation and device fingerprint detection techniques.

Now, let us see how you can configure each technique to detect and manage your bot traffic.

How to upgrade your appliance to Citrix ADC CLI-based bot management configuration

If you are upgrading your appliance from an older version (Citrix ADC release 13.0 build 58.32 or earlier), you must first manually convert the existing bot management configuration to the Citrix ADC CLI based bot management configuration only once. Complete the following steps to manually convert your bot management configuration.

After upgrading to the latest version connect to the upgrade tool “upgrade_bot_config.py” by using the following command

At the command prompt, type:

shell "/var/python/bin/python /netscaler/upgrade_bot_config.py > /var/bot_upgrade_commands.txt"
Run the configuration using the following command.

At the command prompt, type:

batch -f /var/bot_upgrade_commands.txt
Save the upgraded configuration.

save ns config

Configure Citrix ADC CLI-based bot management

The bot management configuration enables you to bind one or more bot detection techniques to a specific bot profile. You begin the process by enabling the bot management feature on your appliance. Once you enable, you import the bot signature file into the appliance. After import, you must create a bot profile. You then create a bot policy with the bot profile bound to it for evaluating the incoming traffic as bot and bind the policy globally or to a virtual server.

Note:

If you are upgrading your appliance from an older version, you must first manually convert the existing bot management configuration. For more information, see How to upgrade to Citrix ADC CLI-based bot management configuration section.

You must complete the following steps to configure Citrix ADC-based bot management:

Enable bot management
Import bot signature
Add bot profile
Bind bot profile
Add bot policy
Bind bot policy
Configure bot settings

Enable bot management

Before you can begin, ensure that the Bot Management feature is enabled on the appliance. If you have a new Citrix ADC or VPX, you must enable the feature before you configure it. If you are upgrading a Citrix ADC or VPX appliance from an earlier version of the Citrix ADC software version to the current version, you must need to enable the feature before you configure it. At the command prompt, type:

enable ns feature Bot

Import bot signature

You can import the default signature bot file and bind it to the bot profile. At the command prompt, type:

import bot signature [<src>] <name> [-comment <string>] [-overwrite]

Where, src. Local path to and name of, or URL (protocol, host, path, and file name) for, the file in which to store the imported signature file. Note: The import fails if the object to be imported is on an HTTPS server that requires client certificate authentication for access. Maximum Length: 2047 name. Name to assign to the bot signature file object on the Citrix ADC. This is a mandatory argument. Maximum Length: 31 comment. Any comments to preserve information about the signature file object. Maximum Length: 255. Overwrite. Overwrites the existing file. Note: Use the overwrite option to update the content in the signature file. Alternately, use the update bot signature <name> command to update the signature file on the Citrix ADC appliance

Example

import bot signature http://www.example.com/signature.json signaturefile -comment commentsforbot –overwrite

Note:

You can use the overwrite option to update the content in the signature file. Also, you can use the update bot signature <name> command to update the signature file in the Citrix ADC appliance.

Add bot profile

A bot profile is a collection of profile settings to configure bot management on the appliance. You can configure the settings to perform bot detection.

At the command prompt, type:

Example:

add bot profile profile1 -signature signature -errorURL http://www.example.com/error.html -trapURL /trap.html -whitelist ON -blacklist ON -ratelimit ON -deviceFingerprint ON -deviceFingerprintAction drop -ipReputation ON -trap ON

Bind bot profile

After you create a bot profile, you must bind the bot detection mechanism to the profile.

At the command prompt, type:

Example:

The following example is for binding the IP reputation detection technique to a specific bot profile.

bind bot profile profile5 -ipReputation -category BOTNET -enabled ON -action drop -logMessage message

Add bot policy

You must add the bot policy for evaluating bot traffic.

At the command prompt, type:

add bot policy <name> -rule <expression> -profileName <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]

Where,

Name. Name for the bot policy. Must begin with a letter, number, or the underscore character (_), and must contain only letters, numbers, and the hyphen (-), period (.) pound (#), space ( ), at (@), equals (=), colon (:), and underscore characters. Can be changed after the bot policy is added.

Rule. Expression that the policy uses to determine whether to apply the bot profile on the specified request. This is a mandatory argument. Maximum Length: 1499

profileName. Name of the bot profile to apply if the request matches this bot policy. This is a mandatory argument. Maximum Length: 127

undefAction. Action to perform if the result of policy evaluation is undefined (UNDEF). An UNDEF event indicates an internal error condition. Maximum Length: 127

Comment. Any type of information about this bot policy. Maximum Length: 255

logAction. Name of the log action to use for requests that match this policy. Maximum Length: 127

Example:

add bot policy pol1 –rule "HTTP.REQ.HEADER(\"header\").CONTAINS(\"custom\")" - profileName profile1 -undefAction drop –comment commentforbotpolicy –logAction log1

Bind bot policy global

At the command prompt, type:

bind bot global -policyName <string> -priority <positive_integer> [-gotoPriorityExpression <expression>][-type ( REQ_OVERRIDE | REQ_DEFAULT )] [-invoke (-labelType ( vserver | policylabel ) -labelName <string>) ]

Example:

bind bot global –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Bind bot policy to a virtual server

At the command prompt, type:

bind lb vserver <name>@ ((<serviceName>@ [-weight <positive_integer>] ) | <serviceGroupName>@ | (-policyName <string>@ [-priority <positive_integer>] [-gotoPriorityExpression <expression>] [-type ( REQUEST | RESPONSE )] [-invoke (<labelType> <labelName>) ] ) | -analyticsProfile <string>@)

Example:

bind lb vserver lb-server1 –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Configure bot settings

You can customize the default settings if necessary. At the command prompt, type:

set bot settings [-defaultProfile <string>] [-javaScriptName <string>] [-sessionTimeout <positive_integer>] [-sessionCookieName <string>] [-dfpRequestLimit <positive_integer>] [-signatureAutoUpdate ( ON | OFF )] [-signatureUrl <URL>] [-proxyServer <ip_addr|ipv6_addr|*>] [-proxyPort <port|*>]

Where,

defaultProfile. Profile to use when a connection does not match any policy. Default setting is “, which sends unmatched connections back to the Citrix ADC without attempting to filter them further. Maximum Length: 31

javaScriptName. Name of the JavaScript that the BotNet feature uses in response. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the Citrix ADC CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

sessionTimeout. Session times out, in seconds, after which a user session is terminated.

Minimum value: 1, Maximum value: 65535

sessionCookieName. Name of the SessionCookie that the BotNet feature uses it for tracking. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the Citrix ADC CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

dfpRequestLimit. Number of requests to allow without bot session cookie if device fingerprint is enabled.

Minimum value: 1, Maximum Value: 4294967295

signatureAutoUpdate. Flag used to enable/disable bot auto update signatures.

Possible values: ON, OFF

Default value: OFF

signatureUrl. URL to download the bot signature mapping file from the server.

Default value: https://nsbotsignatures.s3.amazonaws.com/BotSignatureMapping.json.

Maximum Length: 2047

ProxyServer. Proxy Server IP to get updated signatures from AWS.

proxyPort. Proxy Server Port to get updated signatures from AWS. Default value: 8080

Example:

set bot settings –defaultProfile profile1 –javaScriptName json.js –sessionTimeout 1000 –sessionCookieName session

Configuring bot management by using Citrix ADC GUI

You can configure Citrix ADC bot management by first enabling the feature on the appliance. Once you enable, you can create a bot policy to evaluate the incoming traffic as bot and send the traffic to the bot profile. Then, you create a bot profile and then bind the profile to a bot signature. As an alternative, you can also clone the default bot signature file and use the signature file to configure the detection techniques. After creating the signature file, you can import it into the bot profile.

Bot Management Page

Enable bot management feature
Configure bot management settings
Clone Citrix bot default signature
Import Citrix bot signature
Configure bot signature settings
Create bot profile
Create bot policy

Enable bot management feature

Complete the following steps to enable bot management:

On the navigation pane, expand System and then click Settings.
On the Configure Advanced Features page, select the Bot Management check box.
Click OK, and then click Close.

Configure bot management settings for device fingerprint technique

Complete the following step to configure the device fingerprint technique:

Navigate to Security > Citrix Bot Management.
In the details pane, under Settings click Change Citrix Bot Management Settings.
In the Configure Citrix Bot Management Settings, set the following parameters.
1. Default Profile. Select a bot profile.
2. JavaScript Name. Name of the JavaScript file that bot management uses in its response to the client.
3. Session Timeout. Timeout in seconds after which the user session is terminated.
4. Session Cookie. Name of the session cookie that the bot management system uses for tracking.
5. Device Fingerprint Request Limit. Number of requests to allow without a bot session cookie, if device fingerprint is enabled
Click OK.

Clone bot signature file

Complete the following step to clone the bot signature file:

Navigate to Security > Citrix Bot Management and Signatures.
In Citrix Bot Management Signatures page, select the default bot signatures record and click Clone.
In the Clone Bot Signature page, enter a name and edit the signature data.
Click Create.

Import bot signature file

If you have your own signature file, then you can import it as a file, text, or URL. Perform the following the steps to import the bot signature file:

Navigate to Security > Citrix Bot Management and Signatures.
On the Citrix Bot Management Signatures page, import the file as URL, File, or text.
Click Continue.
On the Import Citrix Bot Management Signature page, set the following parameters.
1. Name. Name of the bot signature file.
2. Comment. Brief description about the imported file.
3. Overwrite. Select the check box to allow overwriting of data during file update.
4. Signature Data. Modify signature parameters
Click Done.

Configure bot allow list by using Citrix ADC GUI

This detection technique enables you to bypass URLs that you configure an allowed listed one. Complete the following step to configure an allow list URL:

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click White List.
In the White List section, set the following parameters:
1. Enabled. Select the check box to validate the allow list URLs as part of the detection process.
2. Configure Types. Configure an allow list URL. The URL is bypassed during bot detection. Click Add to add a URL to the bot allow list.
3. In the Configure Citrix Bot Management Profile Whitelist Binding page, set the following parameters:
  1. Type. URL type can be an IPv4 address, subnet IP address, or an IP address matching a policy expression.
  2. Enabled. Select the check box to validate the URL.
  3. Value. URL address.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the log.
  6. Comments. Brief description about the allow list URL.
  7. Click OK.
Click Update.
Click Done.

Configure bot block list by using Citrix ADC GUI

This detection technique enables you to drop the URLs that you configure as block listed one. Complete the following step to configure a block list URL.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click Black List.
In the Black List section, set the following parameters:
1. Enabled. Select the check box to validate block list URLs as part of the detection process.
2. Configure Types. Configure a URL to be part of the bot block list detection process. These URLs are dropped during bot detection. Click Add to add a URL to the bot block list
3. In the Configure Citrix Bot Management Profile Blacklist Binding page, set the following parameters.
  1. Type. URL type can be an IPv4 address, subnet IP address, or IP address.
  2. Enabled. Select the check box to validate the URL.
  3. Value. URL address.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the login.
  6. Comments. Brief description about the block list URL.
  7. Click OK.
Click Update.
Click Done.

Configure IP reputation by using Citrix ADC GUI

This configuration is a pre-requisite for the bot IP reputation feature. The detection technique enables you to identify if there is any malicious activity from an incoming IP address. As part of the configuration, we set different malicious bot categories and associate a bot action to each of it. Complete the following step to configure the IP reputation technique.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click IP Reputation.
On the IP Reputation section, set the following parameters:
1. Enabled. Select the check box to validate incoming bot traffic as part of the detection process.
2. Configure Categories. You can use the IP reputation technique for incoming bot traffic under different categories. Based on the configured category, you can drop or redirect the bot traffic. Click Add to configure a malicious bot category.
3. In the Configure Citrix Bot Management Profile IP Reputation Binding page, set the following parameters:
  1. Category. Select a malicious bot category from the list. Associate a bot action based on category.
  2. Enabled. Select the check box to validate the IP reputation signature detection.
  3. Bot action. Based on the configured category, you can assign no action, drop, redirect, mitigation, or CAPTCHA action.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the log.
  6. Comments. Brief description about the bot category.
Click OK.
Click Update.
Click Done.

Configure bot rate limit by using Citrix ADC GUI

This detection technique enables you to block bots based on the number of requests received within a predefined time from a client IP address, a session, or a configured resource (for example, from a URL). Complete the following step to configure the rate limit technique.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click Rate Limit.
On the Rate Limit section, set the following parameters:
1. Enabled. Select the check box to validate the incoming bot traffic as part of the detection process.
2. Session. Rate limit requests based on a session. Click Add to configure rate limit requests based on a session.
3. In the Configure Citrix Bot Management Signature Rate Limit page, set the following parameters.
  1. Category. Select a malicious bot category from the list. Associate an action based on the category.
  2. Enabled. Select the check box to validate the incoming bot traffic.
  3. Bot action. Choose a bot action for the selected category.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the log.
  6. Comments. Brief description about the bot category.
  7. Click OK.
Click Update.
Click Done.

Configure device fingerprint technique by using Citrix ADC GUI

This detection technique sends a java script challenge to the client and extracts the device information. Based on device information, the technique drops or bypasses the bot traffic. Follow the steps to configure the detection technique.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click Device Fingerprint.
In the Device Fingerprint section, set the following parameters:
1. Enabled. Set this option to enable the rule.
2. Configuration. For the given device fingerprint, assign no action, drop, or redirect, mitigation, or CAPTCHA action.
3. Log. Select the check box to store log entries.
Click Update.
Click Done.

Configure device fingerprint

Configure device fingerprint technique for mobile (Android) applications

Device fingerprint technique detects an incoming traffic as a bot by inserting a JavaScript script in the HTML response to the client. The JavaScript script when invoked by the browser, it collects browser and client attributes and sends a request to the appliance. The attributes are examined to determine whether the traffic is a Bot or a human.

The detection technique is further extended to detect bots on a mobile (Android) platform. Unlike web applications, in mobile (Android) traffic, bot detection based on JavaScript script do not apply. To detect bots in a mobile network, the technique uses a bot mobile SDK which is integrated with mobile applications on the client-side. The SDK intercepts the mobile traffic, collects device details, and sends the data to the appliance. On the appliance side, the detection technique examines the data and determines whether the connection is from a Bot or a human.

How device fingerprint technique for mobile application works

The following steps explain the bot detection workflow to detect if a request from a mobile device is from a human or a bot.

When a user interacts with a mobile application, the device behavior is recorded by the bot mobile SDK.
Client sends a request to Citrix ADC appliance.
When sending the response, the appliance inserts a bot session cookie with session details, and parameters to collect client parameters.
When the mobile application receives the response, the Citrix bot SDK which is integrated with the mobile application validates the response, retrieves the recorded device fingerprint parameters, and sends it to the appliance.
The device fingerprint detection technique on the appliance side validates the device details and updates the bot session cookie if it is a suspected bot or not.
When the cookie is expired or device fingerprint protection prefers to validate and collect device parameters periodically, the whole procedure or challenge is repeated.

Pre-requisite

To get started with the Citrix ADC device fingerprint detection technique for mobile applications, you must download and install the bot mobile SDK in your mobile application.

Configure fingerprint detection technique for mobile (Android) applications by using the CLI

At the command prompt, type:

set bot profile <profile name> -deviceFingerprintMobile ( NONE | Android )

Example:

set bot profile profile 1 –deviceFingerprintMobile Android

Configure device fingerprint detection technique for mobile (Android) applications by using the GUI

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a file and click Edit.
On the Citrix Bot Management Profile page, click Device Fingerprint under Profile Settings.
In the Configure Bot Mobile SDK section, select the mobile client type.
Click Update and Done.

Bot management device fingerprint detection technique for mobile applications

Configure bot log expression

If the client is identified as a bot, the Citrix bot management enables you to capture additional information as log messages. The data can be the name of the user who requested the URL, the source IP address, and the source port from which the user sent the request or data generated from an expression. To perform custom logging, you must configure a log expression in the bot management profile.

Bind the log expression in the bot profile by using the CLI

At the command prompt, type:

bind bot profile <name> (-logExpression -name <string> -expression <expression> [-enabled ( ON | OFF )]) -comment <string>

Example:

bind bot profile profile1 –logExpression exp1 –expression HTTP.REQ.URL –enabled ON -comment "testing log expression"

Bind log expression to bot profile by using the GUI

Navigate to Security > Citrix Bot Management > Profiles.
On the Citrix Bot Management Profiles page, select Bot Log Expressions from Profile Settings section.
In the Bot Log Expression Settings* section, click **Add.
In the Configure Citrix Bot Management Profile Bot Log Expression Binding page, set the following parameters.
1. Log Expression Name. Name of the log expression.
2. Expression. Enter the log expression.
3. Enabled. Enable or disable the log expression binding.
4. Comments. A brief description about the bot log expression binding.
ClickOK and Done.

Bot log expression

Configure bot trap technique

The Citrix bot trap technique randomly or periodically inserts a trap URL in the client response. You can also create a trap URL list and add URLs for that The URL appears invisible and not accessible if the client is a human user. However, if the client is an automated bot, the URL is accessible and when accessed, the attacker is categorized as bot and any subsequent request from the bot is blocked. The trap technique is effective in blocking attacks from bots.

The trap URL is an alpha-numeric URL of configurable length and it is auto-generated at configurable interval. Also the technique allows you to configure a trap injection URL for top visited websites or frequently visited websites. By doing this, you can mandate the purpose of injecting the bot trap URL for requests matching the trap injection URL.

Note:

Although the bot trap URL is auto-generated, the Citrix ADC bot management still allows you to configure a customized trap URL in the bot profile. This is done to strengthen the bot detection technique and make it harder for attackers to access the trap URL.

To complete the bot trap configuration, you must complete the following steps.

Enable bot trap URL
Configure bot trap URL in bot profile
Bind bot trap injection URL to bot profile
Configure bot trap URL length and interval in bot settings

Enable bot trap URL protection

Before you can begin, you must ensure the Bot trap URL protection is enabled on the appliance. At the command prompt, type:

enable ns feature Bot

Configure bot trap URL in bot profile

You can configure the bot trap URL and specify a trap action in the bot profile.
At the command prompt, type:

add bot profile <name> -trapURL <string> -trap ( ON | OFF ) -trapAction <trapAction>

Where,

trapURL. URL that Bot protection uses as the Trap URL. Maximum Length: 127

trap. Enable bot trap detection. Possible values: ON, OFF, Default value: OFF

trapAction. Action to be taken based bot detection. Possible values: NONE, LOG, DROP, REDIRECT, RESET, MITIGATION. Default value: NONE

Example:

add bot profile profile1 -trapURL www.bottrap1.com trap ON -trapAction RESET

Bind bot trap injection URL to bot profile

You can configure the bot trap injection URL and bind it to the bot profile. At the command prompt, type:

bind bot profile <profile_name> trapInjectionURL –url <url> -enabled ON|OFF -comment <comment>

Where,

URL. Request URL regex pattern for which the bot trap URL is inserted. Maximum Length: 127

Example:

bind bot profile profile1 trapInjectionURL –url www.example.com –enabled ON –comment insert a trap URL randomly

Configure bot trap URL length and interval in bot settings

You can configure the bot trap URL length and also set the interval to auto generate the bot trap URL. At the command prompt, type:

set bot settings -trapURLAutoGenerate ( ON | OFF ) –trapURLInterval <positive_integer> -trapURLLength <positive_integer>

Where,

trapURLInterval. Time in seconds after which the bot trap URL is updated. Default value: 3600, Minimum value: 300, Maximum value: 86400

trapURLLength. Length of the auto-generated bot trap URL. Default value: 32, Minimum value: 10, Maximum value: 255

Example:

set bot settings -trapURLAutoGenerate ON –trapURLInterval 300 -trapURLLength 60

Configure bot trap URL by using the GUI

Navigate to Security > Citrix Bot Management > Profiles.
In the Citrix Bot Management Profiles page, click Edit to configure the bot trap URL technique.
In the Create Citrix Bot Management Profile page, enter the bot trap URL in the general section.
In the Create Citrix Bot Management Profile page, click Bot Trap from Profile Settings.
In the Bot Trap section, set the following parameters.

a. Enabled. Select the check box to enable bot trap detection b. Description. Brief description about the URL. c. Configure Actions. Action to be taken for bot detected by bot trap access.
In Configure Trap Insertion URLs section, click Add.
In the Configure Citrix Bot Management Profile Bot Trap Binding page, set the following parameters.
1. Trap URL. Type the URL that you want to confirm as the bot trap Injection URL.
2. Enabled. Enable or disable bot trap injection URL.
3. Comment. A brief description about the trap injection URL.
In the Signature Settings section, click Bot Trap.
In the Bot Trap section, set the following parameters:
1. Enabled. Select the check box to enable bot trap detection.
2. In the Configure section, set the following parameters.
  1. Action. Action to be taken for bot detected by bot trap access.
  2. Log. Enable or disable logging for bot trap binding.
Click Update and Done.

Configure bot trap URL settings

Complete the following steps to configure bot trap URL settings:

Navigate to Security > Citrix Bot Management.
In the details pane, under Settings click Change Citrix Bot Management Settings.
In the Configure Citrix Bot Management Settings, set the following parameters.
1. Trap URL Interval. Time in seconds after which the bot trap URL is updated.
2. Trap URL Length. Length of the auto-generated bot trap URL.
Click OK and Done.

Bot trap technique for bot trap URL settings

Client IP policy expression for bot detection

The Citrix bot management now enables you to configure an advanced policy expression to extract the client IP address from an HTTP request header, HTTP request body, HTTP request URL, or using an advanced policy expression. The extracted value can be used by a bot detection mechanism (such as TPS, bot trap, or rate limit) to detect if the incoming request is a bot.

Note:

If you have not configured a client IP expression, the default or existing source client IP address is used for bot detection. If an expression is configured, then the evaluation result provides the client IP address that can be used for bot detection.

You can configure and use the client IP expression to extract the actual client IP address if the incoming request is coming through a proxy server and if the client IP address is present in the header. By adding this configuration, the appliance can use the bot detection mechanism in providing more security to software clients and servers.

Configure client IP policy expression in bot profile by using the CLI

At the command prompt, type:

add bot profile <name> [-clientIPExpression <expression>]

Example:

add bot profile profile1 –clientIPExpression 'HTTP.REQ.HEADER("X-Forwarded-For") ALT CLIENT.IP.SRC.TYPECAST_TEXT_T'

Configure client IP policy expression in bot profile by using the GUI

Navigate to Security > Citrix Bot Management > Profiles.
In the details pane, click Add.
In the Create Citrix Bot Management Profile page, set the Client IP Expression.
Click Create and Close.

Configure CAPTCHA for IP reputation and device fingerprint detection

CAPTCHA is an acronym that stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”. CAPTCHA is designed to test if an incoming traffic is from a human user or an automated bot. CAPTCHA helps to block automated bots that cause security violations to web applications. In the ADC appliance, CAPTCHA uses the challenge-response module to identify if the incoming traffic is from a human user and not an automated bot.

Configure bot static signatures

This detection technique enables you to identify the user agent info from the browser details. Based on user agent information, the bot is identified as a bad or a good bot and then you assign a bot action to it. Follow the steps below to configure static signature technique:

On the navigation pane, expand Security > Citrix Bot Management > Signatures.
On the Citrix Bot Management Signatures page, select a signature file and click Edit.
On the Citrix Bot Management Signature page, go to Signature Settings section and click Bot Signatures.
In the Bot Signatures section, set the following parameters:
1. Configure Static Signatures. This section has a list of bot static signature records. You can select a record and click Edit to assign a bot action to it.
2. Click OK.
Click Update Signature.
Click Done.

Bot Static Signature

Bot static signature delineation

Citrix ADC bot management protects your web application against bots. Bot static signatures help in identifying good and bad bots based on request parameters such as user-agent in the incoming request.
The list of signatures in the file is huge and also new rules get added and stale ones are removed periodically. As an administrator, you might want to search for a specific signature or list of signatures under a category. To filter signatures easily, the bot signature page provides an enhanced search capability. The search function enables you to find signature rules and configure its property based on one or more signature parameters like action, signature ID, developer and signature name.

Action. Select a bot action that you prefer to configure for a specific category of signature rules. Following are the available action types:

Enable Selected. Enable all the selected signature rules.
Disable Selected. Disable all the selected signatures rules.
Drop Selected. Apply action as “Drop” to all the selected signature rules.
Redirect Selected. Apply action as “Redirect” to all the selected signature rules.
Reset Selected. Apply action as “Reset” to all the selected signature rules.
Log Selected. Apply action as “Log” to all the selected signature rules.
Remove Drop Selected. Unset the drop action to all the selected signature rules.
Remove Redirect Selected. Unset the redirect action to all the selected signature rules.
Remove Reset Selected. Unset the reset action to all the selected signature rules.
Remove Log Selected. Unset the log action to all the selected signature rules.

Category. Select a category to filter signature rules accordingly. Following is the list of categories that are available for sorting signature rules.

Action. Sort based on bot action.
Category. Sort based on bot category.
Developer. Sort based on the host company publisher.
Enabled. Sort based on signature rules that are enabled.
Id. Sort based on signature rule ID.
Log. Sort based on signature rules that has logging enabled.
Name. Sort based on signature rule name.
Type. Sort based on signature type.
Version. Sort based on signature rule version.

Search bot static signature rules based on action and category types by using the Citrix ADC GUI

Navigate to Security > Citrix Bot Management > Signature.
In the details page, click Add.
In the Citrix Bot Management Signatures page, click edit in the Static Signature section.
In the Configure Static Signature section, select a signature action from the drop-down list.
Use the search function to select a category and filter rules accordingly.
Click Update.

Edit bot static signature rule property by using the Citrix ADC GUI

Navigate to Security > Citrix Bot Management > Signature.
In the details page, click Add.
In the Citrix Bot Management Signatures page, click edit in the Static Signature section.
In the Configure Static Signature section, select an action from the drop-down list.
Use the search function to select a category and filter rules accordingly.
From the static signature list, select a signature to modify its property.
Click OK to confirm.

How CAPTCHA works in Citrix ADC bot management

In Citrix ADC bot management, CAPTCHA validation is configured as a policy action to be run after bot policy is evaluated. The CAPTCHA action is available only for IP reputation and device fingerprint detection techniques. Following are the steps to understand how CAPTCHA works:

If a security violation is observed during IP reputation or device fingerprint bot detection, the ADC appliance sends a CAPTCHA challenge.
The client sends the CAPTCHA response.
The appliance validates the CAPTCHA response and if the CAPTCHA is valid, the request is allowed and it is forwarded to the back-end server.
If the CATCHA response is invalid, the appliance sends a new CAPTCHA challenge until the maximum number of attempts is reached.
If the CAPTCHA response is invalid even after the maximum number of attempts, the appliance drops or redirects the request to the configured error URL.
If you have configured log action, then the appliance stores the request details in the ns.log file.

Configure CAPTCHA settings by using the Citrix ADC GUI

The bot management CAPTCHA action is supported only for IP reputation and device fingerprint detection techniques. Complete the following steps to configure the CAPTCHA settings.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a profile and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click CAPTCHA.
In CAPTCHA Settings section, click Add to configure CAPTCHA settings to the profile:
In the Configure Citrix Bot Management CAPTCHA page, set the following parameters.
1. URL. Bot URL for which the CAPTCHA action is applied during IP reputation and device fingerprint detection techniques.
2. Enabled. Set this option to enable CAPTCHA support.
3. Grace time. Duration until when no new CAPTCHA challenge is sent after the current valid CAPTCHA response is received.
4. Wait time. Duration taken for the ADC appliance to wait until the client sends the CAPTCHA response.
5. Mute Period. Duration for which the client which sent an incorrect CAPTCHA response must wait until allowed to try next. During this mute period, the ADC appliance does not allow any requests. Range: 60–900 seconds, Recommended: 300 seconds
6. Request Length limit. Length of the request for which the CAPTCHA challenge is sent to the client. If the length is greater than the threshold value, the request is dropped. Default value is 10–3000 bytes.
7. Retry Attempts. Number of attempts the client is allowed to retry to solve the CAPTCHA challenge. Range: 1–10, Recommended: 5.
8. No Action/Drop/Redirect action to be taken if the client fails the CAPTCHA validation.
9. Log. Set this option to store request information from the client when response CAPTCHA fails. The data is store in ns.log file.
10. Comment. A brief description about the CAPTCHA configuration.
Click OK and Done.
Navigate to Security > Citrix Bot Management > Signatures.
On the Citrix Bot Management Signatures page, select a signature file and click Edit.
On the Citrix Bot Management Signature page, go to Signature Settings section and click Bot Signatures.
In the Bot Signatures section, set the following parameters:
Configure Static Signatures. Select a bot static signature record and click Edit to assign a bot action to it.
Click OK.
Click Update Signature.
Click Done.

Bot static signature

Auto update for bot signatures

The bot static signature technique uses a signature lookup table with a list of good bots and bad bots. The bots are categorized based on user-agent string and domain names. If the user-agent string and domain name in incoming bot traffic matches a value in the lookup table, a configured bot action is applied. The bot signature updates are hosted on the AWS cloud and the signature lookup table communicates with the AWS database for signature updates. The auto signature update scheduler runs every 1-hour to check the AWS database and updates the signature table in the Citrix ADC appliance.

The signature auto update URL to configure is, https://nsbotsignatures.s3.amazonaws.com/BotSignatureMapping.json

Note:

You can also configure a proxy server and periodically update signatures from the AWS cloud to the appliance through the proxy. For proxy configuration, you must set the proxy IP address and port address in the bot settings.

How bot signature auto update works

The following diagram shows how the bot signatures are retrieved from the AWS cloud, updated on Citrix ADC, and viewed on Citrix ADM for signature update summary.

Bot signature auto update

The bot signature auto-update scheduler does the following:

Retrieves the mapping file from the AWS URI.
Checks the latest signatures in the mapping file with the existing signatures in the ADC appliance.
Downloads the new signatures from AWS and verifies the signature integrity.
Updates the existing bot signatures with the new signatures in the bot signature file.
Generates an SNMP alert and sends the signature update summary to Citrix ADM.

Configure bot signature auto update

For configuring bot signature auto update, complete the following steps:

Enable bot signature auto update

You must enable the auto update option in the bot settings on the ADC appliance. At the command prompt, type:

set bot settings –signatureAutoUpdate ON

Configure proxy server settings (optional)

If you are accessing the AWS signature database through a proxy server, you must configure the proxy server and port. set bot settings –proxyserver –proxyport

Example:

set bot settings –proxy server 1.1.1.1 –proxyport 1356

Configure bot signature auto update using the Citrix ADC GUI

Complete the following steps to configure bot signature auto update:

Navigate to Security > Citrix Bot Management.
In the details pane, under Settings click Change Citrix Bot Management Settings.
In the Configure Citrix Bot Management Settings, select the Auto Update Signature check box.
Click OK and Close.

Create bot management profile

A bot profile is a collection of bot management settings that are used for detecting the bot type. In a profile, you determine how the Web App Firewall applies each of its filters (or checks) to bot traffic to your websites, and responses from them.

Complete the following steps to configure the bot profile:

Navigate to Security > Citrix Bot Management > Profiles.
In the details pane, click Add.
In the Create Citrix Bot Management Profile page, set the following parameters.
1. Name. Bot profile name.
2. Signature. Name of the bot signature file.
3. Error URL. URL for redirects.
4. Comment. Brief description about the profile.
Click Create and Close.

Create bot policy

The bot policy controls the traffic going to the bot management system and also to control the bot logs sent to the auditlog server. Follow the procedure to configure the bot policy.

Navigate to Security > Citrix Bot Management > Bot Policies.
In the details pane, click Add.
In the Create Citrix Bot Management Policy page, set the following parameters.
1. Name. Name of the Bot policy.
2. Expression. Type the policy expression or rule directly in the text area.
3. Bot Profile. Bot profile to apply the bot policy.
4. Undefined Action. Select an action that you prefer to assign.
5. Comment. Brief description about the policy.
6. Log Action. Audit log message action for logging bot traffic. For more information about audit log action, see Audit logging topic.
Click Create and Close.

Configure bot management profile

Bot Transactions Per second (TPS)

The Transactions Per Second (TPS) bot technique detects incoming traffic as a bot if the number of requests per second (RPS) and percentage increase in RPS exceeds the configured threshold value. The detection technique protects your web applications from automated bots that can cause web scraping activities, brute forcing login, and other malicious attacks.

Note:

The bot technique detects an incoming traffic as bot only if both the parameters are configured and if both values increase beyond the threshold limit. Let us consider a scenario, where the appliance receives many requests coming from a specific URL and you want the Citrix ADC bot management to detect if there is a bot attack. The TPS detection technique examines the number of requests (configured value) coming from the URL within 1 second and the percentage increase (configured value) in the number of requests received within 30 minutes. If the values exceed the threshold limit, the traffic is considered as bot and the appliance runs the configured action.

Configure bot transactions per second (TPS) technique

To configure TPS, you must complete the following steps:

Enable bot TPS
Bind TPS settings to bot management profile

Bind TPS settings to bot management profile

Once you enable the bot TPS feature, you must bind the TPS settings to the bot management profile.

At the command prompt, type:

Example:

bind bot profile profile1 -tps -type RequestURL -threshold 1 -percentage 100000 -action drop -logMessage log

Enable bot transaction per second (TPS)

Before you can begin, you must ensure the Bot TPS feature is enabled on the appliance. At the command prompt, type:

set bot profile profile1 –enableTPS ON

Configure bot transactions per second (TPS) by using the Citrix ADC GUI

Complete the following steps to configure bot transactions per second:

Navigate to Security > Citrix Bot Management > Profiles.
In the Citrix Bot Management Profiles page, select a profile and click Edit.
In the Create Citrix Bot Management Profile page, click TPS under Signature Settings section.
In the TPS section, enable the feature and click Add.
In Configure Citrix Bot Management Profile TPS Binding page, set the following parameters.
1. Type. Input types allowed by the detection technique. Possible values: SOURCE IP, GEOLOCATION, HOST, URL.
  
  SOURCE_IP – TPS based on client IP address.
  
  GEOLOCATION – TPS based on the client’s geographic location.
  
  HOST - TPS based on client requests forwarded to a specific back-end server IP address.
  
  URL – TPS based on client requests coming from a specific URL.
2. Fixed Threshold. Maximum number of requests allowed from a TPS input type within 1 second time interval.
3. Percentage Threshold. Maximum percentage increase in requests from a TPS input type within 30 minute time interval.
4. Action. Action to be taken for bot detected by TPS binding.
5. Log. Enable or disable logging for TPS binding.
6. Log Message. Message to log for bot detected by TPS binding. Maximum Length: 255.
7. Comments. A brief description about the TPS configuration. Maximum Length: 255
Click OK and then Close.

Bot management transactions per second section

Bot detection based on mouse and keyboard dynamics

To detect bots and mitigate web scraping anomalies, the Citrix ADC bot management uses an enhanced bot detection technique based on mouse and keyboard behavior. Unlike conventional bot techniques that require direct human interaction (for example, CAPTCHA validation), the enhanced technique passively monitors the mouse and the keyboard dynamics. The Citrix ADC appliance then collects the real-time user data and analyses behavioral between a human and a bot.

The passive bot detection using mouse and keyboard dynamics has the following benefits over existing bot detection mechanisms:

Provides continuous monitoring throughout the user session, and eliminates single checkpoint.
Requires no human interaction and it is completely transparent to users.

How bot detection using mouse and keyboard dynamics works

The bot detection technique using keyboard and mouse dynamics consists of two components, a webpage logger and bot detector. The webpage logger is a JavaScript that records keyboard and mouse movements when a user is performing a task on the webpage (for example, filling a registration form). The logger then sends the data in batches to the Citrix ADC appliance. The appliance then stores the data as a KM record and sends it to the bot detector on the Citrix ADM server, which analyses if the user is a human or bot.

The following steps explain how the components interact with each other:

The Citrix ADC admin configures policy expression through ADM Stylebook, CLI, or NITRO or any other method.
The URL is set in the bot profile when the admin enables the feature on the appliance.
When a client sends a request, the Citrix ADC appliance tracks the session and all requests in the session.
The appliance inserts a JavaScript (webpage logger) in the response if the request matches the configured expression on the bot profile.
The JavaScript then collects all the keyboard, mouse activity and sends the KM data in a POST URL (transient).
The Citrix ADC appliance stores the data and sends it to the Citrix ADM server at the end of the session. Once the appliance receives the complete data of a POST request, the data is sent it to ADM server.
The CAS platform analyses the data and based on the analysis, the result is available on the Citrix ADM GUI.

The JavaScript logger records the following mouse and keyboard movements:

Keyboard events – all events
Mouse events - mouse move, mouse up, mouse down
Clipboard events - paste
Custom events - autofill, autofillcancel
timestamp of each event

Configure bot detection using mouse and keyboard dynamics

The Citrix ADC bot management configuration includes enabling or disabling keyboard and mouse-based detection feature, and configures the JavaScript URL in the bot profile. Complete the following steps to configure bot detection using mouse and keyboard dynamics:

Enable keyboard and mouse-based detection
Configure expression to decide when the JavaScript can be injected in the HTTP response

Enable keyboard mouse-based bot detection

Before you begin the configuration, ensure you have enabled the keyboard and mouse-based bot detection feature on the appliance.

At the command prompt, type:

add bot profile <name> -KMDetection ( ON | OFF )

Example:

add bot profile profile1 –KMDetection ON

Configure bot expression for JavaScript insertion

Configure bot expression to evaluate the traffic and insert JavaScript. The JavaScript is inserted only if the expression is evaluated as true.

At the command prompt, type:

bind bot profile <name> -KMDetectionExpr -name <string> -expression <expression> -enabled ( ON | OFF ) –comment <string>

Example:

bind bot profile profile1 -KMDetectionExpr -name test -expression http.req.url.startswith("/testsite") -enabled ON

Configure JavaScript file name inserted in the HTTP response for keyboard-mouse based bot detection

To collect the user action details, the appliance sends a JavaScript file name in the HTTP response. The JavaScript file collects all the data in a KM record and sends it to the appliance.

At the command prompt, type:

set bot profile profile1 – KMJavaScriptName <string>

Example:

set bot profile profile1 –KMJavaScriptName script1

Configure behavior biometrics size

You can configure the maximum size of mouse and keyboard behavior data that can be sent as KM record to the appliance and processed by ADM server.

At the command prompt, type:

set bot profile profile1 -KMEventsPostBodyLimit <positive_integer>

Example:

set bot profile profile1 – KMEventsPostBodyLimit 25

After you have configured the Citrix ADC appliance to configure the JavaScript and collect keyboard and mouse behavior biometrics, the appliance sends the data to the Citrix ADM server. For more information on how the Citrix ADM server detects bots from behavior biometrics, see Bot Violations topic.

Configure keyboard and mouse bot expression settings by using the GUI

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a profile and click Edit.
In the Keyboard and mouse based bot detection section, set the following parameters:
1. Enable detection. Select the check box to detect the bot based keyboard and mouse dynamics behavior.
2. Event post body limit. Size of the keyboard and mouse dynamics data sent by the browser to be processed by the Citrix ADC appliance.
Click OK.
On the Citrix Bot Management Profile page, go to Profile Settings section and click Keyboard and Mouse Based Bot Expression Settings.
In the Keyboard and Mouse Based Bot Expression Settings section, click Add.
In the Configure Citrix Bot Management Profile Bot Keyboard and Mouse Expression Binding page, set the following parameters:
1. Expression Name. Name of the bot policy expression for detection keyboard and mouse dynamics.
2. Expression. Bot policy expression.
3. Enabled. Select the check box to enable the keyboard and bot keyboard and mouse expression binding.
4. Comments. A brief description about the bot policy expression and its binding to the bot profile.
5. Click OK and Close.
In the Keyboard and Mouse Based Bot Expression Settings section, Click Update.

The official version of this content is in English. Some of the Citrix documentation content is machine translated for your convenience only. Citrix has no control over machine-translated content, which may contain errors, inaccuracies or unsuitable language. No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, suitability, or correctness of any translations made from the English original into any other language, or that your Citrix product or service conforms to any machine translated content, and any warranty provided under the applicable end user license agreement or terms of service, or any other agreement with Citrix, that the product or service conforms with any documentation shall not apply to the extent that such documentation has been machine translated. Citrix will not be held responsible for any damage or issues that may arise from using machine-translated content.

Bot Detection

April 27, 2021

Contributed by:

April 27, 2021

Contributed by:

How to upgrade your appliance to Citrix ADC CLI-based bot management configuration
Configure Citrix ADC CLI-based bot management
Configuring bot management by using Citrix ADC GUI
Configure bot management settings for device fingerprint technique
Configure bot allow list by using Citrix ADC GUI
Configure bot block list by using Citrix ADC GUI
Configure IP reputation by using Citrix ADC GUI
Configure bot rate limit by using Citrix ADC GUI
Configure device fingerprint technique by using Citrix ADC GUI
Configure device fingerprint technique for mobile (Android) applications
How device fingerprint technique for mobile application works
Configure bot log expression
Configure bot trap technique
Configure CAPTCHA for IP reputation and device fingerprint detection
Configure bot static signatures
Create bot management profile
Create bot policy
Bot detection based on mouse and keyboard dynamics

Bot Detection

How to upgrade your appliance to Citrix ADC CLI-based bot management configuration

Configure Citrix ADC CLI-based bot management

Enable bot management

Import bot signature

Example

Add bot profile

Bind bot profile

Add bot policy

Bind bot policy global

Bind bot policy to a virtual server

Configure bot settings

Configuring bot management by using Citrix ADC GUI

Enable bot management feature

Configure bot management settings for device fingerprint technique

Clone bot signature file

Import bot signature file

Configure bot allow list by using Citrix ADC GUI

Configure bot block list by using Citrix ADC GUI

Configure IP reputation by using Citrix ADC GUI

Configure bot rate limit by using Citrix ADC GUI

Configure device fingerprint technique by using Citrix ADC GUI

Configure device fingerprint technique for mobile (Android) applications

How device fingerprint technique for mobile application works

Pre-requisite

Configure fingerprint detection technique for mobile (Android) applications by using the CLI

Configure device fingerprint detection technique for mobile (Android) applications by using the GUI

Configure bot log expression

Bind the log expression in the bot profile by using the CLI

Bind log expression to bot profile by using the GUI

Configure bot trap technique

Enable bot trap URL protection

Configure bot trap URL in bot profile

Bind bot trap injection URL to bot profile

Configure bot trap URL length and interval in bot settings

Configure bot trap URL by using the GUI

Configure bot trap URL settings

Client IP policy expression for bot detection

Configure client IP policy expression in bot profile by using the CLI

Configure client IP policy expression in bot profile by using the GUI

Configure CAPTCHA for IP reputation and device fingerprint detection

Configure bot static signatures

Bot static signature delineation

Search bot static signature rules based on action and category types by using the Citrix ADC GUI

Edit bot static signature rule property by using the Citrix ADC GUI

How CAPTCHA works in Citrix ADC bot management

Configure CAPTCHA settings by using the Citrix ADC GUI

Auto update for bot signatures

How bot signature auto update works

Configure bot signature auto update

Enable bot signature auto update

Configure proxy server settings (optional)

Configure bot signature auto update using the Citrix ADC GUI

Create bot management profile

Create bot policy

Bot Transactions Per second (TPS)

Configure bot transactions per second (TPS) technique

Bind TPS settings to bot management profile

Enable bot transaction per second (TPS)

Configure bot transactions per second (TPS) by using the Citrix ADC GUI

Bot detection based on mouse and keyboard dynamics

How bot detection using mouse and keyboard dynamics works

Configure bot detection using mouse and keyboard dynamics

Enable keyboard mouse-based bot detection

Configure bot expression for JavaScript insertion

Configure JavaScript file name inserted in the HTTP response for keyboard-mouse based bot detection

Configure behavior biometrics size

Configure keyboard and mouse bot expression settings by using the GUI

In this article