Citrix ADC

Bot Detection

January 7, 2021

Contributed by:

S S

The Citrix ADC bot management system uses six different techniques to detect the incoming bot traffic. The techniques are used as detection rules to detect the bot type. The techniques are bot allow list, bot block list, IP reputation, device fingerprinting, rate limiting, bot trap, TPS, and CAPTCHA.

Note:

Bot management supports a maximum of 32 configuration entities for block list, allow list, and rate limiting techniques.

Bot white list. A customized list of IP addresses, subnets, and policy expressions that can be bypassed as an allowed list.

Bot black list. A customized list of IP addresses, subnets, and policy expressions that has to be blocked from accessing your web applications.

IP reputation. This rule detects if the incoming bot traffic is from a malicious IP address.

Device fingerprint. This rule detects if the incoming bot traffic has the device fingerprint ID in the incoming request header and browser attributes of an incoming client bot traffic.

Limitation:

Java Script must be enabled in the client browser.

Does not support in headless browsers.

Does not work for XML responses.

Rate limit. This rule rate limits multiple requests coming from the same client.

Bot trap. Detects and blocks automated bots by advertising a trap URL in the client response. The URL appears invisible and not accessible if the client is a human user. The detection technique is effective in blocking attacks from automated bots.

TPS. Detects incoming traffic as bots if the maximum number of requests and percentage increase in requests exceeds the configured time interval.

CAPTCHA. This rule uses a CAPTCHA for mitigating bot attacks. A CAPTCHA is a challenge-response validation to determine if the incoming traffic is from a human user or an automated bot. The validation helps block automated bots that cause security violations to web applications. You can configure CAPTCHA as a bot action in IP reputation and device fingerprint detection techniques.

Now, let us see how you can configure each technique to detect and manage your bot traffic.

How to upgrade your appliance to Citrix ADC CLI-based bot management configuration

If you are upgrading your appliance from an older version (Citrix ADC release 13.0 build 58.32 or earlier), you must first manually convert the existing bot management configuration to the Citrix ADC CLI based bot management configuration only once. Complete the following steps to manually convert your bot management configuration.

After upgrading to the latest version connect to the upgrade tool “upgrade_bot_config.py” by using the following command

At the command prompt, type:

shell "/var/python/bin/python /netscaler/upgrade_bot_config.py > /var/bot_upgrade_commands.txt"
Run the configuration using the following command.

At the command prompt, type:

batch -f /var/bot_upgrade_commands.txt
Save the upgraded configuration.

save ns config

Configure Citrix ADC CLI-based bot management

The bot management configuration enables you to bind one or more bot detection techniques to a specific bot profile. You begin the process by enabling the bot management feature on your appliance. Once you enable, you import the bot signature file into the appliance. After import, you must create a bot profile. You then create a bot policy with the bot profile bound to it for evaluating the incoming traffic as bot and bind the policy globally or to a virtual server.

Note:

If you are upgrading your appliance from an older version, you must first manually convert the existing bot management configuration. For more information, see How to upgrade to Citrix ADC CLI-based bot management configuration section.

You must complete the following steps to configure Citrix ADC-based bot management:

Enable bot management
Import bot signature
Add bot profile
Bind bot profile
Add bot policy
Bind bot policy
Configure bot settings

Enable bot management

Before you can begin, ensure that the Bot Management feature is enabled on the appliance. If you have a new Citrix ADC or VPX, you must enable the feature before you configure it. If you are upgrading a Citrix ADC or VPX appliance from an earlier version of the Citrix ADC software version to the current version, you must need to enable the feature before you configure it. At the command prompt, type:

enable ns feature Bot

Import bot signature

You must import the default signature bot file and bind it to the bot profile. At the command prompt, type:

import bot signature [<src>] <name> [-comment <string>] [-overwrite]

Where, src. Local path to and name of, or URL (protocol, host, path, and file name) for, the file in which to store the imported signature file. Note: The import fails if the object to be imported is on an HTTPS server that requires client certificate authentication for access. Maximum Length: 2047 name. Name to assign to the bot signature file object on the Citrix ADC. This is a mandatory argument. Maximum Length: 31 comment. Any comments to preserve information about the signature file object. Maximum Length: 255. Overwrite. Overwrites the existing file. Note: Use the overwrite option to update the content in the signature file. Alternately, use the update bot signature <name> command to update the signature file on the Citrix ADC appliance

Example

import bot signature http://www.example.com/signature.json signaturefile -comment commentsforbot –overwrite

Note:

You can use the overwrite option to update the content in the signature file. Also, you can use the update bot signature <name> command to update the signature file in the Citrix ADC appliance.

Add bot profile

A bot profile is a collection of profile settings to configure bot management on the appliance. You can configure the settings to perform bot detection.

At the command prompt, type:

Example:

add bot profile profile1 -signature signature -errorURL http://www.example.com/error.html -trapURL /trap.html -whitelist ON -blacklist ON -ratelimit ON -deviceFingerprint ON -deviceFingerprintAction drop -ipReputation ON -trap ON

Bind bot profile

After you create a bot profile, you must bind the bot detection mechanism to the profile.

At the command prompt, type:

Example:

The following example is for binding the IP reputation detection technique to a specific bot profile.

bind bot profile profile5 -ipReputation -category BOTNET -enabled ON -action drop -logMessage message

Add bot policy

You must add the bot policy for evaluating bot traffic.

At the command prompt, type:

add bot policy <name> -rule <expression> -profileName <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]

Where,

Name. Name for the bot policy. Must begin with a letter, number, or the underscore character (_), and must contain only letters, numbers, and the hyphen (-), period (.) pound (#), space ( ), at (@), equals (=), colon (:), and underscore characters. Can be changed after the bot policy is added.

Rule. Expression that the policy uses to determine whether to apply the bot profile on the specified request. This is a mandatory argument. Maximum Length: 1499

profileName. Name of the bot profile to apply if the request matches this bot policy. This is a mandatory argument. Maximum Length: 127

undefAction. Action to perform if the result of policy evaluation is undefined (UNDEF). An UNDEF event indicates an internal error condition. Maximum Length: 127

Comment. Any type of information about this bot policy. Maximum Length: 255

logAction. Name of the log action to use for requests that match this policy. Maximum Length: 127

Example:

add bot policy pol1 –rule "HTTP.REQ.HEADER(\"header\").CONTAINS(\"custom\")" - profileName profile1 -undefAction drop –comment commentforbotpolicy –logAction log1

Bind bot policy global

At the command prompt, type:

bind bot global -policyName <string> -priority <positive_integer> [-gotoPriorityExpression <expression>][-type ( REQ_OVERRIDE | REQ_DEFAULT )] [-invoke (-labelType ( vserver | policylabel ) -labelName <string>) ]

Example:

bind bot global –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Bind bot policy to a virtual server

At the command prompt, type:

bind lb vserver <name>@ ((<serviceName>@ [-weight <positive_integer>] ) | <serviceGroupName>@ | (-policyName <string>@ [-priority <positive_integer>] [-gotoPriorityExpression <expression>] [-type ( REQUEST | RESPONSE )] [-invoke (<labelType> <labelName>) ] ) | -analyticsProfile <string>@)

Example:

bind lb vserver lb-server1 –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Configure bot settings

You can customize the default settings if necessary. At the command prompt, type:

set bot settings [-defaultProfile <string>] [-javaScriptName <string>] [-sessionTimeout <positive_integer>] [-sessionCookieName <string>] [-dfpRequestLimit <positive_integer>] [-signatureAutoUpdate ( ON | OFF )] [-signatureUrl <URL>] [-proxyServer <ip_addr|ipv6_addr|*>] [-proxyPort <port|*>]

Where,

defaultProfile. Profile to use when a connection does not match any policy. Default setting is “, which sends unmatched connections back to the Citrix ADC without attempting to filter them further. Maximum Length: 31

javaScriptName. Name of the JavaScript that the BotNet feature uses in response. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the Citrix ADC CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

sessionTimeout. Session times out, in seconds, after which a user session is terminated.

Minimum value: 1, Maximum value: 65535

sessionCookieName. Name of the SessionCookie that the BotNet feature uses it for tracking. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore (_) symbols. The following requirement applies only to the Citrix ADC CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31

dfpRequestLimit. Number of requests to allow without bot session cookie if device fingerprint is enabled.

Minimum value: 1, Maximum Value: 4294967295

signatureAutoUpdate. Flag used to enable/disable bot auto update signatures.

Possible values: ON, OFF

Default value: OFF

signatureUrl. URL to download the bot signature mapping file from the server.

Default value: https://nsbotsignatures.s3.amazonaws.com/BotSignatureMapping.json.

Maximum Length: 2047

ProxyServer. Proxy Server IP to get updated signatures from AWS.

proxyPort. Proxy Server Port to get updated signatures from AWS. Default value: 8080

Example:

set bot settings –defaultProfile profile1 –javaScriptName json.js –sessionTimeout 1000 –sessionCookieName session

Configuring bot management by using Citrix ADC GUI

You can configure Citrix ADC bot management by first enabling the feature on the appliance. Once you enable, you can create a bot policy to evaluate the incoming traffic as bot and send the traffic to the bot profile. Then, you create a bot profile and then bind the profile to a bot signature. As an alternative, you can also clone the default bot signature file and use the signature file to configure the detection techniques. After creating the signature file, you can import it into the bot profile.

Bot Management Page

Enable bot management feature
Configure bot management settings
Clone Citrix bot default signature
Import Citrix bot signature
Configure bot signature settings
Create bot profile
Create bot policy

Enable bot management feature

Complete the following steps to enable bot management:

On the navigation pane, expand System and then click Settings.
On the Configure Advanced Features page, select the Bot Management check box.
Click OK, and then click Close.

Configure bot management settings for device fingerprint technique

Complete the following step to configure the device fingerprint technique:

Navigate to Security > Citrix Bot Management.
In the details pane, under Settings click Change Citrix Bot Management Settings.
In the Configure Citrix Bot Management Settings, set the following parameters.
1. Default Profile. Select a bot profile.
2. JavaScript Name. Name of the JavaScript file that bot management uses in its response to the client.
3. Session Timeout. Timeout in seconds after which the user session is terminated.
4. Session Cookie. Name of the session cookie that the bot management system uses for tracking.
5. Device Fingerprint Request Limit. Number of requests to allow without a bot session cookie, if device fingerprint is enabled
Click OK.

Clone bot signature file

Complete the following step to clone the bot signature file:

Navigate to Security > Citrix Bot Management and Signatures.
In Citrix Bot Management Signatures page, select the default bot signatures record and click Clone.
In the Clone Bot Signature page, enter a name and edit the signature data.
Click Create.

Import bot signature file

If you have your own signature file, then you can import it as a file, text, or URL. Perform the following the steps to import the bot signature file:

Navigate to Security > Citrix Bot Management and Signatures.
On the Citrix Bot Management Signatures page, import the file as URL, File, or text.
Click Continue.
On the Import Citrix Bot Management Signature page, set the following parameters.
1. Name. Name of the bot signature file.
2. Comment. Brief description about the imported file.
3. Overwrite. Select the check box to allow overwriting of data during file update.
4. Signature Data. Modify signature parameters
Click Done.

Configure bot allow list by using Citrix ADC GUI

This detection technique enables you to bypass URLs that you configure an allowed listed one. Complete the following step to configure an allow list URL:

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click White List.
In the White List section, set the following parameters:
1. Enabled. Select the check box to validate the allow list URLs as part of the detection process.
2. Configure Types. Configure an allow list URL. The URL is bypassed during bot detection. Click Add to add a URL to the bot allow list.
3. In the Configure Citrix Bot Management Profile Whitelist Binding page, set the following parameters:
  1. Type. URL type can be an IPv4 address, subnet IP address, or an IP address matching a policy expression.
  2. Enabled. Select the check box to validate the URL.
  3. Value. URL address.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the log.
  6. Comments. Brief description about the allow list URL.
  7. Click OK.
Click Update.
Click Done.

Configure bot block list by using Citrix ADC GUI

This detection technique enables you to drop the URLs that you configure as block listed one. Complete the following step to configure a block list URL.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click Black List.
In the Black List section, set the following parameters:
1. Enabled. Select the check box to validate block list URLs as part of the detection process.
2. Configure Types. Configure a URL to be part of the bot block list detection process. These URLs are dropped during bot detection. Click Add to add a URL to the bot block list
3. In the Configure Citrix Bot Management Profile Blacklist Binding page, set the following parameters.
  1. Type. URL type can be an IPv4 address, subnet IP address, or IP address.
  2. Enabled. Select the check box to validate the URL.
  3. Value. URL address.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the login.
  6. Comments. Brief description about the block list URL.
  7. Click OK.
Click Update.
Click Done.

Configure IP reputation by using Citrix ADC GUI

This configuration is a pre-requisite for the bot IP reputation feature. The detection technique enables you to identify if there is any malicious activity from an incoming IP address. As part of the configuration, we set different malicious bot categories and associate a bot action to each of it. Complete the following step to configure the IP reputation technique.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click IP Reputation.
On the IP Reputation section, set the following parameters:
1. Enabled. Select the check box to validate incoming bot traffic as part of the detection process.
2. Configure Categories. You can use the IP reputation technique for incoming bot traffic under different categories. Based on the configured category, you can drop or redirect the bot traffic. Click Add to configure a malicious bot category.
3. In the Configure Citrix Bot Management Profile IP Reputation Binding page, set the following parameters:
  1. Category. Select a malicious bot category from the list. Associate a bot action based on category.
  2. Enabled. Select the check box to validate the IP reputation signature detection.
  3. Bot action. Based on the configured category, you can assign no action, drop, redirect, mitigation, or CAPTCHA action.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the log.
  6. Comments. Brief description about the bot category.
Click OK.
Click Update.
Click Done.

Configure bot rate limit by using Citrix ADC GUI

This detection technique enables you to block bots based on the number of requests received within a predefined time from a client IP address, a session, or a configured resource (for example, from a URL). Complete the following step to configure the rate limit technique.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click Rate Limit.
On the Rate Limit section, set the following parameters:
1. Enabled. Select the check box to validate the incoming bot traffic as part of the detection process.
2. Session. Rate limit requests based on a session. Click Add to configure rate limit requests based on a session.
3. In the Configure Citrix Bot Management Signature Rate Limit page, set the following parameters.
  1. Category. Select a malicious bot category from the list. Associate an action based on the category.
  2. Enabled. Select the check box to validate the incoming bot traffic.
  3. Bot action. Choose a bot action for the selected category.
  4. Log. Select the check box to store log entries.
  5. Log Message. Brief description of the log.
  6. Comments. Brief description about the bot category.
  7. Click OK.
Click Update.
Click Done.

Configure device fingerprint technique by using Citrix ADC GUI

This detection technique sends a java script challenge to the client and extracts the device information. Based on device information, the technique drops or bypasses the bot traffic. Follow the steps to configure the detection technique.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a signature file and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click Device Fingerprint.
In the Device Fingerprint section, set the following parameters:
1. Enabled. Set this option to enable the rule.
2. Configuration. For the given device fingerprint, assign no action, drop, or redirect, mitigation, or CAPTCHA action.
3. Log. Select the check box to store log entries.
Click Update.
Click Done.

Configure device fingerprint

Configure device fingerprint technique for mobile (Android) applications

Device fingerprint technique detects an incoming traffic as a bot by inserting a JavaScript script in the HTML response to the client. The JavaScript script when invoked by the browser, it collects browser and client attributes and sends a request to the appliance. The attributes are examined to determine whether the traffic is a Bot or a human.

The detection technique is further extended to detect bots on a mobile (Android) platform. Unlike web applications, in mobile (Android) traffic, bot detection based on JavaScript script do not apply. To detect bots in a mobile network, the technique uses a bot mobile SDK which is integrated with mobile applications on the client-side. The SDK intercepts the mobile traffic, collects device details, and sends the data to the appliance. On the appliance side, the detection technique examines the data and determines whether the connection is from a Bot or a human.

How device fingerprint technique for mobile application works

The following steps explain the bot detection workflow to detect if a request from a mobile device is from a human or a bot.

When a user interacts with a mobile application, the device behavior is recorded by the bot mobile SDK.
Client sends a request to Citrix ADC appliance.
When sending the response, the appliance inserts a bot session cookie with session details, and parameters to collect client parameters.
When the mobile application receives the response, the Citrix bot SDK which is integrated with the mobile application validates the response, retrieves the recorded device fingerprint parameters, and sends it to the appliance.
The device fingerprint detection technique on the appliance side validates the device details and updates the bot session cookie if it is a suspected bot or not.
When the cookie is expired or device fingerprint protection prefers to validate and collect device parameters periodically, the whole procedure or challenge is repeated.

Pre-requisite

To get started with the Citrix ADC device fingerprint detection technique for mobile applications, you must download and install the bot mobile SDK in your mobile application.

Configure fingerprint detection technique for mobile (Android) applications by using the CLI

At the command prompt, type:

set bot profile <profile name> -deviceFingerprintMobile ( NONE | Android )

Example:

set bot profile profile 1 –deviceFingerprintMobile Android

Configure device fingerprint detection technique for mobile (Android) applications by using the GUI

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a file and click Edit.
On the Citrix Bot Management Profile page, click Device Fingerprint under Profile Settings.
In the Configure Bot Mobile SDK section, select the mobile client type.
Click Update and Done.

Bot management device fingerprint detection technique for mobile applications

Configure bot trap technique

The Citrix bot trap technique randomly or periodically inserts a trap URL in the client response. You can also create a trap URL list and add URLs for that The URL appears invisible and not accessible if the client is a human user. However, if the client is an automated bot, the URL is accessible and when accessed, the attacker is categorized as bot and any subsequent request from the bot is blocked. The trap technique is effective in blocking attacks from bots.

The trap URL is an alpha-numeric URL of configurable length and it is auto-generated at configurable interval. Also the technique allows you to configure a trap injection URL for top visited websites or frequently visited websites. By doing this, you can mandate the purpose of injecting the bot trap URL for requests matching the trap injection URL.

Note:

Although the bot trap URL is auto-generated, the Citrix ADC bot management still allows you to configure a customized trap URL in the bot profile. This is done to strengthen the bot detection technique and make it harder for attackers to access the trap URL.

To complete the bot trap configuration, you must complete the following steps.

Enable bot trap URL
Configure bot trap URL in bot profile
Bind bot trap injection URL to bot profile
Configure bot trap URL length and interval in bot settings

Enable bot trap URL protection

Before you can begin, you must ensure the Bot trap URL protection is enabled on the appliance. At the command prompt, type:

enable ns feature Bot

Configure bot trap URL in bot profile

You can configure the bot trap URL and specify a trap action in the bot profile.
At the command prompt, type:

add bot profile <name> -trapURL <string> -trap ( ON | OFF ) -trapAction <trapAction>

Where,

trapURL. URL that Bot protection uses as the Trap URL. Maximum Length: 127

trap. Enable bot trap detection. Possible values: ON, OFF, Default value: OFF

trapAction. Action to be taken based bot detection. Possible values: NONE, LOG, DROP, REDIRECT, RESET, MITIGATION. Default value: NONE

Example:

add bot profile profile1 -trapURL www.bottrap1.com trap ON -trapAction RESET

Bind bot trap injection URL to bot profile

You can configure the bot trap injection URL and bind it to the bot profile. At the command prompt, type:

bind bot profile <profile_name> trapInjectionURL –url <url> -enabled ON|OFF -comment <comment>

Where,

URL. Request URL regex pattern for which the bot trap URL is inserted. Maximum Length: 127

Example:

bind bot profile profile1 trapInjectionURL –url www.example.com –enabled ON –comment insert a trap URL randomly

Configure bot trap URL length and interval in bot settings

You can configure the bot trap URL length and also set the interval to auto generate the bot trap URL. At the command prompt, type:

set bot settings -trapURLAutoGenerate ( ON | OFF ) –trapURLInterval <positive_integer> -trapURLLength <positive_integer>

Where,

trapURLInterval. Time in seconds after which the bot trap URL is updated. Default value: 3600, Minimum value: 300, Maximum value: 86400

trapURLLength. Length of the auto-generated bot trap URL. Default value: 32, Minimum value: 10, Maximum value: 255

Example:

set bot settings -trapURLAutoGenerate ON –trapURLInterval 300 -trapURLLength 60

Configure bot trap URL by using the GUI

Navigate to Security > Citrix Bot Management > Profiles.
In the Citrix Bot Management Profiles page, click Edit to configure the bot trap URL technique.
In the Create Citrix Bot Management Profile page, enter the bot trap URL in the general section.
In the Create Citrix Bot Management Profile page, click Bot Trap from Profile Settings.
In the Bot Trap section, set the following parameters.

a. Enabled. Select the check box to enable bot trap detection b. Description. Brief description about the URL. c. Configure Actions. Action to be taken for bot detected by bot trap access.
In Configure Trap Insertion URLs section, click Add.
In the Configure Citrix Bot Management Profile Bot Trap Binding page, set the following parameters.
1. Trap URL. Type the URL that you want to confirm as the bot trap Injection URL.
2. Enabled. Enable or disable bot trap injection URL.
3. Comment. A brief description about the trap injection URL.
In the Signature Settings section, click Bot Trap.
In the Bot Trap section, set the following parameters:
1. Enabled. Select the check box to enable bot trap detection.
2. In the Configure section, set the following parameters.
  1. Action. Action to be taken for bot detected by bot trap access.
  2. Log. Enable or disable logging for bot trap binding.
Click Update and Done.

Configure bot trap URL settings

Complete the following steps to configure bot trap URL settings:

Navigate to Security > Citrix Bot Management.
In the details pane, under Settings click Change Citrix Bot Management Settings.
In the Configure Citrix Bot Management Settings, set the following parameters.
1. Trap URL Interval. Time in seconds after which the bot trap URL is updated.
2. Trap URL Length. Length of the auto-generated bot trap URL.
Click OK and Done.

Bot trap technique for bot trap URL settings

Configure CAPTCHA for IP reputation and device fingerprint detection

CAPTCHA is an acronym that stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”. CAPTCHA is designed to test if an incoming traffic is from a human user or an automated bot. CAPTCHA helps to block automated bots that cause security violations to web applications. In the ADC appliance, CAPTCHA uses the challenge-response module to identify if the incoming traffic is from a human user and not an automated bot.

Configure bot static signature by using Citrix ADC GUI

This detection technique enables you to identify the user agent info from the browser details. Based on user agent information, the bot is identified as a bad or a good bot and then you assign a bot action to it. Follow the steps to configure the static signature technique.

How CAPTCHA works in Citrix ADC bot management

In Citrix ADC bot management, CAPTCHA validation is configured as a policy action to be run after bot policy is evaluated. The CAPTCHA action is available only for IP reputation and device fingerprint detection techniques. Following are the steps to understand how CAPTCHA works:

If a security violation is observed during IP reputation or device fingerprint bot detection, the ADC appliance sends a CAPTCHA challenge.
The client sends the CAPTCHA response.
The appliance validates the CAPTCHA response and if the CAPTCHA is valid, the request is allowed and it is forwarded to the back-end server.
If the CATCHA response is invalid, the appliance sends a new CAPTCHA challenge until the maximum number of attempts is reached.
If the CAPTCHA response is invalid even after the maximum number of attempts, the appliance drops or redirects the request to the configured error URL.
If you have configured log action, then the appliance stores the request details in the ns.log file.

Configure CAPTCHA settings by using the Citrix ADC GUI

The bot management CAPTCHA action is supported only for IP reputation and device fingerprint detection techniques. Complete the following steps to configure the CAPTCHA settings.

Navigate to Security > Citrix Bot Management and Profiles.
On the Citrix Bot Management Profiles page, select a profile and click Edit.
On the Citrix Bot Management Profile page, go to Signature Settings section and click CAPTCHA.
In CAPTCHA Settings section, click Add to configure CAPTCHA settings to the profile:
In the Configure Citrix Bot Management CAPTCHA page, set the following parameters.
1. URL. Bot URL for which the CAPTCHA action is applied during IP reputation and device fingerprint detection techniques.
2. Enabled. Set this option to enable CAPTCHA support.
3. Grace time. Duration until when no new CAPTCHA challenge is sent after the current valid CAPTCHA response is received.
4. Wait time. Duration taken for the ADC appliance to wait until the client sends the CAPTCHA response.
5. Mute Period. Duration for which the client which sent an incorrect CAPTCHA response must wait until allowed to try next. During this mute period, the ADC appliance does not allow any requests. Range: 60–900 seconds, Recommended: 300 seconds
6. Request Length limit. Length of the request for which the CAPTCHA challenge is sent to the client. If the length is greater than the threshold value, the request is dropped. Default value is 10–3000 bytes.
7. Retry Attempts. Number of attempts the client is allowed to retry to solve the CAPTCHA challenge. Range: 1–10, Recommended: 5.
8. No Action/Drop/Redirect action to be taken if the client fails the CAPTCHA validation.
9. Log. Set this option to store request information from the client when response CAPTCHA fails. The data is store in ns.log file.
10. Comment. A brief description about the CAPTCHA configuration.
Click OK and Done.
Navigate to Security > Citrix Bot Management > Signatures.
On the Citrix Bot Management Signatures page, select a signature file and click Edit.
On the Citrix Bot Management Signature page, go to Signature Settings section and click Bot Signatures.
In the Bot Signatures section, set the following parameters:
Configure Static Signatures. Select a bot static signature record and click Edit to assign a bot action to it.
Click OK.
Click Update Signature.
Click Done.

Bot static signature

Auto update for bot signatures

The bot static signature technique uses a signature lookup table with a list of good bots and bad bots. The bots are categorized based on user-agent string and domain names. If the user-agent string and domain name in incoming bot traffic matches a value in the lookup table, a configured bot action is applied. The bot signature updates are hosted on the AWS cloud and the signature lookup table communicates with the AWS database for signature updates. The auto signature update scheduler runs every 1-hour to check the AWS database and updates the signature table in the Citrix ADC appliance.

The signature auto update URL to configure is, https://nsbotsignatures.s3.amazonaws.com/BotSignatureMapping.json

Note:

You can also configure a proxy server and periodically update signatures from the AWS cloud to the appliance through the proxy. For proxy configuration, you must set the proxy IP address and port address in the bot settings.

How bot signature auto update works

The following diagram shows how the bot signatures are retrieved from the AWS cloud, updated on Citrix ADC, and viewed on Citrix ADM for signature update summary.

Bot signature auto update

The bot signature auto-update scheduler does the following:

Retrieves the mapping file from the AWS URI.
Checks the latest signatures in the mapping file with the existing signatures in the ADC appliance.
Downloads the new signatures from AWS and verifies the signature integrity.
Updates the existing bot signatures with the new signatures in the bot signature file.
Generates an SNMP alert and sends the signature update summary to Citrix ADM.

Configure bot signature auto update

For configuring bot signature auto update, complete the following steps:

Enable bot signature auto update

You must enable the auto update option in the bot settings on the ADC appliance. At the command prompt, type:

set bot settings –signatureAutoUpdate ON

Configure proxy server settings (optional)

If you are accessing the AWS signature database through a proxy server, you must configure the proxy server and port. set bot settings –proxyserver –proxyport

Example:

set bot settings –proxy server 1.1.1.1 –proxyport 1356

Configure bot signature auto update using the Citrix ADC GUI

Complete the following steps to configure bot signature auto update:

Navigate to Security > Citrix Bot Management.
In the details pane, under Settings click Change Citrix Bot Management Settings.
In the Configure Citrix Bot Management Settings, select the Auto Update Signature check box.
Click OK and Close.

Create bot management profile

A bot profile is a collection of bot management settings that are used for detecting the bot type. In a profile, you determine how the Web App Firewall applies each of its filters (or checks) to bot traffic to your websites, and responses from them.

Complete the following steps to configure the bot profile:

Navigate to Security > Citrix Bot Management > Profiles.
In the details pane, click Add.
In the Create Citrix Bot Management Profile page, set the following parameters.
1. Name. Bot profile name.
2. Signature. Name of the bot signature file.
3. Error URL. URL for redirects.
4. Comment. Brief description about the profile.
Click Create and Close.

Create bot policy

The bot policy controls the traffic going to the bot management system and also to control the bot logs sent to the auditlog server. Follow the procedure to configure the bot policy.

Navigate to Security > Citrix Bot Management > Bot Policies.
In the details pane, click Add.
In the Create Citrix Bot Management Policy page, set the following parameters.
1. Name. Name of the Bot policy.
2. Expression. Type the policy expression or rule directly in the text area.
3. Bot Profile. Bot profile to apply the bot policy.
4. Undefined Action. Select an action that you prefer to assign.
5. Comment. Brief description about the policy.
6. Log Action. Audit log message action for logging bot traffic. For more information about audit log action, see Audit logging topic.
Click Create and Close.

Configure bot management profile

Bot Transactions Per second (TPS)

The Transactions Per Second (TPS) bot technique detects incoming traffic as a bot if the number of requests per second (RPS) and percentage increase in RPS exceeds the configured threshold value. The detection technique protects your web applications from automated bots that can cause web scraping activities, brute forcing login, and other malicious attacks.

Note:

The bot technique detects an incoming traffic as bot only if both the parameters are configured and if both values increase beyond the threshold limit. Let us consider a scenario, where the appliance receives many requests coming from a specific URL and you want the Citrix ADC bot management to detect if there is a bot attack. The TPS detection technique examines the number of requests (configured value) coming from the URL within 1 second and the percentage increase (configured value) in the number of requests received within 30 minutes. If the values exceed the threshold limit, the traffic is considered as bot and the appliance runs the configured action.

Configure bot transactions per second (TPS) technique

To configure TPS, you must complete the following steps:

Enable bot TPS
Bind TPS settings to bot management profile

Bind TPS settings to bot management profile

Once you enable the bot TPS feature, you must bind the TPS settings to the bot management profile.

At the command prompt, type:

Example:

bind bot profile profile1 -tps -type RequestURL -threshold 1 -percentage 100000 -action drop -logMessage log

Enable bot transaction per second (TPS)

Before you can begin, you must ensure the Bot TPS feature is enabled on the appliance. At the command prompt, type:

set bot profile profile1 –enableTPS ON

Configure bot transactions per second (TPS) by using the Citrix ADC GUI

Complete the following steps to configure bot transactions per second:

Navigate to Security > Citrix Bot Management > Profiles.
In the Citrix Bot Management Profiles page, select a profile and click Edit.
In the Create Citrix Bot Management Profile page, click TPS under Signature Settings section.
In the TPS section, enable the feature and click Add.
In Configure Citrix Bot Management Profile TPS Binding page, set the following parameters.
1. Type. Input types allowed by the detection technique. Possible values: SOURCE IP, GEOLOCATION, HOST, URL.
  
  SOURCE_IP – TPS based on client IP address.
  
  GEOLOCATION – TPS based on the client’s geographic location.
  
  HOST - TPS based on client requests forwarded to a specific back-end server IP address.
  
  URL – TPS based on client requests coming from a specific URL.
2. Fixed Threshold. Maximum number of requests allowed from a TPS input type within 1 second time interval.
3. Percentage Threshold. Maximum percentage increase in requests from a TPS input type within 30 minute time interval.
4. Action. Action to be taken for bot detected by TPS binding.
5. Log. Enable or disable logging for TPS binding.
6. Log Message. Message to log for bot detected by TPS binding. Maximum Length: 255.
7. Comments. A brief description about the TPS configuration. Maximum Length: 255
Click OK and then Close.

Bot management transactions per second section

The official version of this content is in English. Some of the Citrix documentation content is machine translated for your convenience only. Citrix has no control over machine-translated content, which may contain errors, inaccuracies or unsuitable language. No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, suitability, or correctness of any translations made from the English original into any other language, or that your Citrix product or service conforms to any machine translated content, and any warranty provided under the applicable end user license agreement or terms of service, or any other agreement with Citrix, that the product or service conforms with any documentation shall not apply to the extent that such documentation has been machine translated. Citrix will not be held responsible for any damage or issues that may arise from using machine-translated content.

Bot Detection

January 7, 2021

Contributed by:

S S

January 7, 2021

Contributed by:

S S

How to upgrade your appliance to Citrix ADC CLI-based bot management configuration
Configure Citrix ADC CLI-based bot management
Configuring bot management by using Citrix ADC GUI
Configure bot management settings for device fingerprint technique
Configure bot allow list by using Citrix ADC GUI
Configure bot block list by using Citrix ADC GUI
Configure IP reputation by using Citrix ADC GUI
Configure bot rate limit by using Citrix ADC GUI
Configure device fingerprint technique by using Citrix ADC GUI
Configure device fingerprint technique for mobile (Android) applications
How device fingerprint technique for mobile application works
Configure bot trap technique
Configure CAPTCHA for IP reputation and device fingerprint detection
Configure bot static signature by using Citrix ADC GUI
Create bot management profile
Create bot policy

Bot Detection

How to upgrade your appliance to Citrix ADC CLI-based bot management configuration

Configure Citrix ADC CLI-based bot management

Enable bot management

Import bot signature

Example

Add bot profile

Bind bot profile

Add bot policy

Bind bot policy global

Bind bot policy to a virtual server

Configure bot settings

Configuring bot management by using Citrix ADC GUI

Enable bot management feature

Configure bot management settings for device fingerprint technique

Clone bot signature file

Import bot signature file

Configure bot allow list by using Citrix ADC GUI

Configure bot block list by using Citrix ADC GUI

Configure IP reputation by using Citrix ADC GUI

Configure bot rate limit by using Citrix ADC GUI

Configure device fingerprint technique by using Citrix ADC GUI

Configure device fingerprint technique for mobile (Android) applications

How device fingerprint technique for mobile application works

Pre-requisite

Configure fingerprint detection technique for mobile (Android) applications by using the CLI

Configure device fingerprint detection technique for mobile (Android) applications by using the GUI

Configure bot trap technique

Enable bot trap URL protection

Configure bot trap URL in bot profile

Bind bot trap injection URL to bot profile

Configure bot trap URL length and interval in bot settings

Configure bot trap URL by using the GUI

Configure bot trap URL settings

Configure CAPTCHA for IP reputation and device fingerprint detection

Configure bot static signature by using Citrix ADC GUI

How CAPTCHA works in Citrix ADC bot management

Configure CAPTCHA settings by using the Citrix ADC GUI

Auto update for bot signatures

How bot signature auto update works

Configure bot signature auto update

Enable bot signature auto update

Configure proxy server settings (optional)

Configure bot signature auto update using the Citrix ADC GUI

Create bot management profile

Create bot policy

Bot Transactions Per second (TPS)

Configure bot transactions per second (TPS) technique

Bind TPS settings to bot management profile

Enable bot transaction per second (TPS)

Configure bot transactions per second (TPS) by using the Citrix ADC GUI

In this article