Bot Detection

The Citrix ADC bot management system uses six different techniques to detect the incoming bot traffic. The techniques are used as detection rules to detect the bot type. The techniques are bot white list, bot black list, IP reputation, device fingerprinting, rate limiting, and bot signatures.

Note

Bot management supports a maximum of 32 configuration entities for black list, white list, and rate limiting techniques.

Bot white list. An customized list of IP addresses, subnets, and policy expressions that can be bypassed as a white list.

Bot black list. A customized list of IP addresses, subnets, and policy expressions that has to be blocked from accessing your web applications.

IP reputation. This rule detects if the incoming bot traffic is from a malicious IP address.

Device fingerprinting. This rule detects if the incoming bot traffic has device fingerprint ID in the incoming request header and browser attributes of an incoming client bot traffic.

Rate limiting. This rule rate limits multiple requests coming from the same client.

Bot signatures. This bot rule detects and blocks bots based on signature detection. Helps prevent unauthorized URLs that scrape websites, brute force logins, and bots that probe for vulnerabilities.

Now, let us see how you can configure each technique to detect and manage your bot traffic.

Configure bot management

For configuring bot management, you must complete the following steps:

  1. Enable bot management
  2. Import bot signature
  3. Add bot profile
  4. Add bot policy
  5. Bind bot policy global
  6. Configure bot settings

Enable bot management

Before you can begin, ensure that the Bot Management feature is enabled on the appliance. If you have a new Citrix ADC or VPX, you must enable the feature before you configure it. If you are upgrading a Citrix ADC or VPX appliance from an earlier version of the Citrix ADC software version to the current version, you must need to enable the feature before you configure it. At the command prompt, type:

enable ns feature Bot

Import bot signature

You must import the default signature bot file and bind it to the bot profile. At the command prompt, type:

import bot signature [<src>] <name> [-comment <string>] [-overwrite]

Where, src. Local path to and name of, or URL (protocol, host, path, and file name) for, the file in which to store the imported signature file. Note: The import fails if the object to be imported is on an HTTPS server that requires client certificate authentication for access. Maximum Length: 2047 name. Name to assign to the bot signature file object on the Citrix ADC. This is a mandatory argument. Maximum Length: 31 comment. Any comments to preserve information about the signature file object. Maximum Length: 255. overwrite. Overwrites the existing file. Note: Use the overwrite option to update the content in the signature file. Alternately, use the “update bot signature <name>” command to update the signature file on the Citrix ADC appliance

Example

import bot signature http://www.example.com/signature.json signaturefile -comment commentsforbot –overwrite

Note You can use the overwrite option to update the content in the signature file. Also, you can use the “update bot signature <name>” command to update the signature file in the Citrix ADC appliance.

Add bot profile

A bot detection profile is a collection of bot settings and signature rules to detect bots and protect your appliance from attacks. The bot type can a good bot, bad bot or undetectable bot. The bot signature file is bound to the bot detection profile. The six bot detection categories include bot white list, bot black list, device fingerprinting, IP reputation, rate limiting, and static signature to detect bot traffic into your appliance.

At the command prompt, type:

add bot profile <name> [-signature <string>] [-errorURL <string>] [-comment <string>]

Example:

add bot profile profile1 –signature signaturefile –errorURL www.badbot.com –comment commentsforBot

Add bot policy

You must add the bot policy for evaluating bot traffic. At the command prompt, type:

add bot policy <name> -rule <expression> -profileName <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]

Where, Name. Name for the bot policy. Must begin with a letter, number, or the underscore character (_), and must contain only letters, numbers, and the hyphen (-), period (.) pound (#), space ( ), at (@), equals (=), colon (:), and underscore characters. Can be changed after the bot policy is added. Rule. Expression that the policy uses to determine whether to apply bot profile on the specified request. This is a mandatory argument. Maximum Length: 1499 profileName. Name of the bot profile to apply if the request matches this bot policy. This is a mandatory argument. Maximum Length: 127 undefAction. Action to perform if the result of policy evaluation is undefined (UNDEF). An UNDEF event indicates an internal error condition. Maximum Length: 127 comment. Any type of information about this bot policy. Maximum Length: 255 logAction. Name of the messagelog action to use for requests that match this policy. Maximum Length: 127

Example:

add bot policy pol1 –rule "HTTP.REQ.HEADER(\"header\").CONTAINS(\"custom\")" - profileName profile1 -undefAction drop –comment commentforbotpolicy –logAction log1

Bind bot policy global

At the command prompt, type:

bind bot global -policyName <string> -priority <positive_integer> [-gotoPriorityExpression <expression>][-type ( REQ_OVERRIDE | REQ_DEFAULT )] [-invoke (-labelType ( vserver | policylabel ) -labelName <string>) ]

Example:

Bind bot global –policyName pol1 –priority 100 –gotoPriorityExpression NEXT -type REQ_OVERRIDE

Bot settings

You can customize the default settings if necessary. At the command prompt, type:

set bot settings [-defaultProfile <string>] [-javaScriptName <string>] [-sessionTimeout] positive_integer>] [-sessionCookieName <string>] [-dfpRequestLimit <positive_integer>]

Where, defaultProfile. Profile to use when a connection does not match any policy. Default setting is “ “, which sends unmatched connections back to the Citrix ADC without attempting to filter them further. Maximum Length: 31 javaScriptName. Name of the JavaScript that the BotNet feature uses in response. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore () symbols. The following requirement applies only to the Citrix ADC CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31 sessionTimeout. Timeout, in seconds, after which a user session is terminated. Minimum value: 1, Maximum value: 65535 sessionCookieName. Name of the SessionCookie that the BotNet feature uses it for tracking. Must begin with a letter or number, and can consist of from 1 to 31 letters, numbers, and the hyphen (-) and underscore () symbols. The following requirement applies only to the Citrix ADC CLI: If the name includes one or more spaces, enclose the name in double or single quotation marks (for example, “my cookie name” or ‘my cookie name’). Maximum Length: 31 dfpRequestLimit. Number of requests to allow without bot session cookie if device fingerprint is enabled Minimum value: 1, Maximum Value: 4294967295

Example:

set bot settings –defaultProfile profile1 –javaScriptName json.js –sessionTimeout 1000 –sessionCookieName session

Configuring bot management by using Citrix ADC GUI

You can configure Citrix ADC bot management by first enabling the feature on the appliance. Once you enable, you can create a bot policy to evaluate the incoming traffic as bot and send the traffic to the bot profile. Then, you create a bot profile and then bind the profile to a bot signature. As an alternative, you can also clone the default bot signature file and use the signature file to configure the detection techniques. After creating the signature file, you can import it into the bot profile. All these steps are performed in the below sequence:

Bot Management Page

  1. Enable bot management feature
  2. Configure bot management settings
  3. Clone Citrix bot default signature
  4. Import Citrix bot signature
  5. Configure bot signature settings
  6. Create bot profile
  7. Create bot policy

Enable bot management feature

Follow the steps given below to enable bot management:

  1. On the navigation pane, expand System and then click Settings.
  2. On the Configure Advanced Features page, select the Bot Management check box.
  3. Click OK, and then click Close.

    Enabling bot management

Configure bot management settings for device fingerprint technique

Follow the steps given below to configure device fingerprint technique:

  1. Navigate to Security > Citrix Bot Management.
  2. In the details pane, under Settings click Change Citrix Bot Management Settings.
  3. In the Configure Citrix Bot Management Settings, set the following parameters.

    1. Default Profile. Select a bot profile.
    2. JavaScript Name. Name of the JavaScript file that bot management uses in its response to the client.
    3. Session Timeout. Timeout in seconds after which the user session is terminated.
    4. Session Cookie. Name of the session cookie that the bot management system uses for tracking.
    5. Device Fingerprint Request Limit. Number of requests to allow without a bot session cookie, if device fingerprint is enabled

    Bot management settings

  4. Click OK.

Clone bot signature file

Follow the steps given below to clone bot signature file:

  1. Navigate to Security > Citrix Bot Management and Signatures.
  2. In Citrix Bot Management Signatures page, select the default bot signatures record and click Clone.
  3. In the Clone Bot Signature page, enter a name and edit the signature data.
  4. Click Create.

    Clone bot signature file

Import bot signature file

If you have your own signature file, then you can import it as a file, text, or URL. Perform the following the steps to import the bot signature file:

  1. Navigate to Security > Citrix Bot Management and Signatures.
  2. On the Citrix Bot Management Signatures page, import the file as URL, File, or text.
  3. Click Continue.

    Import bot signature file

  4. On the Import Citrix Bot Management Signature page, set the following parameters.
    1. Name. Name of the bot signature file.
    2. Comment. Brief description about the imported file.
    3. Overwrite. Select the check box to allow overwriting of data during file update.
    4. Signature Data. Modify signature parameters
  5. Click Done.

    Import bot signature file

Configure bot white list by using Citrix ADC GUI

This detection technique enables you to bypass URLs that you configure a white listed one. Follow the steps below to configure a white list URL:

  1. Navigate to Security > Citrix Bot Management and Signatures.
  2. On the Citrix Bot Management Signatures page, select a signature file and click Edit.
  3. On the Citrix Bot Management Signature page, go to Signature Settings section and click White List.
  4. In the White List section, set the following parameters:
    1. Enabled. Select the check box to validate the white list URLs as part of the detection process.
    2. Configure Types. Configure a whitelist URL. The URL is bypassed during bot detection. Click Add to add a URL to the bot white list.
    3. In the Configure Citrix Bot Management Signature White List page, set the following parameters:
      1. Type. URL type can be an IPv4 address, subnet IP address or an IP address matching a policy expression.
      2. Enabled. Select the check box to validate the URL.
      3. Value. URL address.
      4. Log. Select the check box to store log entries.
      5. Log Message. Brief description of the log.
      6. Comments. Brief description about the white list URL.
      7. Click OK.

    Configure bot white list

  5. Click Update Signature.
  6. Click Done.

    Configure bot white list

Configure bot black list by using Citrix ADC GUI

This detection technique enables you to drop the URLs that you configure as black listed one. Follow the steps below to configure a black list URL.

  1. Navigate to Security > Citrix Bot Management and Signatures.
  2. On the Citrix Bot Management Signatures page, select a signature file and click Edit.
  3. On the Citrix Bot Management Signature page, go to Signature Settings section and click Black List.
  4. In the Black List section, set the following parameters:

    1. Enabled. Select the check box to validate black list URLs as part of the detection process.
    2. Configure Types. Configure a URL to be part of the bot black list detection process. These URLs are dropped during bot detection. Click Add to add a URL to the bot black list
    3. In the Configure Citrix Bot Management Signature Black List page, set the following parameters.

      1. Type. URL type can be an IPv4 address, subnet IP address, or IP address.
      2. Enabled. Select the check box to validate the URL.
      3. Value. URL address.
      4. Log. Select the check box to store log entries.
      5. Log Message. Brief description of the login.
      6. Comments. Brief description about the black list URL.
      7. Click OK.

    Configuring bot black list

  5. Click Update Signature.
  6. Click Done.

    Configuring bot black list

Configuring IP reputation by using Citrix ADC GUI

This detection technique enables you to identify if there is any malicious activity from an incoming IP address. As part of the configuration, we set different malicious bot categories and associate a bot action to each of it. Follow the steps below to configure IP reputation technique.

  1. Navigate to Security > Citrix Bot Management and Signatures.
  2. On the Citrix Bot Management Signatures page, select a signature file and click Edit.
  3. On the Citrix Bot Management Signature page, go to Signature Settings section and click IP Reputation.
  4. On the IP Reputation section, set the following parameters:
    1. Enabled. Select the check box to validate incoming bot traffic as part of the detection process.
    2. Configure Categories. YYou can use IP reputation technique for incoming bot traffic under different categories. Based on the configured category, you can drop or redirect the bot traffic. Click Add to configure a malicious bot category.
    3. In the Configure Citrix Bot Management Signature IP Reputation page, set the following parameters:

      1. Category. Select a malicious bot category from the list. Associate a bot action based on category.
      2. Enabled. Select the check box to validate the IP reputation signature detection.
      3. Bot action. Select a bot action to perform for the selected category.
      4. Log. Select the check box to store log entries.
      5. Log Message. Brief description of the log.
      6. Comments. Brief description about the bot category.
  5. Click OK.
  6. Click Update Signature.
  7. Click Done.

    Configure IP Reputation

Configuring bot rate limit by using Citrix ADC GUI

This detection technique enables you to block bots based on the number of requests received within a predefined time from a client IP address, a session, or a configured resource (for example, from a URL). Follow the steps below to configure rate limit technique.

  1. Navigate to Security > Citrix Bot Management and Signatures.
  2. On the Citrix Bot Management Signatures page, select a signature file and click Edit.
  3. On the Citrix Bot Management Signature page, go to Signature Settings section and click Rate Limit.
  4. On the Rate Limit section, set the following parameters:
    1. Enabled. Select the check box to validate the incoming bot traffic as part of the detection process.
    2. Session. Rate limit requests based on a session. Click Add to configure rate limit requests based on a session.
    3. In the **Configure Citrix Bot Management Signature Rate Limit page, set the following parameters.
      1. Category. Select a malicious bot category from the list. Associate an action based on the category.
      2. Enabled. Select the check box to validate the IP reputation signature detection.
      3. Bot action. Choose a bot action for the selected category.
      4. Log. Select the check box to store log entries.
      5. Log Message. Brief description of the log.
      6. Comments. Brief description about the bot category.
      7. Click OK.

Configure Rate Limit

  1. Click Update Signature.
  2. Click Done.

    Configure Rate Limit

Configuring device fingerprinting by using Citrix ADC GUI

This detection technique sends a java script file to the browser and extracts the device fingerprinting ID in the post body message. Based on the extracted details, the technique drops or bypasses the bot traffic. Follow the steps to configure the detection technique.

  1. Navigate to Security > Citrix Bot Management and Signatures.
  2. On the Citrix Bot Management Signatures page, select a signature file and click Edit.
  3. On the Citrix Bot Management Signature page, go to Signature Settings section and click Device Fingerprint.
  4. In the Device Fingerprint section, set the following parameters:

    1. Enabled. Set this option to enable the rule.
    2. Configuration. For the given device fingerprint, assign no action, drop, or redirect action.
    3. Log. Select the check box to store log entries.
  5. Click Update Signature.
  6. Click Done.

Configure device fingerprint

Configuring bot static signature by using Citrix ADC GUI

This detection technique enables you to identify the user agent info from the browser details. Based on user agent information, the bot is identified as a bad or a good bot and then you assign a bot action to it. Follow the steps below to configure static signature technique.

  1. Navigate to Security > Citrix Bot Management > Signatures.
  2. On the Citrix Bot Management Signatures page, select a signature file and click Edit.
  3. On the Citrix Bot Management Signature page, go to Signature Settings section and click Bot Signatures.
  4. In the Bot Signatures section, set the following parameters:

    1. Configure Static Signatures. Select a bot static signature record and click Edit to assign a bot action to it.
    2. Click OK.
  5. Click Update Signature.
  6. Click Done.

Configure static signature

Create bot management profile

A bot profile is a collection of bot management settings that are used for detecting the bot type. In a profile, you determine how the Web App Firewall applies each of its filters (or checks) to bot traffic to your websites, and responses from them.

Follow the steps given below to configure the bot profile:

  1. Navigate to Security > Citrix Bot Management > Profiles.
  2. In the details pane, click Add.
  3. In the Create Citrix Bot Management Profile page, set the following parameters.

    1. Name. Bot profile name.
    2. Signature. Name of the bot signature file.
    3. Error URL. URL for redirects.
    4. Comment. Brief description about the profile.
  4. Click Create and Close.

Configure bot management profile

Create bot policy

The bot policy controls the traffic going to the bot management system and also to control the bot logs sent to the auditlog server. Follow the procedure to configure the bot policy.

  1. Navigate to Security > Citrix Bot Management > Bot Policies.
  2. In the details pane, click Add.
  3. In the Create Citrix Bot Management Policy page, set the following parameters.
    1. Name. Name of the Bot policy.
    2. Expression. Type the policy expression or rule directly in the text area.
    3. Bot Profile. Bot profile to apply the bot policy.
    4. Undefined Action. Select an action that you prefer to assign.
    5. Comment. Brief description about the policy.
    6. Log Action. Audit log message action for logging bot traffic. For more information about audit log action, see Audit logging topic.
  4. Click Create and Close.

Configure bot management profile