Product Documentation

URL List

The URL List feature enables you to control access to customized URL lists (up to one million entries). The feature filters websites by applying a URL filtering policy bound to a virtual server.

As an administrator, you must import the URL List into the Citrix ADC appliance. This imported list is internally stored as a Policy data set called a URL Set. The appliance then applies a unique fast URL matching algorithm to the incoming URL requests. If the incoming URL request matches an entry in the set, the appliance applies the associated policy action to control access.

URL List Types

Each entry in a URL set can include a URL and, optionally, its metadata (URL category, category groups, or any other related data). For URLs with a metadata, the appliance uses a policy expression that evaluates the metadata. For more information, see URL Sets.

A Citrix ADC appliance supports two types of URL lists: Custom (with or without URL metadata) and IWF (Internet Watch Foundation).

Custom URL List. You can create a customized URL set of up to 1,000,000 URL entries and import it as a text file into your appliance. The list can contain URLs with or without metadata (which could be like a URL category). TheCitrix ADC platform automatically detects whether metadata is present. It also supports storing the imported lists securely. For more information, see URL Set.

IWF URL List. You can import an externally managed list maintained by the IWF. To access this list, you need special access credentials, and you have to use a secure procedure. Citrix ADC has built-in support for the IWF list, making it simple to use.

You can host the URL list and configure the Citrix ADC appliance to periodically update the list without requiring manual intervention. Once the URL list is updated, the appliance can automatically detect the metadata and the categories by using policy expressions to evaluate each incoming URL and then apply actions such as allow, block, redirect, or notify the user.

URL List Policy Expressions

The following table describes the basic expressions you can use to evaluate incoming traffic. After you import an URL List to the appliance, it is called a URL Set.

Expression Operation
<URL expression>.URLSET_MATCHES_ANY(<URLSET>) Evaluates to TRUE if the URL exactly matches any entry in the URL set.
<URL expression>.GET_URLSET_METADATA(<URLSET>) The GET_URLSET_METADATA() expression returns the associated metadata if the URL exactly matches any pattern within the URL set. An empty string is returned if there is no match.
<URL expression>.GET_ URLSET_METADATA(<URLSET>).EQ(<METADATA>) Evaluates to TRUE if the matched metadata is equal to <METADATA>.
<URLexpression>.GET_URLSET_METADATA(<URLSET>).TYPECAST_LIST_T(‘,’).GET(0).EQ(<CATEGORY>) Evaluates to TRUE if the matched metadata is at the beginning of the category. This pattern can be used to encode separate fields within metadata, but match only the 1<sup>st</sup> field.
HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL) Joins the host and URL parameters, which can then be used as a <URL expression> for matching.

URL List Policy Actions

The most common enforcement action for URLs that match a URL list is to restrict access. Create a URL list policy with a desired URL list matching expression and enforcement action. The policy group usage depends upon the incoming traffic type (HTTP or HTTPS) and the virtual server configured on the appliance. You can use a Responder policy for HTTP traffic or a Video Optimization policy for HTTPS traffic. Specify actio ns to apply to the URLs that match the expressions in the policies. The following table lists the available actions.

Action Type Policy Description
ALLOW Responder Allow the request to access the target URL.
REDIRECT Responder Redirect the request to the URL specified as the target.
DENY Responder Deny the request.
RESET Responder, VideoOptimization Reset the connection.
DROP Responder, VideoOptimization Drop the connection.

Prerequisites

To configure URL List feature, make sure you have configured the following server.

DNS Server for DNS Requests

You must configure a DNS server if you import a URL Set from a hostname URL.

At the command prompt, type:

add dns nameServer ((<IP> [-local]) | <dnsVserverName>) [-state (ENABLED | DISABLED )] [-type <type>] [-dnsProfileName <string>]

Example:

add dns nameServer 10.140.50.5

Configuring a URL List

To configure a URL List, do the following:

  • Import a custom URL list
  • Set up IWF URL list support
  • Configure a URL list for HTTP traffic.
    • Add a URL List action.
    • Add a URL List policy.
    • Add an HTTP load balancing virtual server for HTTP traffic.
    • Bind the URL List policy to the HTTP load Balancing virtual server for HTTP traffic.
  • Configure a URL list for HTTPS traffic.
    • Add a video optimization detection policy.
    • Add an SSL-bridge load balancing virtual server for HTTPS traffic.
    • Bind the video optimization detection policy to the SSL-bridge load balancing virtual server for HTTPS traffic

Importing a custom URL list

To import a URL set, see See URL Set.

Setting up IWF URL List Support

To set up IWF URL list support, you need access credentials. The IWF gives each member the unique access credentials required for setup. The access credentials include a user name password.

At the command prompt, type:

import urlset iwfset -url https://<*IWFUser*>:<*IWFPass*>@servicebak.iwf.org.uk/ -private Done

Note

If the IWF changes the URL for importing a list, you can contact the Citrix ADC support team.

Configuring a URL List for HTTP traffic

The Citrix ADC appliance supports HTTP and HTTPS traffic. To configure a load balancing virtual server for HTTP traffic and bind URL list policies to the server, do the following:

  • Add URL List actions.
  • Add URL List policies.
  • Add an HTTP load balancing virtual server for HTTP traffic
  • Bind the URL List policies to the HTTP load balancing virtual server for HTTP traffic

To add a URL list action

At the command prompt, type the following:

add responder action <name> <type> (<target> | <htmlpage>) [-comment <string>] [-responseStatusCode <positive_integer>] [-reasonPhrase <string>]

Example:

add responder action act_iwf_url respondwith '"HTTP/1.1 451 Unavailable For Legal ReasonsrnrnURL is IWF NOT authorizedn"'

To add a URL list policy

At the command prompt, type the following:

add responder policy <name> <rule> <action> [<undefAction>] [-comment <string>] [-logAction <string>] [-appflowAction <string>]

Example:

add responder policy pol_iwf_http 'HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URLSET_MATCHES_ANY("iwfset")' act_iwf_url

To add a HTTP load balancing virtual server for HTTP traffic

At the command prompt, type the following:

add vserver <name> [-td <positive_integer>] <serviceType> [-cltT imeout <secs>]

Example:

add lb vserver vsrv-HTTP HTTP * 80 -persistenceType NONE -cltTimeout 120

To bind URL list policy to HTTP load balancing virtual server

At the command prompt, type the following:

bind lb vserver <vServerName> -policyName <string> [-priority <positive_integer>]

Example:

bind lb vserver vsrv-HTTP -policyName pol_iwf_http -priority 10 -type REQUEST

Configuring URL List for HTTP traffic

The Citrix ADC appliance supports HTTP and HTTPS traffic. To configure a SSL-bridge load balancing virtual server for HTTPS traffic and bind URL list policies to the server, do the following:

  • Add URL List actions.
  • Add URL List policies.
  • Add a SSL-bridge load balancing virtual server for HTTP traffic
  • Bind the URL List policies to the  SSL-bridge load balancing virtual server for HTTP traffic

To add a URL List policy for HTTPS traffic

At the command prompt, type:

add videooptimization detectionpolicy <name> -rule <expression> -action <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]

Example:

add videooptimization detectionpolicy pol_iwf_https -rule CLIENT.SSL.DETECTED_DOMAIN.URLSET_MATCHES_ANY("iwfset") -action RESET

To add a SSL-bridge load balancing virtual server

At the command prompt type:

add vserver <name> [-td <positive_integer>] <serviceType> [-cltT imeout <secs>]

Example:

add lb vserver vsrv-HTTPS SSL_BRIDGE * 443 -persistenceType NONE -cltTimeout 180

To bind URL List policy with SSL-bridge load balancing by using the CLI

At the command prompt type:

bind lb vserver <vServerName> -policyName <string> [-priority <positive_integer>]

Example:

bind lb vserver vsrv-HTTPS -policyName pol_iwf_https -priority 20 -type REQUEST

Configuring a URL List by using the GUI

The GUI enables you to:

  • Import a URL list.
  • Add a URL list.
  • Configure URL list actions.
  • Configure URL list policies for HTTP traffic.
  • Add an HTTP load balancing virtual server for HTTP traffic.
  • Add an SSL-bridge load balancing virtual server for HTTPS traffic.
  • Bind URL list policies to the HTTP load balancing virtual server.
  • Bind a URL list policies to the SSL-bridge load balancing virtual server.

To import a URL list

  1. In the navigation pane, expand AppExpert > URL Sets.
  2. In the details pane, click Import.
  3. On the Configure URL Set page, set the following parameters.
    1. Name. Name of the URL set.
    2. URL. Web address of the location at which to access the URL Set.
    3. Overwrite. Overwrite a previously imported URL set.
    4. Delimiter. Character sequence that delimits a CSV file record.
    5. Row Separator. Row separator used in the CSV file.
    6. Interval. Interval in seconds, rounded off to the nearest 15 minutes, at which the URL set is updated.
    7. Private Set. Option to prevent exporting the URL set
    8. Canary URL. Internal URL for testing whether the content of the URL set is to be kept confidential. The maximum length of the URL is 2047 characters
  4. Click Create, and then Close.

To add a URL list

  1. In the navigation pane, expand AppExpert > URL Sets.
  2. In the details pane, click Add.
  3. On the Create URL Set page, set the following parameters.
    1. Name. The name of the URL set that was given when it was imported.
    2. Comments. A short description about the URL set.
  4. Click Create.

To configure a URL list action

  1. In the navigation pane, navigate to AppExpert > Responder > Action.
  2. In the details pane, click Add.
  3. On the Create Responder Action page, set the following parameters.
    1. Name. Name of the URL List policy action.
    2. Type. Select an action type.
    3. Expression. Use the expression editor to create the policy expression.
    4. Comments. A short description about the policy action.
  4. Click Create and Close.

To configure a URL list policy

  1. In the navigation pane, expand AppExpert > Responder > Policies.
  2. In the details pane, click Add.
  3. On the Create Responder Policy page, set the following parameters.
    1. Name. Name of the URL List policy action.
    2. Action. Select the URL List action that you prefer to associate with the policy.
    3. Log Action. Select the log action.
    4. AppFlow. Select an AppFlow action.
    5. Expression. Use the expression editor to create the policy expression.
    6. Comments. A short description about the policy.
  4. Click Create and Close.

To add an HTTP load balancing virtual server

  1. Navigate to the Traffic Management > Load Balancing > Virtual Servers page.
  2. In the details pane, click Add.
  3. On the Load Balancing Virtual Server screen, set the following parameters:
    1. Name. Name of the load balancing virtual server.
    2. Protocol. Choose protocol type as HTTP.
    3. IP Address Type. IP addressable type.
    4. IP Address. IP 4 or IP6 IP address assigned to the virtual server.
    5. Port. Port number of the virtual server.
  4. Click OK to continue with the configuration of other, optional, parameters. For more information, see Creating a Virtual Server.

To bind a URL List policy to the HTTP load balancing virtual server

  1. Navigate to Traffic Management > Load Balancing > Virtual Servers screen.
  2. In the details pane, select the load balancing virtual server and click Edit.
  3. In the Advanced Setting section, click Policies.
  4. In the Policies section, click the + icon to access the Policies slider.
  5. In the Policies section, set the following parameters.
    1. Choose Policy. Select a URL categorization policy from the drop-down list.
    2. Choose Type. Select the policy type as Request.
  6. Click Continue.
  7. Select the URL List policy from the list and click Close.

To add URL List policy for HTTPS traffic

  1. Log on to the Citrix ADC appliance and navigate to Configuration > Optimization > Video Optimization > Detection.
  2. On the Detection page, click the Video Optimization Detection Policies link.
  3. On the Video Optimization Detection Policies page, click Add.
  4. On the Create Video Optimization Detection Policy page, set the following parameters.
    1. Name. Name of the optimization policy
    2. Expression. Configure policy using custom expressions.
    3. Action. Optimization action associated with the policy to handle the incoming video traffic.
    4. UNDEF Action. Undefined event if the incoming request does not match the optimization policy.
    5. Comment. A short description of the policy.
    6. Log Action. Select an audit log action that specifies the action to be performed for the log messages.
  5. Click Create and Close.

To add a SSL-bridge load balancing virtual server for HTTPS traffic

  1. Navigate to the Traffic Management > Load Balancing > Virtual Servers page.
  2. In the details pane, click Add.
  3. On the Load Balancing Virtual Server screen, set the following parameters:
    1. Name. Name of the load balancing virtual server.
    2. Protocol. Select protocol type as SSL-bridge.
    3. IP Address Type. IP address type: IPv4 or IPv6.
    4. IP Address. IPv4 or IP6vIP address assigned to the virtual server.
    5. Port. Port number of the virtual server.
  4. Click OK to continue with the configuration of other, optional, parameters. For more information, see “Creating a Virtual Server” topic.

To bind a URL List Policy to the SSL-bridge load balancing virtual server

  1. Navigate to the Traffic Management > Load Balancing > Virtual Servers screen.
  2. In the details pane, select the SSL-bridge load balancing virtual server and click Edit.
  3. In the Advanced Setting section, click Policies.
  4. In the Policies section, click the + icon to access the Policies slider.
  5. Set the following parameters.
    1. Choose Policy. Select video detection policy from the drop-down list.
    2. Choose Type. Select the policy type as Request.
  6. Click Continue.
  7. Select the video detection policy from the list and click Close.

Configuring Audit Log Messaging

Audit logging enables you to review a condition or a situation in any phase of URL List process. When a Citrix ADC appliance receives an incoming URL, if the responder policy has an URL Set advanced policy expression, the audit log feature collects URL Set information in the URL and stores the details as a log message for any target allowed by audit logging.

The log message contains the following information:

  1. Timestamp.
  2. Log message type.
  3. The predefined log levels (Critical, Error, Notice, Warning, Informational, Debug, Alert, and Emergency).
  4. Log message information, such as URL set name, policy action, URL.

To configure audit logging for URL List feature, you must complete the following tasks:

  1. Enable Audit Log.
  2. Create Audit Log message action.
  3. Set URL List responder policy with Audit Log message action.

For more information, see Audit Logging.

URL List Semantics

The following table lists the URL Match patterns and describes how the URLs within a URL list are matched against the incoming-request URLs. For example, the pattern www.example.com/bar matches only with one page at www.example.com/bar. To match all the pages whose URL starts with ‘ www.example.com/bar’, you would add an asterisk (*) to the end of the URL.

Semantics URL Pattern Matched Unmatched
Subdomain matching domain.com domain.com; [www.domain.com](http://www.domain.com/;sub.one.domain.com yourdomain.com; wwwdomain.com
URL matching, exact path domain.com/example/bar/index.html domain.com/example/bar/index.html; www.domain.com/example/bar/index.html; s.domain.com/example/bar/index.html wwwdomaincom/example/bar/index.html; domain.com/example/bar/index.html/one.jpg
URL matching, exact path domain.com/example/bar/index.html domain.com/example/bar/index.html?key=value; www.domain.com/example/bar/index.html?; s.domain.com/example/bar/index.html wwwdomaincom/example/bar/index.html; domain.com/example/bar/index.html/one.jpg
URL matching, subpath matching domain.com/example/bar/ domain.com/example/bar/;domain.com/example/bar/index.html; www.domain.com/example/bar/index.html; domain.com/example/bar/index.html/one.jpg wwwdomaincom/example/bar/index.html