ADC

URL Categorization

Note:

URL categorisation in the URL filtering feature is deprecated in this release.

URL Categorization restricts user access to specific websites and website categories. As a subscribed service in collaboration with NetSTAR, the feature enables enterprise customers to filter web traffic using a commercial categorization database. The NetSTAR database has a vast number (billions) of URLs classified into different categories, such as social networking, gambling, adult content, new media, and shopping. In addition to categorization, each URL has a reputation score kept up to date based on the site’s historical risk profile. We can use NetSTAR data to filter the traffic by configuring advanced policies based on categories, category groups (such as Terrorism, Illegal drugs), or site-reputation scores.

For example, you might block access to dangerous sites, such as sites infected with malware, or selectively restrict access to adult content or entertainment streaming media.

How URL Categorization Works

The following figure shows how the NetScaler URL Filtering service is integrated with a commercial URL Categorization database and cloud services for frequent updates.

How URL categorization works

The components interact as follows:

  1. Client sends internet bound URL request.
  2. A NetScaler policy attempts to evaluate the request in terms of categorization details (such as category, category group, and site-reputation score) retrieved from the URL categorization database. If the database returns the category details, the process jumps to step 5.
  3. If the database does not return categorization details, the request is sent to a cloud-based lookup service maintained by a URL categorization vendor. However, the appliance does not wait for a response. Instead, it marks the URL as Uncategorized and jumps to step 5. However, it continues to monitor the cloud query feedback and uses it to update the cache so that future requests can benefit from the cloud lookup.
  4. The NetScaler appliance receives the URL category details (category, category group, and reputation score) from the cloud-based service and stores it in the cloud cache.
  5. If the policy allows the URL, the request is sent to the origin server. Otherwise, the appliance drops or redirects the request, or responds with a custom HTML page.
  6. The origin server responds with the requested data to the NetScaler appliance.
  7. The appliance sends the response to the client.

You can use the URL filtering feature to detect sites that violate safe internet usage mandates issued by the government and implement policies to block these sites. Sites that host adult content, streaming media, or social networking identified as unsafe for children or banned as illegal.

Prerequisites

The feature works on Telco platforms with the purchase of a basic CBM license and CBM Premium license and for other NetScaler platforms, the feature works with the purchase of a CNS Premium license.

Note: In addition to a Basic CBM license and a CBM Premium license, the appliance must have a URL Threat Intelligence license with a subscription service for 1 year or 3 years. Before enabling and configuring the feature, you must install the following licenses:

License support for Telco platforms:

  • CBM_TXXX_SERVER_Retail.lic
  • CBM_TPRE_SERVER_Retail.lic
  • CNS_WEBF_SSERVER_Retail.lic

Where XXX is the throughput, for example, NetScaler T1000.

License support for other NetScaler platforms:

  • CNS_XXX_SERVER_PLT_Retail.lic

Where XXX is the throughput.

URL Categorization Policy Expressions

The following table lists the different URL categorization policy expressions for identifying incoming URLs and applies a configured action.

Expression Operation
<text>. URL_CATEGORIZE (<min_reputation>, <max_reputation>) Returns a URL_CATEGORY object. Reputation score is a number from 1 to 4. To get objects all reputation scores use 0.0 as <min reputation> and . If is greater than 0, the returned object does not contain a category with reputation lower than . If is greater than 0, the returned object does not contain a category with reputation higher than . If the category fails to resolve in a timely manner, the undef value is returned.
. CATEGORY Returns the category string for this object. If the URL does not have a category, or if the URL is malformed, the returned value is “Uncategorized”.
. GROUP Returns a string identifying the object’s category group. This is a higher level grouping of categories, which is useful in operations that require less detailed information about the URL category. If the URL does not have a category, or if the URL is malformed, the returned value is “Uncategorized”.
. REPUTATION Returns the reputation score as a number from 1 to 4, where 4 indicates the riskiest reputation. If the category is “Uncategorized,” the reputation value is 2.

Sample Policy Expressions

Policy Policy Expressions
Policy to select requests for URLs that are in the Search Engine category add responder policy p1 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). CATEGORY.EQ(“Search Engine”)
Policy to select requests for URLs that are in the Adult category group add responder policy p1 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). GROUP.EQ(“Adult”)’
Policy to select requests for Search Engine URLs with a reputation score equal to 4. add responder policy p2 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(4,0). CATEGORY.EQ(“Search Engine”)’
Policy to select requests for Search Engine and Shopping URLs add policy patset good_categories; bind policy good_categories “Search Engine”; bind policy good_categories “Shopping”; add responder policy p3 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). CATEGORY .EQUALS_ANY(“good_categories”)
Policy to select requests for Search Engine URLs with a reputation score equal to 4. add responder policy p5 ‘CLIENT.SSL.DETECTED_DOMAIN.URL_CATEGORIZE(4,0). CATEGORY.EQ(“Search Engine”)

URL Categorization Policy Actions

A URL filtering policy evaluates the traffic to identify requests belonging to a particular category. The following table lists the actions that you can assign to a URL filtering policy.

Policy Action Policy Group Description
ALLOW Responder Allow the incoming request to access the target URL
REDIRECT Responder Redirect the incoming request to the URL specified as the target.
DENY Responder Deny incoming request.
RESET Responder, VideoOptimization Reset connection.
DROP Responder, VideoOptimization Drop connection.

Note

For encrypted traffic, the VideoOptimization policy includes actions that implement the URL Filtering actions.

Configuring URL Categorization

To configure URL categorization, begin by enabling the URL Filtering feature. You must then configure the cache memory limits, categorization policy, and virtual servers for HTTP and HTTPS traffic. Configuring URL Categorization by using the CLI.

To use the CLI configure URL categorization on a NetScaler appliance, do the following:

  • Set up URL Categorization.
    • Enable the URL Filtering feature.
    • Configure shared memory to limit cache memory.
    • Configure URL categorization parameters.
  • Configure URL categorization for HTTP traffic.
    • Add URL categorization actions.
    • Add URL categorization policies.
    • Add a load balancing virtual server for HTTP traffic.
    • Bind URL categorization policies to the load balancing virtual server.
  • Configure URL categorization for HTTPS traffic.
    • Add URL categorization policies.
    • Add a SSL-Bridge load balancing virtual server.
    • Bind URL categorization policies to the load balancing virtual server.

Setting up URL Categorization

To set up the feature, you must enable the URL Categorization feature, configure the filtering parameters and set the shared memory limit.

To enable URL Filtering feature

At the command prompt, type:

enable ns feature URLFiltering VideoOptimization Responder IC SSL AppFlow

To configure shared memory limit

At the command prompt, type:

set cache parameter [-memLimit <megaBytes>]
<!--NeedCopy-->

Where memLimit is the memory limit for caching.

Example:

set cache parameter -memLimit 10

To configure URL categorization parameters

At the command prompt, type:

set urlfiltering parameter [-HoursBetweenDBUpdates <positive_integer>] [-TimeOfDayToUpdateDB <HH:MM>]
<!--NeedCopy-->

*Example:

set urlfiltering parameter -HoursBetweenDBUpdates 3 -TimeOfDayToUpdateDB 03:00

Configuring URL categorization for HTTP traffic

To configure the URL categorization feature for HTTP traffic, you must configure a loading balancing virtual server, add URL categorization policies and bind the policies to the virtual server. By doing so, the virtual server receives the HTTP traffic and based on policy evaluation, the system assigns a filtering action.

To add URL categorization action for HTTP traffic

At the command prompt, type:

add responder action <name> <type> (<target> | <htmlpage>) [-comment <string>] [-responseStatusCode <positive_integer>] [-reasonPhrase <string>]

Example:

add responder action act_url_categorize respondwith "\"HTTP/1.1 200 OK\r\n\r\n\" + HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0).CATEGORY + \"\n\""

To add URL categorization policy for HTTP traffic

At the command prompt, type:

add responder policy <name> <rule> <action> [<undefAction>] [-comment <string>] [-logAction <string>] [-appflowAction <string>]

Example:

add responder policy pol_url_categorize_http "HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0).GROUP.EQ(\"Adult\") || HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0).GROUP.EQ(\"Gambling\")" RESET

To add an HTTP load balancing virtual server

If a virtual server for HTTP traffic is not already configured, at the command prompt, type:

add lb vserver <name> [-td <positive_integer>] <serviceType> [-clt Timeout <secs>]

Example:

add lb vserver vsrv-HTTP HTTP * 80 -persistenceType NONE -cltTimeout 120

To bind URL categorization policy with load balancing virtual server

At the command prompt, type:

bind lb vserver <name> -policyName <string> [-priority <positive_integer>]

Example:

bind lb vserver vsrv-HTTP -policyName pol_url_categorize_http -priority 10 -gotoPriorityExpression END -type REQUEST

Configuring URL categorization for HTTPS traffic

To configure the URL categorization feature for HTTPS traffic, you must configure a SSL-bridge loading balancing virtual server, add URL categorization policies and bind the policies to the SSL-bridge virtual server. By doing so, the server receives the HTTPS traffic and based on policy evaluation, the system assigns a filtering action.

To add URL categorization policy for HTTPS traffic

At the command prompt, type:

add videooptimization detectionpolicy <name> -rule <expression> -action <string> [-undefAction <string>] [-comment <string>] [-logAction <string>]

Example:

add videooptimization detectionpolicy pol_url_categorize_https_block_adult –rule "CLIENT.SSL.DETECTED_DOMAIN.URL_CATEGORIZE(0,0).CATEGORY.EQ("Adult")' –action RESET

To add SSL-Bridge load balancing virtual server

At the command prompt, type:

add lb vserver <name> [-td <positive_integer>] <serviceType> [-cltT imeout <secs>]

Example:

add lb vserver vsrv-HTTPS SSL_BRIDGE * 443 -persistenceType NONE -cltTimeout 180

To bind categorization policy with SSL-Bridge virtual server

At the command prompt, type:

bind lb vserver <name> -policyName <string> [-priority <positive_integer>]

Example:

bind lb vserver vsrv-HTTPS -policyName pol_url_categorize_https_block_adult -priority 20 -type REQUEST

Configuring URL Categorization by using the GUI

The GUI enables you to:

  • Enable the URL Categorization feature.
  • Add URL Categorization actions for HTTP traffic.
  • Add URL Categorization policies for HTTP traffic.
  • Add URL Categorization policies for HTTPS traffic.
  • Add a load balancing virtual server for HTTP traffic.
  • Add an SSL bridge load balancing virtual server for HTTPS traffic.
  • Bind URL Categorization policies to the load balancing virtual server.
  • Bind URL Categorization policies to the SSL-Bridge load balancing virtual server.
  • Configure shared memory limit.
  • Configure URL categorization parameters.

To enable URL categorization

  1. In the navigation pane, expand System and then click Settings.
  2. On the Settings page, click Configure Advanced Features link.
  3. On the Configure Advanced Features page, select URL Filtering check box.
  4. Click OK and Close.

To add a URL categorization action

  1. In the navigation pane, expand AppExpert > Responder > Action.
  2. In the details pane, click Add.
  3. On the Create Responder Action page, set the following parameters.
    1. Name. Name of the URL categorization policy action.
    2. Type. Select an action type.
    3. Expression. Use the expression editor to create the policy expression.
    4. Comments. A short description of the policy action.
  4. Click Create and Close.

To add a URL categorization policy for HTTP traffic

  1. In the navigation pane, expand AppExpert > Responder > Policies.
  2. On the details pane, click Add.
  3. On the Create Responder Policy page, set the following parameters.
    1. Name. Name of the URL categorization policy action.
    2. Action. Select the URL Categorization action that you prefer to associate with the policy.
    3. Log Action. Select the log action.
    4. AppFlow. Select an AppFlow action.
    5. Expression. Use the expression editor to create the policy expression.
    6. Comments. A short description about the policy action.
  4. Click Create and Close.

To add a categorization policy for HTTPS traffic

  1. Log on to the NetScaler appliance and navigate to Configuration > Optimization > Video Optimization > Detection.
  2. On the Detection page, click the Video Optimization Detection Policies link.
  3. On the Video Optimization Detection Policies page, click Add.
  4. On the Create Video Optimization Detection Policy page, set the following parameters.
    1. Name. Name of the optimization policy
    2. Expression. Configure policy using custom expressions.
    3. Action. Optimization action associated with the policy to handle the incoming video traffic.
    4. UNDEF Action. Undefined event if the incoming request does not match the optimization policy.
    5. Comment. A short description about the policy.
    6. Log Action. Select an audit log action that specifies the action to be performed for the log messages.
  5. Click Create and Close.

To add a load balancing virtual server for HTTP traffic

  1. Navigate to the Traffic Management > Load Balancing > Virtual Servers page.
  2. In the details pane, click Add.
  3. On the Load Balancing Virtual Server page set the following parameters:
    1. Name. Name of the load balancing virtual server.
    2. Protocol. Choose protocol type as HTTP.
    3. IP Address Type. IPv4 or IPv6.
    4. IP Address. IPv4 or IPv6, VIP address assigned to the virtual server.
    5. Port. Port number of the virtual server.
  4. Click OK to continue with the configuration of other, optional, parameters.
  5. Click Create and Close.

To add an SSL-bridge load balancing virtual server

  1. Navigate to the Traffic Management > Load Balancing > Virtual Servers page.
  2. On the details pane, click Add.
  3. On the Load Balancing Virtual Server page, set the following parameters:
    1. Name. Name of the load balancing virtual server.
    2. Protocol. Select protocol type as SSL-bridge.
    3. IP Address Type. IP addressable type.
    4. IP Address. IP 4 or IP6 IP address assigned to the virtual server.
    5. Port. Port number of the virtual server.
  4. Choose OK to continue configuration other optional parameters.
  5. Click Create and then Close.

To bind a URL categorization policy to the HTTP load balancing virtual server

  1. Navigate to Traffic Management > Load Balancing > Virtual Servers page.
  2. On the details pane, select the load balancing virtual server and click Edit.
  3. In the Advanced Setting section, click Policies.
  4. In the Policies section, click the + icon to access the Policies slider.
  5. Set the following parameters.
    1. Choose Policy. Select URL categorization policy from the drop-down list.
    2. Choose Type. Select the policy type as Request.
  6. Click Continue.
  7. Select the URL categorization policy from the list and click Close.

To bind a categorization policy to the SSL-bridge load balancing virtual server

  1. Navigate to Traffic Management > Load Balancing > Virtual Servers screen.
  2. In the details pane, select the SSL-bridge load balancing virtual server and click Edit.
  3. In the Advanced Setting section, click Policies.
  4. In the Policies section, click + icon to access the Policies slider.
  5. In the Policies section, set the following parameters.
    1. Choose Policy. Select video detection policy from the drop-down list.
    2. Choose Type. Select the policy type as Request.
  6. Click Continue.
  7. Select the video detection policy from the list and click Close.

To configure the shared memory limit

  1. Sign on to the appliance and navigate to Optimization > Integrated Caching.
  2. In the details pane, click Change cache settings link.
  3. On the Cache Global Settings page, set the following parameters.
    1. Memory Usage Limit (MB).
    2. Active memory Usage Limit.
    3. Via Header.
    4. Maximum Post Body Length to be Cached
    5. Global Undefined-Result Action
    6. Enable HA Object Persist
    7. Verify Cached Object Persist
    8. Prefetches
  4. Click OK and Close.

To configure URL categorization parameters

  1. Sign to the appliance and navigate to Security.
  2. On the details pane, click Change URL filtering settings link.
  3. In the Configuring URL Filtering Params page, set the following parameters.
    1. Hours Between DB Updates. URL Filtering hours between database updates. Minimum value: 0 and Maximum value: 720.
    2. Time of Day to Update DB. URL Filtering time of day to update the database.
  4. Click OK and Close.

Configuring Audit Log Messaging

When a NetScaler appliance receives an incoming URL, if the responder policy has a URL Filtering expression, the audit log feature collects categorization information and displays it as log messages to any target audit log server that is configured. The info is logged.

  • Source IP address (the IP address of the client that made the request).
  • Destination IP address (the IP address of the requested server).
  • Requested URL containing the schema, the host, and the domain name (http://www.example.com).
  • URL category that the URL filtering framework returns.
  • URL category group that the URL filtering framework returned.
  • URL reputation number that the URL filtering framework returned.
  • Audit log action taken by URL Categorization policy.

To configure audit logging for the URL List feature, you must complete the following tasks:

  1. Enable Audit Log.
  2. Create Audit Log message action.
  3. Set URL List responder policy with Audit Log message action.

For more information, see Audit Logging topic.

Storing Failure Errors Using SYSLOG Messaging

At any stage of the URL Filtering process, if there is a system-level failure, the NetScaler appliance uses the audit log mechanism to store logs in the ns.log file. The errors are stored as text messages in SYSLOG format so that, an administrator can view it later in a chronological order of event occurrence. These logs are also sent to an external SYSLOG server for archival. For more information, see article CTX229399.

For example, if a failure occurs when you initialize the URL Filtering SDK, the error message is stored in the following messaging format.

Oct 3 15:43:40 <local0.err> ns URLFiltering[1349]: Error initializing NetStar SDK (SDK error=-1). (status=1).

The NetScaler appliance stores the error messages under four different failure categories:

  • Download failure. If an error occurs when you try to download the categorization database.
  • Integration failure. If an error occurs when you integrate an update into the existing categorization database.
  • Initialization failure. If an error occurs when you initialize the URL Categorization feature, set categorization parameters, or end a categorization service.
  • Retrieval failure. If an error occurs when the appliance retrieves the categorization details of the request.

URL Reputation Score

The URL Categorization feature provides policy-based control to restrict blacklisted URLs. You can control access to websites based on URL category, reputation score, or URL category and reputation score. If a network administrator monitors a user accessing highly risky websites, he or she can use a responder policy bound to the URL reputation score to block such risky websites.

Upon receiving an incoming URL request, the appliance retrieves the category and reputation score from the URL categorization database. Based on the reputation score returned by the database, the appliance assigns a reputation rating for websites. The value can range from 1 to 4, where 4 is the riskiest type of websites, as shown in the following table.

URL Reputation Rating Reputation Comment
1 Clean site.
2 Unknown site.
3 Potentially dangerous or affiliated to dangerous site.
4 Malicious site.