ADC

URL categorization

Note:

URL categorisation in the URL filtering feature is deprecated in this release.

URL Categorization restricts user access to specific websites and website categories. As a subscribed service in collaboration with NetSTAR, the feature enables enterprise customers to filter web traffic using a commercial categorization database. The NetSTAR database has a vast number (billions) of URLs classified into different categories, such as social networking, gambling, adult content, new media, and shopping. In addition to categorization, each URL has a reputation score kept up to date based on the site’s historical risk profile. We can use NetSTAR data to filter the traffic by configuring advanced policies based on categories, category groups (such as Terrorism, Illegal drugs), or site-reputation scores.

For example, you might block access to dangerous sites, such as sites known to be infected with malware. You might also selectively restrict access to content such as adult content or entertainment streaming media for enterprise users. You can also capture the user’s transactional details and outbound traffic details for monitoring web traffic analytics on the NetScaler Console server.

NetScaler uploads or downloads data from the pre-configured NetSTAR device nsv10.netstar-inc.com and incompasshybridpc.netstar-inc.com is used as a cloud host by default for cloud-categorization requests. These URLs must be accessible through the firewall for URL filtering to work properly. The appliance uses its NSIP address as a source IP address and 443 as the destination port for communication.

How URL categorization works

The following figure shows how a NetScaler URL categorization service is integrated with a commercial URL Categorization database and cloud services for frequent updates.

How URL Categorization works

The components interact as follows:

  1. A client sends an internet bound URL request.

  2. The SSL forward proxy applies a policy enforcement to the request based on the category details, such as, category, category group, and site-reputation score. The category details are retrieved from the URL categorization database. If the database returns the category details, the process jumps to step 5.

  3. If the database misses the categorization details, the request is sent to a cloud-based lookup service maintained by a URL categorization vendor. However, the appliance does not wait for a response, instead, the URL is marked as uncategorized and a policy enforcement is performed (jump to step 5). The appliance continues to monitor the cloud query feedback and updates the cache so that future requests can benefit from the cloud lookup.

  4. The ADC appliance receives the URL category details (category, category group, and reputation score) from the cloud-based service and stores it in the categorization database.

  5. The policy allows the URL and the request is sent to the origin server. Otherwise, the appliance drops, redirects, or responds with a custom HTML page.

  6. The origin server responds with the requested data to the ADC appliance.

  7. The appliance sends the response to the client.

Use Case: Internet usage under corporate compliance for enterprises

You can use the URL Filtering feature to detect and implement compliance policies to block sites that violate corporate compliance. For example, sites such as adult, streaming media, social networking which can be deemed nonproductive or consume excess internet bandwidth in an enterprise network. Blocking access to these websites can improve employee productivity, reduce operating costs for bandwidth usage, and reduce the overhead of network consumption.

Prerequisites

The URL Categorization feature works on a NetScaler platform only if it has an optional subscription service with URL filtering capabilities and threat intelligence for SSL forward proxy. The subscription allows customers to download the latest threat categorizations for websites and then enforce those categories to the SSL forward proxy. Before enabling and configuring the feature, you must install the following licenses:

  • CNS_WEBF_SSERVER_Retail.lic

  • CNS_XXXX_SERVER_PLT_Retail.lic

Where, XXXXX is the platform type, for example: V25000

Responder policy expressions

The following table lists the different policy expressions that you can use to verify if an incoming URL must be allowed, redirected, or blocked.

  1. <text>. URL_CATEGORIZE (<min_reputation>, <max_reputation>) - Returns a URL_CATEGORY object. If <min_reputation> is greater than 0, the returned object does not contain a category with a reputation lower than <min_reputation>. If <max_reputation> is greater than 0, the returned object does not contain a category with a reputation higher than <max_reputation>. If the category fails to resolve in a timely manner, the undef value is returned.
  2. <url_category>. CATEGORY() - Returns the category string for this object. If the URL does not have a category, or if the URL is malformed, the returned value is “Unknown.”
  3. <url_category>. CATEGORY_GROUP() - Returns a string identifying the object’s category group. This grouping is a higher level grouping of categories, which is useful in operations that require less detailed information about the URL category. If the URL does not have a category, or if the URL is malformed, the returned value is “Unknown.”
  4. <url_category>. REPUTATION() - Returns the reputation score as a number from 0 to 5, where 5 indicates the riskiest reputation. If there is the category “Unknown”, the reputation value is 1.

Policy types:

  1. Policy to select requests for URLs that are in the Search Engine category - add responder policy p1 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). CATEGORY().EQ(“Search Engine”)
  2. Policy to select requests for URLs that are in the Adult category group - add responder policy p1 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). CATEGORY_GROUP().EQ(“Adult”)’
  3. Policy to select requests for Search Engine URLs with a reputation score lower than 4 - add responder policy p2 ‘HTTP.REQ.HOSTNAME.APPEND (HTTP.REQ.URL).URL_CATEGORIZE(4,0).HAS_CATEGORY(“Search Engine”)
  4. Policy to select requests for Search Engine and Shopping URLs - add responder policy p3 ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0).CATEGORY().EQ (“good_categories”)
  5. Policy to select requests for Search Engine URLs with a reputation score equal to or greater than 4 - add responder policy p5 ‘CLIENT.SSL.DETECTED_DOMAIN.URL_CATEGORIZE(4,0). CATEGORY().EQ(“Search Engines”)
  6. Policy to select requests for URLs that are in the Search Engine category and compare them with a URL Set - ‘HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URL_CATEGORIZE(0,0). CATEGORY().EQ(“Search Engine”) && HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URLSET_MATCHES_ANY(“u1”)’

Responder policy types

There are two types of policies used in a URL Categorization feature and each of these policy types is explained the following table:

Policy Type Description
URL Category Categorize web traffic and based on evaluation result blocks, allows, or redirects traffic.
URL Reputation Score Determines the reputation score of the website and allows you to control access based on the reputation score threshold level set by the administrator.

Configure URL categorization

To configure URL categorization on a NetScaler appliance, do the following:

  1. Enable URL filtering.
  2. Configure a proxy server for Web traffic.
  3. Configure SSL interception for Web traffic in explicit mode.
  4. Configure shared memory to limit cache memory.
  5. Configure URL categorization parameters.
  6. Configure URL categorization by using the Citrix SSL forward proxy wizard.
  7. Configure URL categorization parameters by using the SSL forward proxy wizard.
  8. Configure seed database path and cloud server name

Step 1: Enabling URL Filtering

To enable URL categorization, enable the URL filtering feature and enable modes for URL categorization.

To enable URL Categorization by using the CLI

At the command prompt, type:

enable ns feature URLFiltering

disable ns feature URLFiltering

Step 2: Configure a proxy server for web traffic in explicit mode

The NetScaler appliance supports transparent and explicit proxy virtual servers. To configure a proxy virtual server for SSL traffic in explicit mode, do the following:

  1. Add a proxy server.
  2. Bind an SSL policy to the proxy server.

To add a proxy server by using the CLI

At the command prompt, type:

add cs vserver <name> [-td <positive_integer>] <serviceType> [-cltTimeout <secs>]

Example:

add cs vserver starcs PROXY 10.102.107.121 80 -cltTimeout 180

Bind an SSL policy to a proxy virtual server by using the CLI

bind ssl vserver <vServerName> -policyName <string> [-priority <positive_integer>]

Step 3: Configure SSL interception for HTTPS traffic

To configure SSL interception for HTTPS traffic, do the following:

  1. Bind a CA certificate-key pair to the proxy virtual server.
  2. Configure the default SSL profile with SSL parameters.
  3. Bind a front-end SSL profile to the proxy virtual server and enable SSL interception in the front-end SSL profile.

To bind a CA certificate-key pair to the proxy virtual server by using the CLI

At the command prompt, type:

bind ssl vserver <vServerName> -certkeyName <certificate-KeyPairName> -CA –skipCAName

To configure the default SSL profile by using the CLI

At the command prompt, type:

set ssl profile <name> -denySSLReneg <denySSLReneg> -sslInterception (ENABLED | DISABLED) -ssliMaxSessPerServer positive_integer>

Bind a front-end SSL profile to a proxy virtual server by using the CLI

At the command prompt, type:

set ssl vserver <vServer name> -sslProfile ssl_profile_interception

Step 4: Configure shared memory to limit cache memory

To configure shared memory to limit cache memory by using the CLI

At the command prompt, type:

set cache parameter [-memLimit <megaBytes>]

Where, the memory limit configured for caching is set as 10 MB.

Step 5: Configure URL categorization parameters

To configure the URL categorization parameters by using the CLI

At the command prompt, type:

set urlfiltering parameter [-HoursBetweenDBUpdates <positive_integer>] [-TimeOfDayToUpdateDB <HH:MM>]

Example:

set urlfiltering parameter –urlfilt_hours_betweenDB_updates 20

Step 6: Configure URL Categorization by using the Citrix SSL forward proxy wizard

  1. Log on to the NetScaler appliance and navigate to Security > SSL Forward Proxy page.
  2. In the details pane, do one of the following:
    1. Click SSL Forward Proxy Wizard to create a new configuration.
    2. Select an existing configuration and click Edit.
  3. In the URL Filtering section, click Edit.
  4. Select the URL Categorization check box to enable the feature.
  5. Select a URL Categorization policy and Click Bind.
  6. Click Continue and then Done.

For more information about URL Categorization policy, see How to Create a URL Categorization Policy.

Step 7: Configuring URL Categorization parameters by using an SSL forward proxy Wizard

  1. Log on to NetScaler appliance and navigate to Security > URL Filtering.
  2. In the URL Filtering page, click Change URL filtering settings link.
  3. In the Configuring URL Filtering Params page, specify the following parameters.
    1. Hours Between DB Updates. URL Filtering hours between database updates. Minimum value: 0 and Maximum value: 720.
    2. Time of Day to Update DB. URL Filtering time of day to update database.
    3. Cloud Host. The URL path of the cloud server.
    4. Seed DB Path. The URL path of the seed database lookup server.
  4. Click OK and Close.

Sample Configuration:

enable ns feature LB CS SSL IC RESPONDER AppFlow URLFiltering

enable ns mode FR L3 Edge USNIP PMTUD

set ssl profile ns_default_ssl_profile_frontend -denySSLReneg NONSECURE -sslInterception ENABLED -ssliMaxSessPerServer 100

add ssl certKey swg_ca_cert -cert ns_swg_ca.crt -key ns_swg_ca.key

set cache parameter -memLimit 100

add cs vserver starcs PROXY 10.102.107.121 80 -cltTimeout 180

add responder action act1 respondwith "\"HTTP/1.1 200 OK\r\n\r\n\" + http.req.url.url_categorize(0,0).reputation + \"\n\""

add responder policy p1 "HTTP.REQ.URL.URL_CATEGORIZE(0,0).CATEGORY.eq(\"Shopping/Retail\") || HTTP.REQ.URL.URL_CATEGORIZE(0,0).CATEGORY.eq(\"Search Engines & Portals

\")" act1

bind cs vserver starcs_PROXY -policyName p1 -priority 10 -gotoPriorityExpression END -type REQUEST

add dns nameServer 10.140.50.5

set ssl parameter -denySSLReneg NONSECURE -defaultProfile ENABLED -sigDigestType RSA-MD5 RSA-SHA1 RSA-SHA224 RSA-SHA256 RSA-SHA384 RSA-SHA512 -ssliErrorCache ENABLED

-ssliMaxErrorCacheMem 100000000

add ssl policy pol1 -rule "client.ssl.origin_server_cert.subject.  URL_CATEGORIZE(0,0).CATEGORY.eq(\"Search Engines & Portals\")"" -action INTERCEPT

add ssl policy pol3 -rule "client.ssl.origin_server_cert.subject.ne(\"citrix\")" -action INTERCEPT

add ssl policy swg_pol -rule "client.ssl.client_hello.SNI.URL_CATEGORIZE(0,0).CATEGORY.ne(\"Uncategorized\")" -action INTERCEPT

set urlfiltering parameter -HoursBetweenDBUpdates 3 -TimeOfDayToUpdateDB 03:00
<!--NeedCopy-->

Configure seed database path and cloud server name

You can now configure the seed database path and cloud lookup server name for manual setting of the cloud lookup server name and the seed database path. To do this, two new parameters, “CloudHost” and “SeedDBPath”, are added to the URL filtering parameter.

At the command prompt, type:

set urlfiltering parameter [-HoursBetweenDBUpdates <positive_integer>] [-TimeOfDayToUpdateDB <HH:MM>] [-LocalDatabaseThreads <positive_integer>] [-CloudHost <string>] [-SeedDBPath <string>]

Example:

set urlfiltering parameter -HoursBetweenDBUpdates 3 -TimeOfDayToUpdateDB 03:00 –CloudHost localhost –SeedDBPath /mypath

The Communication between a NetScaler appliance and NetSTAR might require a domain name server. You can test using a simple console or telnet connection from the appliance.

Example:

root@ns# telnet nsv10.netstar-inc.com 443
Trying 1.1.1.1...
Connected to nsv10.netstar-inc.com.
Escape character is '^]'.

root@ns# telnet incompasshybridpc.netstar-inc.com 443
Trying 10.10.10.10...
Connected to incompasshybridpc.netstar-inc.com.
Escape character is '^]'.
<!--NeedCopy-->

Configure audit log messaging

Audit logging enables you to review a condition or a situation in any phase of the URL Categorization process. When a NetScaler appliance receives an incoming URL, if the responder policy has a URL Filtering expression, the audit log feature collects URL Set information in the URL. It stores the information as log messages for any target allowed by audit logging.

  • Source IP address (the IP address of the client that made the request).

  • Destination IP address (the IP address of the requested server).

  • Requested URL containing the schema, the host, and the domain name (http://www.example.com).

  • URL category that the URL filtering framework returns.

  • URL category group that the URL filtering framework returned.

  • URL reputation number that the URL filtering framework returned.

  • Audit log action taken by the policy.

To configure audit logging for a URL List feature, you must complete the following tasks:

  1. Enable Audit Log.
  2. Create Audit Log message action.
  3. Set URL List responder policy with Audit Log message action.

For more information, see Audit Logging topic.

Storing failure errors using SYSLOG messaging

At any stage of the URL Filtering process, if there is a system-level failure, the ADC appliance uses the audit log mechanism to store logs in the ns.log file. The errors are stored as text messages in SYSLOG format so that, an administrator can view it later in a chronological order of event occurrence. These logs are also sent to an external SYSLOG server for archival. For more information, see article CTX229399.

For example, if a failure occurs when you initialize the URL Filtering SDK, the error message is stored in the following messaging format.

Oct 3 15:43:40 <local0.err> ns URLFiltering[1349]: Error initializing NetStar SDK (SDK error=-1). (status=1).

The NetScaler appliance stores the error messages under four different failure categories:

  • Download failure. If an error occurs when you try to download the categorization database.

  • Integration failure. If an error occurs when you integrate an update into the existing categorization database.

  • Initialization failure. If an error occurs when you initialize the URL Categorization feature, set categorization parameters, or end a categorization service.

  • Retrieval failure. If an error occurs when the appliance retrieves the categorization details of the request.

Configure SNMP traps for NetSTAR events

The URL Filtering feature generates SNMP traps, if the following conditions occur:

  • NetSTAR database update fails or succeeds.
  • NetSTAR SDK initialization fails or succeeds.

The appliance has a set of conditional entities called SNMP alarms. When a condition in the SNMP alarm is met, the appliance generates traps and sends it to a specified trap destination. For example, if the NetSTAR SDK initialization fails, an SNMP OID 1.3.6.1.4.1.5951.1.1.0.183 is generated and sent to the trap destination.

For the appliance to generate traps, you must first enable and configure SNMP alarms. Then, you specify the trap destination to which the appliance sends the generated trap messages

Enable an SNMP alarm

The NetScaler appliance generates traps only for SNMP alarms that are enabled. Some alarms are enabled by default, but you can disable them.

When you enable an SNMP alarm, the URL filtering feature generates trap messages when a success or failure event occurs. Some alarms are enabled by default.

To enable an SNMP alarm by using the command line interface:

At the command prompt, type the following commands to set the parameters and verify the configuration:

enable snmp alarm <trapName> show snmp alarm <trapName>

To enable an SNMP alarm by using the NetScaler GUI

  1. Navigate to System > SNMP > Alarms, and select the alarm.
  2. Click Actions and select Enable.

Configure SNMP alarm by using the CLI

At the command prompt, type the following commands to set the parameters and verify the configuration:

set snmp alarm <trapName> [-thresholdValue <positive_integer> [-normalValue <positive_integer>]] [-time <secs>] [-state ( ENABLED | DISABLED )] [-severity <severity>] [-logging ( ENABLED | DISABLED )]

Example:

set snmp alarm URL-FIL-DB-UPDATE-STATUS –state ENABLED set snmp alarm URL-FIL-INIT-SDK –state ENABLED

Configure SNMP alarms by using the GUI

Navigate to System > SNMP > Alarms, select an alarm, and configure the alarm parameters.

For more information about SNMP traps, see SNMP topic