In AppExpert
In NetScaler 12.0
In Citrix ADC
In All products
Product Documentation
In AppExpert
In NetScaler 12.0
In Citrix ADC
In All products
NetScaler 13.0 NetScaler 12.1 NetScaler 12.0 NetScaler 11.1 NetScaler 10.5
View PDF

Configuring URL Set

January 17, 2020
Contributed by:  G S

You can perform the following tasks to configure a URL set and restrict URLs on a Citrix ADC platform:

  1. Import a URL set (download and encrypt it). Importing a URL set in a Citrix ADC appliance allows you to download the URL file, adding the file to the appliance, and then encrypting the file. Until you add the URL set to the system, it will not be visible to the user.

You can download a set in the following ways:

  • Download a URL set once from a remote server and specify it as http://myserver.com/file_with_urlset.csv

  • add a file under /var/tmp/ path inside ADC and use the command, as in the example below:

> shell cat /var/tmp/test_urlset.csv
example.com
google.com
> import policy urlset top10
k -url local:test_urlset.csv -delimiter "," -rowSeparator "n" -interval 10 -privateSet -canaryUrl http://www.in.gr
Done

The imported URL set is further categorized into different categories and category groups in the database. This is valid only if categories exist in the metadata of the URL set file.

Note: There can be a chance that you might have URL patterns without metadata.

Once you have imported the file, you can update, delete or display file properties. After the file is pushed into the appliance, you can modify the entries by adding further rows as it remains static.

The imported set is then stored in an encrypted file format on the Citrix ADC directory. The imported list contains millions of URL entries. Otherwise, the appliance returns an error message saying that the value exceeds the limit. If the imported URL set has blacklisted entries with metadata, the metadata it is detected by the appliance when it is imported.

Once you import a URL set and add it into the appliance, the URL set is available for advanced policies to identify the correct URL set during incoming URL evaluation. HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URLSET_MATCHES_ANY(<URL set name>)  

  1. Updating a URL set on the Citrix ADC appliance. Once you have pushed the file into the appliance, at this interval you can manually update a URL file by using command line interface.

  2. Exporting a URL set. If you prefer a backup of the URL set, you can export the list of URL patterns and save a copy of it to a destination URL. Before you export, check whether the URL set is marked as private. If is marked private, the URL set cannot be exported. Export functionality does not work with private set. So a new urlset myurl would be imported without privateset defined, and then it would be exported to another file in a local path, as below:

> shell touch /var/tmp/test_urlset_export.csv
Done
> shell cat /var/tmp/test_urlset_export.csv
Done
> shell cat /var/tmp/test_urlset.csv
example.com
google.com
Done
> export urlset myurl -url local:test_urlset_export.csv
> import urlset myurl -url local:test_urlset.csv
Done
(a non-private urlset is imported)

  1. Removing a URL set. If you want to delete a URL set of blacklisted entries, you can use the remove command to delete the URL set from the Citrix ADC appliance.

  2. Displaying a URL set. You can display the properties of a URL set by using the show command.

Example:

show urlset
Name:  top100            PatternCount: 100         Delimiter:          RowSeparator:            Interval:  0
 Done

Import a URL set with meta by using the command line interface

At command prompt, type:

import policy urlset <name> [-overwrite] [–delimiter <character>] [-rowSeparator <character>] [-url] <url> [-interval <seconds>] [-privateSet] [-canaryUrl <URL>]

Where,

delimiter is a CSV file record with default value set as 44.

rowSeparator is a CSV file row separator with default value set as 10.

Interval is the time interval in secs, rounded to the nearest 15 minutes at which the update of urlset occurs.

CanaryURL is a URL used for testing when contents of the urlset is kept confidential.

Example

import policy urlset top10k -url local:test_urlset.csv -delimiter "," -rowSeparator "n" -interval 10 -privateSet -canaryUrl http://www.in.gr

Perform explicit subdomain match for an imported URL set

You can now perform an explicit subdomain match for an imported URL set. To do this, a new parameter, “subdomainExactMatch” is added to the “import policy URLset” command. When you enable the parameter, the URL Filtering algorithm performs an explicit subdomain match. For example, if the incoming URL is “news.example.com” and if the entry in the URL set is “example.com”, the algorithm does not match the URLs.

At the command prompt, type: import policy urlset <name> [-overwrite] [-delimiter <character>][-rowSeparator <character>] -url [-interval <secs>] [-privateSet][-subdomainExactMatch] [-canaryUrl <URL>]

Example:

import policy urlset forth_urlset -url local:test_urlset.csv -interval 3600

To show URL set by using the command line interface

At the command prompt, type:

show urlset <name>

To export an URL set by using the command line interface

At the command prompt, type:

export urlset <name> <url>

To add an URL set by using the command line interface

At the command prompt, type:

add urlset <urlset_name>

To update an URL set by using the command line interface

At the command prompt, type:

update urlset <name>

To remove a URL set command by using the command line interface

At the command prompt, type:

remove urlset <name>

Example:

Note:

Before you import or export a URLset, you must make sure the test_urlset_export.csv and test_urlset.csv files are created and available under /var/tmp directory.

import policy urlset -url local:test_urlset.csv  -delimiter "," -rowSeparator "n" -interval 10 -privateSet -overwrite –canaryUrl http://www.in.gr  

add policy urlset top10k

update policy urlset top10k

sh policy urlset

sh policy urlset top10k

export policy urlset urlset1 -url local:test_urlset_export.csv

import policy urlset top10k -url local:test_urlset.csv –privateSet

add policy urlset top10k

update policy urlset top10k

show policy urlset top10k

Display imported URL sets

You can now display imported URL sets in addition to added URL sets. To do this, a new parameter “imported” is added to the “show urlset” command. If you enable this option, the appliance displays all imported URL sets and distinguishes the imported URL sets from the added URL sets.

At the command prompt, type:

show policy urlset [<name>] [-imported]

Example:

show policy urlset -imported

To import a URL set by using the GUI

Navigate to AppExpert > URL Sets, click Import to download the URL set.

To add a URL set by using the GUI

Navigate to AppExpert > URL Sets, click Add to create a URL set file for the downloaded URL set.

To edit a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Edit to modify.

To Update a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Update URL Set to update URL set with the latest modifications made to the file.

To Export a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Export URL Set to export the URL patterns in a set to a destination URL and save it in that location.

Configuring URL Set

January 17, 2020
Contributed by:  G S

In this article

Edit this article