Configuring URL Set

You can perform the following tasks to configure a URL set and restrict URLs on a NetScaler platform:

1.Import a URL set (download and encrypt it). Importing a URL set in a NetScaler appliance allows you to download the URL file, adding the file to the appliance, and then encrypting the file. Until you add the URL set to the system, it will not be visible to the user.

You can download a set in the following ways:

• Download a URL set once from a specific URL using HTTP and HTTPS supported for the file download.

• Download a URL set using FTP.

• Downloading a URL set periodically, using a scheduler that periodically downloads or imports URL sets for example, IWF URL set. The time interval is set in seconds for example, *http://10.29.102.200/urls.txt -interval 3600*.

Sample URL Set without a meta data

http://10.102.145.135/top10k

Sample URL Set with a meta data

http://10.102.145.135/blacklists/audio-video/categorized_av

The imported URL set is further categorized into different categories and category groups in the database. This is valid only if categories exist in the metadata of the URL set file.

Note: There can be a chance that you might have URL patterns without metadata.

Once you have downloaded the file, it is pushed into the appliance and at this point of interval, you can update, delete or display file properties. After the file is pushed into the appliance, you can modify the entries by adding further rows as it remains static.

The imported set is then stored in an encrypted file format on the NetScaler directory. The imported list contains millions of URL entries. Otherwise, the appliance returns an error message saying that the value exceeds the limit. If the imported URL set has blacklisted entries with metadata, the metadata it is detected by the appliance when it is imported.

Once you import a URL set and add it into the appliance, the URL set is available for advanced policies to identify the correct URL set during incoming URL evaluation. HTTP.REQ.HOSTNAME.APPEND(HTTP.REQ.URL).URLSET_MATCHES_ANY(<URL set name>)  

2.Updating a URL set on the NetScaler appliance. Once you have pushed the file into the appliance, at this interval you can manually update a URL file by using command line interface.

3.Exporting a URL set. If you prefer a backup of the URL set, you can export the list of URL patterns and save a copy of it to a destination URL. Before you export, check whether the URL set is marked as private. If is marked private, the URL set cannot be exported.

4.Removing a URL set. If you want to delete a URL set of blacklisted entries, you can use the remove command to delete the URL set from the NetScaler appliance.

5.Displaying a URL set. You can display the properties of a URL set by using the show command.

Example:

show urlset


Name:  top100            PatternCount: 100         Delimiter:          RowSeparator:            Interval:  0

 Done

To import a URL set with meta by using the command line interface

At command prompt, type:

import urlset <name> [-overwrite] [–delimiter <character>] [-rowSeparator <character>] [-url] <url> [-interval <seconds>] [-privateSet] [-canaryUrl <URL>]

Where,

delimiter is a CSV file record with default value set as 44.

rowSeparator is a CSV file row separator with default value set as 10.

Interval is the time interval in secs, rounded to the nearest 15 minutes at which the update of urlset occurs.

CanaryURL is a URL used for testing when contents of the urlset is kept confidential.

To show URL set by using the command line interface

At the command prompt, type:

show urlset <name>

To export an URL set by using the command line interface

At the command prompt, type:

export urlset <name> <url>

To add an URL set by using the command line interface

At the command prompt, type:

add urlset <urlset_name>

To update an URL set by using the command line interface

At the command prompt, type:

update urlset <name>

To remove a URL set command by using the command line interface

At the command prompt, type:

remove urlset <name>

Example

import policy urlset top1k -url http://10.78.79.80/alytra/top1k.csv  -delimiter "," -rowSeparator "n" -interval 10 -privateSet -overwrite –canaryUrl http://www.in.gr  

add policy urlset top1k

update policy urlset top1k

sh policy urlset

sh policy urlset top1k

export policy urlset urlset1 -url http://www.example.com/PUT_file_1
import policy urlset top10k -url http://10.102.145.135/top10k -private

add policy urlset top10k

update policy urlset top10k

show policy urlset top10k

To import a URL set by using the GUI

Navigate to AppExpert > URL Sets, click Import to download the URL set.

To add a URL set by using the GUI

Navigate to AppExpert > URL Sets, click Add to create a URL set file for the downloaded URL set.

To edit a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Edit to modify.

To Update a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Update URL Set to update URL set with the latest modifications made to the file.

To Export a URL set by using the GUI

Navigate to AppExpert > URL Sets, select a URL set and click Export URL Set to export the URL patterns in a set to a destination URL and save it in that location.