PoC Guide: Web Application Firewall Deployment

Overview

This proof of concept (PoC) guide is designed to help you quickly deploy Citrix Web App Firewall (WAF) either standalone or as a part of an existing ADC deployment to protect web applications and services. This guide covers some of the basics of Citrix WAF, deployment best practices, and next steps for your WAF projects. This guide will NOT cover every protection available nor will it cover all web technologies available today.

These instructions cover cloning an application or server that is already being proxied by an ADC appliance allowing administrators to build policies and rules without disrupting existing traffic. This is not the only way to deploy a PoC for Web Application Firewall, but it is easier than other methods.

Conceptual Architecture

A Web Application Firewall operates at Layer 7, the application layer, unlike traditional network firewalls that operate at Layer 3 or Layer 4. It understands web and application protocols like HTTP, XML, SQL, and HTML.

The Citrix WAF provides both positive and negative security models for the widest protection possible. The negative security model employs vulnerability signatures to prevent known attacks. It also allows for rapid initial deployment and provides a mechanism for "Virtual Patching" of applications by importing the results from third party assessment tools.

The positive security model defines the traffic patterns and user behaviors that are allowed and blocks everything else. Dynamic profiling is used by the WAF engine to build rule sets for the behavior that is acceptable.

Prerequisites

This guide assumes the following is already configured on your Citrix ADC or standalone Citrix WAF:

A basic understanding of networking concepts and managing the Citrix ADC
The ADC appliance is already deployed and accessible on the network
A suitable license has been applied - either standalone WAF or for ADC, a Premium license is required
An NSIP has been assigned, and is accessible for management
A SNIP has been assigned and can communicate with one or more services that are protected
An internal only IP address and DNS entry for another VIP (the service may already be load balanced by the ADC, ensure that a separate IP and DNS are available)

Quote

Note:

Using a different DNS name and URL to access an application MAY require adding configuration to the application to make it aware of the new name
A Basic understanding of the architecture and structure of the application or service that you are protecting. For example, if the site processes credit card information determines if credit card protections are enabled, or if the application uses an SQL database for storage determines if SQLi protections are used
A basic understanding of web protocols and technologies such as HTTP, HTML, regular expressions, JavaScript and more, depending on the application that is being protected
A Basic understanding of layer 4 load balancing and configuration on the Citrix ADC
A dedicated syslog server - while local syslogs are available, having a dedicated syslog server configured is recommended. Alternatively, Citrix ADM with Security Insight provides detailed analytics into the WAF engine and application security

WAF Proof of Concept Considerations

More than likely, the web service or application that we want to protect is already being proxied by the ADC for load balancing or other traffic management. During the WAF PoC deployment we do NOT want to disrupt the application functionality or availability for existing users. To accomplish this, a duplicate virtual server is made with the only difference being that the duplicate virtual server has WAF policies bound to it.

This method also gives users the ability to test the application to ensure that it is functioning correctly and to build WAF rules for acceptable traffic.

Sizing the ADC deployment needs to be considered before enabling Web Application Firewall as it can introduce significant resource consumption. The best way to estimate sizing is to use the web server to analyze and estimate the request sizes, response sizes, and number of forms. Citrix ADC can be used to gather basic web sizing metrics, but only if the application is already being proxied:


show httpband -type RESPONSE
show httpband -type REQUEST

Certain protections require significantly more resources than others, thus referred to as Advanced protections. Advanced protections consume more resources because they are remembering Form Fields, URLs, Cookies, and other data in the web transaction. The WAF engine validates that the data does not change between transactions to ensure that requests and responses are not tampered with and as such consume more appliance memory. Some advanced protections support Sessionless checking as noted in the configuration - sessionless protection requires more CPU resources rather than memory.

As an example, an appliance with Advanced policies is able to process approximately one tenth as much traffic as an identical appliance with only Basic protections. Refer to the Citrix ADC data sheets for the supported WAF throughput capabilities for each different appliance.

Recommended Protections

These security checks can usually be enabled in the WAF profile. Most of these checks require learned rules to be applied before enabling blocking; do NOT enable BLOCK by default, without deploying learned rules first.

Quote

Note:

This list is NOT exhaustive. Some protections are not compatible with some applications and some protections do not add any benefit to other applications.

Cookie Consistency
- Allows session cookies from the application to be encrypted or proxied by the WAF engine
- Cookie Consistency prevents a malicious actor from tampering with cookies
- Cookie Consistency is an Advanced protection
  
  Quote
  
  Note:
  
  Do NOT enable this setting when users are actively using the application, as any non-encrypted session cookies sent will not be allowed
- More information in Citrix Prod Docs
Buffer Overflow
- Prevents attacks against insecure OS or web server software that can behave unpredictably when receiving data that is larger than it can handle
- Buffer Overflow detects if the URL, cookies, or headers are longer than the specified maximum size
- More information in Citrix Prod Docs
Form Field Consistency Check
- Validates that web forms returned by users have not been modified before being sent to the server
- Form Field Consistency is an Advanced protection with sessionless support
- More information in Citrix Prod Docs
SQL Injection
- Blocks or transforms SQL special keywords and characters from the POST body, header, and cookies
- Only applicable if the web application uses an SQL database
- More information in Citrix Prod Docs
Cross-Site Scripting Check (XSS)
- Checks the headers and POST bodies of requests for cross-site script attacks
- WAF can either block the request completely or transform the offending scripts to prevent them from running
- More information in Citrix Prod Docs
Deny URL
- Blocks connections to URLs that are commonly accessed by hackers and malicious code or scripts
- Custom Deny URL Rules can be added in the Relaxation Rules > Deny URL UI, by clicking Edit then Add
- More information in Citrix Prod Docs
Start URL (URL Closure)
- Prevents direct access to random URLs on a site (forceful browsing) through bookmarks, external links, page jumping, or manually entering URLs
- Start URL is an Advanced protection with sessionless support; learned rules MUST be deployed before blocking based on Start URL
- More information in Citrix Prod Docs

Initial WAF Configuration

There are two options when configuring WAF protection:

Option 1: Use the Web App Firewall configuration Wizard to create a WAF profile and WAF policy
- The wizard guides you through building an initial signature set and other security checks and settings as follows:
  - In the web UI, navigate to: Security > Citrix Web App Firewall
  - Choose a name for the WAF profile and the profile type (Web, XML, or JSON depending on the type of application you are protecting)
  - Specify the expression that determines which traffic is processed by the WAF engine
  - Select the signatures that apply protections for known vulnerabilities - there are options to use an existing signature set or create a new set. To choose which signatures to apply to this protection profile, there are two editor options offering basic or advanced capabilities.
  - There are navigation options on both editors to select the signatures then enable, block, log, or stats. Watch out for multiple pages of signatures and ensure that all are selected. The Basic editor is easier to get started with, while the advanced editor offers more capabilities
  Quote
  
  Note:
  
  Ensure that the signatures are marked as ENABLED prior to moving on to the next step
  - Choose the Deep Protections that are applicable to the application. It is recommended to start by enabling logging, stats, and learning; do NOT enable blocking initially as many protections require some number of learned rules
  Quote
  IMPORTANT:
  
  If the WAF wizard is used to create a profile and policy, it does become bound GLOBALLY! If the filtering expression is left as default (true) then ALL traffic is processed by the WAF engine, likely leading to applications misbehaving. The recommended action is to unbind the policy from the Default Global bind point and instead bind to the LB virtual server that is created later. To unbind a WAF policy from the Global bind point, do the following:
  - Navigate to Security > Citrix Web App Firewall > Policies > Firewall
  - Choose Policy Manager, then choose Default Global
  - Highlight the WAF policy listed, and choose Unbind
Option 2: Alternatively, WAF protection can be applied to an application or site by manually creating a WAF profile, WAF policy, and applying a signature set
1. Create a signature set for the application
  1. Navigate to Security > Citrix Web App Firewall > Signatures
  2. Highlight the Default Signatures then press Add to create a copy of the default signature set
  3. Select the appropriate signatures depending on the application or site being protected and ensure that the signatures are set to ENABLED
2. Create a Web App Firewall Profile
  1. Navigate to Security > Citrix Web App Firewall > Profiles
  2. Give the profile a descriptive name then select Web, XML, or JSON depending on the type of application you are protecting
  3. Select Basic to apply the basic default settings
  4. Once created, highlight the new Profile, and press Edit
  5. Open Security Checks and deselect BLOCK on all items
  6. Open Profile Settings
    - Select the previously created Signature Set in the Bound Signatures setting
    - If the back-end application server supports chunked requests, enable Streaming to improve performance
    - Modify any additional settings based on the application or website being protected
3. Create a Web App Firewall Policy
  1. Navigate to Security > Citrix Web App Firewall > Policies > Firewall
  2. Press Add to create a new policy
  3. Give the policy a descriptive name and select the WAF Profile created earlier
  4. Create a traffic expression that determines which traffic is processed by the WAF engine

Regardless of the method used, a basic Web App Firewall configuration is now available to be bound. The final step is to update the signature file.

Navigate to Security > Citrix Web App Firewall > Signatures
Highlight the newly created signature set and press Select Action > Auto Update Settings
Enable Signatures Auto Update and ensure that the update URL is populated

Quote

Note:

Signature update requires that the ADC can resolve DNS names and access the public internet from the NSIP

Once auto-update is configured, ADC checks for updates once per hour. The signatures can be updated manually as well and a manual update needs to be run once after the WAF profile and policy are created

To verify if the ADC appliance is able to reach the signature updates, the following command is run from the CLI (exit to shell):


shell
curl -I https://s3.amazonaws.com/NSAppFwSignatures/SignaturesMapping.xml

The system should return an HTTP/1.1 200 OK

The most recent signature update documentation is available on Citrix Prod Docs - updates via RSS are also available.

By default, Common Event Format (CEF) logging is not enabled, with the WAF using native logging format. Additionally, the WAF enginge can include GeoLocation information for requests. For ease of troubleshooting, it is recommended to enable CEF logging and GeoLocation by navigating to Security > Citrix Web App Firewall > Change Engine Settings. Check the boxes for CEF Logging and Geo-Location Logging. Alternatively, these settings can be enabled in via the CLI as follows:


set appfw settings GeoLocationLogging ON CEFLogging ON

Clone Virtual Server Configuration

Once the Web App Firewall Profile and Policy have been created, it needs to be bound to the load balancing virtual server. If the application or service is not already configured on the ADC, it needs to be configured from scratch. If the application is already configured, it needs to be cloned.

To quickly clone an existing virtual server, do the following:

Navigate to Traffic Management > Load Balancing > Virtual Servers

Highlight the existing virtual server and press Add to create a new virtual server with similar settings
- Give the new virtual server a new name and different IP address
- Ensure that the port and protocol are correct then press OK
- Bind services or service groups then press Continue
- Modify the load balancing method, persistence, and any other Traffic settings needed to match the existing virtual server
- If this virtual server is an HTTPS virtual server, ensure that an appropriate certificate is bound and a secure Cipher Suite is bound

Next, the WAF policy needs to be bound to this new virtual server:

Navigate to Traffic Management > Load Balancing > Virtual Servers
Highlight the newly created virtual server and press Edit
In the right Advanced Settings column, choose Policies
Press the + (plus) icon to add a policy
- Choose an App Firewall policy with a type of Request
- The default binding details can be left alone

User Acceptance Testing and Initial Troubleshooting

At this point, users can begin testing the application using the new URL, accessing the web application with WAF policies bound. Depending on the complexity of the application, multiple users across different business units need to use the application to generate an acceptable amount of traffic. This traffic is used by the learning engine to see all parts of the application and to ensure there are no false positive blocks that prevent application functionality.

During initial testing, information pertaining to the offending request can be difficult to find, but there a few settings that assist with this. The first is to set a custom HTML Error Object that presents diagnostic information whenever a request is blocked. If an error object is not set, a user is often be presented with a blank page that does not provide any information.

Quote

Note:

This is recommended to be removed or changed once the WAF protected application is exposed to a non-testing audience.

Set the HTML Error Object to present diagnostic information
- Go to the WAF Profile, then expand Profile Settings
- Select the HTML Error Object and click the add button (or plus) to create a new object
- In the 'Import HTML Error Page' choose Text
- Give the error object a name and paste the following code:

<html>
<head>
<title>Application Firewall Block Page</title>
</head>
<body>
<h1><B>Your request has been blocked by a security policy<B><BR></H1>
<H3>Access has been blocked - if you feel this is in error, please contact the site
administrators quoting the following: </H3> <UL>
<li>NS Transaction ID: ${NS_TRANSACTION_ID}
<li>AppFW Session ID: ${NS_APPFW_SESSION_ID}
<li>Violation Category: ${NS_APPFW_VIOLATION_CATEGORY}
<li>Violation Details: ${NS_APPFW_VIOLATION_LOG} </UL>
</body>
</html>

The resulting error page presents valuable diagnostic information, including the violation category and log details. This information allows the administrator to validate if the block needs to be relaxed due to a false positive.

Quote

Important:

Some WAF blocks only prevent certain elements of the page from loading, for example an image, CSS formatting, script or button\form functionality may not work. These blocks are still considered issues, though the diagnostic error page does not show. Use the syslog viewer (see the example below) to find the offending signature or security check that is causing the issue.

The Syslog Messages that are generated by the WAF engine help understand blocked sessions when the diagnostic error page is not available

Go to System > Auditing > Syslog messages

Under the Filter By column, select Module and choose APPFW
Press apply

Some violations can be immediately relaxed from the Syslog viewer if there are false positive blocks

From the Syslog Messages view (preceding), find a Syslog Message that is a rule violation
Check the box in the top right corner of the message
Select the Action menu then choose deploy
Alternatively, you can modify the relaxation rule before deploying it

Quote

Note:

A false-positive block from a signature violation often means that the web application is not following standards or is written poorly. The immediate action to allow the application to function is to apply a relaxation rule, but the long term goal is to fix the application itself. Relaxation rules on poorly written applications can allow malicious content through by means of allowing the application to function.

By turning on stats for all items, the appliance gathers counters anytime a rule or signature is violated. This is useful in the initially as it allows counters to be cleared to see if any new blocked requests occurred as a result of the WAF policy.

From the Dashboard tab of the ADC UI, set the stats view to Application Firewall to view all of the associated stat counts. Stats can be cleared from this view as well.

Alternatively, stats can be viewed from the shell command line of the appliance by using the following command:


nsconmsg -d stats | grep appfw

Be aware that the Web Application Firewall follows web standards and by default, malformed requests are blocked. To change this behavior, do the following:

Citrix WAF uses a strict HTTP check
By default invalid, non-RFC compliant HTTP requests are blocked
- To modify non-RFC compliant request handling, go to Security > Citrix Web App Firewall then Change Engine Settings
- See the following code for the CLI method to set only logging and stat generation for malformed requests
  
  Quote
  
  Note:
  
  This setting is global. If you disable the block option, the WAF engine bypasses application firewall processing for any non-RFC compliant requests. This is not recommended.


set appfw settings -malformedReqAction log stats

Security Check Learning

Learned data is used to create rules for the positive security model of the WAF engine - a few quick details about learned rules:

Stored in RegEx format and can be exported to CSV for analysis or migration
Are propagated in an HA pair of appliances
Security checks can be left in learning mode permanently, but is not necessary
Gathers more useful data the more traffic is processed during UAT
It is strongly recommended to configure Trusted Learning Clients for learning when the appliance is receiving traffic from both known and unknown clients. This setting ensures that malicious traffic is not allowed into a learned rule.

Learned engine runs as the aslearn process and rules are stored in SQLite format in the following location:


/var/nslog/asl/appfwprofile.db

Learning mode can be enabled permanently, but the learning database has a size limit of 20 MB. The learning engine does not add new data once the database reaches this size.

From a process perspective, learning occurs as follows:

Anytime that a user is accessing the new application with a bound WAF policy containing enabled learning checks, learned rules are generated. By using a duplicate virtual server approach, production traffic is not blocked while users test the protected application. Additionally, all traffic that is passed through the WAF engine at this point will generate learned rules.

The typical flow for building relaxation rules and ensuring the application is working as expected without false positive blocks occurs continuously:

This process must be repeated anytime there is a change to an application or workflow where rules were not previously created.

To view the learned data using the UI, do the following:

Navigate to Security > Citrix Web App Firewall > Profiles, select the App Firewall profile, then choose Edit
On the right column, choose Learned Rules
Each of the security checks that support learning are displayed with a few options:
- Select Edit to view and deploy the rules that have been learned for a specific security check

Once rules are deployed, they are moved from learned rules to relaxation rules
From the learned rules UI, the rules can be exported to CSV using the Export Learned Data button before moving rules to relaxation rules
The Visualizer is also a valuable tool especially for Start URL checks to visualize rules and a URL map of the protected application

Once the learned rules have been deployed to relaxation rules, the security check can be set to Block.

Quote

Note:

If enough traffic has not been processed or all of the application paths have not been seen by the WAF engine, there will likely be false positive blocks. The learned rule process can be repeated until the application behaves as expected.

Import Export

The entire configuration of a Web Application Firewall profile can be replicated across multiple appliances. This process will capture all bound objects such as the HTML error object, XML error object, WSDL\XML schema, signatures, relaxation rules and so on. This can be particularly useful when migrating from a test\dev environment to a production environment or to back up the WAF configuration before making changes.

Quote

Note:

Exported configurations require that both appliances be on the same build.

To export the WAF profile using the GUI, on the source appliance, navigate to Security > Citrix Web App Firewall > Profiles > Select the WAF profile, then choose Action > Export The export process may take some time to generate the export file. Once it has been archived, a WAFProfile.tgz file is downloaded to the client computer.

To import the WAF profile using the GUI, on the destination appliance, navigate to Security > Citrix Web App Firewall > Profiles > Select Action > Import. Select 'Import From' File, then select 'Choose File' to locate the previously downloaded archive. Any issues with the import process are logged to:


/var/log/ns.log

Quote

Note:

The import process does NOT create a WAF policy or binding.

Dynamic Profiling

The Dynamic Profiling feature was added to the WAF Engine in version 13.0 and allows for learned data to be automatically deployed to relaxation rules automatically after a configured threshold. An alert is sent to administrators to allow them to skip the data if it is not valid, otherwise it is added automatically. By default, dynamic profiling is disabled for all supported security checks - to enable it and configure the settings, do the following:

Navigate to Security > Citrix Web App Firewall > Profiles, select the App Firewall profile, then choose Edit
On the right column, choose Dynamic Profiling
- Each security check can be enabled or disabled from this UI
- The dynamic profiling settings for this WAf profile can also be modified by selecting Settings

When using dynamic profiling, it is strongly recommended to enable Trusted Learning Clients to ensure that only valid application traffic is generating rules. These clients can be added from:

Learned Rules UI
Dynamic Profiling UI

Considerations When Moving to Production

The most important consideration with using the duplicate virtual server method is that any of the learned rules that include URLs do not apply if the application URL changes. For example, the Start URL and CSRF Form Tagging rules include URLs.

The best way to migrate to the production virtual server would be to turn off blocking for security checks and relearn and deploy rules that include URLs. Alternatively, the learned rules could be exported then readded after modifying the URL. It would also be very important to enable Trusted Learning Clients if they have not already been added.

Lastly, if the PoC is being run on a test platform that has different resources than production, it is important to consider the resource requirements that change by enabling Web Application Firewall.