Tech Paper: Analyzing Citrix Provisioning RAM Cache with Overflow

This tech paper provides information about accurately determining the RAM cache size when using the feature RAM cache with overflow to disk.

RAM cache with overflow to disk is a Citrix Provisioning (PVS) feature where vDisk writes are written to Windows nonpaged pool RAM first.
Once the user-specified RAM cache size has reached its specified size, PVS flushes the RAM cache content to disk to create room for new data.
The RAM cache size fluctuates based on workload pattern and other variations. PoolMon is a tool to take a snapshot of the current RAM cache usage size by looking up pooltag VhdR.

For additional information about this PVS feature, refer here about using RAM cache with overflow.

Important

The tools described in this article are for administrators with advanced knowledge of Provisioning Services. This information can be used to help debug performance-related problems which go beyond using commonly used tools and processes, including the Process Monitor (ProcMon). With this information, you have a better understanding of how the PVS driver functions.

Memory pool monitor

PoolMon (poolmon.exe) refers to the Memory Pool Monitor tool from Microsoft.
It is used to display data (memory allocations from the system paged and nonpaged kernel pools, and the memory pools used for Terminal Services sessions) collected by an OS. This data is grouped by the pool allocation tag.

With nonpaged pool memory, you can use the PoolMon tool to verify the existence of the pooltag denoted by VhdR. VhdR is used for RAM cache allocation. This tag, along with the pooltag VhdL, is useful when creating scripts to help analyze data associated with the RAM cache in nonpaged pool memory.

Tip

Developers and testers typically use PoolMon to detect memory leaks when a driver is created, driver code is altered, or to stress test the driver.
PoolMon can also be used in each stage of the testing process to verify a driver’s pattern of memory allocation and free operations, including being used to determine how much pool memory the driver is using at any given time. For more information on using the Memory Pool Monitor, refer to the Microsoft documentation.

Using the Windows Performance Analyzer

The Windows Performance Analyzer (WPA) is a tool that allows you to create graphs and data tables related to events (specifically, event tracing for Windows) that are recorded by the Windows Performance Recorder (WPR).
Use the WPA to help identify performance bottlenecks while debugging issues related to the PVS driver, the storage stack, and performance related problems that occur when writing to the VHDX disk.
With these tools, you can run assessments and open any event trace log file for analysis. Refer to the Microsoft Developer Network site for more information about the Windows Performance Analyzer.

How the Windows Performance Analyzer works with Provisioning Services

PVS generates events that are captured by the Event Tracing for Windows (ETW) mechanism. This functionality provides a way to trace and log events that are raised by user mode applications and kernel mode drivers. ETW is implemented in the Windows OS and provides an easy way for developers to use a set of event tracing features. For more information, refer here.

Installing the Windows Performance Analyzer

WPA needs to be installed on the master image.

The WPA is part of the latest SDK for the Windows Desktop OS. You can selectively install the Performance Toolkit which includes both the WPA and WPR:

After installing the WPA and the WPR, use the WPR to simulate PVS disk and file I/O activity. Once this traffic is created, analyze the data using the WPA. To do these actions:

1. Launch the WPR on the Target Device and click Add Profiles.

2. In the Add Profiles screen, browse to the PVS-specific template or profile. This permits you to receive the events generated by the PVS event provider. The profile to use is available here. After importing the profile, return to the WPR screen and select any additional options you want to analyze and click the Start button:

   

After adding the options and clicking Start, you can simulate PVS activity.
In this example, a new write cache with a small memory buffer (128 MB) is created. A larger file (279 MB) is copied to C:\Users\User\Documents\test.bin to force the PVS driver to write some data to the nonpaged pool to verify what happens when failover occurs, which starts writing to the local disk (for example, D:\vdiskdif.vhdx).
After copying the file and forcing the buffer to exceed capacity, you can stop the capture process in WPR and open the results using WPA.

1. Using the WPA, open the Graph Explorer, expand System Activity and select Generic Events.
   Using the following screen as an example, view the contents in the sections WriteData and WriteRamData. This information displays the exact count of files being written to C:\vDisk (2419 files), including the VHDX file on the D: drive (348 files):

   

Tip

The WriteData is less than the value displayed because it is cached in RAM and has not been flushed to the disk.

1. Return to the Graph Explorer screen, and expand File IO and Count by Type.
   The following image illustrates the reduction of I/O (file count) and the duration of time that it takes between writing to C:\Users\User\Documents\test.bin and the spillover write cache file D:\vdiskdif.vhdx.
   Using this data, you can view potential performance bottlenecks, and effectively rule out the PVS filter driver as a problem:

   

2. After viewing the file count and duration of time between writes (between the log file and the spillover write cache), you can go further in the debugging process to understand where data is written initially (and where it ends up) using disk offsets.
   In the Windows Performance Analyzer, open the Graph Explorer expand System Activity and select Generic Events.
   Modify the column view to enable the WPA tool to display the data transition in the various storage layers.
   For further debugging, return to the PVS environment and set the RAM cache buffer to 0 MB, then re-run the recorder (WPR) and analyzer (WPA) tools.
   The following image illustrates how spillover to disk occurs: