To ensure that the XenApp and XenDesktop Site database is always available, Citrix recommends starting with a fault-tolerant SQL Server deployment, by following high availability best practices from Microsoft. (The Databases section in the System requirements article lists the SQL Server high availability features supported in XenApp and XenDesktop.) However, network issues and interruptions may result in users not being able to connect to their applications or desktops.
The Local Host Cache (LHC) feature allows connection brokering operations in a XenApp or XenDesktop Site to continue when an outage occurs. An outage occurs when the connection between a Delivery Controller and the Site database fails in an on-premises Citrix environment.
Although this Local Host Cache implementation shares the name of the Local Host Cache feature in XenApp 6.x and earlier XenApp releases, there are significant improvements. This implementation is more robust and immune to corruption. Maintenance requirements are minimized, such as eliminating the need for periodic dsmaint commands. This Local Host Cache is an entirely different implementation technically; read on to learn how it works.
As of the 7.16 release, the connection leasing feature (a predecessor high availability feature in earlier releases) is removed from XenApp and XenDesktop, and is no longer available.
The following graphic illustrates the Local Host Cache components and communication paths during normal operations.
During normal operations:
The following graphic illustrates the changes in communications paths if the principal broker loses contact with the Site database (an outage begins):
When an outage begins:
The transition between normal and outage mode does not affect existing sessions; it affects only the launching of new sessions.
In the unlikely event that an outage begins during a synchronization, the current import is discarded and the last known configuration is used.
The event log provides information about synchronizations and outages. See the "Monitor" section below for details.
You can also intentionally trigger an outage; see the "Force an outage" section below for details about why and how to do this.
Among its other tasks, the CSS routinely provides the High Availability Service with information about all Controllers in the zone. (If your deployment does not contain multiple zones, this action affects all Controllers in the Site.) Having that information, each High Availability Service knows about all peer High Availability Services.
The High Availability Services communicate with each other on a separate channel. They use an alphabetical list of FQDN names of the machines they're running on to determine (elect) which High Availability Service will be in charge of brokering operations in the zone if an outage occurs. During the outage, all VDAs re-register with the elected High Availability Service. The non-elected High Availability Services in the zone will actively reject incoming connection and VDA registration requests.
If an elected High Availability Service fails during an outage, another High Availability Service is elected to take over, and VDAs will re-register with the newly-elected High Availability Service.
During an outage, if a Controller is restarted:
If you power off a Controller during normal operations and then power it on during an outage, Local Host Cache cannot be used on that Controller if it is elected as the primary broker.
The event log provides information about elections. See the "Monitor" section below.
There is no time limit imposed for operating in outage mode. However, restore the site to normal operation as quickly as possible.
What is unavailable or changes during an outage
Local Host Cache is supported for server-hosted applications and desktops, and static (assigned) desktops.
By default, power-managed desktop VDAs in pooled Delivery Groups (created by MCS or PVS) that have the "ShutdownDesktopsAfterUse" property enabled are placed into maintenance mode when an outage occurs. You can change this default, to allow those desktops to be used during an outage. However, you cannot rely on the power management during the outage. (Power management resumes after normal operations resume.) Also, those desktops might contain data from the previous user, because they have not been restarted.
To override the default behavior, you must enable it site-wide and for each affected Delivery Group. Run the following PowerShell cmdlets.
Set-BrokerSite -ReuseMachinesWithoutShutdownInOutageAllowed $true
Set-BrokerDesktopGroup -Name "<name>" -ReuseMachinesWithoutShutdownInOutage $true
Enabling this feature in the Site and the Delivery Groups does not affect how the configured “ShutdownDesktopsAfterUse” property works during normal operations.
The LocalDB service can use approximately 1.2 GB of RAM (up to 1 GB for the database cache, plus 200 MB for running SQL Server Express LocalDB). The High Availability Service can use up to 1 GB of RAM if an outage lasts for an extended interval with many logons occurring (for example, 12 hours with 10K users). These memory requirements are in addition to the normal RAM requirements for the Controller, so you might need to increase the total amount of RAM capacity.
Note that if you use a SQL Server Express installation for the Site database, the server will have two sqlserver.exe processes.
CPU core and socket configuration
A Controller's CPU configuration, particularly the number of cores available to the SQL Server Express LocalDB, directly affects Local Host Cache performance, even more than memory allocation. This CPU overhead is observed only during the outage period when the database is unreachable and the High Availability service is active.
While LocalDB can use multiple cores (up to 4), it’s limited to only a single socket. Adding more sockets will not improve the performance (for example, having 4 sockets with 1 core each). Instead, Citrix recommends using multiple sockets with multiple cores. In Citrix testing, a 2x3 (2 sockets, 3 cores) configuration provided better performance than 4x1 and 6x1 configurations.
As users access resources during an outage, the LocalDB grows. For example, during a logon/logoff test running at 10 logons per second, the database grew by one MB every 2-3 minutes. When normal operation resumes, the local database is recreated and the space is returned. However, sufficient space must be available on the drive where the LocalDB is installed to allow for the database growth during an outage. Local Host Cache also incurs additional I/O during an outage: approximately 3 MB of writes per second, with several hundred thousand reads.
During an outage, one High Availability Service handles all the connections, so in Sites (or zones) that load balance among multiple Controllers during normal operations, the elected High Availability Service might need to handle many more requests than normal during an outage. Therefore, CPU demands will be higher. Every High Availability Service in the Site (zone) must be able to handle the additional load imposed by LocalDB and all of the affected VDAs, because the High Availability Service elected during an outage could change.
During an outage, load management within the Site may be affected. Load evaluators (and especially, session count rules) may be exceeded.
During the time it takes all VDAs to re-register with a High Availability Service, that service might not have complete information about current sessions. So, a user connection request during that interval could result in a new session being launched, even though reconnection to an existing session was possible. This interval (while the "new" High Availability Service acquires session information from all VDAs during re-registration) is unavoidable. Note that sessions that are connected when an outage starts are not impacted during the transition interval, but new sessions and session reconnections could be.
This interval occurs whenever VDAs must re-register:
You can decrease the interval by lowering the Citrix Broker Protocol's HeartbeatPeriodMs registry value (default = 600000 ms, which is 10 minutes). This heartbeat value is double the interval the VDA uses for pings, so the default value results in a ping every 5 minutes.
For example, the following command changes the heartbeat to five minutes (300000 milliseconds), which results in a ping every 2.5 minutes:
New-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer -Name HeartbeatPeriodMs
-PropertyType DWORD –Value 300000
Use caution when changing the heartbeat value. Increasing the frequency results in greater load on the Controllers during both normal and outage. modes.
The interval cannot be eliminated entirely, no matter how quickly the VDAs register.
The time it takes to synchronize between High Availability Services increases with the number of objects (such as VDAs, applications, groups). For example, synchronizing 5000 VDAs might take ten minutes of more to complete. See the "Monitor" section below for information about synchronization entries in the event log.
For Local Host Cache to work correctly, the PowerShell execution policy on each Controller must be set to RemoteSigned, Unrestricted, or Bypass.
The Microsoft SQL Server Express LocalDB that Local Host Cache uses is installed automatically when you install a Controller or upgrade a Controller from a version earlier than 7.9. There is no administrator maintenance needed for the LocalDB. Only the High Availability Service communicates with this database. You cannot use PowerShell cmdlets to change anything about this database. The LocalDB cannot be shared across Controllers.
The SQL Server Express LocalDB database software is installed regardless of whether Local Host Cache is enabled.
To prevent its installation, install or upgrade the Controller using the XenDesktopServerSetup.exe command, and include the /exclude "Local Host Cache Storage (LocalDB)" option. However, keep in mind that the Local Host Cache feature will not work without the database, and you cannot use a different database with the High Availability Service.
Note: Installation of this LocalDB database has no effect on whether or not you install SQL Server Express for use as the site database.
During a new installation of XenApp and XenDesktop (version 7.16 or later), Local Host Cache is enabled. After an upgrade (to version 7.16 or later), Local Host Cache is enabled if there are fewer than 10,000 VDAs in the entire deployment.
To enable Local Host Cache, enter:
Set-BrokerSite -LocalHostCacheEnabled $true
To determine whether Local Host Cache is enabled, enter:
Check that the LocalHostCacheEnabled property is True.
To disable Local Host Cache, enter:
Set-BrokerSite -LocalHostCacheEnabled $false
Remember: As of version 7.16, connection leasing (the feature that preceded Local Host Cache beginning with version 7.6) is removed from XenApp and XenDesktop, and is no longer available.
You might want to deliberately force a database outage.
To force an outage, edit the registry of each server containing a Delivery Controller. In HKLM\Software\Citrix\DesktopServer\LHC, set OutageModeForced to 1. This instructs the broker to enter outage mode, regardless of the state of the database. (Setting the value to 0 takes the server out of outage mode.)
Event logs indicate when synchronizations and outages occur.
Config Synchronizer Service
During normal operations, the following events can occur when the CSS copies and exports the broker configuration and imports it to the LocalDB using the High Availability Service.
High Availability Service
Several troubleshooting tools are available when an synchronization import to the LocalDB fails and a 505 event is posted.
CDF tracing: Contains options for the ConfigSyncServer and BrokerLHC modules. Those options, along with other broker modules, will likely identify the problem.
Report: You can generate and provide a report that details the failure point. This report feature affects synchronization speed, so Citrix recommends disabling it when not in use.
To enable and produce a CSS trace report, enter:
New-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer\LHC -Name EnableCssTraceMode -PropertyType DWORD -Value 1
The HTML report is posted at C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Temp\CitrixBrokerConfigSyncReport.htmll
After the report is generated, disable the reporting feature:
Set-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer\LHC -Name EnableCssTraceMode -Value 0
Export the broker configuration: Provides the exact configuration for debugging purposes.
Export-BrokerConfiguration | Out-File < file-pathname>
For example, Export-BrokerConfiguration | Out-File C:\BrokerConfig.xml.