Local Host Cache
The Local Host Cache feature enables connection brokering operations in a Citrix Virtual Apps and Desktops service deployment to continue when a Cloud Connector cannot communicate with Citrix Cloud. Local Host Cache engages when the network connection is lost for 60 seconds.
With Local Host Cache, users who are connected when an outage occurs can continue working uninterrupted. Reconnections and new connections experience minimal connection delays.
Each resource location must have a customer-deployed on-premises StoreFront. Local Host Cache works only in resource locations containing an on-premises StoreFront.
Local Host Cache includes the following information, which is a subset of the information in the main database:
- Identities of users and groups who are assigned rights to resources published from the Site.
- Identities of users who are currently using, or who have recently used, published resources from the Site.
- Identities of VDA machines (including Remote PC Access machines) configured in the Site.
- Identities (names and IP addresses) of client Citrix Workspace app machines being actively used to connect to published resources.
It also contains information for currently active connections that were established while the main database was unavailable:
- Results of any client machine endpoint analysis performed by Citrix Workspace app.
- Identities of infrastructure machines (such as Citrix Gateway and StoreFront servers) involved with the site.
- Dates, times, and types of recent activity by users.
How it works
During normal operations
- The Brokering Principal (also known as the Citrix Remote Broker Provider Service) on a Cloud Connector accepts connection requests from StoreFront. The Brokering Principal communicates with Citrix Cloud to connect users with VDAs that are registered with the Cloud Connector.
- The Citrix Config Synchronizer Service (CSS) checks with the broker in Citrix Cloud approximately every minute to see if any configuration changes were made. Those changes can be administrator-initiated (such as changing a Delivery Group property) or system actions (such as machine assignments).
If a configuration change occurred since the previous check, the CSS synchronizes (copies) information to a secondary broker on the Cloud Connector(The secondary broker is also known as the High Availability Service, or HA broker, as shown in the preceding figure).
All configuration data is copied, not just items that changed since the previous check. The CSS imports the configuration data into a Microsoft SQL Server Express LocalDB database on the Cloud Connector. This database is referred to as the Local Host Cache database. The CSS ensures that the information in the Local Host Cache database matches the information in the site database in Citrix Cloud. The Local Host Cache database is re-created each time synchronization occurs.
Microsoft SQL Server Express LocalDB (used by the Local Host Cache database) is installed automatically when you install a Cloud Connector. The Local Host Cache database cannot be shared across Cloud Connectors. You do not need to back up the Local Host Cache database. It is recreated every time a configuration change is detected.
- If no changes occurred since the last check, the configuration data is not copied.
During an outage
When an outage begins:
- The secondary broker starts listening for and processing connection requests.
- When the outage begins, the secondary broker does not have current VDA registration data, but as soon as a VDA communicates with it, a registration process is triggered. During that process, the secondary broker also gets current session information about that VDA.
- While the secondary broker is handling connections, the Brokering Principal continues to monitor the connection to Citrix Cloud. When the connection is restored, the Brokering Principal instructs the secondary broker to stop listening for connection information, and the Brokering Principal resumes brokering operations. The next time a VDA communicates with the Brokering Principal, a registration process is triggered. The secondary broker removes any remaining VDA registrations from the previous outage. The CSS resumes synchronizing information when it learns that configuration changes have occurred in Citrix Cloud.
In the unlikely event that an outage begins during a synchronization, the current import is discarded and the last known configuration is used.
The event log indicates when synchronizations and outages occur.
There is no time limit imposed for operating in outage mode.
You can also intentionally trigger an outage. See Force an outage for details about why and how to do this.
Resource locations with multiple Cloud Connectors
Among its other tasks, the CSS routinely provides the secondary broker with information about all Cloud Connectors in the resource location. Having that information, each secondary broker knows about all peer secondary brokers running on other Cloud Connectors in the resource location.
The secondary brokers communicate with each other on a separate channel. They use an alphabetical list of FQDN names of the machines they’re running on to determine (elect) which secondary broker will be in charge of brokering operations in the zone if an outage occurs. During the outage, all VDAs re-register with the elected secondary broker. The non-elected secondary brokers in the zone actively reject incoming connection and VDA registration requests.
If an elected secondary broker fails during an outage, another secondary broker is elected to take over, and VDAs register with the newly elected secondary broker.
During an outage, if a Cloud Connector is restarted:
- If that Cloud Connector is not the elected broker, the restart has no impact.
- If that Cloud Connector is the elected broker, a different Cloud Connector is elected, causing VDAs to register. After the restarted Cloud Connector powers on, it automatically takes over brokering, which causes VDAs to register again. In this scenario, performance can be affected during the registrations.
The event log provides information about elections.
What is unavailable during an outage, and other differences
There is no time limit imposed for operating in outage mode. However, if the outage is due to loss of Citrix Cloud connectivity from their resource location, customers should restore connectivity from their resource location as quickly as possible.
During an outage:
- You cannot use Studio or run PowerShell commands.
- Monitoring data is not sent to Citrix Cloud during an outage. So, the Monitor functions (Director) do not show activity from an outage interval.
- Hypervisor credentials cannot be obtained from the Host Service. All machines are in the unknown power state, and no power operations can be issued. However, VMs on the host that are powered-on can be used for connection requests.
- An assigned machine can be used only if the assignment occurred during normal operations. New assignments cannot be made during an outage.
- Automatic enrollment and configuration of Remote PC Access machines is not possible. However, machines that were enrolled and configured during normal operation are usable.
- Server-hosted applications and desktop users may use more sessions than their configured session limits, if the resources are in different zones.
- Users can launch applications and desktops only from registered VDAs in the zone containing the currently active/elected broker. Launches across zones (from a broker in one zone to a VDA in a different zone) are not supported during an outage.
- If a site database outage occurs before a scheduled restart begins for VDAs in a Delivery Group, the restarts begin when the outage ends. This can have unintended results. For more information, see Scheduled restarts delayed due to database outage.
Each resource location must have a customer-deployed on-premises StoreFront. Local Host Cache works only in resource locations containing an on-premises StoreFront. It does not support Workspace.
You can ensure the availability of resources (apps and desktops) during an outage in two ways:
- Publish the resources in every resource location in your deployment.
Publish the resources to at least one resource location. Then use the following procedure to enable the advanced health check feature in each StoreFront store.
- Upgrade the StoreFront installation in each resource location to minimum version 1912 CU1. For guidance, see the StoreFront documentation.
For each StoreFront store, enable the advanced health check option. In the store’s
- After you update the file, manually restart IIS. Repeat the web.config file update and IIS restart for other stores.
Application and desktop support
Local Host Cache works only with customer-deployed StoreFront. It does not support Workspace.
Local Host Cache supports server-hosted applications and desktops, and static (assigned) desktops.
Local Host Cache supports desktop VDAs in pooled Delivery Groups, as follows.
By default, power-managed desktop VDAs in pooled Delivery Groups (created by MCS or Citrix Provisioning) that have the
ShutdownDesktopsAfterUseproperty enabled are placed into maintenance mode when an outage occurs. You can change this default, to allow those desktops to be used during an outage.
However, you cannot rely on the power management during the outage. (Power management resumes after normal operations resume.) Also, those desktops might contain data from the previous user, because they have not been restarted.
To override the default behavior, it must be enabled site-wide and for each affected Delivery Group. Place a support call to have it enabled site-wide (this command cannot be run using the Remote PowerShell SDK).
Set-BrokerSite -ReuseMachinesWithoutShutdownInOutageAllowed $true
For each affected Delivery Group, run the following PowerShell command.
Set-BrokerDesktopGroup -Name "name" -ReuseMachinesWithoutShutdownInOutage $true
Enabling this feature in the site and the Delivery Groups does not affect how the configured
ShutdownDesktopsAfterUseproperty works during normal operations.
RAM size considerations
SQL Server Express LocalDB can use up to 1.2 GB of RAM (up to 1 GB for the database cache, plus 200 MB for running SQL Server Express LocalDB). The High Availability Service (the Local Host Cache broker) can use up to 1 GB of RAM if an outage lasts for an extended interval with many logons occurring (for example, 12 hours with 10K users). These memory requirements are in addition to the normal RAM requirements for the Cloud Connector. Consider increasing the total amount of RAM capacity.
CPU core and socket configuration considerations
A Cloud Connector’s CPU configuration, particularly the number of cores available to the SQL Server Express LocalDB, directly affects Local Host Cache performance, even more than memory allocation. This CPU overhead is observed only during the outage period when the database is unreachable and the Local Host Cache broker is active.
While SQL Server Express LocalDB can use multiple cores (up to 4), it’s limited to only a single socket. Adding more sockets does not improve the performance (for example, having 4 sockets with 1 core each). Instead, Citrix recommends using multiple sockets with multiple cores. In Citrix testing, a 2x3 (2 sockets, 3 cores) configuration provided better performance than 4x1 and 6x1 configurations.
As users access resources during an outage, the Local Host Cache database grows. For example, during a logon/logoff test running at 10 logons per second, the database grew by 1 MB every 2 to 3 minutes. When normal operation resumes, the Local Host Cache database is recreated when a configuration change is detected. The Local Host Cache broker must have sufficient space on the drive where the Local Host Cache database is installed to allow for the database growth during an outage. Local Host Cache also incurs additional I/O during an outage: approximately 3 MB of writes per second, with several hundred thousand reads.
During an outage, the elected Local Host Cache broker handles all the connections. In resource locations that load balance among multiple Cloud Connectors during normal operations, the elected Local Host Cache broker might need to handle many more requests than normal during an outage. Therefore, CPU demands are higher. Every Local Host Cache broker in the resource location must be able to handle the additional load and all affected VDAs, because the elected broker can change.
- In a deployment with one resource location, up to 10,000 VDAs can be handled effectively during an outage.
- In a VDI deployment with more than one resource location, up to 10,000 VDAs in each resource location can be handled effectively during an outage, to a maximum of 40,000 VDAs in the deployment. For example, each of the following deployments can be handled effectively during an outage:
- A deployment with four secondary resource locations, each containing 10,000 VDAs.
- A deployment with seven secondary resource locations, one containing 10,000 VDAs, and six containing 5,000 VDAs each.
During an outage, load management may be affected. Load evaluators (and especially, session count rules) can be exceeded.
During the time it takes all VDAs to register with a Local Host Cache broker, that broker might not have complete information about current sessions. So, a user connection request during that interval might result in starting a new session, even though reconnection to an existing session was possible. This interval (while the new Local Host Cache broker acquires session information from all VDAs during registration) is unavoidable. When an outage starts, connected sessions are not impacted during the transition interval, but new sessions and session reconnections can be.
This interval occurs whenever VDAs must register with a different Local Host Cache broker, which occurs in the following scenarios:
- An outage starts: When migrating from a Brokering Principal to a Local Host Cache broker.
- Local Host Cache broker failure during an outage: When migrating from a Local Host Cache broker that failed to a newly elected Local Host Cache broker.
- Recovery from an outage: When normal operations resume, and the Brokering Principal resumes control.
The time it takes to synchronize configuration increases with the number of objects (such as VDAs, applications, groups). For example, synchronizing 5,000 VDAs might take ten minutes or more to complete.
Differences from XenApp 6.x releases
Although this Local Host Cache implementation shares the name of the Local Host Cache feature in XenApp 6.x and earlier XenApp releases, this Local Host Cache is an entirely different implementation technically. This implementation has significant improvements, is more robust, is more immune to corruption, and requires less maintenance.
Manage Local Host Cache
In a Citrix Virtual Apps and Desktops service deployment, Local Host Cache is always enabled. You don’t have to do anything else to configure or manage it.
As noted previously, the Microsoft SQL Server Express LocalDB database is installed automatically when you install a Cloud Connector in a resource location. Do not attempt to disable or remove it. Citrix updates the Cloud Connector regularly. If you disable or remove the SQL Server Express LocalDB software manually, the next Cloud Connector update replaces it.
Verify that Local Host Cache is working
To verify that Local Host Cache is set up and working correctly:
- Verify that the resource location contains a local StoreFront that points to the Cloud Connectors in that resource location.
- Ensure that synchronization imports complete successfully. Check the event logs.
- Ensure that the Local Host Cache database was created on each Cloud Connector. This confirms that the High Availability Service can take over, if needed.
- On the Cloud Connector server, browse to
- Verify that
- On the Cloud Connector server, browse to
- Force an outage on all Cloud Connectors in the resource location. After you’ve verified that Local Host Cache works, remember to place all of the Cloud Connectors back into normal mode. This can take approximately 15 minutes.
Event logs indicate when synchronizations and outages occur. In event viewer logs, outage mode is referred to as HA mode.
Config Synchronizer Service
During normal operations, the following events can occur when the CSS imports the configuration data into the Local Host Cache database using the Local Host Cache broker.
- 503: The Citrix Config Sync Service received an updated configuration. This event occurs each time an updated configuration is received from Citrix Cloud. It indicates the start of the synchronization process.
- 504: The Citrix Config Sync Service imported an updated configuration. The configuration import completed successfully.
- 505: The Citrix Config Sync Service failed an import. The configuration import did not complete successfully. If a previous successful configuration is available, it will be used if an outage occurs. However, it will be out-of-date from the current configuration. If there is no previous configuration available, the service cannot participate in session brokering during an outage. In this case, see the Troubleshoot section, and contact Citrix Support.
- 507: The Citrix Config Sync Service abandoned an import because the system is in outage mode and the Local Host Cache broker is being used for brokering. The service received a new configuration, but the import was abandoned because an outage occurred. This is expected behavior.
High Availability Service
This service is also known as the Local Host Cache broker.
- 3502: An outage occurred and the Local Host Cache broker is performing broker operations.
- 3503: An outage was resolved and normal operations have resumed.
- 3504: Indicates which Local Host Cache broker is elected, plus other Local Host Cache brokers involved in the election.
Force an outage
You might want to deliberately force an outage.
- If your network is going up and down repeatedly. Forcing an outage until the network issues are resolved prevents continuous transition between normal and outage modes (and the resulting frequent VDA registration storms).
- To test a disaster recovery plan.
- To help ensure that Local Host Cache is working correctly.
To force an outage, edit the registry of each Cloud Connector server. In
HKLM\Software\Citrix\DesktopServer\LHC, create and set
1. This instructs the Local Host Cache broker to enter outage mode, regardless of the state of the connection to Citrix Cloud. Setting the value to
0 takes the Local Host Cache broker out of outage mode.
To verify events, monitor the
Current_HighAvailabilityService log file in
Several troubleshooting tools are available when a synchronization import to the Local Host Cache database fails and a 505 event is posted.
CDF tracing: Contains options for the ConfigSyncServer and BrokerLHC modules. Those options, along with other broker modules, can identify the problem.
Report: If a synchronization import fails, you can generate a report. This report stops at the object causing the error. This report feature affects synchronization speed, so Citrix recommends disabling it when not in use.
To enable and produce a CSS trace report, enter the following command:
New-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer\LHC
-Name EnableCssTraceMode -PropertyType DWORD -Value 1
The HTML report is posted at:
After the report is generated, enter the following command to disable the reporting feature:
Set-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer\LHC
-Name EnableCssTraceMode -Value 0
Local Host Cache
In this article
- Data content
- How it works
- What is unavailable during an outage, and other differences
- StoreFront requirement
- Application and desktop support
- RAM size considerations
- CPU core and socket configuration considerations
- Storage considerations
- Performance considerations
- Differences from XenApp 6.x releases
- Manage Local Host Cache
- Verify that Local Host Cache is working
- Event logs
- Force an outage