Using Local Host Cache for Nondisruptive Database Upgrades

The Local Host Cache (LHC) feature allows connection brokering operations in a Citrix Virtual Apps and Desktops site to continue when an outage occurs.
The following procedure demonstrates how LHC can be used to perform a nondisruptive upgrade of the Site when there are no secondary zones.

Before continuing it is recommended to review the Local Host Cache feature, its requirements and limitations: https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/manage-deployment/local-host-cache.html

There is also a Tech Zone guide on Local Host Cache sizing and scaling which can be found here: https://docs.citrix.com/en-us/tech-zone/design/design-decisions/local-host-cache-sizing-scaling.html

Disclaimer: Perform these steps in a test environment before implementing them in a live production environment to be sure you are familiar with the process and prepared for any environment specific issues or questions that arise. It is also recommended to use the latest LTSR Cumulative Update (CU) available as there are several fixes related to LHC that can benefit your environment.

Overview

1. Configure the environment for this procedure.
2. Determine the primary broker.
3. Force an outage to trigger the Local Host Cache feature.
4. Allow VDAs to re-register with the elected secondary brokers.
5. Perform the product upgrade on an unelected secondary broker.
6. Perform the mandatory site upgrade including database upgrade.
7. Perform product upgrades on any remaining unelected secondary brokers.
8. Exit the outage and Local Host Cache mode.
9. Allow VDAs to re-register with newly upgraded Delivery Controllers.
10. Perform product upgrade on last remaining Delivery Controller (previously elected secondary broker).
11. Return the environment to default configuration.

Procedure

1. Check to determine whether Local Host Cache is enabled using the following PowerShell cmdlet.

   Get-BrokerSite

   Look for LocalHostCacheEnabled : True

   

   If false, enable Local Host Cache.

   Set-BrokerSite -LocalHostCacheEnabled $true -ConnectionLeasingEnabled $false

   This cmdlet also disables the connection leasing feature. Do not enable both Local Host Cache and connection leasing.

2. By default, power-managed desktop VDAs in pooled Delivery Groups that have the "ShutdownDesktopsAfterUse" property enabled are placed into maintenance mode when an outage occurs. To override the default behavior, you must enable it site-wide and for each affected Delivery Group. Run the following PowerShell cmdlets.

   Set-BrokerSite -ReuseMachinesWithoutShutdownInOutageAllowed $true

   Set-BrokerDesktopGroup -Name "<Delivery Group Name>"- ReuseMachinesWithoutShutdownInOutage $true

3. If the Broker Service has been configured to use custom VDA, StoreFront, or StoreFront TLS ports, perform the following steps to ensure the high availability (HA) service is also configured with the correct custom ports.

   • Verify the current Broker Service port settings on each Broker by issuing the following command: %programfiles%\Citrix\Broker\Service\BrokerService.exe -show

     

   • Verify the current HA Service port settings on each Broker by issuing the following command: %programfiles%\Citrix\Broker\Service\HighAvailabilityService.exe -show

     

   • If the VDA, StoreFront or StoreFront TLS ports listed for the HA Service do not match the Broker Service, use the appropriate command line switches listed to set the HA Service port settings to match accordingly:

   %programfiles%\Citrix\Broker\Service\HighAvailabilityService.exe -VdaPort <port>
   %programfiles%\Citrix\Broker\Service\HighAvailabilityService.exe -StoreFrontPort <port>
   %programfiles%\Citrix\Broker\Service\HighAvailabilityService.exe -StoreFrontTlsPort <port>

   

   Note:

   It is expected that the SDK port is different between the Broker Service and HA Service.

   When changing the StoreFront port of the Broker Service, the StoreFront port of the HA service is updated to match automatically. However, the service receiving the automatic update will still need to be restarted manually to begin using the new port.

4. During an outage, the elected secondary broker handles all the connections. When the outage begins, the secondary broker has no current VDA registration data, but when a VDA communicates with it, a re-registration process is triggered. During that process, the secondary broker also gets current session information about that VDA. To accelerate the re-registration of the VDAs from the default 5-minute interval to a 1 minute interval, this setting must be applied to all Controllers in the site.

   New-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer -Name HeartbeatPeriodMs -PropertyType DWORD -Value 60000

5. To monitor the VDA re-registrations launch Citrix Studio and click the Configuration > Controllers node and view the number of VDAs registered with primary brokers. Leave the Citrix Studio open to see the counts fall to zero as the VDAs re-register with the elected secondary broker during the outage. Note that you cannot use Citrix Studio to view the count of VDAs registered with the secondary broker.

6. To force the outage and enter into LHC mode, edit the registry of each Delivery Controller.

   New-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer\LHC -Name OutageModeForced -PropertyType DWORD -Value 1

7. To determine if the outage has been triggered and each primary broker has entered LHC mode, go to the Application node of the Event logs on each Controller and look for the following event from the Citrix High Availability Service.

   3502: The Citrix High Availability Service has become active and will broker user request for sessions until the issue discovered with the normal brokering activity is resolved.

   

8. Confirm all VDAs have re-registered with the elected secondary broker by refreshing the Controllers node in Citrix Studio. All VDAs have likely re-registered when the primary brokers show zero registered VDAs.

9. The secondary brokers use the alphabetical list of FQDNs of the machines that they're running on to determine (elect) which secondary broker will be in charge of brokering operations in the zone if an outage occurs. To confirm which secondary broker has been elected look for the following event from the Citrix High Availability Service in the Windows Event Application Logs.

   3504: The Citrix High Availability Service 'FQDN of elected Controller' has become the elected instance among its peers (list of peer Controller FQDNs).

   

10. Choose one of the unelected peer Controllers and perform the product upgrade on the unelected Controller.

11. From the newly upgraded Controller launch Citrix Studio and perform the mandatory site upgrade including the database upgrade.

    

12. Perform product upgrades on the remaining unelected peer Controllers. Be sure to not disrupt the elected Controller which is still managing all new and active connections in the environment.

13. After all unelected Controllers have been upgraded, it is time to bring the site out of the outage and transition out of LHC mode. To remove the forced outage trigger, edit the registry of each Controller. The key can also be deleted if preferred.

    Set-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer\LHC -Name OutageModeForced -Value 0

14. To confirm if the site is out of the outage mode, on each Controller look for the following events from the Citrix Broker Service in the Application event log.

    3004: The Citrix Broker Service successfully connected to the XenDesktop database.

    

    3500: The Citrix Broker Service has detected that the issue with communication with the database has been resolved and will resume normal brokering activity using configuration in the main site database.

    

15. Refresh the Controllers node from Citrix Studio to watch the VDAs re-register with the upgraded Controllers. Confirm all VDAs have re-registered successfully.

16. Perform the product upgrade on the last remaining Controller which served as the elected secondary broker during the outage.

17. (This step is optional.) Set the VDA registration interval back to the default value of 5 minutes by modifying the registry on each Controller (the key can also be deleted if preferred).

    Set-ItemProperty -Path HKLM:\SOFTWARE\Citrix\DesktopServer -Name HeartbeatPeriodMs -PropertyType DWORD -Value 300000

18. (This step is optional.) Use the following cmdlets if you want to change back to the default behavior of the power-managed Delivery Groups.

    Set-BrokerSite  -ReuseMachinesWithoutShutdownInOutageAllowed $false
    Set-BrokerDesktopGroup -Name "<Delievery Group Name>" -ReuseMachinesWithoutShutdownInOutage $false

The nondisruptive upgrade using Local Host Cache should now be complete.