Tuning XenMobile Operations

The performance and stability of XenMobile operations involves many settings across XenMobile and depends on your NetScaler and SQL Server database configuration. This article focuses on the settings that admins most often configure, related to the tuning and optimization of XenMobile. Citrix recommends that you evaluate each of the settings in this article before deploying XenMobile.

Important:

These guidelines assume that the XenMobile server CPU and RAM is adequate for the number of devices. For more information about scalability, see Scalability and performance.

The following server properties globally apply to operations, users, and devices across an entire XenMobile instance. A change to some server properties requires a restart of each XenMobile server node. XenMobile notifies you when a restart is required.

These tuning guidelines apply to both clustered and non-clustered environments.

hibernate.c3p0.idle_test_period

This XenMobile server property, a Custom Key, determines the idle time in seconds before a connection is automatically validated. Configure the key as follows. Default is 30.

  • Key: Custom Key
  • Key: hibernate.c3p0. idle_test_period
  • Value: 120
  • Display name: hibernate.c3p0. idle_test_period
  • Description: Hibernate idle test period

hibernate.c3p0.max_size

This Custom Key determines the maximum number of connections that XenMobile can open to the SQL Server database. XenMobile uses the value you specify for this custom key as an upper limit. The connections open only if you need them. Base your settings on the capacity of your database server.

Note the following equation in a clustered configuration. Your c3p0 connection multiplied by the number of nodes equals your actual maximum number of connections that XenMobile can open to the SQL Server database.

In clustered and non-clustered configuration, setting the value too high with an undersized SQL Server can cause resource issues on the SQL side during peak load. Setting the value too low means you might not be able to take advantage of the SQL resources available.

Configure the key as follows. Default is 1000.

  • Key: hibernate.c3p0.max_size
  • Value: 1000
  • Display name: hibernate.c3p0.max_size
  • Description: DB connections to SQL

hibernate.c3p0.min_size

This Custom Key determines the minimum number of connections that XenMobile opens to the SQL Server database. Configure the key as follows. Default is 100.

  • Key: hibernate.c3p0.min_size
  • Value: 100
  • Display name: hibernate.c3p0.min_size
  • Description: DB connections to SQL

hibernate.c3p0.timeout

This Custom Key determines the idle time-out. If you use database cluster failover, Citrix recommends that you add this Custom Key and set it to reduce the idle time-out. Default is 120.

  • Key: Custom Key
  • Key: hibernate.c3p0.timeout
  • Value: 120
  • Display name: hibernate.c3p0.timeout
  • Description: Database idle timeout

Push Services Heartbeat Interval

This setting determines how frequently an iOS device checks if an APNs notification is not delivered in the interim. Increasing the APNs heartbeat frequency can optimize database communications. Too large a value can add unnecessary load. This setting applies only to iOS. Default is 20 hours.

If you have many iOS devices in your environment, the heartbeat interval can lead to higher load than necessary. Security actions, such as selective wipe, lock, and full wipe, do not rely on this heartbeat. The reason is that an APNs notification is sent to the device when these actions are executed. This value governs how quickly a policy updates after Active Directory Group membership changes. As such, it is often suitable to increase this value to something between 12 and 20 hours to reduce load.

iOS MDM APNS connection pool size

An APNs connection pool that is too small can negatively affect APNs activity performance when you have more than 100 devices. Performance issues include slower deployment of apps and policies to devices and slower device registration. Default is 1. We recommend that you increase this value by 1 for about every 400 devices, up to a maximum value of 15.

auth.ldap.connect.timeout

To compensate for slow LDAP responses, Citrix recommends that you add server properties for the following Custom Key.

  • Key: Custom Key
  • Key: auth.ldap.connect.timeout
  • Value: 60000
  • Display name: auth.ldap.connect.timeout
  • Description: LDAP connection timeout

auth.ldap.read.timeout

To compensate for slow LDAP responses, Citrix recommends that you add server properties for the following Custom Key.

  • Key: Custom Key
  • Key: auth.ldap.read.timeout
  • Value: 60000
  • Display name: auth.ldap.read.timeout
  • Description: LDAP read timeout

Other Server Optimizations

     
Server Property Default Setting Why Change This Setting?
Background Deployment 1,440 minutes The frequency for background policy deployments, in minutes. Applies only to always-on connections for Android devices. Increasing the frequency of policy deployments reduces server load. Recommended setting is 1440 (24 hours).
Background Hardware Inventory 1,440 minutes The frequency for background hardware inventory, in minutes. Applies only to always-on connections for Android devices. Increasing the frequency of hardware inventory reduces server load. Recommended setting is 1440 (24 hours).
Interval for check deleted Active Directory user 15 minutes The standard sync time for Active Directory is 15 minutes. The value 0 prevents XenMobile from checking for deleted Active Directory users. Recommended setting is 15 minutes.
MaxNumberOfWorker 3 The number of threads used when importing many VPP licenses. Defaults to 3. If you need further optimization, you can increase the number of threads. However, be aware that with a larger number of threads, such as 6, a VPP import results in high CPU usage.

Optimizing Deployment Scheduling for Android Devices

You can schedule deployments for Android devices using the Google Firebase Cloud Messaging (FCM).

Enabling FCM for your XenMobile environment allows for near real-time notifications to Android devices, similar to APNs for iOS devices. With FCM configured, when XenMobile needs to connect to a device for a policy update, selective wipe, and so on, the XenMobile server sends a notification message to the FCM server to forward the request to the client device. After the device receives the notification from FCM, the device connects back to XenMobile for further instructions. Keep in mind that this method relies on third-party servers (Google) and therefore is subject to service interruptions outside the control of your IT department or Citrix Support.

For information on how to register with the FCM service, refer to XenMobile and Firebase Cloud Messaging (FCM) Configuration.

If using FCM for Android, be aware of the following XenMobile server properties.

  • FCM API Key: The key created in the Google Developers Console.
  • FCM Sender ID: The Project Number in the Google Developers Console.
  • FCM Registration ID TTL: The delay, in days, before renewing the device FCM registration ID. Defaults to 10.
  • FCM Heartbeat Interval: Defaults to 20 hours.

How to check deadlocks in a SQL DB and delete historical data

When you see deadlocks, run the following query to see the deadlocks. Then, a database administrator or Microsoft SQL team can confirm the information.

SQL Query

SELECT

db.name DB_Service,

tl.request_session_id,

wt.blocking_session_id,

OBJECT_NAME(p.OBJECT_ID) BlockedObjectName,

tl.resource_type,

h1.TEXT AS RequestingText,

h2.TEXT AS BlockingTest,

tl.request_mode

FROM sys.dm_tran_locks AS tl

INNER JOIN sys.databases db ON db.database_id = tl.resource_database_id

INNER JOIN sys.dm_os_waiting_tasks AS wt ON tl.lock_owner_address = wt.resource_address

INNER JOIN sys.partitions AS p ON p.hobt_id = tl.resource_associated_entity_id

INNER JOIN sys.dm_exec_connections ec1 ON ec1.session_id = tl.request_session_id

INNER JOIN sys.dm_exec_connections ec2 ON ec2.session_id = wt.blocking_session_id

CROSS APPLY sys.dm_exec_sql_text(ec1.most_recent_sql_handle) AS h1

CROSS APPLY sys.dm_exec_sql_text(ec2.most_recent_sql_handle) AS h2

GO

Clean up the database

Important:

Back up your database before you make changes to tables.

  1. Run the following query to check the historical data.

    select COUNT(*) as total_record from dbo.EWDEPLOY_HISTO;
    select COUNT(*) as total_record from dbo.EWSESS;
    select COUNT(*) as total_record from dbo.EWAUDIT;
    
  2. Delete the data from the preceding three tables.

    Note:

    You may not see historical data in a table. If so, skip running the truncate query for that particular table.

    truncate TABLE dbo.EWDEPLOY_HISTO;
    truncate TABLE dbo.EWSESS;
    truncate TABLE dbo.EWAUDIT;
    
  3. Unblock the SELECT queries which were blocked due to deadlocks. This step takes care of further deadlocks.

    ALTER DATABASE <database_name> SET       READ_COMMITTED_SNAPSHOT ON WITH ROLLBACK IMMEDIATE
    
  4. By default, database cleanup is seven days for retaining session retention and audit retention data, which is high for many users. Change the cleanup value to 1 or 2 days. In server properties, make the following change:

    zdm.dbcleanup.sessionRetentionTimeInDays = 1 day
    zdm.dbcleanup.deployHistRetentionTimeInDays = 1 day
    zdm.dbcleanup.auditRetentionTimeInDays=1 day
    

Clean up orphans in the KEYSTORE table

If XenMobile nodes have poor performance, check if the KEYSTORE table is too large. XenMobile stores enrollment certificates in the ENROLLMENT_CERTIFICATE and KEYSTORE tables. When you delete or re-enroll devices, the certificates in the ENROLLMENT_CERTIFICATE table get deleted. Entries in the KEYSTORE table remain, which can cause performance issues. Perform the following procedure to clean the orphans from the KEYSTORE table.

Important:

Back up your database before you make changes to tables.

  1. Run the following query to check the historical data.

    select COUNT(*) from KEYSTORE
    
  2. Check for orphans in the KEYSTORE table with the following query.

    WITH cte(KEYSTORE_ID)
    AS (SELECT KEYSTORE_ID
        FROM ENROLLMENT_CERTIFICATE
        UNION
        SELECT CA_KEYSTORE_ID
        FROM LDAP_CONFIG
        UNION
        SELECT CLIENT_KEYSTORE_ID
        FROM LDAP_CONFIG
        UNION
        SELECT KEYSTORE_ID
        FROM SAML_SERVICE_PROVIDER
        UNION
        SELECT KEYSTORE_ID
        FROM SERVER_CERTIFICATE)
    SELECT keystore.id
    FROM keystore
        LEFT JOIN cte ON keystore.id = cte.KEYSTORE_ID
    WHERE KEYSTORE_ID IS NULL;
    
  3. Clear the orphans using the following query.

    WITH cte(KEYSTORE_ID)
        AS (SELECT KEYSTORE_ID
            FROM ENROLLMENT_CERTIFICATE
            UNION
            SELECT CA_KEYSTORE_ID
            FROM LDAP_CONFIG
            UNION
            SELECT CLIENT_KEYSTORE_ID
            FROM LDAP_CONFIG
            UNION
            SELECT KEYSTORE_ID
            FROM SAML_SERVICE_PROVIDER
            UNION
            SELECT KEYSTORE_ID
            FROM SERVER_CERTIFICATE)
        DELETE FROM keystore
        WHERE id IN
        (
            SELECT keystore.id
            FROM keystore
                LEFT JOIN cte ON keystore.id = cte.KEYSTORE_ID
            WHERE KEYSTORE_ID IS NULL AND keystore.TYPE = 'X_509'
        );
    
  4. Add an index to the KEYSTORE table to improve search efficiency.

    DROP INDEX "KEYSTORE_NAME_IDX" ON "KEYSTORE";
    ALTER TABLE "KEYSTORE" ALTER COLUMN "NAME" NVARCHAR(255) NULL;
    CREATE INDEX "KEYSTORE_NAME_IDX" ON "KEYSTORE"("NAME") INCLUDE ("ID", "TYPE", "CONTENT", "PASSWORD", "PUBLICLY_TRUSTED", "DESCRIPTION", "ALIAS", "MODIFICATION_DATE");