Deploy NetScaler MAS in High Availability Mode

A high availability (HA) deployment of two NetScaler MAS servers can provide uninterrupted operation in any transaction. An HA pair of NetScaler MAS servers is in active-passive mode. When two NetScaler MAS servers are configured in active-passive mode, both servers have the same configuration. This active-passive deployment type is ideal for disaster recovery.

Important

To access NetScaler MAS 12.0 build 57.x or later versions of 12.0 using HTTPS:

If you have configured a NetScaler appliance to load balance NetScaler MAS in a high availability mode, update the NetScaler settings to allow all ciphers (in the backend) to access NetScaler MAS.  

In this type of active-passive deployment, one NetScaler MAS server is configured as the primary node and the other as the secondary node. If, for any reason, the primary node goes down, the secondary node takes over.

The secondary node listens to the heartbeat messages that the primary node sends through the NetScaler MAS database. If the secondary node does not receive the heartbeats for a specific period, the secondary node performs an SSH-based check on the primary node. If the heartbeat and SSH based check fails, the primary node is considered to be down. In such a scenario, the secondary node takes over as the primary node (a process called failover). The whole process requires about 15–20 seconds, depending on hardware infrastructure and network.

A failover occurs if one of the following conditions is encountered:

  • Primary server loses connectivity in the LAN.

  • Primary server encounters a critical software issue.

  • Primary server encounters a hardware issue, such as power failure.

  • The administrator manually forces the primary server to shut down and the secondary server takes over.

The following figure shows a high availability deployment of two NetScaler MAS servers in active-passive mode.

localized image

When configuring your NetScaler MAS deployment in HA mode, installing both the server nodes in the same subnet is recommended.

Note

You can also configure your NetScaler MAS deployment in HA mode with unified GUI access. For more information, see Configuring Load Balancing of the NetScaler MAS HA Pair on a NetScaler Instance.

Points to Note:

  • You can access an HA node from the GUI by entering either the primary or secondary IP address. You can see the marking as Passive for the secondary node. If you have configured load balancing in NetScaler MAS with unified access, enter the load balancing virtual server IP address to view and manage your NetScaler MAS HA setup.

  • Although you can make configuration changes on both the primary and secondary nodes, Citrix recommends you to make changes only on the active node.

  • In a high availability setup, all configuration files are synchronized automatically from the primary node to the secondary node at an interval of one minute.

  • Database synchronization happens instantly and is taken care by the backup and disaster recovery (BDR) software.

  • In a high availability setup, all system settings configuration done on the primary node propagates automatically to the secondary. For all other commands, the database synchronization takes care the propagation from the primary to secondary and you need not run these commands separately.

  • As part of the upgrade process, all configurations are updated. Therefore, all NetScaler instances are now managed by the active node.

  • After you upgrade the NetScaler MAS to active-passive mode, some NetScaler instances continue to send traffic to the passive node for approximately 5 minutes. As a result, the incoming traffic is lost for that duration.

  • Consider that you had configured a load balancing virtual server for unified management access of your NetScaler MAS servers. After you upgrade the NetScaler MAS HA pair to active-passive mode, you have to run the following command on the NetScaler appliance to update the load balancing configuration:

     add lb monitor MAS_Monitor TCP-ECV -send "GET/mas_health HTTP/1.1\r\nAccept-Encoding:identity\r\nUser-Agent NetScaler-Monitor\r\nConnection: close\r\n\r\n\" -recv "{\"statuscode\":0, "is_passive\":0}" -LRTM DISABLED
    
  • In a high availability setup, the heartbeat messages set up through Nitro commands and Database synchronization.

  • NetScaler MAS does not support forced synchronization. You cannot force the synchronization from either the primary or the secondary node.

  • Automatic HA synchronization is enabled by default. You cannot enable or disable automatic HA synchronization after you deploy the HA pair.

  • In a high availability setup, you must open the following ports in both the NetScaler MAS servers:

    • For ICMP (ping) – (No reserved port) To detect network reachability.

    • For NITRO communications (TCP) - Port 443.

    • For synchronization - Port 22.

    • For Database synchronization - Port 5454.

  • You cannot use Nitro calls to force the primary to stay primary and the secondary to stay secondary.

  • Always use the GUI to perform the NetScaler MAS upgrade.

  • In a high availability setup you can initiate the upgrade on either of the nodes by their GUI. The other node is automatically upgraded. Citrix recommends you to always use the primary node for the upgrade process.
  • After completion of the upgrade process, either node can act as a primary node. No data is lost during the upgrade.

  • You can gracefully shut down a node in HA setup so that DB gracefully shuts down. At the command prompt, type shutdown –p now to shut down the node.

  • Do not perform a hard reboot in a high availability setup.

  • It is highly recommended to deploy the NetScaler MAS HA nodes in the same datacenter.

  • NetScaler MAS in both single and HA mode can manage and monitor NetScaler instances that are deployed in different datacenters having latency less than or equal to 400 ms.

  • The latency between two NetScaler MAS HA nodes deployed in different data centers is less than or equal to 10 ms.

Recommended precautions:

  • Back up the NetScaler MAS server before you upgrade.

  • When upgrading NetScaler MAS servers in a high availability setup, do not make any configuration changes on either of the nodes.

    Warning

    Do not refresh your browser until the upgrade process is successfully completed. It might take a few minutes for the upgrade process to finish.

Prerequisites

Before you set up HA for NetScaler MAS, note the following requirements:

  • The NetScaler MAS active-passive HA deployment mode is supported from NetScaler MAS version 12.0 build 51.24.

  • In an HA setup, both nodes must run the same version of NetScaler MAS System software.

  • You have to have downloaded the NetScaler Management and Analytics System image file.

Citrix recommends that you set CPU priority in virtual machine properties to the highest level. High CPU priority improves scheduling behavior and network latency.

The following table lists the minimum requirements for the virtual computing resources that XenServer, VMWare ESX, or Microsoft Hyper-V must provide for each component of this deployment.

Component Minimum Requirement
RAM 32 GB
  Note: The default value is 8 GB. Citrix recommends that you increase the default value to 32 GB for better performance.
Virtual CPU 8 CPUs
  Note: The default is 2 CPUs. Citrix recommends that you increase the default value to 8 CPUs for better performance.
Storage space Citrix recommends using solid-state drive (SSD) technology for NetScaler MAS deployments. The default value is 120 GB. Actual storage requirement depends on NetScaler MAS sizing estimation. If your NetScaler MAS storage requirement exceeds 120 GB, you to have to attach an additional disk. Note. You can add only one additional disk. Citrix recommends you to estimate storage and attach additional disk at the time of initial deployment. For more information, see How to Attach an Additional Disk to NetScaler MAS.
Virtual Network Interfaces 1
Throughput 1 Gbps
Hypervisor Requirements
XenServer 6.2, 6.5
VMWare ESX 5.5, 6.0
Microsoft Hyper-V 2012 R2
Linux - KVM Ubuntu, Fedora

Installing NetScaler MAS in high availability mode

Obtain the NetScaler MAS image file from the Citrix download site. Installing a NetScaler MAS in HA mode involves the following steps:

  1. Provisioning the first server node

  2. Provisioning the second server node

  3. Deploying the two server nodes in HA mode

Provisioning the first server node

To begin provisioning the NetScaler MAS HA setup, install the first NetScaler MAS server. Use the image file that you downloaded from the Citrix download site.

  1. Import the image file to your hypervisor, and then from the Console tab configure the initial network configuration options as explained on the following screen:

    localized image

  2. After specifying the required IP addresses, select the deployment type as NetScaler MAS Server. If you do not select any option, by default, it is deployed as a server.

    localized image

  3. The deployment console prompts you to select the server deployment (as Standalone). Enter No to confirm the deployment as HA pair.

    localized image

  4. The console prompts you to select the (first server node). Enter Yes to confirm the node as the first server node.

    localized image

  5. The console prompts you to restart the server. Enter Yes to restart.

    localized image

Provisioning the second server node

After provisioning the first NetScaler MAS server, provision the second server. You can use the same image file that you used to install the first server. You can also obtain that same version of the image from the Citrix download site.

  1. Import the image file to your hypervisor, and then from the Console tab configure the initial network configuration options as explained on the following screen:

    localized image

  2. After specifying the required IP addresses, select the deployment type as NetScaler MAS server. If you do not select any option, by default, it is deployed as a server.

    localized image

  3. The deployment console prompts you to select the server deployment (as Standalone). Enter No to confirm the deployment as HA pair.

    localized image

  4. The console then prompts you to select the (first server node). Enter NO to confirm the node as the second server node.

    localized image

  5. Enter the first server’s IP address and password, and reboot the node when the console prompts.

    localized image

Deploying the two servers in high availability mode

To complete the installation process of the two server nodes as an HA pair, deploy these nodes from the GUI of the first server node that you configured. Internal communication between the two servers starts when you deploy the first server node.

  1. In a web browser, type the IP address of the first NetScaler MAS server node (for example, http://10.102.29.52).

  2. In the User Name and Password fields, enter the administrator credentials.

  3. Select the deployment type as Two Servers deployed in High Availability Mode, and click Next.

    localized image

  4. On the System tab, navigate to Deployment and click Deploy.

    localized image

  5. A confirmation message appears. Click Yes.

    localized image

    After you deploy the NetScaler MAS in HA mode, either the first server node or the second server node can be an active node. The active node is identified by the star symbol. In the following figure, the 10.102.29.53 node is shown as the active node. The other node acts as a passive node and is ready to take over when the active node becomes unavailable.

    localized image

Configuring load balancing of the NetScaler MAS HA pair on a NetScaler instance

You can configure the NetScaler MAS in HA mode with unified GUI access. Both the primary and secondary servers must be connected to a load balancing virtual server hosted in a NetScaler instance. The load balancing virtual server sends the requests to the primary NetScaler MAS server. The primary node accepts connections and manages all the devices. All the communications like AppFlow, SNMP, LogStream, and Syslog are managed by the primary node.

You can access an HA node through its IP address or through the load balancing virtual server’s IP address. If you use the load balancing virtual server’s IP address, the GUI of the active node appears. The service state of the passive node is shown as down.

localized image

To configure load balancing of the NetScaler MAS HA pair by using the NetScaler command line:

  1. On a workstation or laptop, open an SSH connection to the instance by using an SSH client, such as PuTTY.

  2. Log on to the NetScaler instance. In User Name and Password, type the administrator credentials. The defaults are nsroot and nsroot, respectively.

  3. At the command prompt, type:

add lb vserver <load_balancer_name> HTTP <vserver_ip> 80 -persistenceType SOURCEIP

add server <MAS_server_1_name> <MAS_server_1_ip>

add server <MAS_server_2_name> <MAS_server_2_ip>

add service <service_1_name> <MAS_server_1_name> HTTP 80 -maxReq 1

add service <service_2_name> <MAS_server_2_name> HTTP 80 -maxReq 1

add lb monitor MAS_Monitor TCP-ECV -send "GET /mas_health HTTP/1.1\r\nAccept-Encoding: identity\r\nUser-Agent: NetScaler-Monitor\r\nConnection: close\r\n\r\n\" -recv "{\"statuscode\":0, \"is_passive\":0}​" -LRTM DISABLED

bind service <service_1_name> -monitorName <Monitor name>

bind service <service_2_name> -monitorName <Monitor name>

bind lb vserver <load_balancer_name> <service_1_name>

bind lb vserver <load_balancer_name> <service_2_name>

To configure the NetScaler MAS HA pair by using the NetScaler GUI:

  1. In a web browser, type the IP address of the NetScaler instance (for example, http://192.168.100.1).

  2. In the User Name and Password fields, enter the administrator credentials.

  3. On the Configuration tab, navigate to Traffic Management > Load Balancing > Virtual Servers and select Add.

  4. Enter the name and IP address for the load balancing virtual server in the Name and IP Address fields.

    Note

    The IP address you enter becomes the load balancing virtual server’s virtual IP (VIP) address. You can then use the VIP address to access the HA pair once the configuration is complete.

  5. In the Protocol field, choose the communication protocol for the load balancing virtual server.

  6. In the Port field, enter the virtual server’s port number. Then, click OK.

  7. Click Persistence and select SourceIP as the persistence type.

  8. To add load balancing services for the NetScaler MAS HA pair, select the Services and Service Groups tab. Select Load Balancing Virtual Server Service Binding to add new services.

    localized image

    Alternatively, to add a load balancing service, navigate to Traffic Management > Load Balancing > Services, and click Add.

    localized image

  9. To set a threshold limit for a service, navigate to Traffic Management > Load Balancing > Services. On the Services page, click the service you want to modify. On the Load Balancing Service page, click Thresholds & Timeouts under the Advanced Settings section on the right of the page. Under Threshold, enter 1 as the value for Max Requests and click OK.

    localized image

    localized image

  10. Add the services for the first and second NetScaler MAS by clicking the plus sign (+) and clicking Bind.

    localized image

  11. Bind a monitor to each service by navigating to Configuration > Traffic Management > Load Balancing > Services. Then, select the monitor type as tcp-ecv for each of the two services.

    localized image

Disabling HA on a NetScaler MAS HA pair

You can disable high availability on a NetScaler MAS HA pair and convert the nodes to stand-alone NetScaler MAS servers. Select one of the NetScaler MAS servers to retain all the data, and remove the other server node from the HA configuration.

To disable High Availability via GUI:

  1. In a web browser, type the IP address of the NetScaler MAS server node that you want to retain the data (for example, http://10.102.29.53).

  2. In the User Name and Password fields, enter the administrator credentials.

  3. On the System tab, navigate to Deployment and click Break HA.

    localized image

    The other server node restarts, and the node on which you issued the command goes out of service for a while. You can remove the other load balancing virtual server and replace its virtual IP address in all the configurations where it was used.

    When the server node that you removed from the HA configuration restarts, all its configurations and settings are deleted. Access its console again to choose a deployment type. The following screen on the console appears automatically.

    localized image

    Select the deployment type as NetScaler MAS Server to start the deployment process again.

    Note

    When breaking HA, data present on the node where you initiate Break HA is retained. You can break HA from either the primary or the secondary node. Citrix recommends doing Break HA from primary node. Only the data that has been synced to the secondary node is saved.

Redeploying High Availability on a NetScaler MAS

After you break the HA in NetScaler MAS to a standalone deployment, you can redeploy the HA in NetScaler MAS. Redeploying HA is similar to the first-time deployment of HA.

To redeploy HA in NetScaler MAS:

  1. Log on to the secondary node using an SSH client.

  2. Use nsrecover as the user name and enter the password that you have set. Run the deployment_type.py script in the secondary node.

    Note: After breaking the HA, you cannot log on to the secondary node console using nsroot as the user name, since the secondary node is down.

  3. Alternately, you can also log on to the hypervisor console and run the deployment_type.py script in the secondary node.

  4. Configure the secondary node. Perform the procedure from step 2 available at Provisioning the Second Server Node.

  5. Deploy the two servers in HA mode.

  6. Configure load balancing for NetScaler MAS HA pair

    Important

    Configure SNMP requests to be sent to individual nodes, if you use an external SNMP server in a NetScaler MAS HA setup. There are no responses if the SNMP server is configured to send requests to the load balancing virtual server’s IP address.