Product Documentation

High Availability Deployment

Oct 22, 2017

A high availability (HA) deployment of two NetScaler MAS servers can provide uninterrupted operation in any transaction. An HA pair of NetScaler MAS servers is in active-passive mode. When two NetScaler MAS servers are configured in active-passive mode, both servers have the same configuration. This active-passive deployment type is ideal for disaster recovery.

In this type of active-passive deployment, one NetScaler MAS server is configured as the primary node and the other as the secondary node. If, for any reason, the primary node goes down, the secondary node takes over.

The secondary node listens to the heartbeat messages that the primary node sends through the NetScaler MAS database. If the secondary node does not receive the heartbeats for a specific period of time, the secondary node performs an SSH based check of NetScaler MAS processes on the primary node. If the heartbeat and SSH based check fails, the primary node is considered to be down and the secondary node takes over as the primary node (a process called failover). The whole process requires about 15-20 seconds, depending on hardware infrastructure and network. 

A failover occurs if one of the following conditions is encountered:

  • Primary server loses connectivity in the LAN.
  • Primary server encounters a critical software issue.
  • Primary server encounters a hardware issue, such as power failure.
  • The administrator manually forces the primary server to shut down and the secondary server takes over.

The following figure shows a high availability deployment of two NetScaler MAS servers in active-passive mode. 

localized image

When configuring your NetScaler MAS deployment in HA mode, installing both the server nodes in the same subnet is recommended. 

Note: You can also configure your NetScaler MAS deployment in HA mode with unified GUI access. For more information, see Configuring Load Balancing of the NetScaler MAS HA Pair on a NetScaler Instance.                   

Points to Note

  • The upgrade process changes the active-active HA setup to an active-passive HA setup.
  • You can access an HA node from the GUI by entering either the primary or secondary IP address. You can see the marking as “Passive” for the secondary node. If you have configured load balancing in NetScaler MAS with unified access, you can enter the load balancing virtual server IP address to view and manage your NetScaler MAS HA setup.
  • Although you can make configuration changes on both the primary and secondary nodes, making them only on the active (primary) node is strongly recommended.
  • In a high availability setup, all configuration files are synchronized automatically from the primary node to the secondary node at an interval of one minute. Database synchronization happens instantly and is taken care by the backup and disaster recovery (BDR) software.
  • In a high availability setup, all system settings configuration done on the primary node propagates automatically to the secondary. For all other commands, the database synchronization takes care the propagation from the primary to secondary and you need not run these commands separately.
  • As part of the upgrade process, all configurations are updated such that all NetScaler instances are now managed by the active node. This is applicable to all configurations such as SNMP, Syslog, Analytics, and so on.
  • After you upgrade the NetScaler MAS to active-passive mode, some NetScaler instances will still continue to send traffic to the passive node for approximately 5 minutes. As a result, the incoming traffic will be lost for that duration.
  • If you had configured a load balancing virtual server for unified management access of your NetScaler MAS servers, after you upgrade the NetScaler MAS HA pair to active-passive mode you have to  run the following command on the NetScaler appliance to update the load balancing configuration:
    add lb monitor MAS_Monitor TCP-ECV -send "GET /mas_health HTTP/1.1\r\nAccept-Encoding: identity\r\nUser-Agent: NetScaler-Monitor\r\nConnection: close\r\n\r\n\"" -recv "{\"statuscode\":0, \"is_passive\":0}" -LRTM DISABLED
  • In a high availability setup, the heartbeat messages set up through Nitro commands and Database synchronization.
  • NetScaler MAS does not support forced synchronization. You cannot force the synchronization from either the primary or the secondary node.
  • Automatic HA synchronization is enabled by default. You cannot enable or disable automatic HA synchronization after you deploy the HA pair.
  • In a high availability setup, you must open the following ports in both the NetScaler MAS servers:
    • For ICMP (ping) – (No reserved port) To detect network reachability.
    • For NITRO communications (TCP) - Port 443.
    • For synchronization - Port 22. 
    • For Database synchronization - Port 5454.
  • You cannot use Nitro calls to force the primary to stay primary and the secondary to stay secondary.
  • It is recommended not to configure HA nodes in different subnets in NetScaler MAS.
  • When upgrading from active-active to active-passive, always use the same software version and build number on both the primary and the secondary server.
  • Always use the GUI to perform the NetScaler MAS upgrade.
  • In a high availability setup, when you initiate the upgrade on either of the nodes via GUI, the other node is automatically upgraded. However, always using the primary node for the upgrade is recommended.
  • After completion of the upgrade process, either node can act as a primary node. No data is lost during the upgrade.
  • You can gracefully shutdown a node in HA setup so that DB gracefully shuts down. At the command prompt, type shutdown –p now to shut down the node.
  • A hard reboot should not be performed in a high availability setup.
  • It is highly recommended to deploy the NetScaler MAS HA nodes in the same datacenter.
  • NetScaler MAS HA and NetScaler MAS single server nodes can manage and monitor NetScaler instances that are deployed in different datacentres having latency less than or equal to 150ms.

Recommended Precautions:

  • Back up the NetScaler MAS server before you upgrade.
  • When upgrading NetScaler MAS servers in a high availability setup, do not make any configuration changes on either of the nodes.

Warning

Do not refresh your browser until the upgrade process is successfully completed. It might take a few minutes for the upgrade process to finish.

Prerequisites

Before you set up HA for NetScaler MAS, note the following requirements:

  • The NetScaler MAS active-passive HA deployment mode is supported from NetScaler MAS version 11.1 build 54.14.
  • In an HA setup, both nodes must run the same version of NetScaler MAS System software.
  • You have to have downloaded the NetScaler Management and Analytics System image file.

For production use of the NetScaler Management and Analytics System, Citrix recommends that you set CPU priority (in virtual machine properties) to the highest level, to improve scheduling behavior and network latency.

The following table lists the minimum requirements for the virtual computing resources that XenServer, VMWare ESX, or Microsoft Hyper-V must provide for each component of this deployment. 

Component

Minnimum Requirement

RAM

32 GB

Note: The default value is 8 GB. Citrix recommends that you increase the default value to 32 GB for better performance.

Virtual CPU

8 CPUs

Note: The default is 2 CPUs. Citrix recommends that you increase the default value to 8 CPUs for better performance.

Storage space

The default value is 120 GB. Actual storage requirement will depend on NetScaler MAS sizing estimation.

If your NetScaler MAS storage requirement exceeds 120 GB, you to have to attach an additional disk. Note that you can add only one additional disk.

Citrix recommends you to estimate storage and attach additional disk at the time of initial deployment.

For more information, see How to Attach an Additional Disk to NetScaler MAS.

Virtual Network Interfaces

1

Throughput

1 Gbps

Hypervisor Requirements

XenServer

6.2, 6.5

VMWare ESX

5.5, 6.0

Microsoft Hyper-V

2012 R2

Linux - KVM

Ubuntu, Fedora

Installing NetScaler MAS in HA Mode

Obtain the NetScaler MAS image file from the Citrix download site. Installing a NetScaler MAS in HA mode involves the following steps:

  1. Provisioning the first server node
  2. Provisioning the second server node
  3. Deploying the two server nodes in HA mode

Provisioning the First Server Node

To begin provisioning the NetScaler MAS HA setup, install the first NetScaler MAS server. Use the image file that you downloaded from the Citrix download site.

  1. Import the image file to your hypervisor, and then from the Console tab configure the initial network configuration options as explained on the following screen:
localized image
2. After specifying the required IP addresses, select the deployment type as NetScaler MAS Server. If you do not select any option, by default, it’ll be deployed as a server.
 
localized image

3. The deployment console prompts you to select the server deployment (as Standalone). Enter No to confirm the deployment as HA pair.

localized image

4. The console prompts you to select the (first server node). Enter Yes to confirm the node as the first server node.

localized image

5. The console prompts you to restart the server. Enter Yes to restart.

localized image

Provisioning the Second Server Node

After provisioning the first NetScaler MAS server, provision the second server. On your hypervisor, use the same image file that you used to install the first server, or obtain that same version of the image from the Citrix download site.

  1. Import the image file to your hypervisor, and then from the Console tab configure the initial network configuration options as explained on the following screen:
localized image

2. After specifying the required IP addresses, select the deployment type as NetScaler MAS server. If you do not select any option, by default, it will be deployed as a server.

localized image

3. The deployment console prompts you to select the server deployment (as Standalone). Enter No to confirm the deployment as HA pair.

localized image

4. The console then prompts you to select the (first server node). Enter NO to confirm the node as the second server node.

localized image

5. Enter the first server’s IP address and password, and reboot the node when the console prompts.

localized image

Deploying the two servers in HA mode

To complete the installation process of the two server nodes as an HA pair, you have to deploy these nodes from the GUI of the first server node that you configured. Internal communication between the two servers starts when you deploy the first server node.

  1. In a web browser, type the IP address of the first NetScaler MAS server node (for example, http://10.102.29.52).
  2. In the User Name and Password fields, enter the administrator credentials.
  3. Select the deployment type as Two Servers deployed in High Availability Mode, and click Next.
localized image

4. On the System tab, navigate to Deployment and click Deploy.

localized image

5. A confirmation message appears. Click Yes.

localized image

After you deploy the NetScaler MAS in HA mode, either the first server node or the second server node can be an active node. The active node is identified by the star symbol as shown in the following figure, which shows the node at 10.102.29.53 as the active node. The other node acts as a passive node and is ready to take over as soon as the active node becomes unavailable.

localized image

Configuring Load Balancing of the NetScaler MAS HA Pair on a NetScaler Instance

You can configure the NetScaler MAS in HA mode with unified GUI access. Both the primary and secondary servers must be connected to a load balancing virtual server hosted in a NetScaler instance. The load balancing virtual server sends the requests to the primary NetScaler MAS server. The primary node accepts connections and manages all the devices. All the communications like AppFlow, SNMP, LogStream, Syslog, and so on are managed by the primary node.

You can access an HA node through its IP address or through the load balancing virtual server's IP address. If you use the load balancing virtual server's IP address, the GUI of the active node appears. The service state of the passive node is shown as down.

localized image

To configure load balancing of the NetScaler MAS HA pair by using the NetScaler command line

1. On a workstation or laptop, open an SSH connection to the instance by using an SSH client, such as PuTTY.

2. Log on to the NetScaler instance. In User Name and Password, type the administrator credentials. The defaults are nsroot and nsroot, respectively.

3. At the command prompt, type:

add lb vserver <Load Balancer name> HTTP <Vserver_ip> 80 -persistenceType SOURCEIP

add server <NMAS Server 1 name> < NMAS Server 1_ip>

add server <NMAS Server 2 name> < NMAS Server 2_ip>

add service <Service 1 name> <NMAS Server 1 name> HTTP 80 -maxReq 1

add service <Service 2 name> <NMAS Server 2 name> HTTP 80 -maxReq 1

add lb monitor MAS_Monitor TCP-ECV -send "GET /mas_health HTTP/1.1\r\nAccept-Encoding: identity\r\nUser-Agent: NetScaler-Monitor\r\nConnection: close\r\n\r\n\"" -recv "{\"statuscode\":0, \"is_passive\":0}​" -LRTM DISABLED

bind service <Service 1 name> -monitorName <Monitor name>

bind service <Service 2 name> -monitorName <Monitor name>

bind lb vserver <Load Balancer name> <Service 1 name>

bind lb vserver <Load Balancer name> <Service 2 name>

To configure the NetScaler MAS HA pair by using the NetScaler GUI

1. In a web browser, type the IP address of the NetScaler instance (for example, http://192.168.100.1).

2. In the User Name and Password fields, enter the administrator credentials.

3. On the Configuration tab, navigate to Traffic Management > Load Balancing > Virtual Servers and select Add.

4. Enter the name and IP address for the load balancing virtual server in the Name and IP Address fields, respectively.
Note: The IP address you enter becomes the load balancing virtual server’s virtual IP (VIP) address and can be used to access the HA pair once the configuration is complete

5. In the Protocol field, choose the communication protocol for the load balancing virtual server.

6. In the Port field, enter the virtual server's port number. Then, click OK.

7. Click Persistence and select SourceIP as the persistence type.

8. To add load balancing services for the NetScaler MAS HA pair, select the Services and Service Groups tab, and then select Load Balancing Virtual Server Service Binding to add new services.

localized image

Alternatively, to add a load balancing service, navigate to Traffic Management > Load Balancing > Services, and click Add.

localized image

9. To set a threshold limit for a service, navigate to Traffic Management > Load Balancing > Services. On the Services page, click the service you want to modify. On the Load Balancing Service page, click Thresholds & Timesouts under the Advanced Settings section on the right of the page. Under Threshold, enter 1 as the value for Max Requests and click OK.

localized image
localized image

10. Add the services for the first and second NetScaler MAS by clicking the plus sign (+) and clicking Bind.

localized image

11. Bind a monitor to each service by navigating to Configuration > Traffic Management > Load Balancing > Services and selecting the monitor type as tcp-ecv for each of the two services. 

localized image

Disabling HA on a NetScaler MAS HA pair

You can disable high availability on a NetScaler MAS HA pair and convert the nodes to stand-alone NetScaler MAS servers. In the GUI, you can select one of the NetScaler MAS servers to retain all the data, and remove the other server node from the HA configuration.

To disable High Availability via GUI:

  1. In a web browser, type the IP address of the NetScaler MAS server node that you want to retain the data (for example, http://10.102.29.53).
  2. In the User Name and Password fields, enter the administrator credentials.
  3. On the System tab, navigate to Deployment and click Break HA.
localized image

The other server node restarts, and the node on which you issue the command goes out of service for a while. You can remove the other load balancing virtual server and replace its VIP address in all the configurations in which it was used.

When the server node that you remove from the HA configuration restarts, all its configurations and settings are deleted. You have to access its console to choose a deployment type. The following screen on the console appears automatically.

localized image

Select the deployment type as NetScaler MAS Server to start the deployment process again. 

Note

When breaking HA, data on the node where you initiate "Break HA" is retained. You can break HA from either the primary or the secondary node, but doing so from primary node avoids losing data that has not been synced to the secondary node.

Redeploying High Availability on a NetScaler MAS

After you break the HA in NetScaler MAS to a standalone deployment, you can redeploy the HA in NetScaler MAS. Redeploying HA is similar to the first-time deployment of HA.

To redeploy HA in NetScaler MAS

1. On a workstation or laptop, open an SSH connection to the appliance by using an SSH client. Log in to the appliance using nsrecover as the user name and enter the password that you have set. Run the deployment_type.py script in the secondary node.

Note: After breaking the HA, you cannot log in to the secondary node console using nsroot as the user name, since the secondary node is down.

2. Alternately, you can also log in to the hypervisor console and run the deployment_type.py script in the secondary node.

3. In the GUI, access the primary node of the HA pair. Navigate to System > Deployment and then click the Deploy button. The system reboots and the HA pair is established.

Important

If you use an external SNMP server in a NetScaler MAS HA setup, you have to configure SNMP requests to be sent to individual nodes. You will not get responses if the SNMP server is configured to send requests to the load balancing virtual server's IP address.