Product Documentation

Configure high availability deployment

High Availability (HA) refers to a system that is always available to a user without any interruption to the services. High availability setup is crucial during system downtime, network or application failures, and is a key requirement to any enterprise. A high availability deployment of two NetScaler MAS nodes in active-passive mode with same configurations provides uninterrupted operations.

Important

To access NetScaler MAS 12.1 build 48.18 or later versions using HTTPS:

If you have configured a NetScaler appliance to load balance NetScaler MAS in a high availability mode, first remove the NetScaler appliance. Then, configure a floating IP address to access NetScaler MAS in high availability mode.

The following are the benefits of high availability deployment in NetScaler MAS:

  • An improved mechanism to monitor heartbeats between the primary and secondary node.

  • Provides physical streaming replication of database instead of a logical bi-directional replication.

  • Ability to configure the floating IP address on the primary node to eliminate the need of separate NetScaler load balancer.

  • Provides easy access to the NetScaler MAS user interface using the floating IP address.

  • NetScaler MAS user interface is provided only on the primary node. By using the primary node, you can eliminate the risk of accessing and making changes to the secondary node.

  • Configuring the floating IP address handles the failover situation and reconfiguring the instances is not required.

  • Provides built in ability to detect and handle split-brain situation.

The following table describes the terms used in high availability deployment.

Terms Description
Primary node First node registered in the high availability deployment.
Secondary node Second node registered in the high availability deployment.
Heartbeat A mechanism used to exchange messages between primary and secondary node in the high availability setup. The messages determine status and health of the application on each individual node.
Floating IP address A floating IP is an IP address that can be instantly moved from one node to another in the same subnet. Internally it is set up as an alias on the network interface of the primary node. If there is a failover, the floating IP address is seamlessly moved from the old primary to the new one. It is useful in high availability setup because it allows clients to communicate with the high availability nodes using a single IP address.

Components of high availability architecture

The following figure displays the architecture of two NetScaler MAS nodes deployed in high availability mode.

localized image

In high availability deployment, one NetScaler MAS node is configured as the primary node (MAS 1) and the other as the secondary node (MAS 2). If the primary node goes down due to any reason, the secondary node takes over as the new primary.

Monitoring tool

Monitoring tool is an internal process used to monitor, alert, and handle failover situations. The tool is active and running on each node in high availability. It is responsible for starting subsystems, initiating database on both the nodes, deciding on the primary, or secondary node if there is a failover, and so on.

Primary node (MAS 1)

The primary node accepts connections and manages the instances. All processes such as AppFlow, SNMP, LogStream, syslog, and so on is managed by the primary node. The NetScaler MAS user interface access is available on primary node. The floating IP address is configured on the primary node.

Secondary node (MAS 2)

The secondary node listens to the heartbeat messages sent from the primary node. Database on the secondary node is in read-replica mode only. None of the processes are active in the secondary node and the NetScaler MAS user interface is not accessible on the secondary node.

Physical streaming replication

The primary and secondary nodes synchronize through heartbeat mechanism. With the physical streaming replication of database, the secondary node starts in read-replica mode. The secondary node listens to the heartbeat messages received from the primary node. If the secondary node does not receive any heartbeats for a time period of 180 seconds, the primary node is considered to be down. Then, the secondary node takes over as the primary node.

Heartbeat messages

Heartbeat messages are User Datagram Packets (UDP) that are sent and received between primary and secondary node. It monitors all subsystems of NetScaler MAS and database to exchange information about the node state, health, processes, and so on. The information is shared between the high availability nodes every second. Notifications are sent as alerts to the administrator if there is a failover or break up of high availability states.

Floating IP address

The floating IP address is associated with the primary node in the high availability setup. It is an alias given to the primary node IP address, that the client can use to connect to NetScaler MAS in the primary node. Since the floating IP address is configured on the primary node, the instance reconfiguration is not required in case of failover. The instances reconnect to the same IP address to reach the new primary.

Key points to note

  • In a high availability setup, both the NetScaler MAS nodes must be deployed in active-passive mode. They must be on the same subnets using the same software version and build, and have same configurations.

  • Floating IP address:

    • Floating IP address is configured on the primary node.

    • Instances need not be reconfigured if there is a failover.

    • You can access a high availability node from the user interface, either by using the primary node IP or floating IP address.

Note

Citrix recommends that you use the floating IP address to access the user interface.

  • Database:

    • In a high availability setup, all configuration files are synchronized automatically from the primary node to the secondary node at an interval of one minute.

    • Database synchronization happens instantly by physical replication of database.

    • Database on secondary node is in read-replica mode.

  • NetScaler MAS upgrade:

    • Internal processes implicitly upgrades NetScaler MAS from the earlier versions.

Note

After the upgrade is successful, you must configure the floating IP address.

  • UDP default port 5005 is available on both the nodes for heartbeats to be sent and for messages to be received.

  • You can make configuration changes on both the primary and secondary nodes. Citrix recommends making them on the primary node.

  • MAC address
    The setting for the “MAC Address Changes” option in a hypervisor affects the traffic that a virtual machine receives. Allow MAC address changes to be enabled on the virtual switch so that the floating IP address moves seamlessly to the new primary node after failover. For example, when deploying NetScaler MAS on a high availability on Vmware ESXi, make sure you accept changes to MAC address. ESXi now allows requests to change the active MAC address to other than the initial MAC address.

    localized image

Prerequisites

Before you set up high availability for NetScaler MAS nodes, note the following prerequisites:

  • The NetScaler MAS high availability deployment is supported from NetScaler MAS version 12.0 build 51.24.

  • Download the NetScaler Management and Analytics System image file (.xva) from the Citrix download site: https://www.citrix.co.in/downloads/

Citrix recommends that you set CPU priority (in virtual machine properties) at the highest level to improve scheduling behavior and network latency.

The following table lists the minimum requirements for the virtual computing resources:

Component Requirement
RAM 32 GB
  Note: The default value is 8 GB. Citrix recommends that you increase the default value to 32 GB for better performance.
Virtual CPU 8 CPUs
  Note: The default is 2 CPUs. Citrix recommends that you increase the default value to 8 CPUs for better performance.
Storage Space Citrix recommends using solid-state drive (SSD) technology for NetScaler MAS deployments. The default value is 120 GB. Actual storage requirement depends on NetScaler MAS sizing estimation. If your NetScaler MAS storage requirement exceeds 120 GB, you have to attach an additional disk. Note: You can add only one additional disk. Citrix recommends you to estimate storage and attach additional disk at the time of initial deployment. For more information, see How to Attach an Additional Disk to NetScaler MAS.
Virtual network interfaces 1
Throughput 1 Gbps or 100 Mbps
Hypervisor Versions
Citrix XenServer 6.2 and 6.5
VMware ESXi 5.5 and 6.0
Microsoft Hyper-V 2012 R2
Linux KVM Ubuntu and Fedora

To set up NetScaler MAS in high availability mode

  1. Register and deploy the first server (primary node).

  2. Register and deploy the second server (secondary node).

  3. Deploy the primary and secondary node for high availability setup.

Registering and deploying the first server (primary node)

To register the first node:

  1. Use the .xva image file downloaded from the Citrix download site and import it in to your hypervisor.

  2. From the Console tab, configure NetScaler MAS with the initial network configurations as displayed in the following image.

    localized image

  3. After the initial network configuration is complete, the system prompts for login. Log on using following credentials – nsrecover/nsroot.

  4. To deploy the primary node, enter /mps/deployment_type.py. The NetScaler MAS deployment configuration menu is displayed as shown in the following image.

    localized image

  5. Select 1 to register NetScaler MAS server as primary node. See 1 in following image.

  6. The console prompts you to select the NetScaler MAS standalone deployment. Enter No to confirm the deployment as high availability. See 2 in following image.

  7. The console prompts you to select the First Server Node. Enter Yes to confirm the node as the first node. See 3 in following image.

  8. The console prompts you to restart the system. Enter Yes to restart. See 4 in following image.

    localized image

    The system restarts and is displayed as the primary node in the NetScaler MAS user interface.

Registering and deploying the second server (secondary node)

  1. Use the .xva image file downloaded from the Citrix download site and import it in to your hypervisor.

  2. From the Console tab, configure NetScaler MAS with the initial network configurations as displayed in the following image.

  3. After the initial network configuration is completed, the system prompts for login. Log on using following credentials – nsrecover/nsroot.

  4. To deploy the secondary node, enter /mps/deployment_type.py. The NetScaler MAS deployment configuration menu is displayed as shown in the following image.

  5. Select 1 to register NetScaler MAS server as secondary node. See 1 in the following image.

  6. The console prompts you to select the NetScaler MAS as standalone deployment. Enter No to confirm the deployment as high availability. See 2 in the following image.

  7. The console prompts you to select the first server node. Enter No to confirm the node as the second server. See 3 in the following image.

  8. The console prompts you to enter the IP address and password of the primary node. See 4 in the following image.

  9. The console prompts you to enter the floating IP address. See 5 in the following image.

  10. The console prompts you to restart the system. Enter Yes to restart. See 6 in the following image.

Note

Floating IP address is mandatory for high availability deployment of nodes.

localized image

The system restarts and is displayed as the secondary node in the NetScaler MAS GUI.

Deploying the primary and secondary node as a high availability pair

After the registration both primary and secondary nodes are displayed on the NetScaler MAS user interface. Deploy these nodes into a high availability pair.

Note

After the high availability deployment is complete, use the floating IP address to access the NetScaler MAS user interface.

To deploy nodes as a high availability pair:

  1. Open a web browser and enter the floating IP address or the IP address of the first NetScaler MAS server node.  

  2. In the user Name and password fields, enter the administrator credentials.

  3. Select the deployment type as Two Servers deployed in High Availability Mode, and click Next.

    localized image

  4. On the System tab, navigate to Deployment and click Deploy.

    localized image

  5. A confirmation message is displayed. Click Yes.

After you deploy the NetScaler MAS in high availability mode, the primary and secondary nodes are displayed as shown in the following image.

localized image

Disabling high availability

You can disable high availability on a NetScaler MAS high availability pair and convert the nodes to standalone NetScaler MAS servers.

Note

Disable high availability from the primary node.

To disable the high availability:

  1. In a web browser, enter the IP address of the NetScaler MAS server primary node.

  2. In the User Name and Password fields, enter the administrator credentials.

  3. On the System tab, navigate to Deployment and click Break HA.

    localized image

A dialogue box is displayed. Click Yes to break the high availability deployment.

Redeploying high availability

After you disable the high availability to a standalone deployment, you can redeploy it to high availability mode again. Redeploying high availability is similar to the first time deployment of high availability. For more details see, Deploying the primary and secondary node as a high availability pair.

High availability failover scenarios

A failover occurs if one of the following conditions is encountered:

  • Node failure: Primary node goes down, no heartbeat is detected from primary node for 180 seconds.

  • Application health failure: Primary node is up and running but one of the NetScaler MAS processes is down.

Split-brain scenario

When there is no communication between both the nodes due to downtime in network link, then:

  • Primary node continues to operate as primary

  • Secondary node takes over as primary because of the failure to receive heartbeats

  • Both the nodes would run their individual database instances

For example, in an enterprise two NetScaler MAS nodes have been deployed as primary and secondary. Due to a possible network link downtime, the communication between the two NetScaler MAS nodes breaks completely. Since there is no heartbeat exchange for over 180 seconds, both the nodes consider themselves to be the primary node. Both nodes act as active nodes and run their own instances of database.

With NetScaler MAS 12.1, this split-brain situation is handled gracefully after the network link and heartbeat is restored. High availability synchronization is restored automatically. The recovery time depends on the data and speed of the link between the nodes.

Note

During the split-brain condition, changes that occurred on the old primary node is reset with the new primary when it is rejoined in high availability. The changes that happened on new primary node during split-brain remains intact.