Graceful shutdown of nodes
In a cluster, if a node leaves the system or if a node joins the system, some of the existing connections (1/Nth connections, where N is the cluster size) at cluster level or specific virtual server level are lost. To address the loss, you must gracefully handle the existing connections. This is done by configuring “retain connections on cluster” option in the CLIP address and specifying timeout interval in the node’s NSIP.
Graceful handling of connections is applicable in two scenarios:
- Cluster upgrade
- New Node Addition
Graceful handling of Nodes in cluster upgrade
To upgrade a cluster, you must upgrade one node at a time. Before upgrading a node, you must set it to passive state and then set it to active state after the upgrade. To avoid terminating existing connections when upgrading the node, shut it down gracefully with a configured timeout interval. Otherwise, 1/Nth (where N is the cluster size) of the cluster’s connections are terminated.
If existing sessions are not completed within the configured timeout interval, they get terminated after the grace time.
Following are the steps to gracefully handle nodes in a cluster upgrade scenario:
Consider a cluster setup of five nodes (n0, n1, n2, n3, n4).
Before you shut down a node, you must configure “retainConnectionsOnCluster” option to retain all existing connections of this node at cluster level or virtual server level for specific time interval.
set cluster instance <clusterID> –retainConnectionsOnCluster YES
set lb vserver <vserver name> –retainConnectionsOnCluster Yes
Now, log on to the NSIP address of node n3 and set the node n3 to PASSIVE with a timeout internal.
set cluster node n3 –state PASSIVE –delay 60
After the grace period expires, close all connections, shut down n3 and reboot the NetScaler appliance.
Upgrade the appliance. Then, with the CLI connected to the appliance’s NSIP address, set the node to ACTIVE.
set cluster node n3 –state ACTIVE
Repeat steps 4 to 6 for all nodes in the cluster.
After all nodes are upgraded and set to ACTIVE, reset the retainConnectionsOnCluster option from the CLIP address.
set cluster instance <clusterID> -retainConnectionsOnCluster NO
set lb vserver <vserver name> –retainConnectionsOnCluster NO
If there is a version mismatch when upgrading a Cluster, cluster propagation is automatically disabled and no commands are allowed on the CLIP.
Graceful handling of nodes during a new node addition
If you have an appliance that is already serving traffic, and you want to add this appliance as a node to a cluster without terminating its existing connections, set the option to retain existing connections either at Global level or specific virtual sever level and save the configuration. Now set the option to retain connections to NO, to allow existing connections from other nodes to be reassigned to the new node.
Following are the steps to gracefully handle nodes if a node newly added:
- You must save the existing configuration that has “retainConnectionsOnCluster” option enabled to retain all existing connections of this node at cluster level or virtual server level for specific time interval.
set cluster instance x – retainConnectionsOnCluster YES
set lb vserver xxxx –retainConnectionsOnCluster Yes
2. Add a new node n5 to the cluster setup.
3. Disable “the retainConnectionOnCluster” option to “NO” for distributing existing connections from other nodes to the newly added node n5.
set cluster instance x – retainConnectionsOnCluster NO
set lb vserver xxxx –retainConnectionsOnCluster NO
Note: The backplane steering depends on the type of traffic distribution mechanism (ECMP, CLAG, and USIP) on a cluster setup. The increase in backplane steering is based on the traffic type.
Configuring graceful shutdown of nodes in a cluster
To configure graceful shutdown of nodes in a cluster, do the following:
- Configure “retainConnectionsonCluster” option at Global (Cluster) level.
- Configure “retainConnectionsonCluster” option at virtual server level.
- Set the node (leaving the system) to the passive state with a graceful timeout interval specified in the node’s NSIP address.
- Monitor the existing connections to make sure all transactions are completed within the grace period.
To retain existing connections at the global (cluster) level by using the command line
You can retain existing connections either at global level or at a specific virtual server level. This option is configured to retain all existing connections at global level. By default, this option is disabled.
At the command prompt type:
set cluster instance <clusterID> –retainConnectionsOnCluster YES set cluster instance 60 – retainConnectionsOnCluster YES
To retain existing connections of a specific virtual server in the cluster by using the command line
This option is configured to retain existing connections specific to a load balancing virtual server. To retain those connections, we enable this option at the virtual server level. By default, this option is disabled.
At the command prompt, type:
set lb vserver <clusterID> –retainConnectionsOnCluster Yes set lb vserver v1 –retainConnectionsOnCluster Yes
To set a cluster node to passive state by using the command line
To set a cluster node to passive state with a gracefully timeout interval. This setting is performed in the node’s NSIP as propagation is disabled during cluster upgrade.
At the command prompt, type:
set cluster node <clusterID> -state passive -backplane <interface_name>@ -priority <positive_integer> -delay <mins> set cluster node 4 –state PASSIVE -delay 60 set cluster instance 60 – retainConnectionsOnCluster YES set lb vserver v1 –retainConnectionsOnCluster Yes set cluster node 4 –state PASSIVE -delay 60
To configure graceful shutdown of nodes by using the NetScaler GUI
- Navigate to Configuration > System > Cluster and click Manage Cluster.
- On the Manage Cluster page, select Retain Connections on Cluster option.
- Click OK, and then click Done.