Product Documentation

Graceful shutdown of nodes

In a cluster, if a node leaves the system or if a node joins the system, some of the existing connections (1/Nth connections, where N is the cluster size) at cluster level or specific virtual server level are lost. To address the loss, you must gracefully handle the existing connections. This is done by configuring “retain connections on cluster” option in the CLIP address and specifying timeout interval in the node’s NSIP.

Graceful handling of connections is applicable in two scenarios:

  1. Cluster upgrade
  2. New Node Addition

Graceful handling of Nodes in cluster upgrade

To upgrade a cluster, you must upgrade one node at a time. Before upgrading a node, you must set it to passive state and then set it to active state after the upgrade. To avoid terminating existing connections when upgrading the node, shut it down gracefully with a configured timeout interval.  Otherwise, 1/Nth (where N is the cluster size) of the cluster’s connections are terminated.

Note

If existing sessions are not completed within the configured timeout interval, they get terminated after the grace time.

Following are the steps to gracefully handle nodes in a cluster upgrade scenario:

  1. Consider a cluster setup of five nodes (n0, n1, n2, n3, n4).

  2. Before you shut down a node, you must configure “retainConnectionsOnCluster” option to retain all existing connections of this node at cluster level or virtual server level for specific time interval.

    example

    On CLIP

    set cluster instance <clusterID> –retainConnectionsOnCluster YES

    OR

    set lb vserver <vserver name> –retainConnectionsOnCluster Yes

  3. Now, log on to the NSIP address of node n3 and set the node n3 to PASSIVE with a timeout internal.

    Example

    set cluster node n3 –state PASSIVE –delay 60

    saveconfig

  4. After the grace period expires, close all connections, shut down n3 and reboot the Citrix ADC appliance.

  5.  Upgrade the appliance. Then, with the CLI connected to the appliance’s NSIP address, set the node to ACTIVE.

    Example

    set cluster node n3 –state ACTIVE

    saveconfig

  6.  Repeat steps 4 to 6 for all nodes in the cluster.

  7.  After all nodes are upgraded and set to ACTIVE, reset the retainConnectionsOnCluster option from the CLIP address.

    Example

    set cluster instance <clusterID> -retainConnectionsOnCluster NO

    OR

    set lb vserver <vserver name> –retainConnectionsOnCluster NO

    saveconfig

Note

If there is a version mismatch when upgrading a Cluster, cluster propagation is automatically disabled and no commands are allowed on the CLIP.

Graceful handling of nodes during a new node addition

If you have an appliance that is already serving traffic, and you want to add this appliance as a node to a cluster without terminating its existing connections, set the option to retain existing connections either at Global level or specific virtual sever level and save the configuration. Now set the option to retain connections to NO, to allow existing connections from other nodes to be reassigned to the new node.

Following are the steps to gracefully handle nodes if a node newly added:

  1.  You must save the existing configuration that has “retainConnectionsOnCluster” option enabled to retain all existing connections of this node at cluster level or virtual server level for specific time interval.

On CLIP

set cluster instance x – retainConnectionsOnCluster YES

OR

set lb vserver xxxx –retainConnectionsOnCluster Yes

2.  Add a new node n5 to the cluster setup.

3.  Disable “the retainConnectionOnCluster” option to “NO” for distributing existing connections from other nodes to the newly added node n5.

On CLIP

set cluster instance x – retainConnectionsOnCluster NO

OR

set lb vserver xxxx –retainConnectionsOnCluster NO

Note: The backplane steering depends on the type of traffic distribution mechanism (ECMP, CLAG, and USIP) on a cluster setup. The increase in backplane steering is based on the traffic type.

Configuring graceful shutdown of nodes in a cluster

To configure graceful shutdown of nodes in a cluster, do the following:

  1. Configure “retainConnectionsonCluster” option at Global (Cluster) level.
  2. Configure “retainConnectionsonCluster” option at virtual server level.
  3. Set the node (leaving the system) to the passive state with a graceful timeout interval specified in the node’s NSIP address.
  4. Monitor the existing connections to make sure all transactions are completed within the grace period.

To retain existing connections at the global (cluster) level by using the command line

You can retain existing connections either at global level or at a specific virtual server level. This option is configured to retain all existing connections at global level.  By default, this option is disabled.

At the command prompt type:

set cluster instance <clusterID> –retainConnectionsOnCluster YES

set cluster instance 60 – retainConnectionsOnCluster YES

To retain existing connections of a specific virtual server in the cluster by using the command line

This option is configured to retain existing connections specific to a load balancing virtual server. To retain those connections, we enable this option at the virtual server level. By default, this option is disabled.

At the command prompt, type:

set lb vserver <clusterID> –retainConnectionsOnCluster Yes

set lb vserver v1 –retainConnectionsOnCluster Yes

To set a cluster node to passive state by using the command line

To set a cluster node to passive state with a gracefully timeout interval. This setting is performed in the node’s NSIP as propagation is disabled during cluster upgrade.

At the command prompt, type:

set cluster node <clusterID> -state passive
-backplane <interface_name>@
-priority <positive_integer>
-delay <mins>

set cluster node 4 –state PASSIVE -delay 60

set cluster instance 60 – retainConnectionsOnCluster YES
set lb vserver v1 –retainConnectionsOnCluster Yes
set cluster node 4 –state PASSIVE -delay 60

To configure graceful shutdown of nodes by using the Citrix ADC GUI

  1. Navigate to Configuration > System > Cluster and click Manage Cluster.
  2. On the Manage Cluster page, select Retain Connections on Cluster option.
  3. Click OK, and then click Done.