This section provides an overview of XenServer networking, including networks, VLANs, and NIC bonds. It also discusses how to manage your networking configuration and troubleshoot it.
vSwitch is the default network stack of XenServer. Follow the instructions in vSwitch networks to configure the Linux network stack.
If you are already familiar with XenServer networking concepts, you can skip ahead to Manage networking for information about the following sections:
Create networks for standalone XenServer hosts
Create private networks across XenServer hosts
Create networks for XenServer hosts that are configured in a resource pool
Create VLANs for XenServer hosts, either standalone or part of a resource pool
Create bonds for standalone XenServer hosts
Create bonds for XenServer hosts that are configured in a resource pool
The term ‘management interface’ is used to indicate the IP-enabled NIC that carries the management traffic. The term ‘secondary interface’ is used to indicate an IP-enabled NIC configured for storage traffic.
XenServer supports up to 16 physical network interfaces (or up to 4 bonded network interfaces) per host and up to 7 virtual network interfaces per VM.
XenServer provides automated configuration and management of NICs using the xe command line interface (CLI). Do not edit the host networking configuration files directly.
When used with a controller appliance, vSwitch networks support open flow and provide extra functionality such as Access Control Lists (ACL). The controller appliance for the XenServer vSwitch is called the vSwitch Controller. The vSwitch Controller lets you monitor your networks through a GUI. The vSwitch Controller:
Supports fine-grained security policies to control the flow of traffic sent to and from a VM.
Provides detailed visibility about the behavior and performance of all traffic sent in the virtual network environment.
A vSwitch greatly simplifies IT administration in virtualized networking environments. All VM configuration and statistics remain bound to the VM even when the VM migrates from one physical host in the resource pool to another. For more information, see vSwitch and Controller.
To determine what networking stack is configured, run the following command:
xe host-list params=software-version
In the command output, look for
network_backend. When the vSwitch is configured as the network stack, the output appears as follows:
When the Linux bridge is configured as the network stack, the output appears as follows:
To revert to the Linux network stack, run the following command:
Restart your host after running this command.
The Linux network stack is not open-flow enabled, does not support Cross Server Private Networks. The XenServer vSwitch Controller does not manage the Linux network stack.
XenServer networking overview
This section describes the general concepts of networking in the XenServer environment.
XenServer creates a network for each physical NIC during installation. When you add a server to a pool, the default networks are merged. This is to ensure all physical NICs with the same device name are attached to the same network.
Typically, you add a network to create an internal network, set up a new VLAN using an existing NIC, or create a NIC bond.
You can configure four different types of networks in XenServer:
External networks have an association with a physical network interface. External networks provide a bridge between a virtual machine and the physical network interface connected to the network. External networks enable a virtual machine to connect to resources available through the server’s physical NIC.
Bonded networks create a bond between two or more NICs to create a single, high-performing channel between the virtual machine and the network.
Single-Server Private networks have no association to a physical network interface. Single-server private networks can be used to provide connectivity between the virtual machines on a given host, with no connection to the outside world.
Cross-Server Private networks extend the single server private network concept to allow VMs on different hosts to communicate with each other by using the vSwitch.
Some networking options have different behaviors when used with standalone XenServer hosts compared to resource pools. This section contains sections on general information that applies to both standalone hosts and pools, followed by specific information and procedures for each.
This section uses three types of server-side software objects to represent networking entities. These objects are:
A PIF, which represents a physical NIC on a host. PIF objects have a name and description, a UUID, the parameters of the NIC they represent, and the network and server they are connected to.
A VIF, which represents a virtual NIC on a virtual machine. VIF objects have a name and description, a UUID, and the network and VM they are connected to.
A network, which is a virtual Ethernet switch on a host. Network objects have a name and description, a UUID, and the collection of VIFs and PIFs connected to them.
XenCenter and the xe CLI allow you to configure networking options. You can control the NIC used for management operations, and create advanced networking features such as VLANs and NIC bonds.
Each XenServer host has one or more networks, which are virtual Ethernet switches. Networks that are not associated with a PIF are considered internal. Internal networks can be used to provide connectivity only between VMs on a given XenServer host, with no connection to the outside world. Networks associated with a PIF are considered external. External networks provide a bridge between VIFs and the PIF connected to the network, enabling connectivity to resources available through the PIF’s NIC.
VLANs, as defined by the IEEE 802.1Q standard, allow a single physical network to support multiple logical networks. XenServer hosts support VLANs in multiple ways.
All supported VLAN configurations are equally applicable to pools and standalone hosts, and bonded and non-bonded configurations.
Using VLANs with management interfaces
Switch ports configured to perform 802.1Q VLAN tagging/untagging are commonly described as ports with a native VLAN or as access mode ports. These switch ports can be used with management interfaces to place management traffic on a desired VLAN.
Management interfaces cannot be assigned to XenServer VLANs via a trunk port.
Using VLANs with virtual machines
Switch ports configured as 802.1Q VLAN trunk ports can be used with the XenServer VLAN features to connect guest virtual network interfaces (VIFs) to specific VLANs. In this case, the XenServer host performs the VLAN tagging/untagging functions for the guest, which is unaware of any VLAN configuration.
XenServer VLANs are represented by additional PIF objects representing VLAN interfaces corresponding to a specified VLAN tag. You can connect XenServer networks to the PIF representing the physical NIC to see all traffic on the NIC. Alternatively, connect networks to a PIF representing a VLAN to see only the traffic with the specified VLAN tag. You can also connect a network such that it only sees the native VLAN traffic, by attaching it to VLAN 0.
For procedures on how to create VLANs for XenServer hosts, either standalone or part of a resource pool, see Creating VLANs.
Using VLANs with dedicated storage NICs
Dedicated storage NICs can be configured to use native VLAN or access mode ports as described in the previous section for management interfaces. Dedicated storage NICs are also known as IP-enabled NICs or secondary interfaces. You can configure dedicated storage NICs to use trunk ports and XenServer VLANs as described in the previous section for virtual machines. For more information, see Configuring a dedicated storage NIC.
Combining management interfaces and guest VLANs on a single host NIC
A single switch port can be configured with both trunk and native VLANs, allowing one host NIC to be used for a management interface (on the native VLAN) and for connecting guest VIFs to specific VLAN IDs.
Jumbo frames can be used to optimize the performance of storage traffic. Jumbo frames are Ethernet frames containing more than 1,500 bytes of payload. Jumbo frames are typically used to achieve better throughput, reducing the load on system bus memory, and reducing the CPU overhead.
XenServer supports jumbo frames only when using vSwitch as the network stack on all hosts in the pool.
Requirements for using jumbo frames
Customers must note the following when using jumbo frames:
Jumbo frames are configured at a pool level
vSwitch must be configured as the network back-end on all hosts in the pool
Every device on the subnet must be configured to use jumbo frames
Enable jumbo frames on a dedicated storage network (recommended)
Enabling jumbo frames on the Management network is not a supported configuration
Jumbo frames are not supported for use on VMs
To use jumbo frames, set the Maximum Transmission Unit (MTU) to a value between 1500 and 9216. You can use XenCenter or the xe CLI to set the MTU.
NIC bonds, sometimes also known as NIC teaming, improve XenServer host resiliency and bandwidth by enabling administrators to configure two or more NICs together. NIC bonds logically function as one network card and all bonded NICs share MAC address.
If one NIC in the bond fails, the host’s network traffic is automatically redirected through the second NIC. XenServer supports up to eight bonded networks.
XenServer supports active-active, active-passive, and LACP bonding modes. The number of NICs supported and the bonding mode supported varies according to network stack:
LACP bonding is only available for the vSwitch whereas active-active and active-passive are available for both the vSwitch and Linux bridge.
When the vSwitch is the network stack, you can bond either two, three, or four NICs.
When the Linux bridge is the network stack, you can only bond two NICs.
In the illustration that follows, the management interface is on a bonded pair of NICs. XenServer uses this bond for management traffic.
All bonding modes support failover. However, not all modes allow all links to be active for all traffic types. XenServer supports bonding the following types of NICs together:
NICs (non-management). You can bond NICs that XenServer is using solely for VM traffic. Bonding these NICs not only provides resiliency, but doing so also balances the traffic from multiple VMs between the NICs.
Management interfaces. You can bond a management interface to another NIC so that the second NIC provides failover for management traffic. Although configuring a LACP link aggregation bond provides load balancing for management traffic, active-active NIC bonding does not. You can create a VLAN on bonded NICs and host management interface can be assigned to that VLAN.
Secondary interfaces. You can bond NICs that you have configured as secondary interfaces (for example, for storage). However, for most iSCSI software initiator storage, Citrix recommends configuring multipathing instead of NIC bonding as described in the Designing XenServer Network Configurations.
Throughout this section, the term IP-based storage traffic is used to describe iSCSI and NFS traffic collectively.
You can create a bond if a VIF is already using one of the interfaces that will be bonded: the VM traffic migrates automatically to the new bonded interface.
In XenServer, An additional PIF represents a NIC bond. XenServer NIC bonds completely subsume the underlying physical devices (PIFs).
- Creating a bond that contains only one NIC is not supported.
- NIC bonds are not supported on NICs that carry FCoE traffic.
Key points about IP addressing
Bonded NICs either have one IP address or no IP addresses, as follows:
Management and storage networks.
If you bond a management interface or secondary interface, a single IP address is assigned to the bond. That is, each NIC does not have its own IP address. XenServer treats the two NICs as one logical connection.
When bonds are used for non-VM traffic, for example, to connect to shared network storage or XenCenter for management, configure an IP address for the bond. However, if you have already assigned an IP address to one of the NICs (that is, created a management interface or secondary interface), that IP address is assigned to the entire bond automatically.
If you bond a management interface or secondary interface to a NIC without an IP address, the bond assumes the IP address of the respective interface.
If you bond a tagged VLAN management interface and a secondary interface, the management VLAN is created on that bonded NIC.
VM networks. When bonded NICs are used for VM traffic, you do not need to configure an IP address for the bond. This is because the bond operates at Layer 2 of the OSI model, the data link layer, and no IP addressing is used at this layer. IP addresses for virtual machines are associated with VIFs.
XenServer provides three different types of bonds, all of which can be configured using either the CLI or XenCenter:
Active-Active mode, with VM traffic balanced between the bonded NICs. See Active-active bonding.
Active-Passive mode, where only one NIC actively carries traffic. See Active-passive bonding.
LACP Link Aggregation, in which active and stand-by NICs are negotiated between the switch and the server. See LACP Link Aggregation Control Protocol bonding.
Bonding is set up with an Up Delay of 31,000 ms and a Down Delay of 200 ms. The seemingly long Up Delay is deliberate because of the time some switches take to enable the port. Without a delay, when a link comes back after failing, the bond can rebalance traffic onto it before the switch is ready to pass traffic. To move both connections to a different switch, move one, then wait 31 seconds for it to be used again before moving the other. For information about changing the delay, see Changing the up delay for bonds.
XenServer provides status for bonds in the event logs for each host. If one or more links in a bond fails or is restored, it is noted in the event log. Likewise, you can query the status of a bond’s links by using the
links-up parameter as shown in the following example:
xe bond-param-get uuid=bond_uuid param-name=links-up
XenServer checks the status of links in bonds approximately every five seconds. Therefore, if more links in the bond fail in the five-second window, the failure is not logged until the next status check.
Bonding event logs appear in the XenCenter Logs tab. For users not running XenCenter, event logs also appear in
/var/log/xensource.log on each host.
Active-active is an active/active configuration for guest traffic: both NICs can route VM traffic simultaneously. When bonds are used for management traffic, only one NIC in the bond can route traffic: the other NIC remains unused and provides failover support. Active-active mode is the default bonding mode when either the Linux bridge or vSwitch network stack is enabled.
When active-active bonding is used with the Linux bridge, you can only bond two NICs. When using the vSwitch as the network stack, you can bond either two, three, or four NICs in active-active mode. However, in active-active mode, bonding three, or four NICs is only beneficial for VM traffic, as shown in the illustration that follows.
XenServer can only send traffic over two or more NICs when there is more than one MAC address associated with the bond. XenServer can use the virtual MAC addresses in the VIF to send traffic across multiple links. Specifically:
VM traffic. Provided you enable bonding on NICs carrying only VM (guest) traffic, all links are active and NIC bonding can balance spread VM traffic across NICs. An individual VIF’s traffic is never split between NICs.
Management or storage traffic. Only one of the links (NICs) in the bond is active and the other NICs remain unused unless traffic fails over to them. Configuring a management interface or secondary interface on a bonded network provides resilience.
Mixed traffic. If the bonded NIC carries a mixture of IP-based storage traffic and guest traffic, only the guest and control domain traffic are load balanced. The control domain is essentially a virtual machine so it uses a NIC like the other guests. XenServer balances the control domain’s traffic the same way as it balances VM traffic.
XenServer balances the traffic between NICs by using the source MAC address of the packet. Because, for management traffic, only one source MAC address is present, active-active mode can only use one NIC, and traffic is not balanced. Traffic balancing is based on two factors:
The virtual machine and its associated VIF sending or receiving the traffic
The quantity of data (in kilobytes) being sent.
XenServer evaluates the quantity of data (in kilobytes) each NIC is sending and receiving. If the quantity of data sent across one NIC exceeds the quantity of data sent across the other NIC, XenServer rebalances which VIFs use which NICs. The VIF’s entire load is transferred. One VIF’s load is never split between two NICs.
Though active-active NIC bonding can provide load balancing for traffic from multiple VMs, it cannot provide a single VM with the throughput of two NICs. Any given VIF only uses one of the links in a bond at a time. As XenServer periodically rebalances traffic, VIFs are not permanently assigned to a specific NIC in the bond.
Active-active mode is sometimes described as Source Load Balancing (SLB) bonding as XenServer uses SLB to share load across bonded network interfaces. SLB is derived from the open-source Adaptive Load Balancing (ALB) mode and reuses the ALB functionality to rebalance load across NICs dynamically.
When rebalancing, the number of bytes going over each slave (interface) is tracked over a given period. If a packet to be sent contains a new source MAC address, it is assigned to the slave interface with the lowest utilization. Traffic is rebalanced at regular intervals.
Each MAC address has a corresponding load and XenServer can shift entire loads between NICs depending on the quantity of data a VM sends and receives. For active-active traffic, all the traffic from one VM can be sent on only one NIC.
Active-active bonding does not require switch support for EtherChannel or 802.3ad (LACP).
An active-passive bond routes traffic over only one of the NICs. If the active NIC loses network connectivity, traffic fails over to the other NIC in the bond. Active-passive bonds route traffic over the active NIC. The traffic shifts to the passive NIC if the active NIC fails.
Active-passive bonding is available in the Linux bridge and the vSwitch network stack. When used with the Linux bridge, you can bond two NICs together. When used with the vSwitch, you can only bond two, three, or four NICs together. However, regardless of the traffic type, when you bond NICs in active-passive mode, only one link is active and there is no load balancing between links.
The illustration that follows shows two bonded NICs configured in active-passive mode.
Active-active mode is the default bonding configuration in XenServer. If you are configuring bonds using the CLI, you must specify a parameter for the active-passive mode. Otherwise, an active-active bond is created. You do not need to configure active-passive mode because a network is carrying management traffic or storage traffic.
Active-passive can be a good choice for resiliency as it offers several benefits. With active-passive bonds, traffic does not move around between NICs. Similarly, active-passive bonding lets you configure two switches for redundancy but does not require stacking. If the management switch dies, stacked switches can be a single point of failure.
Active-passive mode does not require switch support for EtherChannel or 802.3ad (LACP).
Consider configuring active-passive mode in situations when you do not need load balancing or when you only intend to send traffic on one NIC.
After you have created VIFs or your pool is in production, be careful about changing bonds or creating bonds.
LACP Link Aggregation Control Protocol bonding
LACP Link Aggregation Control Protocol is a type of bonding that bundles a group of ports together and treats it like a single logical channel. LACP bonding provides failover and can increase the total amount of bandwidth available.
Unlike other bonding modes, LACP bonding requires configuring both sides of the links: creating a bond on the host, and creating a Link Aggregation Group (LAG) for each bond on the switch. See Switch configuration for LACP bonds. You must configure the vSwitch as the network stack to use LACP bonding. Also, your switches must support the IEEE 802.3ad standard.
A comparison of active-active SLB bonding and LACP bonding:
Active-active SLB bonding
- Can be used with any switch on the Hardware Compatibility List.
- Does not require switches that support stacking.
- Supports four NICs.
- Optimal load balancing requires at least one NIC per VIF.
- Storage or management traffic cannot be split on multiple NICs.
- Load balancing occurs only if multiple MAC addresses are present.
- All links can be active regardless of traffic type.
- Traffic balancing does not depend on source MAC addresses, so all traffic types can be balanced.
- Switches must support the IEEE 802.3ad standard.
- Requires switch-side configuration.
- Supported only for the vSwitch.
- Requires a single switch or stacked switch.
XenServer supports two LACP bonding hashing types. The term hashing describes how the NICs and the switch distribute the traffic— (1) load balancing based on IP and port of source and destination addresses and (2) load balancing based on source MAC address.
Depending on the hashing type and traffic pattern, LACP bonding can potentially distribute traffic more evenly than active-active NIC bonding.
You configure settings for outgoing and incoming traffic separately on the host and the switch: the configuration does not have to match on both sides.
Load balancing based on IP and port of source and destination addresses.
This hashing type is the default LACP bonding hashing algorithm. If there is a variation in the source or destination IP or port numbers, traffic from one guest can be distributed over two links.
If a virtual machine is running several applications which use different IP or port numbers, this hashing type distributes traffic over several links. Distributing the traffic gives the guest the possibility of using the aggregate throughput. This hashing type lets one guest use the whole throughput of multiple NICs.
As shown in the illustration that follows, this hashing type can distribute the traffic of two different applications on a virtual machine to two different NICs.
Configuring LACP bonding based on IP and port of source and destination address is beneficial when you want to balance the traffic of two different applications on the same VM. For example, when only one virtual machine is configured to use a bond of three NICs.
The balancing algorithm for this hashing type uses five factors to spread traffic across the NICs: the source IP address, source port number, destination IP address, destination port number, and source MAC address.
Load balancing based on source MAC address.
This type of load balancing works well when there are multiple virtual machines on the same host. Traffic is balanced based on the virtual MAC address of the VM from which the traffic originated. XenServer sends outgoing traffic using the same algorithm as it does in active-active bonding. Traffic coming from the same guest is not split over multiple NICs. As a result, this hashing type is not suitable if there are fewer VIFs than NICs: load balancing is not optimal because the traffic cannot be split across NICs.
Depending on your redundancy requirements, you can connect the NICs in the bond to either the same or separate stacked switches. If you connect one of the NICs to a second, redundant switch and a NIC or switch fails, traffic fails over to the other NIC. Adding a second switch prevents a single point-of-failure in your configuration in the following ways:
When you connect one of the links in a bonded management interface to a second switch, if the switch fails, the management network remains online and the hosts can still communicate with each other.
If you connect a link (for any traffic type) to a second switch and the NIC or switch fails, the virtual machines remain on the network as their traffic fails over to the other NIC/switch.
Use stacked switches when you want to connect bonded NICs to multiple switches and you configured the LACP bonding mode. The term ‘stacked switches’ is used to describe configuring multiple physical switches to function as a single logical switch. You must join the switches together physically and through the switch-management software so the switches function as a single logical switching unit, as per the switch manufacturer’s guidelines. Typically, switch stacking is only available through proprietary extensions and switch vendors may market this functionality under different terms.
If you experience issues with active-active bonds, the use of stacked switches may be necessary. Active-passive bonds do not require stacked switches.
The illustration that follows shows how the cables and network configuration for the bonded NICs have to match.
Switch configuration for LACP bonds
Because the specific details of switch configuration vary by manufacturer, there are a few key points to remember when configuring switches for use with LACP bonds:
The switch must support LACP and the IEEE 802.3ad standard.
When you create the LAG group on the switch, you must create one LAG group for each LACP bond on the host. For example, if you have a five-host pool and you created a LACP bond on NICs 4 and 5 on each host, you must create five LAG groups on the switch. One group for each set of ports corresponding with the NICs on the host.
You may also need to add your VLAN ID to your LAG group.
XenServer LACP bonds require setting the Static Mode setting in the LAG group to be set to Disabled.
As previously mentioned in Switch configuration, stacking switches are required to connect LACP bonds to multiple switches.
Initial networking configuration after setup
The XenServer host networking configuration is specified during initial host installation. Options such as IP address configuration (DHCP/static), the NIC used as the management interface, and hostname are set based on the values provided during installation.
When a host has multiple NICs, the configuration present after installation depends on which NIC is selected for management operations during installation:
PIFs are created for each NIC in the host
The PIF of the NIC selected for use as the management interface is configured with the IP addressing options specified during installation
A network is created for each PIF (“network 0”, “network 1”, and so on)
Each network is connected to one PIF
The IP addressing options are left unconfigured for all PIFs other than the PIF used as the management interface
When a host has a single NIC, the follow configuration is present after installation:
A single PIF is created corresponding to the host’s single NIC
The PIF is configured with the IP addressing options specified during installation and to enable management of the host
The PIF is set for use in host management operations
A single network, network 0, is created
Network 0 is connected to the PIF to enable external connectivity to VMs
When an installation of XenServer is done on a tagged VLAN network, the following configuration is present after installation:
PIFs are created for each NIC in the host
The PIF for the tagged VLAN on the NIC selected for use as management interface is configured with the IP address configuration specified during installation
A network is created for each PIF (for example: network 1, network 2, and so on). Additional VLAN network is created (for example, for Pool-wide network associated with eth0 on VLAN<TAG>)
Each network is connected to one PIF. The VLAN PIF is set for use in host management operations
In both cases, the resulting networking configuration allows connection to the XenServer host by XenCenter, the xe CLI, and any other management software running on separate machines through the IP address of the management interface. The configuration also provides external networking for VMs created on the host.
The PIF used for management operations is the only PIF ever configured with an IP address during XenServer installation. External networking for VMs is achieved by bridging PIFs to VIFs using the network object which acts as a virtual Ethernet switch.
The steps required for networking features such as VLANs, NIC bonds, and dedicating a NIC to storage traffic are covered in the sections that follow.
Changing networking configuration
You can change your networking configuration by modifying the network object. To do so, you run a command that affects either the network object or the VIF.
Modifying the network object
You can change aspects of a network, such as the frame size (MTU), name-label, name-description, purpose, and other values. Use the xe
network-param-set command and its associated parameters to change the values.
When you run the xe
network-param-set command, the only required parameter is
Optional parameters include:
default_locking_mode. See Simplifying VIF locking mode configuration in the Cloud.
purpose. See Adding a purpose to a network.
If a value for a parameter is not given, the parameter is set to a null value. To set a (key, value) pair in a map parameter, use the syntax
Changing the up delay for bonds
Bonding is set up with an Up Delay of 31,000 ms by default to prevent traffic from being rebalanced onto a NIC after it fails. While seemingly long, the up delay is important for all bonding modes and not just active-active.
However, if you understand the appropriate settings to select for your environment, you can change the up delay for bonds by using the procedure that follows.
Set the up delay in milliseconds:
xe pif-param-set uuid=<uuid of bond master PIF> other-config:bond-updelay=<delay in ms>
To make the change take effect, you must unplug and then replug the physical interface:
xe pif-unplug uuid=<uuid of bond master PIF> xe pif-plug uuid=<uuid of bond master PIF>