XenServer

Using HTTP to interact with XenServer

December 9, 2023

Contributed by:

X X

XenServer exposes an HTTP(S) interface on each host that can be used to perform various operations.

Each operation is performed by constructing an HTTP(S) GET or POST call to a dedicated URL path. Parameters are appended to the URL following a question mark (?) and separated by ampersands (&). The calls require authentication by providing a valid management API session reference as a query parameter. For example:

wget https://<server>/services?session_id=<session_opaque_ref>
<!--NeedCopy-->

Note:

Basic auth using a username and password is also available:
wget https://username:password@<server>/services

However, this method is not recommended.

The following sections describe in greater detail some of the available operations.

Importing and exporting VMs as XVA packages

Because the import and export of VMs can take some time to complete, an asynchronous HTTP(S) interface is provided for these operations.

VM Export

To export a VM as an XVA package using the XenServer Management API, construct an HTTP(S) GET call. The call accepts the following query parameters:

Parameter	Description
`session_id`	The opaque reference of the session being used to authenticate.
`uuid`	The `uuid` of the VM; required only if not using the `ref` parameter.
`ref`	The opaque reference of the VM; required only if not using the `uuid` parameter.
`task_id`	(Optional) The opaque reference of a task object with which to monitor the progress of the operation; the task object needs to be created first.

For example, to export a VM to the XVA package vm.xva using the Linux command line utility cURL:

curl https://server/export?session_id=<session_opaque_ref>&task_id=<task_opaque_ref>&uuid=<vm_uuid> -o vm.xva
<!--NeedCopy-->

Note that you can perform this operation on the pool coordinator even if the VM runs on a pool supporter. The request might result in a redirect if the VM’s disks are placed on an SR accessible only by a pool member.

If you want to export only the VM metadata without the disks, use the HTTP(S) handler export_metadata. For example:

curl https://server/export_metadata?session_id=<session_opaque_ref>&ref=<vm_opaque_reference> -o vm_meta.xva
<!--NeedCopy-->

VM Import

To import a VM from an XVA package, construct an HTTP(S) PUT call. The call accepts the following query parameters:

Parameter	Description
`session_id`	The opaque reference of the session being used to authenticate.
`sr_id`	(Optional) The opaque reference of the SR on which the imported VM’s disks will be placed. If not specified, the disks will be imported to `Pool.default_SR`. If the latter if not set, the error `DEFAULT_SR_NOT_FOUND` is returned.
`restore`	(Optional) If `true`, the import is treated as replacing the original VM from which the XVA package was exported. The implication of this currently is that the MAC addresses on the VIFs are exactly as the export was, which will lead to conflicts if the original VM is still running.
`force`	(Optional) If `true`, any checksum failures will be ignored (the default is to destroy the VM if a checksum error is detected).
`task_id`	(Optional) The opaque reference of a task object with which to monitor the progress of the operation; the task object needs to be created first

For example, to import a VM from package vm.xva placing its disks on a specific SR:

curl -T vm.xva http://server/import?session_id=<session_opaque_ref>&sr_id=<sr_opaque_ref>
<!--NeedCopy-->

Or, to import a VM from package vm.xva placing its disks on the default SR, while monitoring the progress of the operation:

curl -T vm.xva http://server/import?session_id=<session_opaque_ref>&task_id=<task_opaque_ref>
<!--NeedCopy-->

To import just the metadata, use the HTTP(S) handler import_metadata:

curl -T vm.xva http://server/import_metadata?session_id=<session_opaque_ref>
<!--NeedCopy-->

Xen Virtual Appliance (XVA) VM Import Format

XenServer supports a human-readable VM input format called XVA. This section describes the syntax and structure of the format.

An XVA is essentially a tar archive containing XML metadata and a set of disk images. A VM represented by an XVA is not intended to be directly executable. The data within an XVA package is intended for either archiving on permanent storage or for being transmitted to a VM server - such as a XenServer host - where it can be extracted and run.

XVA is a hypervisor-neutral packaging format; it is possible to create simple tools to instantiate an XVA VM on any other platform. XVA does not specify any particular runtime format; for example disks may be instantiated as file images, LVM volumes, QCoW images, VMDK or VHD images. An XVA VM may be instantiated any number of times, each instantiation may have a different runtime format.

Unlike other formats like the Open Virtual Appliance (OVA) format, XVA does not:

specify any particular serialization or transport format
provide any mechanism for customizing VMs (or templates) on install
address how a VM may be upgraded post-install
define how multiple VMs, acting as an appliance, may communicate.

The extracted content of an XVA file is a directory containing, at a minimum, a file called ova.xml. This file contains the metadata describing the VM packaged within the XVA. Each disk image is stored within a sub-directory and is referenced from the ova.xml. For example, the XVA describing a VM with two disks would contain the following metadata file and sub-directories:

$ls -al
total 72
drwx------+ 1 user1 Domain Users     0 Nov  7 10:35 .
drwxrwx---+ 1 user1 Domain Users     0 Nov  7 10:35 ..
-r-x------+ 1 user1 Domain Users 56046 Jan  1  1970 ova.xml
drwx------+ 1 user1 Domain Users     0 Nov  7 10:34 Ref_6
drwx------+ 1 user1 Domain Users     0 Nov  7 10:34 Ref_9
<!--NeedCopy-->

The file ova.xml contains the VM metadata in XML-RPC format, for example:

<value>
  <struct>
    <member>
      <name>objects</name>
      <value>
        <array>
          <data>
            <value>
              <struct>
                <member><name>class</name><value>VM</value></member>
                <member><name>id</name><value>Ref:VM</value></member>
                <member>
                  <name>snapshot</name>
                  <value>
                    <struct>
                      <member><name>power_state</name><value>Halted</value></member>
                      <member><name>name_label</name><value>Test VM</value></member>
                      <member><name>memory_static_max</name><value>4294967296</value></member>
                      <member><name>VCPUs_max</name><value>2</value></member>
                      <member><name>VIFs</name><value><array><data><value>Ref:VIF</value></data></array></value></member>
                      <member><name>PV_bootloader</name><value>pygrub</value></member>
                      <member><name>HVM_boot_policy</name><value>BIOS order</value></member>
                      <!-- ... -->
                    </struct>
                  </value>
                </member>
              </struct>
            </value>
            <value>
              <struct>
                <member><name>class</name><value>VIF</value></member>
                <member><name>id</name><value>Ref:VIF</value></member>
                <member>
                  <name>snapshot</name>
                  <value>
                    <!--...-->
                  </value>
                </member>
              </struct>
            </value>
            <!--...-->
          </data>
        </array>
      </value>
    </member>
  </struct>
</value>
<!--NeedCopy-->

A single disk image is represented by a directory containing a sequence of files as follows:

$ls -al Ref_6
total 2054
drwx------+ 1 user1 Domain Users       0 Nov  7 13:40 .
drwx------+ 1 user1 Domain Users       0 Nov  7 10:35 ..
-r-x------+ 1 user1 Domain Users 1048576 Jan  1  1970 00000000
-r-x------+ 1 user1 Domain Users      16 Jan  1  1970 00000000.xxhash
-r-x------+ 1 user1 Domain Users 1048576 Jan  1  1970 00000499
-r-x------+ 1 user1 Domain Users      16 Jan  1  1970 00000499.xxhash
...
<!--NeedCopy-->

Each of the extensionless files contains a block of raw disk image data of size 1MiB. The filename is the block number in decimal. The small size was chosen to be safely under the maximum file size limit of several filesystems. If these files are concatenated, the original image is recovered.

Each of the data block files is accompanied by a file with the same base name and the extension .xxhash, which contains the checksum of the corresponding data block file calculated using the xxHash algorithm.

Note:

In XenServer versions prior to Citrix Hypervisor 8.1, the checksums of the disk image blocks were calculated using the SHA1 algorithm, and the files containing the checksums had the extension .checksum. For example, a directory representing a disk image would contain the following sequence of files:
$ls -al
total 3079
drwx------+ 1 user1 Domain Users       0 Nov  7 14:20 .
drwx------+ 1 user1 Domain Users       0 Nov  7 10:35 ..
-rwx------+ 1 user1 Domain Users 1048576 Aug  4 09:50 00000000000000000000
-rwx------+ 1 user1 Domain Users      40 Aug  4 09:50 00000000000000000000.checksum
-rwx------+ 1 user1 Domain Users 1048576 Aug  4 09:50 00000000000000000001
-rwx------+ 1 user1 Domain Users      40 Aug  4 09:50 00000000000000000001.checksum
-rwx------+ 1 user1 Domain Users 1048576 Aug  4 09:50 00000000000000000003
-rwx------+ 1 user1 Domain Users      40 Aug  4 09:50 00000000000000000003.checksum
...

Getting XenServer Operational Metrics

XenServer records metrics about the operation of various aspects of your XenServer installation. The metrics are stored persistently for long term access and analysis of historical trends.

Metrics available

A wide range of metrics are available for XenServer including: C-State, P-State, IOPS, Latency and many more. Not all of these are turned on by default. The full range of metrics available for both Hosts and VMs are detailed in section Monitor XenServer Performance of XenServer’s documentation. This chapter also details how you can explore these metrics via XenCenter.

The data store for operational metrics

Metrics are stored in Round Robin Databases (RRDs), which are maintained for individual VMs (including the control domain) and the server. Each database consists of multiple Round Robin Archives (RRAs) and is of a fixed size. This is because each RRA is in effect a circular buffer with a predefined maximum capacity.

The VM RRDs are resident on the server on which the VM is running, or the pool coordinator when the VM is not running. This means that the location of the VM must be known in order to retrieve the associated data.

The RRDs are also backed up every day. Metrics are persisted for a maximum of one year, and are stored at different granularities.

RRDs are saved to disk as uncompressed XML. The size of each RRD when written to disk ranges from 200KiB to approximately 1.2MiB when the RRD stores the full year of metrics.

Note:

If metrics cannot be written to disk, for example when a disk is full, metrics will be lost and the last saved version of the RRD will be used.

Data Granularity

Each archive in the database samples its particular metric at a specified granularity. The values are stored at intervals of:

5 seconds for the past 10 minutes
one minute for the past 2 hours
one hour for the past week
one day for the past year

The sampling that takes place every 5 seconds records actual data points, however, the following RRAs contain Consolidated Data Points (CDPs) which are produced by applying the Consolidation Functions (CFs) to a number of actual data points, storing historical data in a far more compressed format. The CFs supported by XenServer are:

AVERAGE
MIN
MAX

Downloading RRDs

Metrics can be downloaded over HTTP(s), for example using wget.

Note:

In early versions of the XenServer Management API, instantaneous operational metrics could be obtained using the VM_metrics, VM_guest_metrics, host_metrics classes and associated methods. These methods have been deprecated in favor of using the HTTP(S) interface described in this chapter to download the metrics from the RRDs on the VMs and servers. The legacy metrics APIs will not return any values.

Metrics can be downloaded in xport style XML or JSON format. See rrddump and rrdxport for information about the XML format. The JSON format has the same structure as the XML.

Starting from xapi version 23.17.0, the server decides which format to return using the HTTP header Accept. To download metrics in XML format use 'Accept: text/xml', while to download metircs in JSON format use 'Accept: application/json'. When both formats are accepted, for example, when using 'Accept: */*' (which is the default for the wget client), the server returns JSON. The content type is provided in the reponse’s headers.

Note:

When no header is specified, for example, when using 'Accept: ', the server returns XML; to retrieve JSON you need to use the query parameter json. The header takes precedence over the query parameter and is the recommended method for deciding the returned format.

Downloading the whole RRD

RRDs can be downloaded from the XenServer host on which they reside by using the HTTP(S) handler registered at /host_rrd or /vm_rrd.

For example, to download a host RRD:

wget https://<server>/host_rrd?session_id=<session_opaque_reference>
<!--NeedCopy-->

To download a VM RRD, you must also specify the VM’s uuid as a query parameter:

wget https://<server>/vm_rrd?session_id=<session_opaque_reference>&uuid=<vm_uuid>
<!--NeedCopy-->

Getting updates from the RRD

So as to not have to download the whole database when you have most of the data already, a query can also be made to retrieve just updates. This is achieved via the HTTP(S) handler /rrd_updates.

Updates are described in relation to a start time which is specified as epoch time (number of seconds since Jan 1st, 1970) in a query parameter named start.

To obtain an update of all VM metrics on a server, the URL would be of the form:

wget http://<server>/rrd_updates?session_id=<session_opaque_reference>&start=<start_time>
<!--NeedCopy-->

The response contains data for every VM resident on the particular host that is being queried. This means that it is not possible to query a particular VM on its own. To differentiate which column in the export is associated with which VM, the legend field is prefixed with the VM’s uuid. The data set also contains a prefix describing the CF used to collect the data (MAX/MIN/AVERAGE).

To obtain host updates too, add the query parameter host=true:

wget http://<server>/rrd_updates?session_id=<session_opaque_reference>&start=<start_time>&host=true
<!--NeedCopy-->

When using RRD updates, you should be aware of the following:

/rrd_updates will only return data for the metrics that are currently being collected. For example, if you have an unplugged VBD, /vm_rrd will return the historical data for that VBD from when it was in use, but /rrd_updates won’t.
The value of the start parameter is used to deduce the granularity of the data /rrd_updates will return. If you specify a shorter time period, you will get more detailed metrics. For example, if you specify a start value that is 9 minutes before “now”, you’ll get 108 rows from the 10 minute RRA, but if you specify 11 minutes before “now”, you’ll get 11 rows from the 2 hour RRA.
Of particular importance is to use the same definition of “now” as the server is using, since the server is likely to be in UTC and your client may be in a different time zone.
The data returned by each call to /rrd_updates will contain a value (in the end XML tag or JSON field) that can be used as the start parameter for the next call.

Additional `/rrd_updates` parameters

Parameter	Value	Description
`cf`	`average\\|min\\|max`	the data consolidation mode
`interval`	`<interval>`	the interval between values to be reported

Note:

By default only average metrics are available. To obtain min and max metrics for a VM, run the following command:
xe pool-param-set uuid=<pool_uuid> other-config:create_min_max_in_new_VM_RRDs

Analysing RRDs

Once you have obtained the requested RRDs, you can use a utility such as rrdtool that will allow you, amongst other forms of analysis, to plot graphs very easily, or you can parse the results yourself.

This might be useful in if you want to perform simple tasks such as determining what the current Network throughput is, or perhaps carrying out your own, more complex analysis on the data points.

Some examples written in Python to demonstrate how you can parse and use the metrics can be found at XenServer Operational Metrics.

Example RRDs

Whole RRDs

Below is an example XML for a whole host RRD. The table highlights certain XML tags of interest.

XML tag	Description
`ds`	A data source field.
`name`	The name of this data source.
`type`	Defines the data source type. Can be `GAUGE`, `COUNTER`, `DERIVE` or `ABSOLUTE`.
`minimal_heartbeat`	Defines the maximum number of seconds before a `ds` value is considered unknown.
`min`	The minimum acceptable value. Values less than this number are considered unknown. This is optional.
`max`	The maximum acceptable value. Values exceeding this number are considered unknown. This is optional.

<?xml version="1.0"?>
<rrd>
  <version>0003</version>
  <step>5</step>
  <lastupdate>1213616574</lastupdate>
  <ds>
    <name>memory_total_kib</name>
    <type>GAUGE</type>
    <minimal_heartbeat>300.0000</minimal_heartbeat>
    <min>0.0</min>
    <max>Infinity</max>
    <last_ds>2070172</last_ds>
    <value>9631315.6300</value>
    <unknown_sec>0</unknown_sec>
  </ds>
  <ds>
   <!-- other data sources - the order of the data sources is important
        and defines the ordering of the columns in the archives below -->
  </ds>
  <rra>
    <cf>AVERAGE</cf>
    <pdp_per_row>1</pdp_per_row>
    <params>
      <xff>0.5000</xff>
    </params>
    <cdp_prep> <!-- This is for internal use -->
      <ds>
        <primary_value>0.0</primary_value>
        <secondary_value>0.0</secondary_value>
        <value>0.0</value>
        <unknown_datapoints>0</unknown_datapoints>
      </ds>
      ...other data sources - internal use only...
    </cdp_prep>
    <database>
     <row>
        <v>2070172.0000</v>  <!-- columns correspond to the DSs defined above -->
        <v>1756408.0000</v>
        <v>0.0</v>
        <v>0.0</v>
        <v>732.2130</v>
        <v>0.0</v>
        <v>782.9186</v>
        <v>0.0</v>
        <v>647.0431</v>
        <v>0.0</v>
        <v>0.0001</v>
        <v>0.0268</v>
        <v>0.0100</v>
        <v>0.0</v>
        <v>615.1072</v>
     </row>
     ...
  </rra>
  ... other archives ...
</rrd>
<!--NeedCopy-->

Here is an example JSON for a whole host RRD:

{
    "version": "0003",
    "step": "5",
    "lastupdate": "1698240426.8141",
    "ds": [
        {
            "name": "memory_total_kib",
            "type": "GAUGE",
            "minimal_heartbeat": "300.0000",
            "min": "0.0",
            "max": "Infinity",
            "last_ds": "33371836",
            "value": "60540111.1548",
            "unknown_sec": "0"
        }
        ... other data sources...
    ],
    "rra": [
        {
            "cf": "AVERAGE",
            "pdp_per_row": "1",
            "params": {
                "xff": "0.5000"
            },
            "cdp_prep": {
                "ds": [
                    {
                        "primary_value": "0.00",
                        "secondary_value": "0.00",
                        "value": "0.0",
                        "unknown_datapoints": "0.00"
                    }
                    ... other data sources - internal use only...
                ]
            },
            "database": [
                [
                    "0.0084",
                    "0.0090",
                    "0.0113",
                    "0.0072",
                    "0.0104",
                    "0.0099",
                    "0.0112",
                    "0.0131",
                    "0.0101",
                    "0.0100",
                    ... other values for the first data source...
                ],
                ... other data sources...
            ]
        },
        ... other archives...
    ]
}
<!--NeedCopy-->

Note that a VM RRD is identically structured, but with different data sources.

RRD updates

Example rrd_updates XML for one VM and no host updates:

<xport>
  <meta>
    <start>1213578000</start>
    <step>3600</step>
    <end>1213617600</end>
    <rows>12</rows>
    <columns>12</columns>
    <legend>
      <entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry> <!-- nb - each data source might have multiple entries for different consolidation functions -->
      <entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
      <entry>AVERAGE:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
      <entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
      <entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
      <entry>MIN:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
      <entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
      <entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
      <entry>MAX:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
      <entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu1</entry>
      <entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:cpu0</entry>
      <entry>LAST:vm:ecd8d7a0-1be3-4d91-bd0e-4888c0e30ab3:memory</entry>
    </legend>
  </meta>
  <data>
    <row>
      <t>1213617600</t>
      <v>0.0</v> <!-- once again, the order or the columns is defined by the legend above -->
      <v>0.0282</v>
      <v>209715200.0000</v>
      <v>0.0</v>
      <v>0.0201</v>
      <v>209715200.0000</v>
      <v>0.0</v>
      <v>0.0445</v>
      <v>209715200.0000</v>
      <v>0.0</v>
      <v>0.0243</v>
      <v>209715200.0000</v>
    </row>
   ...
  </data>
</xport>
<!--NeedCopy-->

Example rrd_updates JSON for one VM and no host updates:

{
    "meta": {
        "start": "1213660800",
        "step": "86400",
        "end": "1698192000",
        "rows": "366",
        "columns": "101",
        "legend": [
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu0",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu1",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu2",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu3",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu4",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu5",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu6",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:cpu7",
            "AVERAGE:vm:b36c4883-7dca-41c6-9f35-ec91826ec29f:memory",
            ...
        ]
    },
    "data": [
        {
            "t": "1698192000",
            "values": [
                "0.0082",
                "0.0080",
                "0.0072",
                "0.0072",
                "0.0069",
                "0.0073",
                "0.0069",
                "0.0076",
                "2724740352.0000",
                "NaN",
                "NaN",
                ...
            ]
        },
        ...
    ]
}
<!--NeedCopy-->

The official version of this content is in English. Some of the Cloud Software Group documentation content is machine translated for your convenience only. Cloud Software Group has no control over machine-translated content, which may contain errors, inaccuracies or unsuitable language. No warranty of any kind, either expressed or implied, is made as to the accuracy, reliability, suitability, or correctness of any translations made from the English original into any other language, or that your Cloud Software Group product or service conforms to any machine translated content, and any warranty provided under the applicable end user license agreement or terms of service, or any other agreement with Cloud Software Group, that the product or service conforms with any documentation shall not apply to the extent that such documentation has been machine translated. Cloud Software Group will not be held responsible for any damage or issues that may arise from using machine-translated content.

Was this helpful

Using HTTP to interact with XenServer

December 9, 2023

Contributed by:

X X

December 9, 2023

Contributed by:

X X

Using HTTP to interact with XenServer

Importing and exporting VMs as XVA packages

VM Export

VM Import

Xen Virtual Appliance (XVA) VM Import Format

Getting XenServer Operational Metrics

Metrics available

The data store for operational metrics

Data Granularity

Downloading RRDs

Downloading the whole RRD

Getting updates from the RRD

Additional /rrd_updates parameters

Analysing RRDs

Example RRDs

Whole RRDs

RRD updates

In this article

Additional `/rrd_updates` parameters