Troubleshoot VM problems

Citrix provides two forms of support:

  • Free, self-help support on the Citrix website
  • Paid-for Support Services, which you can purchase from the Support Site.

With Citrix Technical Support, you can open a Support Case online or contact the support center by phone if you experience technical difficulties.

The Citrix Support site hosts several resources that might be helpful to you if you experience unusual behavior, crashes, or other problems. Resources include: Support Forums, Knowledge Base articles, and product documentation.

If you see unusual VM behavior, this section aims to help you solve the problem. This section describes where application logs are located and other information that can help your Citrix Hypervisor Solution Provider track and resolve the issue.

Important:

Follow the troubleshooting information in this section only under the guidance of your Citrix Hypervisor Solution Provider or the Support Team.

Vendor Updates: Keep your VMs up-to-date with operating system vendor-supplied updates. The vendor might have provided fixes for VM crashed and other failures.

VM crashes

If you are experiencing VM crashes, it is possible that a kernel crash dump can help identify the problem. Reproduce the crash, if possible, and follow this procedure. Consult your guest OS vendor for further investigation on this issue.

Control Linux VM crashdump behavior

For Linux VMs, the crashdump behavior can be controlled through the actions-after-crash parameter. The following are the possible values:

Value Description
preserve Leave the VM in a paused state. (For analysis)
restart No core dump, reboot VM. (This is the default)
destroy No core dump, leave VM halted.

To enable saving of Linux VM crash dumps:

  1. On the Citrix Hypervisor server, determine the UUID of the desired VM by running the following command:

    xe vm-list name-label=name params=uuid --minimal
    
  2. Change the actions-after-crash value using xe vm-param-set; for example, run the following command on dom0:

    xe vm-param-set uuid=vm_uuid actions-after-crash=preserve
    
  3. Crash the VM.

    • For PV guests, run the following command on the VM:

       echo c | sudo tee /proc/sysrq-trigger
      
  4. Execute the dump core command on dom0. For example, run:

    xl dump-core domid filename
    

Control Windows VM crashdump behavior

For Windows VMs, the actions-after-crash parameter cannot control the core dump behavior. By default Windows crash dumps are put into %SystemRoot%\Minidump in the Windows VM itself.

You can configure the VMs dump level by following the menu path My Computer > Properties > Advanced > Startup and Recovery.

Troubleshoot boot problems on Linux VMs

There is a utility script named xe-edit-bootloader in the Citrix Hypervisor server control domain. This script can be used to edit the bootloader configuration of a shutdown Linux VM and fix problems that prevent the VM from booting.

To use this script:

  1. Run the following command:

    xe vm-list
    

    This command ensures that the VM in question is shut down (the value of power-state is halted).

  2. You can use the UUID as follows:

    xe-edit-bootloader -u linux_vm_uuid -p partition_number
    

    Or, you can use the name-label as follows:

    xe-edit-bootloader -n linux_vm_name_label -p partition_number
    

    The partition number represents the slice of the disk which has the filesystem. For the default Debian template, the partition number is 1 since it is the first partition.

  3. You are dropped into an editor with the grub.conf file for the specified VM loaded. Change the file to fix it, and save the file, exit the editor, and start the VM.

Troubleshoot UEFI and Secure Boot problems on Windows VMs

How do I change the screen resolution of the XenCenter console on a UEFI-enabled VM?

To change the screen resolution of the XenCenter console on a UEFI-enabled VM:

  1. Open the Windows Settings
  2. Click the Update & Security button
  3. Under the recovery tab, press the Restart now button.
  4. Navigate to Troubleshoot > Advanced Options > UEFI firmware settings.
  5. Press Restart. During restart, the UEFI settings menu loads.
  6. Navigate to Device Manager > OVMF Platform Configuration. This displays the current screen resolution.
  7. Press Enter to see the screen resolution options.
  8. Use the arrow keys to select the desired screen resolution and press Enter.
  9. Press F10 to save the changes and confirm your choice.
  10. Reboot the VM to see the XenCenter console in an updated screen resolution.

Why can’t I create a UEFI Secure Boot VM?

Check that your VM operating system supports UEFI Secure Boot mode. In Citrix Hypervisor 8.2, only the following operating systems support Secure Boot: Windows 10 (64-bit), Windows Server 2016 (64-bit), Windows Server 2019 (64-bit).

Check that your Citrix Hypervisor server is booted in UEFI mode. You can only create UEFI Secure Boot VMs on a Citrix Hypervisor server that has the Secure Boot certificates present. Secure Boot certificates are only present on servers booted in UEFI mode or servers in the same pool as a server booted in UEFI mode. For more information, see Network Boot.

Check that the UEFI-booted Citrix Hypervisor server is included in the Hardware Compatibility List. Older servers might not include the Secure Boot certificates when booted in UEFI mode.

How do I know if the Citrix Hypervisor server that I create a Secure Boot VM on has the Secure Boot certificates?

If your Citrix Hypervisor server is booted in UEFI mode, the Secure Boot certificates are available on the server. Citrix Hypervisor servers share their certificates with other servers in the same resource pool. If you have a UEFI booted server in your resource pool, all servers in that pool have the Secure Boot certificates available.

Run the following command on a Citrix Hypervisor server:

xe pool-param-get param-name=uefi-certificates uuid=<pool-uuid> | wc -c

If it returns a value that is greater than zero, the Secure Boot certificates are present.

To check that the certificates are valid, run the following command on your Citrix Hypervisor server:

xe pool-param-get uuid=$(xe pool-list --minimal)  param-name=uefi-certificates|base64 -d|tar tv
-rw-r--r-- root/root      1600 2019-11-11 17:09 KEK.auth
-rw-r--r-- root/root      3212 2019-11-11 17:09 db.auth

If the Secure Boot certificates are absent, run the following command on your Citrix Hypervisor server:

ls /sys/firmware/efi/efivars | grep KEK

If this command returns empty, Secure Boot VMs cannot be created on that server because the the required certificates are missing from the UEFI firmware.

Why is my UEFI Secure Boot VM failing to start?

If you see the following messages on the console of your UEFI Secure Boot VM and an alert in XenCenter, the Secure Boot process has failed and your VM does not start.

Console output from UEFI Secure Boot VM that does not start.

This is usually caused by the installation of unsigned drivers into the VM. Investigate what drivers have been updated or installed since the last successful Secure Boot.

You can disable Secure Boot and start the VM in setup mode to remove the unsigned drivers.

Important:

Before doing this, back up your VM by taking a snapshot.

To change a UEFI Secure Boot VM into a UEFI boot VM, run the following command on the Citrix Hypervisor server that hosts the VM:

varstore-sb-state <VM_UUID> setup

After you have fixed your VM, run the following command to re-enable Secure Boot:

varstore-sb-state <VM_UUID> user

Is Secure Boot causing an issue on my Windows VM?

To diagnose whether an issue on your Windows VM is caused by Secure Boot being enabled for the VM, disable Secure Boot and try to reproduce the issue.

To disable Secure Boot, run the following command on the Citrix Hypervisor server that hosts the VM:

varstore-sb-state <VM_UUID> setup

After you have debugged the issue, you can run the following command to re-enable Secure Boot:

varstore-sb-state <VM_UUID> user

How do I run Windows debug on a Secure Boot VM?

You cannot run Windows debug on a Secure Boot VM. To run Windows debug on your VM, you can do one of the following things:

  • Switch your VM to UEFI boot mode by running the following command:

     xe vm-param-set uuid=<UUID> platform:secureboot=false
    

    Reboot the VM.

    After you have debugged the issue, you can run the following command to re-enable Secure Boot:

     xe vm-param-set uuid=<UUID> platform:secureboot=auto
    

    Reboot the VM.

  • Disable Secure Boot by running the following command on the Citrix Hypervisor server that hosts the VM:

     varstore-sb-state <VM_UUID> setup
    

    After you have debugged the issue, you can run the following command to re-enable Secure Boot:

     varstore-sb-state <VM_UUID> user
    

Why are only two NICs showing up for my UEFI VM?

Even if you set up more than two NICs when you created your UEFI-enabled VM, when the VM first starts you only see two NICs. This information displays correctly after the Citrix VM Tools for Windows have been installed in the VM.

Why are my emulated devices showing as different types than expected?

A UEFI Secure Boot VMs use NVME and E1000 for emulated devices. However, when the VM first starts the emulated devices show as different types. This information displays correctly after the Citrix VM Tools for Windows have been installed in the VM.

Why can’t I convert my templates from BIOS mode to UEFI or UEFI Secure Boot mode?

You can only create a UEFI-enabled VM template from a template supplied with Citrix Hypervisor.

Do not use the xe template-param-set command for templates that have something installed on them or templates that you created from a snapshot. The boot mode of these snapshots cannot be changed and, if you attempt to change the boot mode, the VM fails to boot.

How do I check UEFI and UEFI Secure Boot settings?

On the Citrix Hypervisor server where the UEFI or UEFI Secure Boot VM is hosted, run the following commands:

 varstore-ls

This command lists the IDs of the available settings. Use the setting ID in the following command:

varstore-get <setting_ID> <VM_ID> | hexdump -C

The value for the setting is 0 for disabled and 1 for enabled.

Why can’t I use a ‘test’ driver with a Secure Boot VM?

If a customer is also working with a third party to debug and fix issues in their UEFI Secure Boot VM, the third party provide might provide unsigned drivers for test or verification purpose. These drivers will not work in a UEFI Secure Boot VM.

Tell the customer to request a signed driver from the third party. Or the customer can switch their UEFI Secure Boot VM into setup mode to run with the unsigned driver.