Migrating from a standalone Engine to a self-hosted engine

You can convert a standalone oVirt Engine to a self-hosted engine by backing up the standalone Engine and restoring it in a new self-hosted environment.

The difference between the two environment types is explained below:

Standalone Engine Architecture

The oVirt Engine runs on a physical server, or a virtual machine hosted in a separate virtualization environment. A standalone Engine is easier to deploy and manage, but requires an additional physical server. The Engine is only highly available when managed externally with a product such as Red Hat’s High Availability Add-On.

The minimum setup for a standalone Engine environment includes:

  • One oVirt Engine machine. The Engine is typically deployed on a physical server. However, it can also be deployed on a virtual machine, as long as that virtual machine is hosted in a separate environment. The Engine must run on Enterprise Linux 7.

  • A minimum of two hosts for virtual machine high availability. You can use Enterprise Linux hosts or oVirt Nodes (oVirt Node). VDSM (the host agent) runs on all hosts to facilitate communication with the oVirt Engine.

  • One storage service, which can be hosted locally or on a remote server, depending on the storage type used. The storage service must be accessible to all hosts.

Standalone Engine oVirt Architecture
Figure 1. Standalone Engine oVirt Architecture

Self-Hosted Engine Architecture

The oVirt Engine runs as a virtual machine on self-hosted engine nodes (specialized hosts) in the same environment it manages. A self-hosted engine environment requires one less physical server, but requires more administrative overhead to deploy and manage. The Engine is highly available without external HA management.

The minimum setup of a self-hosted engine environment includes:

  • One oVirt Engine virtual machine that is hosted on the self-hosted engine nodes. The Engine Appliance is used to automate the installation of a Enterprise Linux 7 virtual machine, and the Engine on that virtual machine.

  • A minimum of two self-hosted engine nodes for virtual machine high availability. You can use Enterprise Linux hosts or oVirt Nodes (oVirt Node). VDSM (the host agent) runs on all hosts to facilitate communication with the oVirt Engine. The HA services run on all self-hosted engine nodes to manage the high availability of the Engine virtual machine.

  • One storage service, which can be hosted locally or on a remote server, depending on the storage type used. The storage service must be accessible to all hosts.

Self-Hosted Engine oVirt Architecture
Figure 2. Self-Hosted Engine oVirt Architecture

1. Migration Overview

When you specify a backup file during self-hosted engine deployment, the Engine backup is restored on a new virtual machine, with a dedicated self-hosted engine storage domain. Deploying on a fresh host is highly recommended; if the host used for deployment existed in the backed up environment, it will be removed from the restored database to avoid conflicts in the new environment.

At least two self-hosted engine nodes are required for the Engine virtual machine to be highly available. You can add new nodes, or convert existing hosts.

The migration involves the following key steps:

This procedure assumes that you have access and can make changes to the original Engine.

Prerequisites
  • FQDNs prepared for your Engine and the deployment host. Forward and reverse lookup records must both be set in the DNS. The new Engine must have the same FQDN as the original Engine.

  • The management network (ovirtmgmt by default) must be configured as a VM network, so that it can manage the Engine virtual machine.

2. Installing the Self-hosted Engine Deployment Host

A self-hosted engine can be deployed from a oVirt Node or a Enterprise Linux host.

If you plan to use bonded interfaces for high availability or VLANs to separate different types of traffic (for example, for storage or management connections), you should configure them on the host before beginning the self-hosted engine deployment. See Networking Recommendations in the Planning and Prerequisites Guide.

2.1. Installing oVirt Nodes

oVirt Node (oVirt Node) is a minimal operating system based on Enterprise Linux that is designed to provide a simple method for setting up a physical machine to act as a hypervisor in a oVirt environment. The minimal operating system contains only the packages required for the machine to act as a hypervisor, and features a Cockpit web interface for monitoring the host and performing administrative tasks. See http://cockpit-project.org/running.html for the minimum browser requirements.

oVirt Node supports NIST 800-53 partitioning requirements to improve security. oVirt Node uses a NIST 800-53 partition layout by default.

The host must meet the minimum host requirements.

Procedure
  1. Visit the oVirt Node Download page.

  2. Choose the version of oVirt Node to download and click its Installation ISO link.

  3. Write the oVirt Node Installation ISO disk image to a USB, CD, or DVD.

  4. Start the machine on which you are installing oVirt Node, booting from the prepared installation media.

  5. From the boot menu, select Install oVirt Node 4.4 and press Enter.

    You can also press the Tab key to edit the kernel parameters. Kernel parameters must be separated by a space, and you can boot the system using the specified kernel parameters by pressing the Enter key. Press the Esc key to clear any changes to the kernel parameters and return to the boot menu.

  6. Select a language, and click Continue.

  7. Select a keyboard layout from the Keyboard Layout screen and click Done.

  8. Select the device on which to install oVirt Node from the Installation Destination screen. Optionally, enable encryption. Click Done.

    Use the Automatically configure partitioning option.

  9. Select a time zone from the Time & Date screen and click Done.

  10. Select a network from the Network & Host Name screen and click Configure…​ to configure the connection details.

    To use the connection every time the system boots, select the Connect automatically with priority check box. For more information, see Configuring network and host name options in the Enterprise Linux 8 Installation Guide.

    Enter a host name in the Host Name field, and click Done.

  11. Optionally configure Language Support, Security Policy, and Kdump. See Customizing your RHEL installation using the GUI in Performing a standard RHEL installation for _Enterprise Linux 8 for more information on each of the sections in the Installation Summary screen.

  12. Click Begin Installation.

  13. Set a root password and, optionally, create an additional user while oVirt Node installs.

    Do not create untrusted users on oVirt Node, as this can lead to exploitation of local security vulnerabilities.

  14. Click Reboot to complete the installation.

    When oVirt Node restarts, nodectl check performs a health check on the host and displays the result when you log in on the command line. The message node status: OK or node status: DEGRADED indicates the health status. Run nodectl check to get more information. The service is enabled by default.

2.2. Installing Enterprise Linux hosts

A Enterprise Linux host is based on a standard basic installation of Enterprise Linux 8 on a physical server, with the Enterprise Linux Server and oVirt repositories enabled.

For detailed installation instructions, see the Performing a standard EL installation.

The host must meet the minimum host requirements.

Virtualization must be enabled in your host’s BIOS settings. For information on changing your host’s BIOS settings, refer to your host’s hardware documentation.

Do not install third-party watchdogs on Enterprise Linux hosts. They can interfere with the watchdog daemon provided by VDSM.

Although the existing storage domains will be migrated from the standalone Engine, you must prepare additional storage for a self-hosted engine storage domain that is dedicated to the Engine virtual machine.

3. Preparing Storage for oVirt

Prepare storage to be used for storage domains in the new environment. A oVirt environment must have at least one data storage domain, but adding more is recommended.

A data domain holds the virtual hard disks and OVF files of all the virtual machines and templates in a data center, and cannot be shared across data centers while active (but can be migrated between data centers). Data domains of multiple storage types can be added to the same data center, provided they are all shared, rather than local, domains.

You can use one of the following storage types:

Prerequisites
  • Self-hosted engines must have an additional data domain with at least 74 GiB dedicated to the Engine virtual machine. The self-hosted engine installer creates this domain. Prepare the storage for this domain before installation.

  • When using a block storage domain, either FCP or iSCSI, a single target LUN is the only supported setup for a self-hosted engine.

  • If you use iSCSI storage, the self-hosted engine storage domain must use a dedicated iSCSI target. Any additional storage domains must use a different iSCSI target.

  • It is strongly recommended to create additional data storage domains in the same data center as the self-hosted engine storage domain. If you deploy the self-hosted engine in a data center with only one active data storage domain, and that storage domain is corrupted, you cannot add new storage domains or remove the corrupted storage domain. You must redeploy the self-hosted engine.

3.1. Preparing NFS Storage

Set up NFS shares on your file storage or remote server to serve as storage domains on Red Hat Enterprise Virtualization Host systems. After exporting the shares on the remote storage and configuring them in the Red Hat Virtualization Manager, the shares will be automatically imported on the Red Hat Virtualization hosts.

For information on setting up and configuring NFS, see Network File System (NFS) in the Enterprise Linux 7 Storage Administration Guide.

Specific system user accounts and system user groups are required by oVirt so the Engine can store data in the storage domains represented by the exported directories. The following procedure sets the permissions for one directory. You must repeat the chown and chmod steps for all of the directories you intend to use as storage domains in oVirt.

Procedure
  1. Create the group kvm:

    # groupadd kvm -g 36
  2. Create the user vdsm in the group kvm:

    # useradd vdsm -u 36 -g 36
  3. Set the ownership of your exported directory to 36:36, which gives vdsm:kvm ownership:

    # chown -R 36:36 /exports/data
  4. Change the mode of the directory so that read and write access is granted to the owner, and so that read and execute access is granted to the group and other users:

    # chmod 0755 /exports/data

3.2. Preparing iSCSI Storage

oVirt supports iSCSI storage, which is a storage domain created from a volume group made up of LUNs. Volume groups and LUNs cannot be attached to more than one storage domain at a time.

For information on setting up and configuring iSCSI storage, see Online Storage Management in the Enterprise Linux 7 Storage Administration Guide.

If you are using block storage and you intend to deploy virtual machines on raw devices or direct LUNs and to manage them with the Logical Volume Manager, you must create a filter to hide the guest logical volumes. This will prevent guest logical volumes from being activated when the host is booted, a situation that could lead to stale logical volumes and cause data corruption. See https://access.redhat.com/solutions/2662261 for details.

oVirt currently does not support storage with a block size of 4K. You must configure block storage in legacy (512b block) mode.

If your host is booting from SAN storage and loses connectivity to the storage, the storage file systems become read-only and remain in this state after connectivity is restored.

To prevent this situation, add a drop-in multipath configuration file on the root file system of the SAN for the boot LUN to ensure that it is queued when there is a connection:

# cat /etc/multipath/conf.d/host.conf
multipaths {
    multipath {
        wwid boot_LUN_wwid
        no_path_retry queue
    }

3.3. Preparing FCP Storage

oVirt supports SAN storage by creating a storage domain from a volume group made of pre-existing LUNs. Neither volume groups nor LUNs can be attached to more than one storage domain at a time.

oVirt system administrators need a working knowledge of Storage Area Networks (SAN) concepts. SAN usually uses Fibre Channel Protocol (FCP) for traffic between hosts and shared external storage. For this reason, SAN may occasionally be referred to as FCP storage.

For information on setting up and configuring FCP or multipathing on Enterprise Linux, see the Storage Administration Guide and DM Multipath Guide.

If you are using block storage and you intend to deploy virtual machines on raw devices or direct LUNs and to manage them with the Logical Volume Manager, you must create a filter to hide the guest logical volumes. This will prevent guest logical volumes from being activated when the host is booted, a situation that could lead to stale logical volumes and cause data corruption. See https://access.redhat.com/solutions/2662261 for details.

oVirt currently does not support storage with a block size of 4K. You must configure block storage in legacy (512b block) mode.

If your host is booting from SAN storage and loses connectivity to the storage, the storage file systems become read-only and remain in this state after connectivity is restored.

To prevent this situation, add a drop-in multipath configuration file on the root file system of the SAN for the boot LUN to ensure that it is queued when there is a connection:

# cat /etc/multipath/conf.d/host.conf
multipaths {
    multipath {
        wwid boot_LUN_wwid
        no_path_retry queue
    }

3.4. Preparing Gluster Storage

For information on setting up and configuring Gluster Storage, see the Gluster Storage Installation Guide.

3.5. Customizing Multipath Configurations for SAN Vendors

To customize the multipath configuration settings, do not modify /etc/multipath.conf. Instead, create a new configuration file that overrides /etc/multipath.conf.

Upgrading Virtual Desktop and Server Manager (VDSM) overwrites the /etc/multipath.conf file. If multipath.conf contains customizations, overwriting it can trigger storage issues.

Prerequisites
Procedure
  1. To override the values of settings in /etc/multipath.conf, create a new configuration file in the /etc/multipath/conf.d/ directory.

    The files in /etc/multipath/conf.d/ execute in alphabetical order. Follow the convention of naming the file with a number at the beginning of its name. For example, /etc/multipath/conf.d/90-myfile.conf.

  2. Copy the settings you want to override from /etc/multipath.conf to the new configuration file in /etc/multipath/conf.d/. Edit the setting values and save your changes.

  3. Apply the new configuration settings by entering the systemctl reload multipathd command.

    Avoid restarting the multipathd service. Doing so generates errors in the VDSM logs.

Verification steps

If you override the VDSM-generated settings in /etc/multipath.conf, verify that the new configuration performs as expected in a variety of failure scenarios.

For example, disable all of the storage connections. Then enable one connection at a time and verify that doing so makes the storage domain reachable.

Troubleshooting

If a oVirt Node has trouble accessing shared storage, check /etc/multpath.conf and files under /etc/multipath/conf.d/ for values that are incompatible with the SAN.

Additional resources

When overriding /etc/multipath.conf, Do not override the following settings:

user_friendly_names no

This setting controls whether user-friendly names are assigned to devices in addition to the actual device names. Multiple hosts must use the same name to access devices. Disabling this setting prevents user-friendly names from interfering with this requirement.

find_multipaths no

This setting controls whether oVirt Node tries to access all devices through multipath, even if only one path is available. Disabling this setting prevents oVirt from using the too-clever behavior when this setting is enabled.

Avoid overriding the following settings unless required by the storage system vendor:

no_path_retry 4

This setting controls the number of polling attempts to retry when no paths are available. Before oVirt version 4.2, the value of no_path_retry was fail because QEMU had trouble with the I/O queuing when no paths were available. The fail value made it fail quickly and paused the virtual machine. oVirt version 4.2 changed this value to 4 so when multipathd detects the last path has failed, it checks all of the paths four more times. Assuming the default 5-second polling interval, checking the paths takes 20 seconds. If no path is up, multipathd tells the kernel to stop queuing and fails all outstanding and future I/O until a path is restored. When a path is restored, the 20-second delay is reset for the next time all paths fail. For more details, see the commit that changed this setting.

polling_interval 5

This setting determines the number of seconds between polling attempts to detect whether a path is open or has failed. Unless the vendor provides a clear reason for increasing the value, keep the VDSM-generated default so the system responds to path failures sooner.

Before backing up the Engine, ensure it is updated to the latest minor version. The Engine version in the backup file must match the version of the new Engine.

4. Updating the oVirt Engine

Updates to the oVirt Engine are released through the Content Delivery Network.

Procedure
  1. Check if updated packages are available:

    # engine-upgrade-check
  2. Update the setup packages:

    # yum update ovirt\*setup\*
  3. Update the oVirt Engine with the engine-setup script. The engine-setup script prompts you with some configuration questions, then stops the ovirt-engine service, downloads and installs the updated packages, backs up and updates the database, performs post-installation configuration, and starts the ovirt-engine service.

    # engine-setup

    When the script completes successfully, the following message appears:

    Execution of setup completed successfully

    The engine-setup script is also used during the oVirt Engine installation process, and it stores the configuration values supplied. During an update, the stored values are displayed when previewing the configuration, and might not be up to date if engine-config was used to update configuration after installation. For example, if engine-config was used to update SANWipeAfterDelete to true after installation, engine-setup will output "Default SAN wipe after delete: False" in the configuration preview. However, the updated values will not be overwritten by engine-setup.

    The update process might take some time. Do not stop the process before it completes.

  4. Update the base operating system and any optional packages installed on the Engine:

    # yum update

    If any kernel packages were updated, reboot the machine to complete the update.

5. Backing up the Original Engine

Back up the original Engine using the engine-backup command, and copy the backup file to a separate location so that it can be accessed at any point during the process.

For more information about engine-backup --mode=backup options, see Backing Up and Restoring the oVirt Engine in the Administration Guide.

Procedure
  1. Log in to the original Engine and stop the ovirt-engine service:

    # systemctl stop ovirt-engine
    # systemctl disable ovirt-engine

    Though stopping the original Engine from running is not obligatory, it is recommended as it ensures no changes are made to the environment after the backup is created. Additionally, it prevents the original Engine and the new Engine from simultaneously managing existing resources.

  2. Run the engine-backup command, specifying the name of the backup file to create, and the name of the log file to create to store the backup log:

    # engine-backup --mode=backup --file=file_name --log=log_file_name
  3. Copy the files to an external server. In the following example, storage.example.com is the fully qualified domain name of a network storage server that will store the backup until it is needed, and /backup/ is any designated folder or path.

    # scp -p file_name log_file_name storage.example.com:/backup/

After backing up the Engine, deploy a new self-hosted engine and restore the backup on the new virtual machine.

6. Restoring the Backup on a New Self-Hosted Engine

Run the hosted-engine script on a new host, and use the --restore-from-file=path/to/file_name option to restore the Engine backup during the deployment.

If you are using iSCSI storage, and your iSCSI target filters connections according to the initiator’s ACL, the deployment may fail with a STORAGE_DOMAIN_UNREACHABLE error. To prevent this, you must update your iSCSI configuration before beginning the self-hosted engine deployment:

  • If you are redeploying on an existing host, you must update the host’s iSCSI initiator settings in /etc/iscsi/initiatorname.iscsi. The initiator IQN must be the same as was previously mapped on the iSCSI target, or updated to a new IQN, if applicable.

  • If you are deploying on a fresh host, you must update the iSCSI target configuration to accept connections from that host.

Note that the IQN can be updated on the host side (iSCSI initiator), or on the storage side (iSCSI target).

Procedure
  1. Copy the backup file to the new host. In the following example, host.example.com is the FQDN for the host, and /backup/ is any designated folder or path.

    # scp -p file_name host.example.com:/backup/
  2. Log in to the new host.

  3. If you are deploying on oVirt Node, ovirt-hosted-engine-setup is already installed, so skip this step. If you are deploying on Enterprise Linux, install the ovirt-hosted-engine-setup package:

    # yum install ovirt-hosted-engine-setup
  4. Use the tmux window manager to run the script to avoid losing the session in case of network or terminal disruption.

    Install and run tmux:

    # yum -y install tmux
    # tmux
  5. Run the hosted-engine script, specifying the path to the backup file:

    # hosted-engine --deploy --restore-from-file=backup/file_name

    To escape the script at any time, use CTRL+D to abort deployment.

  6. Select Yes to begin the deployment.

  7. Configure the network. The script detects possible NICs to use as a management bridge for the environment.

  8. If you want to use a custom appliance for the virtual machine installation, enter the path to the OVA archive. Otherwise, leave this field empty to use the Engine Appliance.

  9. Specify the FQDN for the Engine virtual machine.

  10. Enter the root password for the Engine.

  11. Enter an SSH public key that will allow you to log in to the Engine as the root user, and specify whether to enable SSH access for the root user.

  12. Enter the virtual machine’s CPU and memory configuration.

    The virtual machine must have the same amount of RAM as the physical machine from which the Engine is being migrated. If you must migrate to a virtual machine that has less RAM than the physical machine from which the Engine is migrated, see https://access.redhat.com/articles/2705841.

  13. Enter a MAC address for the Engine virtual machine, or accept a randomly generated one. If you want to provide the Engine virtual machine with an IP address via DHCP, ensure that you have a valid DHCP reservation for this MAC address. The deployment script will not configure the DHCP server for you.

  14. Enter the virtual machine’s networking details. If you specify Static, enter the IP address of the Engine.

    The static IP address must belong to the same subnet as the host. For example, if the host is in 10.1.1.0/24, the Engine virtual machine’s IP must be in the same subnet range (10.1.1.1-254/24).

  15. Specify whether to add entries for the Engine virtual machine and the base host to the virtual machine’s /etc/hosts file. You must ensure that the host names are resolvable.

  16. Provide the name and TCP port number of the SMTP server, the email address used to send email notifications, and a comma-separated list of email addresses to receive these notifications:

  17. Enter a password for the admin@internal user to access the Administration Portal.

    The script creates the virtual machine. This can take some time if the Engine Appliance needs to be installed.

  18. Select the type of storage to use:

    • For NFS, enter the version, full address and path to the storage, and any mount options.

    • For iSCSI, enter the portal details and select a target and LUN from the auto-detected lists. You can only select one iSCSI target during the deployment, but multipathing is supported to connect all portals of the same portal group.

      To specify more than one iSCSI target, you must enable multipathing before deploying the self-hosted engine. See Enterprise Linux DM Multipath for details. There is also a Multipath Helper tool that generates a script to install and configure multipath with different options.

    • For Gluster storage, enter the full address and path to the storage, and any mount options.

      Only replica 1 and replica 3 Gluster storage are supported. Ensure you configure the volume as follows:

      gluster volume set VOLUME_NAME group virt
      gluster volume set VOLUME_NAME performance.strict-o-direct on
      gluster volume set VOLUME_NAME network.remote-dio off
      gluster volume set VOLUME_NAME storage.owner-uid 36
      gluster volume set VOLUME_NAME storage.owner-gid 36
      gluster volume set VOLUME_NAME network.ping-timeout 30
    • For Fibre Channel, select a LUN from the auto-detected list. The host bus adapters must be configured and connected, and the LUN must not contain any existing data. To reuse an existing LUN, see Reusing LUNs in the Administration Guide.

  19. Enter the Engine disk size.

    The script continues until the deployment is complete.

  20. The deployment process changes the Engine’s SSH keys. To allow client machines to access the new Engine without SSH errors, remove the original Engine’s entry from the .ssh/known_hosts file on any client machines that accessed the original Engine.

When the deployment is complete, log in to the new Engine virtual machine and enable the required repositories.

7. Enabling the oVirt Engine Repositories

Ensure the correct repositories are enabled:

You can check which repositories are currently enabled by running yum repolist.

  1. Enable the javapackages-tools module.

    # yum module -y enable javapackages-tools
  2. Enable the pki-deps module.

    # yum module -y enable pki-deps
  3. Enable version 12 of the postgresql module.

    # yum module -y enable postgresql:12

The oVirt Engine has been migrated to a self-hosted engine setup. The Engine is now operating on a virtual machine on the new self-hosted engine node.

The hosts will be running in the new environment, but cannot host the Engine virtual machine. You can convert some or all of these hosts to self-hosted engine nodes.

8. Reinstalling an Existing Host as a Self-Hosted Engine Node

You can convert an existing, standard host in a self-hosted engine environment to a self-hosted engine node capable of hosting the Engine virtual machine.

Procedure
  1. Click Compute  Hosts and select the host.

  2. Click Management  Maintenance and click OK.

  3. Click Installation  Reinstall.

  4. Click the Hosted Engine tab and select DEPLOY from the drop-down list.

  5. Click OK.

The host is reinstalled with self-hosted engine configuration, and is flagged with a crown icon in the Administration Portal.

After reinstalling the hosts as self-hosted engine nodes, you can check the status of the new environment by running the following command on one of the nodes:

# hosted-engine --vm-status

If the new environment is running without issue, you can decommission the original Engine machine.

Certain portions of this text first appeared in Red Hat Virtualization 4.4-beta Migrating from a standalone Manager to a self-hosted engine. Copyright © 2020 Red Hat, Inc. Licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.