OpenStack Live Migration Failed: How to Diagnose and Fix Common Causes

OpenStack Live Migration and Nova Host Evacuation: What They Are and When to Use Them

Production infrastructure cannot afford unplanned downtime, but with OpenStack, it doesn't have to. OpenStack's live migration feature moves running virtual machine instances from one compute node to another within the same cluster. This keeps services available while the underlying hardware or software is maintained. The Nova compute service orchestrates this process and extends it further through host evacuation, which is a mechanism designed to recover workloads from compute nodes that have become unreachable or failed entirely.

These two mechanisms work together to give operations teams the flexibility to respond to both planned maintenance windows and unplanned node failures without taking workloads offline.

Understanding Nova host evacuation operations

OpenStack Nova provides several commands that handle different migration or evacuation scenarios:

• host-evacuate-live: Live-migrate running instances (virtual machines) across hosts in the same cluster with minimal service disruptions.
• host-evacuate: Evacuate all instances (virtual machines) from a host that has failed due to hardware or other malfunctions. 
• host-server-migrate: Cold-migrate powered-off instances (virtual machines) to other hosts in the cluster.
• live-migration: Moving a live running instance (virtual machine) from one host to another with minimal service disruption during host switchover.
• migrate: Migrate a server where the compute scheduler selects a new host.

As the title indicates, we are mostly interested in host-evacuate-live, which migrates all running instances to other computing nodes. The host-evacuate-live is a powerful and useful one, since it performs host-wide migrations of running workloads. Performing a host-evacuate-live is also the most demanding; other similar actions would undoubtedly work.

Let’s examine an example of migration failure and how to resolve it.

Diagnosing a failed Nova live host evacuation

Cluster architecture and pre-migration state

Our test environment consisted of three control-converged nodes: cc1, cc2, and cc3. Control-converged nodes all provide compute, storage, and networking services to the cluster. All VM instances (virtual machines) were active and running as our initial cluster state on the cc2 host.

Image 1: All instances (virtual machines) hosted on cc2 in the running state

Initiating virtual machine live migration with Nova host-evacuate-live

To begin maintenance on node cc2, we attempted to migrate all running instances from cc2 to cc3 using Nova live evacuation. We migrate all instances (virtual machines) from cc2 with:

# nova host-evacuate-live cc2

The instance status transitioned from 'ACTIVE' to 'MIGRATING' as the migration operation took place.

Image 2: All instances (virtual machines) enter the 'MIGRATING' status

After several seconds, the two instances have successfully migrated to cc3, and their status has transitioned from 'MIGRATING' to 'Active'. We can confirm the operation has concluded successfully by verifying that cc3 became the new instance host.

Image 3: Two instances (virtual machines) have completed migrating, and one instance remains in migration

However, the migration operation times out for our Cirros_Host-Storage instance, resulting in an ERROR status.

Image 4: An instance encountered an 'ERROR' during Nova live migration

This failure could also be confirmed in Horizon, the OpenStack dashboard.

Image 5: Horizon management interface with the failed VM display

Step 2: Diagnosing the migration timeout through OpenStack log analysis

To identify the root cause, we begin by checking the Nova compute log on the source node cc2. We can see that the live migration operation encountered a time out error. 

Live migration failed.: libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainGetobStats)

On the destination node cc3, the libvirtd log showed a similar timeout message.

cc3 libvirtd[39220]: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)

The pattern was now clear: both the source and destination were unable to acquire state change locks during their respective migration operations. The libvirt layer was functioning correctly, but something at the QEMU level was holding these locks and preventing the migration from proceeding. 

Step 3: Investigating QEMU process locks and state

To understand what was blocking the state change operations, we examine the QEMU process on the source node cc2 where the VM was originally running.

# ps awx | grep qemu
635175 ? Sl  51:20 /usr/libexec/qemu-kvm -name guest=instance-00000007,debug-threads=on -S

The corresponding errored out instance VM was shown as paused:

# virsh list
Id Name State
----------------------------------
2 instance-00000007 paused

A paused state was not an expected state, so we connect to the QEMU monitor to find out more. Connecting to the QEMU monitor would reveal the issue in more detail through the qemi-monitor-command 

# virsh qemu-monitor-command --hmp 2 info status

Interestingly enough, the command hung, and it showed timeout operations in the logs. The VNC console was also unresponsive, matching the "time out" messages observed in the logs. Older reports online have suggested that this was a case of thread race conditions among QEMU threads. We can confirm this by cross-checking the gbd and gstack dumps. Fortunately, because libvirt and virsh manage instance configurations in XML, we can safely destroy and restart the VM process without losing configuration data. 

Step 4: Resetting and recovering a stuck QEMU instance

We will first forcibly shut down the instance using the virsh destroy command, entering the paused instance ID of 2.

# virsh destroy 2
Domain '2' destroyed

We can confirm the instance status by listing all instance statuses with virsh list -all: 

# virsh list --all
Id Name State
------------------------------------
- instance-00000007 shut off

The errored-out instance has transitioned from the paused state to a shut off state. We can quickly verify that the stuck QEMU process was terminated: 

# ps awx | grep qemu
3633296 pts/2 S+ 0:00 grep --color=auto qemu

And indeed, the old QEMU process was terminated. With the instance shut off and the QEMU process terminated, we can restart the instance. 

# virsh start instance-00000007
Domain 'instance-00000007' started

Verification:

# virsh list
Id Name State
-----------------------------------
4 instance-00000007 running

Accessing the console confirmed normal operation:

# virsh console 4
Connected to domain 'instance-00000007'
Escape character is ^] (Ctrl + ])

Login and uptime confirmed a healthy state:

login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros-hotstorage-1a0e login: cirros
cirros
Password:
$ hostname
hostname
cirros-hotstorage-1a0e
$ uptime
uptime
 19:25:08 up 12:03,  1 users,  load average: 0.00, 0.00, 0.00
$
 

Step 5: Synchronizing instance state with OpenStack Nova

Finally, we needed to inform OpenStack that the instance was back to a healthy state:

# openstack server set --state active 198e7b81-8267-44dc-a061-0cf63a970e57

Image 6: The previously errored virtual machine is back in a running state in OpenStack Horizon.

Once done, the instance appeared healthy in both CLI and Horizon UI. To complete the process, we reissued the evacuation command to move it to cc3:

We can verify the state of the instance through our web console and confirm that the VM migrated successfully this time, and the node cc2 was finally empty and ready for maintenance. 

Image 7: Verifying instance health through the use of the web-based VNC console

Preventing and resolving live migration failures

In summary, Nova host evacuation live in OpenStack is a critical capability that enables administrators to migrate running virtual machines between compute nodes with minimal downtime. It ensures business continuity, simplifies infrastructure maintenance, and prevents service interruptions during hardware or software updates.

By understanding how to troubleshoot migration failures — from analyzing libvirt and QEMU logs to resetting and restarting instances — operators can maintain high availability and operational stability.

Whether you’re maintaining a production cluster or preparing for scheduled maintenance, mastering live evacuation workflows is essential for a resilient OpenStack environment.