Production infrastructure cannot afford unplanned downtime, but with OpenStack, it doesn't have to. OpenStack's live migration feature moves running virtual machine instances from one compute node to another within the same cluster. This keeps services available while the underlying hardware or software is maintained. The Nova compute service orchestrates this process and extends it further through host evacuation, which is a mechanism designed to recover workloads from compute nodes that have become unreachable or failed entirely.
These two mechanisms work together to give operations teams the flexibility to respond to both planned maintenance windows and unplanned node failures without taking workloads offline.
OpenStack Nova provides several commands that handle different migration or evacuation scenarios:
As the title indicates, we are mostly interested in host-evacuate-live, which migrates all running instances to other computing nodes. The host-evacuate-live is a powerful and useful one, since it performs host-wide migrations of running workloads. Performing a host-evacuate-live is also the most demanding; other similar actions would undoubtedly work.
Let’s examine an example of migration failure and how to resolve it.
Our test environment consisted of three control-converged nodes: cc1, cc2, and cc3. Control-converged nodes all provide compute, storage, and networking services to the cluster. All VM instances (virtual machines) were active and running as our initial cluster state on the cc2 host.
Image 1: All instances (virtual machines) hosted on cc2 in the running state
To begin maintenance on node cc2, we attempted to migrate all running instances from cc2 to cc3 using Nova live evacuation. We migrate all instances (virtual machines) from cc2 with:
# nova host-evacuate-live cc2The instance status transitioned from 'ACTIVE' to 'MIGRATING' as the migration operation took place.
After several seconds, the two instances have successfully migrated to cc3, and their status has transitioned from 'MIGRATING' to 'Active'. We can confirm the operation has concluded successfully by verifying that cc3 became the new instance host.
However, the migration operation times out for our Cirros_Host-Storage instance, resulting in an ERROR status.
Image 4: An instance encountered an 'ERROR' during Nova live migration
This failure could also be confirmed in Horizon, the OpenStack dashboard.
To identify the root cause, we begin by checking the Nova compute log on the source node cc2. We can see that the live migration operation encountered a time out error.
Live migration failed.: libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainGetobStats)On the destination node cc3, the libvirtd log showed a similar timeout message.
cc3 libvirtd[39220]: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)The pattern was now clear: both the source and destination were unable to acquire state change locks during their respective migration operations. The libvirt layer was functioning correctly, but something at the QEMU level was holding these locks and preventing the migration from proceeding.
To understand what was blocking the state change operations, we examine the QEMU process on the source node cc2 where the VM was originally running.
# ps awx | grep qemu
635175 ? Sl 51:20 /usr/libexec/qemu-kvm -name guest=instance-00000007,debug-threads=on -SThe corresponding errored out instance VM was shown as paused:
# virsh list
Id Name State
----------------------------------
2 instance-00000007 paused
A paused state was not an expected state, so we connect to the QEMU monitor to find out more. Connecting to the QEMU monitor would reveal the issue in more detail through the qemi-monitor-command
# virsh qemu-monitor-command --hmp 2 info status
Interestingly enough, the command hung, and it showed timeout operations in the logs. The VNC console was also unresponsive, matching the "time out" messages observed in the logs. Older reports online have suggested that this was a case of thread race conditions among QEMU threads. We can confirm this by cross-checking the gbd and gstack dumps. Fortunately, because libvirt and virsh manage instance configurations in XML, we can safely destroy and restart the VM process without losing configuration data.
We will first forcibly shut down the instance using the virsh destroy command, entering the paused instance ID of 2.
# virsh destroy 2
Domain '2' destroyedWe can confirm the instance status by listing all instance statuses with virsh list -all:
# virsh list --all
Id Name State
------------------------------------
- instance-00000007 shut off
The errored-out instance has transitioned from the paused state to a shut off state. We can quickly verify that the stuck QEMU process was terminated:
# ps awx | grep qemu
3633296 pts/2 S+ 0:00 grep --color=auto qemu
And indeed, the old QEMU process was terminated. With the instance shut off and the QEMU process terminated, we can restart the instance.
# virsh start instance-00000007
Domain 'instance-00000007' started
Verification:
# virsh list
Id Name State
-----------------------------------
4 instance-00000007 running
Accessing the console confirmed normal operation:
# virsh console 4
Connected to domain 'instance-00000007'
Escape character is ^] (Ctrl + ])
Login and uptime confirmed a healthy state:
login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros-hotstorage-1a0e login: cirros
cirros
Password:
$ hostname
hostname
cirros-hotstorage-1a0e
$ uptime
uptime
19:25:08 up 12:03, 1 users, load average: 0.00, 0.00, 0.00
$
Finally, we needed to inform OpenStack that the instance was back to a healthy state:
# openstack server set --state active 198e7b81-8267-44dc-a061-0cf63a970e57Once done, the instance appeared healthy in both CLI and Horizon UI. To complete the process, we reissued the evacuation command to move it to cc3:
# nova host-evacuate-live cc2We can verify the state of the instance through our web console and confirm that the VM migrated successfully this time, and the node cc2 was finally empty and ready for maintenance.
In summary, Nova host evacuation live in OpenStack is a critical capability that enables administrators to migrate running virtual machines between compute nodes with minimal downtime. It ensures business continuity, simplifies infrastructure maintenance, and prevents service interruptions during hardware or software updates.
By understanding how to troubleshoot migration failures — from analyzing libvirt and QEMU logs to resetting and restarting instances — operators can maintain high availability and operational stability.
Whether you’re maintaining a production cluster or preparing for scheduled maintenance, mastering live evacuation workflows is essential for a resilient OpenStack environment.