Save-Restore feature for bhyve x86_64

Save/Restore feature for a running Virtual Machine on bhyve is a very complex operation due to the fact that you must ensure consistency by correlating the state of the guest with the internal data about the guest that is hold in the hypervisor. In order to create a minimal feature to save the state of a virtual machine which is running on ramdisk you must do the following:

• save the virtual machine memory on disk (dump all its memory on disk)
• save the virtual machine internal state from the hypervisor (internal state from userspace and from kernelspace)
• save the state of the devices

In order to restore the state of a virtual machine:

• create an empty virtual machine
• restore its memory
• restore the internal structures from files

In the above presented steps, one of the most tedious operation was saving the internal state of the hypervisor in a file. There were a lot of structures having pointers to each other that needed to be reinitialized and this induced a lot of bugs. Another issue about saving the state of the machine was to completely freeze it to be sure that one gets a consistent state. To increase the user experience while saving the whole memory (imagine you have a 32 GB of RAM VM) we used the copy-on-write mechanism: we freeze the VM, mark all the pages copy-on-write and we unfreeze it. If the VM would write something, a copy of the page will be created, leaving in place the initial state.

This is currently a work-in-progress project. We are at the step of restoring the state of the virtual machine, having trouble with the internal hypervisor registers, specific to a virtual machine.

The project was also sponsored by Matthew Grooms (he paid two students to work with me at this project).