live migration of vTPM enabled VM with swtpm state files in shared storage.

[ifnkhan]

As per Troubleshooting migration of VM with swtpm article, 'swtpm' command line parameters should point to two different directories on the source and destination hosts. But with shared store like NFS, the source and destination host can point to the same swtpm directory. I tested live migration with swtpm directory on NFS and didn't find any issues with source and destination hosts pointing to same swtpm directory on NFS. On the destination host, during live migration the swtpm_setup won't run because the swtpm state directory is already initialized(--printstate returns valid permall). I looked at the live migration code, swtpm on destination host just writes the blobs received from QEMU to a respective file(.permall). With the introduction of atomic writes (temp file rename()/fsysnc() in SWTPM_NVRAM_StoreData()), I believe it is safe to migrate a vTPM enabled VM with swtpm state to the same directory on NFS. The only criteria is the destination host swtpm/libtpms version must be higher. Please correct me if I'm wrong.

Source Host:
Qemu:
-> start_thread ()
--> migration_thread ()
---> qemu_savevm_state_complete_precopy ()
----> vmstate_save ()
-----> vmstate_save_state ()
------> tpm_emulator_pre_save ()
-------> tpm_emulator_get_state_blobs ()      
--------> tpm_emulator_stop_tpm ()                <--- STEP 1. CMD_STOP - mark tpm_running=false in swtpm
--------> tpm_emulator_set_state_blob()           <--- STEP 2. CMD_GET_STATEBLOB
swtpm:
-> _start ()
--> __libc_start_main ()
---> swtpm_main ()
----> mainLoop ()
-----> ctrlchannel_process_fd ()
------> ctrlchannel_return_state ()
------->SWTPM_NVRAM_GetStateBlob ()
-------->SWTPM_NVRAM_LoadData ()
*** After this destination host calls tpm_emulator_post_load() i.e before tpm_emulator_shutdown() on source.
Qemu:
-> main ()                                         <--- QEMU main thread !!!
--> tpm_cleanup ()
---> object_unref ()
----> object_finalize ()
-----> object_deinit ()
------> tpm_emulator_inst_finalize ()
-------> tpm_emulator_shutdown ()                  <--- Sends CMD_SHUTDOWN to swtpm

Destination Host:
Qemu:
-> coroutine_trampoline ()
--> process_incoming_migration_co ()
---> qemu_loadvm_state ()
----> qemu_loadvm_state_main ()
-----> qemu_loadvm_section_start_full ()
------> vmstate_load ()
-------> vmstate_load_state ()
--------> tpm_emulator_post_load ()
---------> tpm_emulator_set_state_blobs ()           <--STEP 1.  command: CMD_SET_STATEBLOB
---------> tpm_emulator_startup_tpm_resume ()        <--STEP 2.  command: CMD_INIT
swtpm:
STEP 1. CMD_SET_STATEBLOB
-> _start ()
--> __libc_start_main ()
---> swtpm_main ()
----> mainLoop ()
-----> ctrlchannel_process_fd ()
------> ctrlchannel_receive_state ()
-------> SWTPM_NVRAM_SetStateBlob ()
STEP 2. CMD_INIT
-> _start ()
--> __libc_start_main ()
---> swtpm_main ()
----> mainLoop ()
-----> ctrlchannel_process_fd ()
------> tpmlib_start()

[stefanberger]

As per Troubleshooting migration of VM with swtpm article, 'swtpm' command line parameters should point to two different directories on the source and destination hosts. But with shared store like NFS, the source and destination host can point to the same swtpm directory.

This is a setup that is not supported: https://lists.nongnu.org/archive/html/qemu-trivial/2021-10/msg00034.html

When the destination overwrites the source I am not sure what will happen then. I am not even sure whether the destination will be able to take the lock on the directory, probably not and it already should fail there.

[stefanberger]

I believe it is safe to migrate a vTPM enabled VM with swtpm state to the same directory on NFS. The only criteria is the destination host swtpm/libtpms version must be higher. Please correct me if I'm wrong.

I never tested it this way and like I said above, we're taking a lock when using the directory backend and two swtpm's sharing the same directory wouldn't be able to take that lock. QEMU migrates the state of the TPM 1.2 or TPM 2 in any case, so there's no need to share the directories and for sure it should never be a requirement that shared storage be set up for migration.

The libtpms version must be on the same or lower level on the source side than on the destination side, that's a requirement for any type of migration and suspend/resume operation.

Are these swtpm related files always labeled with the same SELinux labels on source and destination? I actually don't know whether that is the case and libvirt tells the destination what the label should be like before the migration happens. They would have to be the same if shared directories were to be used. That would need to be checked. Likely it's the same, but one would have to make sure that libvirt doesn't roll the dice on the labels for the destination side, otherwise falling back to the source side would have to possibly cause a relabeling.