This is possible! At least, I’ve seen it described in other people’s setups. Having the integrated GPU should help in this case.
If I remember correctly, you need to make sure the vm shuts down properly so that it shuts down the graphics card properly. Then you can unload the vfio-pci module and load the correct module for your card.