I've been thinking about how to add support for libvirt VMs to propellor. TTBOMK setting up the VMs is a matter of creating some files in /etc, so that part is straightforward; might not want very much abstraction in propellor at all. The interesting part is creating the corresponding disk images.
I first thought that I could just extend propellor's existing support
for generating disk images by debootstrapping in a chroot and then
generating an image based on that chroot. It would just be a matter
of using .qcow2 images rather than .img. But the problem with
this is that once the VM is in use, propellor should not just be
overwriting the .qcow2 file. So something different is needed.
What I have in mind is a conditional property that works something like this:
ifM ( doesFileExist "/path/to/image.qcow2"
, debootstrapTheChrootAndPackIntoQcow2File theHost
, conducts [theHost] `requires` KVM.booted theHost
)
where theHost :: Host and either the user's libvirt config or some
property somewhere ensures it can be SSHed to from localhost.
Does this seem like the right approach?
--spwhitton
That seems like a good plan to me, and nice use of the Conductor module.
Of course,
conductsis a Property, not an IO action and presumably so isdebootstrapTheChrootAndPackIntoQcow2File, so to check if the disk image exists, you'll instead want to use thecheckcombinator. Something like:Perhaps the redundancy in that can be reduced with a new combinator that chooses which action to run.
You may want to also delete the chroot once the disk image is built.
There could also be a minor gotcha with the Conductor module trying to conduct the VM before it's gotten set up yet, at worst this would make propellor display a warning.
Let me know if you need help with this, although I will next be available on July 30th.
Maybe this is obvious, but it's cheap to generate a qcow image "backed" by an existing raw image
To quote from a script I have lying around to let a non-root user boot a root owned, read-only image
@david, but you'd not then want to change the backing raw image, I assume, or does qcow somehow deal with that?
I do think that the conditional property would be a good way for this to work.
I think there could also be VMs where you don't want the overhead of running propellor inside the VM (especially if the emulation is slow, or you don't want to allocate that much memory to the VM, or just have a lot of VMs), and the content is disposable. Then propellor could restart the VM when it changes the disk image.
There's room for multiple ways to do it..
The disk image building side of this looks easy to me, so if you do the libvirt stuff, Sean, I might contribute something.
I've now an adhoc, proof-of-concept libvirt VM provisioned by my config.hs, just using a raw disk image. It turns out that propellor should not be writing the XML configuration file in /etc, but having libvirt generate it. This is because the config is not meant to be directly edited. So, propellor should call virt-install(1) to setup and boot the VM.
My code uses virt-install's --import option, also passing it the location of the disk image generated by propellor. The main problem is that the invocation of virt-install won't return until after the VM first shuts down; the idea is that you are running the OS installer and then you reboot. Possibly using --boot instead of --import will help here; not sure.
We will need the user to specify how much RAM and how many vCPUs to assign to the VM. All the other parameters to virt-install can be determined by looking at the properties of the VM
Host.Otherwise, the user will need to set an IP property on the VM so that it can be conducted. I think we will need to leave the user to do this, as there are so many possible network configurations for libvirt VMs. But we could probably provide helper properties. In particular, the standard setup will be to use
Network.static, though I'm not sure about how to do that with indeterministic interface names.Having slept on it, and also looked at some more of virt-install(1), I have a new design. I'd be grateful for feedback on this, before implementation.
We have two properties:
Libvirt.kvmRunningandLibvirt.kvmRunningConducted, whereThe basic reason for separating these two properties is that for the conducting to work, various network things have to be set up, and there isn't a configuration that it would be sensible to have as a default. More generally, libvirt isn't at all suited to declarative configuration. What propellor can do well is build an image and have libvirt generate a barebones XML configuration file to boot the image. This is what
Libvirt.kvmRunningwill do. After that, we're going to have to leave it up to the user. I suspect that outside of the very simplest cases, they are going to have to make a series of virsh(1) calls, usingflagFileto ensure that it only happens once.Libvirt.kvmRunningwill work like this:virt-install -n vm-hostname --description "vm-hostname VM" --os-type=Linux --os-variant=debian9 --ram=1024 --vcpus=2 --autostart --disk path=/var/lib/libvirt/images/vm-hostname.img,device=disk,bus=virtio --import --print-xml >/tmp/foovirsh define /tmp/foovirt-install --print-xmland thenvirsh defineavoids the problem of virt-install not exiting until after the VM has been shutdown at least once/the virt-viewer instance launched by virt-install has been closedvirsh start vm-hostnameSample usage:
Seems that kvmRunning would need a warning that changes to the propellor configuration of the VM's Host won't affect the VM. Because it's unusual for propellor to only be able to set something up and not change it afterwards.
But kvmRunning is certianly a useful low level property, and combining with other properties like that is good.
Hmm, it's actually possible to mount a qcow2 image using libguestfs-tools. http://ask.xmodulo.com/mount-qcow2-disk-image-linux.html So, propellor could temporarily take down the VM and run inside the qcow2 to update it! Although doing that every time propellor is run seems suboptimal. It could keep the chroot around and only update the qcow2 image if the chroot needed to be updated. I am not sure how I feel about that idea.
We could also make conducting easier to set up, perhaps not needing
mainto be modified to use it.Thanks. I've now implemented enough of this to support my usecase in the libvirt branch of https://git.spwhitton.name/propellor.
There are TODOs, but now seems as good a time as any for someone to implement the qcow2 support . . .
With regard to your idea, taking down the VM and running propellor on the qcow2 file seems like it will not be useful to very many people. ISTM that treating the VM as a host and spinning it is right.
Let's comment out the QCow2 constructor until that case is handled.
With NumVCPUs and MiBMemory both Int, and used in the same property, they could get mixed up. Recommend newtypes.
Would it make sense for defaultNetworkAutostarted to itself run the virsh net-start? It would simplify the example.
It's named kvmDefined; is it actually guaranteed to use kvm and not some other VM?
What happens when osVariant is Nothing and no --os-variant is passed? When osType is Nothing? Is it still likely to work?
Please make osType not have a default case and define it for all the current constructors. That way, the next person to add an Linux distro to propellor won't forget to update it.
The chroot nuking code depends on some implementation details in DiskImage, so I'd be inclined to move it to that module. (Which probably has similar code that can be factored out.)
The "defined" scriptProperty uses a lot of values that are unlikely to contain spaces or other script unsafe stuff, but it would still be good to shellEscape them. (Or it could be rewritten to run virt-install w/o a shell, read its output, and write the file.)
The "started" scriptProperty also needs some escaping. (Although I'd be inclined to parse virsh list in haskell, but up to you.)
Minor: The
& Libvirt.defaultNetworkAutostartedexample line is currently indented by spaces while the other lines use tabs.Minor: I'd use
$in some of the places where you have a parenthesized do block or other multiline block of code.--autostartdoes not actually cause the VM to be marked for autostarting. This is a bug in virt-install, though, as we're using the command line interface as documented -- I confirmed this usingPROPELLOR_DEBUG=1.)I've merged the branch, unsure if this should remain open for anything discussed above, so leaving it for now.