feat(fwcfg): add new DataSource for QEMU fw_cfg device blobs

[hw-claudio]

allow sourcing the data from sysfs as exposed by the linux guest fw_cfg driver in /sys/firmware/qemu_fw_cfg/

• add new cloudinit/sources/DataSourceQemuFwCfg.py
• add unit test test_qemufwcfg.py and list it in test_common.py
• add "QemuFwCfg" to the datasource_list in cloudinit/settings.py
• add "QemuFwCfg" to ds-identify
• add documentation in doc/rtd/reference/datasources/qemufwcfg.rst
• add reference to doc file in doc/rtd/reference/datasources.rst
• add integration test test_qemufwcfg.py

Fixes: GH 3684
Fixes: GH 3133

Proposed Commit Message

feat(fwcfg): add new DataSource for QEMU fw_cfg device blobs

allow sourcing the data from sysfs as exposed by the linux guest fw_cfg driver in /sys/firmware/qemu_fw_cfg/

- add new cloudinit/sources/DataSourceQemuFwCfg.py
- add unit test test_qemufwcfg.py and list it in test_common.py
- add "QemuFwCfg" to the datasource_list in cloudinit/settings.py
- add "QemuFwCfg" to ds-identify
- add documentation in doc/rtd/reference/datasources/qemufwcfg.rst
- add reference to doc file in doc/rtd/reference/datasources.rst
- add integration test test_qemufwcfg.py

Fixes: GH 3684
Fixes: GH 3133

Additional Context

Test Steps

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[holmanb]

@hw-claudio Cloud-init already supports QEMU in several different ways. Can you please tell me why you think that the broader cloud-init community needs this feature?

[hw-claudio]

Hello @holmanb ,
in terms of the requirement from cloud-init itself, I have received this response from @blackboxsw

#3684 (comment)

in terms of the general usefulness of the feature from my perspective, it provides a data source that does not require changing the disk configuration of the VM (.iso), does not require any network capability and any external infrastructure.

It is the most straightforward way to pass the parameters if qemu is used as a hypervisor as can be seen by the limited amount of code needed, since the linux guest driver already prepares the blobs in sysfs to be used.

I am building a very lightweight "peer to peer" virtualization platform, so removing complexity is the advantage for the specific project I am working on.

Once this is added I would propose this to cloudbase-init as well, so that in combination with the virtio fw_cfg drivers this can be used on Windows as well.

[holmanb]

This PR doesn't pass mypy typing, has no integration testing, has no documentation, has no commitment of support from anyone, and there is exactly one prospective project that might use it. This new feature re-implements an existing feature - the value proposition is a simplified deployment story.

Your hope is that upstream will document it, test it, fix bugs, upgrade it across Python versions and Qemu changes, provide releases for it, and maintain it forever.

Is that right?

[hw-claudio]

I saw this feature as already requested, and the feedback from this community was, "yes, we need this, this just needs implementation". Maybe you can explain this apparent disconnect.

If you are actually willing to collaborate, on your points:

1 - mypy typing: I can look into it and fix it.
2 - missing tests: I found unit tests and extended those. Did not see the integration tests, can work on it.
2 - documentation: I can study what is in doc/ and add info about this data source
3 - On the why: the current project I am working on and that I tested with can be irrelevant: this is not my feature request. It has been in the cloud-init project feature requests since 2020.

https://bugs.launchpad.net/bugs/1879294

4 - On the prospective use: there is very clear precedent for this architecture because it's better than what cloud-init does now, ie changing the VM to add ISOs, passing kernel command line strings and misusing SMBIOS serial numbers.

Ignition uses this very same mechanism for the same high level goals:
https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-qemu/

This is being used in production at scale in Fedora CoreOS and RHCOS.
Proxmox started looking at this, you can see discussions about it in the mailing list since 2017.

The linux sysfs driver is now widely available, which means that the required code to maintain in cloud-init is minimal as you can see from the actual code in this PR. I can help with the maintenance of it if necessary.

If however my efforts here are not needed, I understand, please let me know so I can look at alternatives.

Thank you.

[hw-claudio]

I added documentation and an integration test, took some time to get the integration test into the green.
PR Lint and PR Integration are both fine for what I could see.

Let me know if there are more technical issues I should fix. In terms of adoption I cannot do much more beyond what I have mentioned, including the willingness to fixup issues should they arise in the future.