ignition remote configuration not downloaded from http(s):// in Hyper-V

[outofsight]

Describe the bug

I'm trying to provision a CoreOS Hyper-V Virtual Machine with a remote ignition file served by http or https.
Apparently ignition file is not downloaded from web server during boot.
CoreOS is not provisioned: no users, no ssh keys, not even when merging a remote configuration with an inline one.
The same ignition configuration inserted directly in VM works fine.

Reproduction steps

1. Setup Hyper-V VM with fedora-coreos-41.20241215.3.0-hyperv.x86_64.vhdx
2. Add ignition file referring a remote http(s) ignition via kvpctl
3. Start VM

Expected behavior

Ignition should be fetched from web server, CoreOS VM should be provisioned.

Actual behavior

CoreOS VM get network configuration from DHCP.
Web Server log doens't show any request for referenced ignition file.
Not even the ssh keys are installed on CoreOS.

Starting VM in single mode and checking journal:

System details

• Hyper-V Server Core 2019
• fedora-coreos-41.20241215.3.0-hyperv.x86_64.vhdx
• kvpctl v0.9.0

Butane or Ignition config

variant: fcos
version: 1.6.0
ignition:
  config:
    replace:
      source: http://provisioning.mydomain.tld/coreos/default.ign

Additional information

• [["kvpctl.exe get" returns "no kvp found" even when using a non-remote ignition config that works; apparently works only when vm is running]]
• tried also to "merge" a remote ignition with an inline ignition for users and ssh keys, same effect
• about integration services hyper-v reports: "The component protocol version installed in the virtual machine does not match the one expected by the host system"
• provisioning coreOS by a non-remote ignition, logging in, downloading the remote ign with http(s) is successful

[Nemric]

I'm not running FCOS on HyperV but on baremetal and I did the upgrade today from PXE with https provisioning and everything did work as expected
From your screenshot, I can't see any try to fetch some "http://provisioning.mydomain.tld/coreos/default.ign" link
Are you sure you didn't make any mistakes ?

[outofsight]

Exactly, I' can't see any attempt to download the ign file from my web server.
The ignition passed to VM is so basic that I can't see what could be wrong.
Booting the VM and provisioning via PXE could be another thing to try...

[jlebon]

"provider config was empty" means that something went wrong at the Hyper-V level. Can you show the output of kvpctl.exe ... add-ign ...?

/cc @baude who has much more knowledge in KVP internals

[baude]

1. are you running as admin
2. can you tell if the networking in the VM set itself up correctly?

Ill add more after booting into windows momentarily.

[outofsight]

"provider config was empty" means that something went wrong at the Hyper-V level. Can you show the output of kvpctl.exe ... add-ign ...?

/cc @baude who has much more knowledge in KVP internals

the output is regular:
added key: ignition.config.0

I also checked via PowerShell/CIM/WMI, after booting the VM, and ign is installed in KVP

[outofsight]

1. are you running as admin
2. can you tell if the networking in the VM set itself up correctly?

Ill add more after booting into windows momentarily.

If I take a VM checkpoint, with VM configured ad desired but with no ign installed in KVP:

1. I install in KVP an ign without any reference to a remote ign, I start the VM: everything is fine, CoreOS is provisioned as expected, network is working, I can curl the http(s) url of a remote ign;

2. Then I restore checkpoint, I install in KVP the ign above with a reference the remote ign, I start the VM: the ign is not downloaded from the web server, CoreOS is not provisioned.

[baude]

can you sanitize an ign file and attach? I'll attach one from podman-machine which uses a derivative of the image you cite.
example.json

[outofsight]

can you sanitize an ign file and attach? I'll attach one from podman-machine which uses a derivative of the image you cite. example.json

Not sure to understand.

The ign i install in the KVP is the one derived from the butane above:
{"ignition":{"config":{"merge":[{"source":"http://provisioning.portal.mydomain.tld/coreos/default.ign"}]},"version":"3.5.0"}}

The ign served via http(s) at http://provisioning.portal.mydomain.tld/coreos/default.ign is relevant? It's not even downloaded. Access log of web server (caddy) doesn't even show any attempt and any error.

[baude]

The ign served via http(s) at http://provisioning.portal.mydomain.tld/coreos/default.ign is relevant? It's not even downloaded. Access log of web server (caddy) doesn't even show any attempt and any error.

That was my guess. With podman-machine, we have to do some network stuff to make things work in usermode iirc. Note the ign file I provided which has:

      {
        "contents": "[Unit]\nDescription=vsock_network\nAfter=NetworkManager.service\n\n[Service]\nExecStart=/usr/libexec/podman/gvforwarder -preexisting -iface vsock0 -url vsock://2:58264/connect\nExecStartPost=/usr/bin/nmcli c up vsock0\n\n[Install]\nWantedBy=multi-user.target\n",
        "enabled": true,
        "name": "vsock-network.service"
      }

That's why I asked if you could tell if the networking was working.

[outofsight]

About networking the only things I know:

• when I install in KVP my default.ign (no reference to a remote configuration) and the machine is provisioned and I can login on console and via ssh, networking is fine, and I can curl the remote configuration
• when I install in KVP the above "bootstrap.ign" (with reference to a remote configuration), after starting the VM, the VM at some point begin to reply to ping, but the machine is not provisioned; if I intercept GRUP, add single to kernel parameters, and I login, I can see the machine got IP from dhcp elsewhere, so I think networking is fine at this time; I know nothing of the situation before, at the time of ignition fetch;
• DNS log doens't show any query for my provisioning.portal.mydomain.tld from the VM

[outofsight]

I'm not running FCOS on HyperV but on baremetal and I did the upgrade today from PXE with https provisioning and everything did work as expected

Booting the Hyper-V VM with iPXE and provisioning by an https-serverd ign works.
iPXE can also provide as query parameters machine details like mac address and uuid so that the web server can better select the ign served
Just a bit slower, need to download the image from local web server and two reboot.

[outofsight]

I also tried to inject the following ign into the Hyper-V VM to be provisioned.

{"ignition":{"config":{"merge":[{"source":"http://provisioning.portal.mydomain.tld/coreos/.ign"}]},"version":"3.5.0"},"kernelArguments":{"shouldExist":["coreos.autologin"]},"passwd":{"users":[{"name":"core","passwordHash":"$y$j9T$PnBzn7v5/7qn31QCrdAtw/$nZjUiH3BHyLnIuo.Npx4D7/zJ2Dob/fnyOCpDO4xzp3"}]}}

This was an attempt to set a password to login or setup autologin to better diagnose the issue.
Not only the remote config is never downloaded, but also the user passowrd directly injected is never setup.
Basically it seems to me that in presence of an ignition/config/{merge,replace}/source, the ijected ignition is completely ignored.

There is a way to check, after unsuccessful prvisioning, what happened? To check if the network was online at the suitable moment and so on...?

[dustymabe]

when I install in KVP the above "bootstrap.ign" (with reference to a remote configuration), after starting the VM, the VM at some point begin to reply to ping, but the machine is not provisioned; if I intercept GRUP, add single to kernel parameters, and I login, I can see the machine got IP from dhcp elsewhere, so I think networking is fine at this time; I know nothing of the situation before, at the time of ignition fetch;

It would be nice if we could get some logs from boot of the machine when this happens. @baude or @outofsight, do you know if we can get serial console on Hyper-V? If so a full serial console log of the boot would be great! screenshots, not so great.

[outofsight]

when I install in KVP the above "bootstrap.ign" (with reference to a remote configuration), after starting the VM, the VM at some point begin to reply to ping, but the machine is not provisioned; if I intercept GRUP, add single to kernel parameters, and I login, I can see the machine got IP from dhcp elsewhere, so I think networking is fine at this time; I know nothing of the situation before, at the time of ignition fetch;

do you know if we can get serial console on Hyper-V?

If serial means something related to COM ports, I don't think it's possible in Hyper-V Generation 2 VM because virtual hardware doesn't include COM ports. Other way to save and transfer this log would be nice.

[baude]

there is only gui im afraid.

[dustymabe]

maybe a screen capture of the VGA console during boot then? at least then we can pause at different points and should be able to see all output

[outofsight]

maybe a screen capture of the VGA console during boot then? at least then we can pause at different points and should be able to see all output

here

[dustymabe]

In the case of a remote/replace config I would expect to see something like:

Jan 14 19:02:11 localhost.localdomain ignition[862]: fetched referenced config at https://dustymabe.fedorapeople.org/config-no-password.ign with SHA512: aa221903e04c43dfddd3a0f492c21745a381c05e1f814d8aebecffda5644531d4b2319680be694f39af7e751b5200a403e70b11d5bdaabea68cf8496bd8f7471

but I'm not seeing that.

[dustymabe]

coreos.autologin

doesn't work in Fedora CoreOS - see #112

I wonder if you can add a merge config without the kernelArgument part (that complicates things since it will cause a reboot to happen and thus Ignition runs twice) that just sets a password and an ssh key but then references a remote config to merge.

That should let you still get into the system and get a proper journal log.

Maybe without the kernelArgument part you'll have success with the merge config.

[outofsight]

• first attempts were made with a very basic ignition file, just a remote config replace, didn't work, so I tried something else
• even with a passwd section in the ignition embedded in VM, and a "config/merge" with a remote config, password are not applied, I can't login with the password specified
• my dns servers are never queried for domain name of the remote config, no way my remote config can be downloaded

see here

from the recorded screen we can confirm the ignition has been embedded and can be read from Hyper-V KVP store?
after unsuccessful provisioning, I think I can login with single: there is a way to better check what's happened?

[dustymabe]

• first attempt was with a very basic ignition, only with a remote config replace, didn't work

right, but that was with a replace and not a merge with ssh and/or password..

can you try with a merge config with ssh key (just an ssh key) and see if at least the key gets set?

variant: fcos
version: 1.6.0
ignition:
  config:
    merge:
      - source: http://provisioning.mydomain.tld/coreos/default.ign
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - <your_ssh_pubkey>

[outofsight]

right, but that was with a replace and not a merge with ssh and/or password..

I tried also this, with a password, with a ssh-key, with both, and I tried again now
No, no password, no ssh keys are stup

[outofsight]

Any other idea to try?

Not sure if could explain but I noticed that even in a coreos successfully provisioned with an ignition fully embedded in KVP since beginning, hv_vss_daemon and hypervvssd are not running by default, and hyperv-daemons and hyperv-tools are not installed by default.