Mantle test cl.cloudinit.basic sometimes fails on Azure

[ader1990]

Description

[2024-07-30T07:46:44.497Z] --- FAIL: cl.cloudinit.basic (236.01s)
[2024-07-30T07:46:44.497Z]         cluster.go:125: cat: /foo: No such file or directory
[2024-07-30T07:46:44.497Z]         cluster.go:145: "cat /foo" failed: output , status Process exited with status 1

As this issue appears only sometimes, it might be the case that there is a concurrency / transient issue type, stemming from either Ignition or Wagent.

Passing test journalctl logs: https://bincache.flatcar-linux.net/testing/4046.0.0+nightly-20240729-2100/amd64/azure/_kola_temp/azure-2024-07-30-0555-15/cl.cloudinit.basic/ci-4046.0.0-n-f9594f212a/journal.txt
Failing test journalctl logs: https://bincache.flatcar-linux.net/testing/4046.0.0+nightly-20240729-2100/amd64/azure/_kola_temp/azure-2024-07-30-0537-54/cl.cloudinit.basic/ci-4046.0.0-n-8bf0e4f56d/journal.txt

The difference between a passing test and a failing test seems to be how Ignition handles Azure metadata:

PASSING - code branch https://github.com/coreos/ignition/blob/main/internal/providers/azure/azure.go#L106:

Jul 30 05:59:31.922072 ignition[905]: GET http://169.254.169.254/metadata/instance/compute/userData?api-version=2021-01-01&format=text: attempt #1
Jul 30 05:59:32.035327 ignition[905]: GET result: OK
Jul 30 05:59:32.035358 ignition[905]: failed to retrieve userdata from IMDS, falling back to custom data: not a config (empty)

FAILING - code branch https://github.com/coreos/ignition/blob/main/internal/providers/azure/azure.go#L98:

Jul 30 05:47:57.158585 ignition[909]: GET http://169.254.169.254/metadata/instance/compute/userData?api-version=2021-01-01&format=text: attempt #1
Jul 30 05:47:57.315914 ignition[909]: GET result: OK
Jul 30 05:47:57.316082 ignition[909]: config has been read from IMDS userdata

Impact

The issue might hide something more serious and it should be investigated and fixed upstream.

Environment and steps to reproduce

sudo ../../kola run -p azure cl.cloudinit.basic

Expected behavior

cl.cloudinit.basic should not fail

[ader1990]

I could always reproduce the issue by forcing the UseUserData field on True. See: https://github.com/flatcar/mantle/blob/flatcar-master/kola/harness.go#L448.

diff --git a/kola/harness.go b/kola/harness.go
index cb59f5f5..fd9bfde4 100644
--- a/kola/harness.go
+++ b/kola/harness.go
@@ -437,17 +437,6 @@ func RunTests(patterns []string, channel, offering, pltfrm, outputDir string, ss

                // If the version is > 3033, we can safely use user-data instead of custom-data for
                // provisioning the instance on Azure.
-               if !version.LessThan(semver.Version{Major: 3034}) && pltfrm == "azure" {
-                       // Using reflection is a bit hacky, but it seems to be the only way to
-                       // access the field we want to set.
-                       f := reflect.ValueOf(flight).Elem()
-                       api := f.FieldByName("Api")
-                       opts := api.Elem().FieldByName("Opts")
-                       userData := opts.Elem().FieldByName("UseUserData")
-                       // At this point, this field can be set.
-                       userData.SetBool(true)
-               }
-
                versionStr = version.String()

                // one more filter pass now that we know real version
@@ -456,6 +445,12 @@ func RunTests(patterns []string, channel, offering, pltfrm, outputDir string, ss
                        plog.Fatal(err)
                }
        }
+       f := reflect.ValueOf(flight).Elem()
+       api := f.FieldByName("Api")
+       optsField := api.Elem().FieldByName("Opts")
+       userData := optsField.Elem().FieldByName("UseUserData")
+       // At this point, this field can be set.
+       userData.SetBool(true)

        opts := harness.Options{
                OutputDir: outputDir,

./bin/kola run --platform=azure --azure-sku alpha --azure-version 4012.0.1 --azure-resource-group flatcar-bug-1505 cl.cloudinit.basic

--------------------------------------------------------------------------------
--- FAIL: cl.cloudinit.basic (198.72s)
        cluster.go:125: cat: /foo: No such file or directory
        cluster.go:145: "cat /foo" failed: output , status Process exited with status 1
FAIL, output in _kola_temp/azure-2024-07-30-1530-1277544
harness: test suite failed