Issues related to CI reproducibility test

[minghangli-uni]

The current CI repro test relies on results from https://github.com/ACCESS-NRI/model-config-tests/blob/main/tests/resources/access-om3/output000/ocean.stats and checksum values from https://github.com/ACCESS-NRI/model-config-tests/tree/main/tests/resources/access-om3/checksums.

But the checksum values will always remain the same as long as the ocean grid doesn’t change. This is because the test reads everything from the initial conditions, and the outputs are deterministic based on those conditions, so the test is essentially invalid or will “always pass” even when model behavior has changed.

  Step,       Day,  Truncs,      Energy/Mass,      Maximum CFL,  Mean Sea Level,  Total Mass,  Mean Salin, Mean Temp, Frac Mass Err,   Salin Err,    Temp Err

            [days]                 [m2 s-2]           [Nondim]       [m]             [kg]         [PSU]      [degC]       [Nondim]        [PSU]        [degC]

     0,  693135.000,     0, En 3.0745627134675957E-23, CFL  0.00000, SL  1.5112E-10, M 1.36404E+21, S 34.7263, T  3.6362, Me  0.00E+00, Se  0.00E+00, Te  0.00E+00

I suggest a valid check should include metrics from at least Day 2 to capture any divergence.

[dougiesquire]

Yikes, good catch @minghangli-uni. I'm confused about why we're not getting the end-of-run stats in ocean.stats. Taking a look now

[anton-seaice]

I thought I had checked at some point it was possible to fail the repro test, but this makes a lot of sense !

How often are ocean.stats calculated ? i.e. for the repro test, can we turn it onto calculate every timestep ?

[minghangli-uni]

Yikes, good catch @minghangli-uni. I'm confused about why we're not getting the end-of-run stats in ocean.stats. Taking a look now

The repro test only takes 3 hours, so it won’t produce any entries in the ocean.stats file.

[dougiesquire]

I thought (assumed?) ocean.stats would always be written at the end of a run. It looks like we may need to set ENERGYSAVEDAYS to get more frequent stats. Testing this now

[anton-seaice]

We can add a check into the model-config-tests that step is non 0 . And we may as well track the step number as an extra field in the checksum file

[minghangli-uni]

It looks like we may need to set ENERGYSAVEDAYS to get more frequent stats.

Yeah, I haven’t tested it, but setting ENERGYSAVEDAYS=0.125 should work at least for the existing repro check.

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/cosima-twg-meeting-minutes-2025/4067/4