Fixing a restart file issue with Isca

[sit23]

Fix moist-physics restart inconsistency: restore pbltop and gust

Problem

Running Isca's moist idealized physics (e.g. the Frierson aquaplanet test case) for N days in one continuous run gives different results than running the same N days split across multiple restarted legs (e.g. one day at a time). This does not happen for the dry Held-Suarez configuration. This problem was discussed recently with Spencer Hill.

Root cause

In src/atmos_spectral/driver/solo/idealized_moist_phys.F90, two module-level arrays implement a deliberate one-timestep-lagged coupling between the boundary-layer turbulence scheme and other physics, active whenever turb=.true. (the default for realistic moist setups like Frierson):

• pbltop: vert_turb_driver diagnoses the PBL-top height each timestep; that value is fed into damping_driver's sponge-layer calculation on the following timestep.
• gust: vert_turb_driver diagnoses surface gustiness each timestep; that value is fed into surface_flux on the following timestep.

Neither variable was ever written to, or read from, a restart file — this driver performed no restart I/O of its own. gust was given a fixed cold-start default (1.0); pbltop was left entirely uninitialized (allocate() with no initializer — whatever bytes happened to be in that memory).

Consequently, every restarted leg silently discarded the evolved value of these two variables and substituted the cold-start default for exactly one timestep, injecting a real (not roundoff-level) perturbation right at the restart boundary. That perturbation then propagated through the fully coupled nonlinear dynamics and grew over subsequent timesteps.

This is specific to physics configurations that exercise vert_turb_driver/vert_diff (turb=.true.), which is why Held-Suarez — which never calls this code path — was unaffected.

Fix

Added a proper restart round-trip for both variables, following the same lightweight read_data/write_data pattern already used elsewhere in this file's sibling driver (mixed_layer.F90):

• pbltop is now explicitly initialized to 0.0 at allocation (removing the previous reliance on implicit/uninitialized memory content).
• Both variables are read from INPUT/idealized_moist_phys.res.nc at init, if present.
• Both variables are written to RESTART/idealized_moist_phys.res at shutdown.

One file changed: src/atmos_spectral/driver/solo/idealized_moist_phys.F90 (~20 lines).

Verification

Reproduction of the bug, and confirmation of the fix

Built a low-resolution Isca and ran each of the following two ways — once as a single continuous run, once split by a restart — comparing the resulting internal model state (restart files) bit-for-bit:

Configuration	Before fix	After fix
Held-Suarez (dry)	bit-identical	bit-identical
Frierson aquaplanet (moist, default do_diffusivity boundary layer)	diverges (already present after a single post-restart timestep)	bit-identical

Bisection (selectively disabling physics components) confirmed the divergence tracked turb specifically: disabling turb alone made the moist case bit-identical even before this fix; disabling convection or radiation alone did not.

Regression check against master (trip_test)

Ran Isca's trip_test, comparing this branch against master (d1321ac3), for a normal (non-restarted) 3-day run at full production resolution (T42, 8 cores):

held_suarez  : pass
frierson     : pass
bucket_model : pass

All three test cases give bit-identical results to master, confirming this change does not alter any existing single-run behaviour — it only changes behaviour when a restart file is actually present. (This also empirically confirms that pbltop's previously-uninitialized memory happened to read as 0.0 on the platform tested; the fix simply makes that explicit rather than implementation-defined.)

Note for reviewers

While tracing this, I also noticed rough — a separate roughness-length array also passed into vert_turb_driver — is allocated but never assigned anywhere in this file, so it's used with whatever's left in memory. This appears unrelated to the restart bug (the same value would be used consistently throughout any given run, restarted or not) but may be a separate, pre-existing issue worth a follow-up look. Not addressed in this PR.