FAIL ERP_P64x2_D_Ld3...clm-default--clm-all_outputs COMPARE_base_rest

[slevis-lmwg]

FAIL ERP_P36x2_D_Ld3.f10_f10_mg37.I1850Clm50BgcCrop.derecho_gnu.clm-default--clm-all_outputs COMPARE_base_rest
33 fields appear different in h0a and h0i output suggesting diagnostic fields not tested for perfect restarts before. Though LEAFCN_STORAGE is not diagnostic, but maybe something's missing in its initialization. Look at these with ncdiff. Some of this seems related to #3525 because I see many "gamma" fields in the diffs.

Originally posted by @slevis-lmwg in #2061 (comment)

This is a new test introduced with #2061 and the problem is due to putting default "inactive" fields on history that may have not been checked before for perfect restarts. The same problem appears for a FATES test in #3660.

[ekluzek]

The final test for aux_clm. is ERP_P64x2_D_Ld3.f10_f10_mt232.IHistClm60BgcCropCrujra.derecho_gnu.clm-default--clm-all_outputs which of course has the same behavior.

Here's information on the list of diffs:

**tail -n20 /glade/derecho/scratch/erik/cases/ERP_P64x2_D_Ld3.f10_f10_mt232.IHistClm60BgcCropCrujra.derecho_gnu.clm-default--clm-all_outputs.GC.ctsm546delist/run/ERP_P64x2_D_Ld3.f10_f10_mt232.IHistClm60BgcCropCrujra.derecho_gnu.clm-default--clm-all_outputs.GC.ctsm546delist.clm2.h0a.1850-01-04-00000.nc.base.cprnc.out

                 772  -0.225543229497236E+03         -0.725532663194660E+08
                 772  -0.225543229497236E+03         -0.725532663194660E+08
                1824  (    21,     4,     4,     1) (    20,     8,     1,     1)
          avg abs field values:    9.146364921734966E+05
                                   9.146364921734966E+05
************************************************************************************************************************************
  
SUMMARY of cprnc:
 A total number of   2026 fields were compared
          of which     35 had non-zero differences
               and     15 had differences in fill patterns
               and      0 had different dimension sizes
               and      0 had different data types
 A total number of      2 fields could not be analyzed
 A total number of      0 time-varying fields on file 1 were not found on file 2.
 A total number of      0 time-constant fields on file 1 were not found on file 2.
 A total number of      0 time-varying fields on file 2 were not found on file 1.
 A total number of     10 time-constant fields on file 2 were not found on file 1.
  diff_test: the two files seem to be DIFFERENT **

grep RMS /glade/derecho/scratch/erik/cases/ERP_P64x2_D_Ld3.f10_f10_mt232.IHistClm60BgcCropCrujra.derecho_gnu.clm-default--clm-all_outputs.GC.ctsm546delist/run/ERP_P64x2_D_Ld3.f10_f10_mt232.IHistClm60BgcCropCrujra.derecho_gnu.clm-default--clm-all_outputs.GC.ctsm546delist.clm2.h0a.1850-01-04-00000.nc.base.cprnc.out
 RMS ALPHA                            2.9045E-06            NORMALIZED  1.4226E-03
 RMS EOPT                             5.1914E-03            NORMALIZED  5.0983E-03
 RMS GAMMA                            4.5224E-04            NORMALIZED  5.8761E-03
 RMS GAMMAA                           1.3964E-02            NORMALIZED  1.4561E-02
 RMS GAMMAC                           1.0381E-03            NORMALIZED  9.5337E-04
 RMS GAMMAL                           1.0092E-02            NORMALIZED  2.1177E-02
 RMS GAMMAP                           1.6617E-03            NORMALIZED  9.1525E-03
 RMS GAMMAS                           6.2108E-04            NORMALIZED  6.3139E-04
 RMS GAMMAT                           9.1926E-04            NORMALIZED  3.0022E-03
 RMS LEAFCN_STORAGE                   1.6425E-01            NORMALIZED  4.9291E-03
 RMS PAR240_shade                     3.4830E-01            NORMALIZED  4.5462E-03
 RMS PAR240_sun                       1.0386E+00            NORMALIZED  4.9029E-03
 RMS PAR24_shade                      1.6701E-01            NORMALIZED  2.1711E-03
 RMS PAR24_sun                        4.4788E-01            NORMALIZED  2.0322E-03
 RMS PAR_shade                        1.0141E+00            NORMALIZED  1.7038E-02
 RMS PAR_sun                          2.2285E+00            NORMALIZED  9.2302E-03
 RMS QAF                              2.5866E-05            NORMALIZED  3.7664E-03
 RMS RAH1                             3.3380E+01            NORMALIZED  1.0960E-01
 RMS RAH2                             1.0461E+00            NORMALIZED  6.1877E-03
 RMS RAW1                             3.3380E+01            NORMALIZED  1.0960E-01
 RMS RAW2                             1.0461E+00            NORMALIZED  6.1877E-03
 RMS RHAF                             5.6048E-02            NORMALIZED  7.0115E-02
 RMS TAF                              2.8192E-01            NORMALIZED  1.0251E-03
 RMS TOPT                             1.7648E-01            NORMALIZED  5.9874E-04
 RMS UAF                              3.6234E-02            NORMALIZED  9.2844E-02
 RMS UM                               2.3202E-01            NORMALIZED  4.7846E-02
 RMS USTAR                            3.6234E-02            NORMALIZED  9.2844E-02
 RMS VPD                              2.3656E-02            NORMALIZED  4.0938E-02
 RMS currentPatch                     1.3476E-03            NORMALIZED  1.9555E-03
 RMS EFF_POROSITY                     7.5268E-08            NORMALIZED  1.8744E-07
 RMS RRESIS                           3.5471E-02            NORMALIZED  9.8978E-02
 RMS SNO_BW                           6.3679E-02            NORMALIZED  1.5895E-04
 RMS SNO_TK                           7.1675E-05            NORMALIZED  2.2831E-04
 RMS KROOT                            1.0163E-10            NORMALIZED  2.9396E-02
 RMS KSOIL                            8.5730E-07            NORMALIZED  6.7199E-02

[ekluzek]

I had a comment about the Fates test here, but I moved it to #3660

[ekluzek]

Whoops put the FATES test in the wrong issue. So I'll delete the above and move it to the right one.

[ekluzek]

@briandobbins this is the issue with the list of fields. Can we check if any of these are needed for CMIP7?

[wwieder]

Sorry, I may need a bit more context here. Why would we expect time averaged (h0a) and instantaneous (h0i) fields to be identical in this test?

[ekluzek]

@wwieder it's comparing the average file for restart with the base file. And the instantaneous similarly, not average to instantaneous. Both files have the same fields on them so the list of fields that are different is the same for both. So the same 35 for each.

[ekluzek]

@wwieder it's comparing the average file for restart with the base file. And the instantaneous similarly, not average to instantaneous. Both files have the same fields on them so the list of fields that are different is the same for both. So the same 35 for each.

[ekluzek]

@wwieder it's comparing the average file for restart with the base file. And the instantaneous similarly, not average to instantaneous. Both files have the same fields on them so the list of fields that are different is the same for both. So the same 35 for each.