Should set_sim_path() preserve external array structure?

[javgs-bd]

Hi,

I have a MODFLOW 6 simulation whose NPF and STO packages contain arrays stored externally using OPEN/CLOSE. I am currently using FloPy 3.10 and would like to know whether moving to FloPy 3.11 would resolve the behavior described below. I have not found a discussion addressing this specific case.

I am trying to load the simulation and write it to a new workspace using a workflow like this:

sim = flopy.mf6.MFSimulation.load(
    sim_ws=original_sim_ws,
)

sim.set_sim_path(new_sim_ws)
sim.write_simulation()

After writing the simulation to new_sim_ws, FloPy 3.10 replaces the original OPEN/CLOSE references with arrays written internally in the package files.

However, if I first copy the entire model directory, then load and write the simulation without changing sim_ws, the external-array storage is preserved.

In summary:

load from original_sim_ws
    -> write to original_sim_ws
    -> OPEN/CLOSE references are preserved

load from original_sim_ws
    -> set_sim_path(new_sim_ws)
    -> write to new_sim_ws
    -> external arrays are converted to internal arrays

I am not modifying the package data or arrays between loading and writing. The only change is the simulation workspace.

Copying the entire model directory before loading is not ideal because the original workspace also contains files that are not required to run the simulation, such as PEST and PLPROC files.

I have read about lazy_io, but as I understand it, this option does not guarantee preservation of the external-file structure, filenames, and OPEN/CLOSE references when changing sim_ws.

Is this intended behavior or a FloPy bug? Would upgrading to FloPy 3.11 resolve it, or is there a recommended workflow for writing a loaded simulation to a new workspace while preserving its existing external-array storage?

[VolodymyrLinuxovich]

This is intended. set_sim_path calls set_all_data_internal() first, in both 3.10 and 3.11. You can skip it like this.

import os
import flopy

sim = flopy.mf6.MFSimulation.load(sim_ws=original_sim_ws)

os.makedirs(new_sim_ws, exist_ok=True)
sim.simulation_data.mfpath.set_sim_path(new_sim_ws, True)
sim.simulation_data.mfpath.copy_files()
sim.write_simulation()

I tested it and the OPEN/CLOSE files kept their names. Only files the model uses get copied, so your PEST files stay behind.

[wpbonelli]

This is a good workaround. If you're not changing your models I don't know if the cost of loading the simulation is worth the convenience of not filtering files and copying manually, the load may be slow if the model is big.

But this design was deliberate #1157. It doesn't seem ideal from a UX point of view, it could be changed in 4.x. But there were legitimate maintenance reasons for it.

In the meantime, it wouldn't be hard to add a parameter as shorthand for the code above, maybe we could do something like

sim.set_sim_path(path, keep_external=True)

[javgs-bd]

Many thanks @VolodymyrLinuxovich and @wpbonelli.
Yes, a keep_external parameter to set_sim_path would be most welcome