Regional means entrypoint

[chpolste]

Description

Compute regional area-weighted means.

• Area weights are based on the cosine of latitude (valid for rectilinear grids with regular latitude coordinates).
• Areas are specified with a rectangular bounding box in lat and lon.
• The output contains one column/variable per configured region. (not good, see below)

To compute zonal-mean zonal wind for https://charts.ecmwf.int/products/extended-zonal-mean-zonal-wind.

New DataFrameTarget

Introduces pproc.config.targets.DataFrameTarget to collect rows of an output table as they are produced, then write the table in different formats to disk.

• The table structure (index and value columns) is specified in the target configuration.
• Available output formats: csv, netcdf. Still missing covjson.
• flush() does not implement incremental writes but replaces the entire target file every time.
• Checkpointing is available, but previous results can only be recovered from netcdf format.

Open issues

• Setting additional metadata for the target (parameter names, units, information about the domains, etc.).
• I'm not particularly happy with how multiple parameters and regions are handled. I'd prefer to have region as a coordinate/index and split parameters into variables/columns (this should be much closer to CF conventions). At the moment all parameters are mixed in the table unless explicitly separated with a coordinate and regions are split into variables.

Contributor Declaration

By opening this pull request, I affirm the following:

• All authors agree to the Contributor License Agreement.
• The code follows the project's coding standards.
• I have performed self-review and added comments where needed.
• I have added or updated tests to verify that my changes are effective and functional.
• I have run all existing tests and confirmed they pass.

[jinmannwong]

Thanks @chpolste, the PR looks good! I just had a few questions/comments:

• I see in your entrypoint that you are only writing to the file right at the very end, rather than in each subprocess. Is this required?
• I'm wondering if you should create a new target XarrayTarget, or something like that, that inherits from Target and has the path attribute since you are not using the write function from FileTarget to write outputs

[chpolste]

Thanks @jinmannwong, good to know I've not gone off in a completely wrong direction :)

I see in your entrypoint that you are only writing to the file right at the very end, rather than in each subprocess. Is this required?

For NetCDF output it's not required, but a lot easier. I think it would be necessary to know the output coordinates already when creating the target file (maybe not for the metadata but at least the ordering) and they depend on the metadata of all input fields. So they must either be inferred from the FDB/MARS request(s) or the inputs have to be iterated over twice (once for the coords, then again for the data).

For CovJSON, I don't see a good option to meaningfully write partial results without constantly replacing the file to keep the partial files valid. Concurrency will further cause issues with the ordering of data in a text-based format.

This is where I struggled with the checkpointing too. The already computed results need to go somewhere persistent (like the FDB for grib). The best solution I've come up with so far is a temporary sqlite database that keeps pickled dataframe results, so they can be loaded again during recovery and merged into the final output file together with the other results. It's lightweight and supports concurrent inserts.

I'm wondering if you should create a new target XarrayTarget, or something like that, that inherits from Target and has the path attribute since you are not using the write function from FileTarget to write outputs

Sounds good, I'll do that!

[jinmannwong]

Thanks, that makes sense. I didn't realise that the outputs for all steps and parameters are required in the same output file.