feat!: typed, opt-in configuration for icechunk virtual chunk access

[maxrjones]

Summary

This PR types the configuration for Icechunk virtual chunk access authorization. I believe this is worth it because:

• It makes authorization explicit. Previously tte default was to grant the service the ambient AWS credentials for any prefix, which is risky (see https://www.earthmover.io/blog/secure-virtual-chunks).
• It makes it possible to access chunks in GCS and Azure.
• It makes mis-configuration of chunk authorization fail fast during startup, rather than when loading data
• It removes the option to point to chunks referring to file:// (also see https://www.earthmover.io/blog/secure-virtual-chunks).

This is a breaking change, since it requires granting access via the chunk configuration. I think this is the right call, but it does mean that the configuration would need to be updated when new data sources are added, rather than just adding access to the deployment credentials. However, NASA data is stored in a reasonably number of buckets that we could enumerate, so I don't think this is a huge downside.

It also relies on keeping the typed configuration up-to-date with Icechunk's, since they don't expose types for this. I added a test to catch drift and pinned the major version of Icehunk to protect against drift.

Lastly, I add documentation for how to authorize access to chunks.

Testing

• I ran the tests locally, and tested with TEMPO data.

PR checks

• Standard CI runs automatically on each push.
• To run the CDK synth check, add the run-cdk-checks label to this PR.
• If you push more commits after that run completes, remove and re-add the label to run it again.
• To trigger a dev deployment, add the deploy-dev label. It smoke-tests tiles from the native MUR, virtual MUR, and virtual NLDAS Icechunk stores after deployment.

[hrodmn]

Thanks for this change @maxrjones! I think the settings -> icechunk credentials approach you built works well. I admit that I get confused by the credentials API for virtual icechunk stores, but your changes and accompanying documentation are really helpful.

FWIW I don't think we should trigger v1.0 with this change (!feat), it would be fine to merge as a feat: ... since we are pre 1.0 still and we are not yet depending on the service in VEDA.

[hrodmn]

This is a nice guardrail!

[hrodmn]

Do you think the earthmover team would be receptive to moving some of the pydantic models into the icechunk library? It's useful here but does not feel specific to titiler-multidim.

[hrodmn]

is there any reason not to add this as a method on one of the pydantic classes? I not sure where the boundary should be between application code and the pydantic settings, but it could be nice to keep all of the icechunk credentials logic close together instead of spread between reader.py and settings.py.

[maxrjones]

FWIW I don't think we should trigger v1.0 with this change (!feat), it would be fine to merge as a feat: ... since we are pre 1.0 still and we are not yet depending on the service in VEDA.

Sounds good. We could also reduce the impact of this change by incorporating earthaccess-auth at the same time, authorizing access to all the buckets registered in CMR metadata via temporary S3 credentials rather than role-based access. Do you think that's worth it either here or as a separate PR?

[hrodmn]

We could also reduce the impact of this change by incorporating earthaccess-auth at the same time, authorizing access to all the buckets registered in CMR metadata via temporary S3 credentials rather than role-based access. Do you think that's worth it either here or as a separate PR?

I think that sounds like a separate PR. Would that mean we don't need to add the DAAC buckets to the JSON config?