Add env variables to configure manifest cache

[kris-gaudel]

Closes #2952

Rationale for this change

Through discussion in issue #2325, we realized that there was a memory leak in the manifest cache. PR #2951 fixed this memory leak, but we decided that it would be best for developer experience if we developers could configure the cache for their needs.

This includes the ability to disable/enable the manifest cache, and configure its size

Are these changes tested?

Yes - unit tests are included to verify these changes

Are there any user-facing changes?

Yes:

• PYICEBERG_MANIFEST_CACHE_SIZE can be used to configure the size of the manifest cache

[kris-gaudel]

@kevinjqliu @Fokko @rambleraptor Please lmk what you think!

[thomas-pfeiffer]

@kris-gaudel I have a few questions:

• Would it make sense to list the new configuration parameters somewhere in mkdocs/docs/configuration.md, so that it shows up in the documentation as well?
• Would it make sense to allow PYICEBERG_MANIFEST__CACHE__SIZE to be set to 0 to effectively disable the cache?
  • It's mentioned in Deprecate support for cache(self) returning None. Why? tkem/cachetools#352 (comment) that maxsize = 0 can be used to bypass the cache.
  • This would make PYICEBERG_MANIFEST__CACHE__ENABLED obsolete, right?
• What's the added benefit of having a _ManifestCacheManager class?
  • Wouldn't _manifest_cache: LRUCache[str, ManifestFile] = LRUCache(maxsize=int(environ.get("PYICEBERG_MANIFEST__CACHE__SIZE", 128))) do the trick as well? (obviously an oversimplified example)

[Fokko]

Thanks @kris-gaudel for raising this. I think it is reasonable to make this configurable. But I think we should reduce the number of changes and simplify where possible. For example, do we need PYICEBERG_MANIFEST__CACHE__ENABLED or can we set PYICEBERG_MANIFEST__CACHE__SIZE to 0 instead.

[thomas-pfeiffer]

Additional remark:
I suggest to add paragraph to /mkdocs/docs/configuration.md to document the new configuration option. Something like:

## Manifest Caching

PyIceberg by default caches the Iceberg Manifest files locally. 
By default up to the `128` least recently used Manifests are cached, leveraging up to approx. X MB of memory. 
(Indicative calculation only. X MB in total = 128 Manifests * average size of X KB per Manifest. Your Manifest's sizes may be different.) 

The maximum cache size can be customised by setting  `PYICEBERG_MANIFEST_CACHE_SIZE` environment variable to a positive number. 
To deactivate the Manifest caching completely, set the max. cache size to `0`; 
To achieve an unlimited / unbound Manifest cache, set the max. cache size to `math.inf` or something comparable.

@kevinjqliu / @Fokko / @kris-gaudel - Do you have a rough ball-park number for the size of the cached Manifests? I would like to give a rough indication to users how much memory the cache may consume. Or is that a bad idea?

[kris-gaudel]

Sorry for the late responses, it was a hectic week. I will update the docs and address PR comments early this week

[thomas-pfeiffer]

Looks good to me.

[thomas-pfeiffer]

Generally speaking, I think we should have a way to monitor the cache performance.

Since we're using the cachetools.LRUCache directly (and not wrapped via @cached), there is not simple way afaik to use the cache_info() method to get hits and misses stats on our global cache instance. In cachetools itself it is implemented as 2 counter (see https://github.com/tkem/cachetools/blob/5dce86fc5c9c565c6e9c912e2be5d6abb9586a1d/src/cachetools/_cached.py#L17) as per my understanding. Maybe we could leverage a similar approach.

Does it make sense to add it here? Or should we create a separate issue and PR for that?

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[kris-gaudel]

Sorry the last month of work was quite hectic so I haven't gotten a chance to review things until today

I think the use of cachetools should be in a separate PR as it seems like a bit adjacent to the issue, I'd be happy to work on that follow up PR though now as I have more time

[kris-gaudel]

CC @geruh @Fokko @kevinjqliu @thomas-pfeiffer

I think this PR is good to go, lmk what you think!

Should the topic Thomas brought up be made into a separate issue or included here?