DOG-6508: Metrics pickling issue

[Hmnt39]

This PR resolves metrics serialization errors when passing BaseMetrics instances through multiprocessing by changing the runtime to pass the metrics class instead of an instance. The extractor now initializes metrics internally using a singleton pattern to avoid Prometheus registry conflicts.

Added comprehensive tests to verify metrics work correctly across process boundaries.

[gemini-code-assist]

Summary of Changes

Hello @Hmnt39, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors how metrics are handled within the extractor framework, specifically addressing issues related to serializing BaseMetrics instances in multiprocessing scenarios. By transitioning to passing the metrics class and utilizing a singleton pattern for internal initialization, it ensures robust and conflict-free metrics collection, particularly in distributed environments. The changes are supported by extensive new tests that validate the correct behavior of metrics across process boundaries.

Highlights

• Metrics Serialization Fix: Addressed BaseMetrics serialization errors in multiprocessing by changing the runtime to pass the metrics class instead of an instance.
• Singleton Metrics Initialization: Implemented a singleton pattern for metrics initialization within the extractor to prevent Prometheus registry conflicts, ensuring consistent metric state across processes.
• Refactored Metrics Handling: The Extractor and Runtime classes have been refactored to manage metrics instances internally, moving away from passing instances directly, which simplifies usage and improves robustness.
• Comprehensive Testing: Added new and updated existing tests to thoroughly verify metrics functionality, especially across process boundaries in a multiprocessing environment, including a dedicated test for counter increments.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively resolves a serialization issue with BaseMetrics instances in a multiprocessing context by passing the metrics class instead of an instance. The extractor now correctly initializes metrics as a singleton within each process, preventing Prometheus registry conflicts. The changes are well-implemented, and the addition of comprehensive integration tests is a great improvement. I have one suggestion to remove some dead code for better maintainability.

[codecov]

Codecov Report

❌ Patch coverage is 88.23529% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.04%. Comparing base (e1538d0) to head (ca7bacf).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
cognite/extractorutils/unstable/core/base.py	90.00%	1 Missing ⚠️
cognite/extractorutils/unstable/core/runtime.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #498      +/-   ##
==========================================
+ Coverage   80.98%   81.04%   +0.05%     
==========================================
  Files          43       43              
  Lines        4203     4210       +7     
==========================================
+ Hits         3404     3412       +8     
+ Misses        799      798       -1     

Files with missing lines	Coverage Δ
cognite/extractorutils/metrics.py	93.83% <100.00%> (-0.65%)	⬇️
cognite/extractorutils/unstable/core/base.py	79.54% <90.00%> (+0.47%)	⬆️
cognite/extractorutils/unstable/core/runtime.py	70.76% <83.33%> (+0.97%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[devendra-lohar]

LGTM

[devendra-lohar]

/gemini review

[gemini-code-assist]

Code Review

This pull request effectively resolves a metric serialization issue encountered in multiprocessing environments by refactoring the system to pass the metrics class instead of an instance. The introduction of a singleton pattern for metrics initialization is a robust solution to prevent Prometheus registry conflicts. The changes are well-implemented across the core base and runtime modules. Furthermore, the addition of comprehensive and well-designed tests, including an integration test that verifies the fix across process boundaries and new validation tests, significantly improves the reliability and test coverage of the metrics functionality. The code quality is high, and the approach taken is solid.

[rbtcollins]

🦄

[rbtcollins]

🦄