Implemented MLTransform generate vocab Dataflow benchmark#38215
Implemented MLTransform generate vocab Dataflow benchmark#38215aIbrahiim wants to merge 1 commit into
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new performance benchmark for the MLTransform vocabulary generation pipeline in Apache Beam. It includes the implementation of the benchmark pipeline, comprehensive unit and integration tests, and the necessary updates to the website's performance tracking infrastructure to visualize the results. Additionally, it improves the robustness of the existing Dataflow cost benchmark utility by adding support for diverse numeric metric types. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
R: @damccorm |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
aIbrahiim
force-pushed
the
ml-transform-generate-vocab
branch
from
021405e to
5ba07ba
Compare
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new batch-only vocabulary generation example using MLTransform and ComputeAndApplyVocabulary, complete with unit tests, integration tests, and performance benchmarks. Additionally, it enhances the DataflowCostBenchmark utility to support a wider range of metric value types from Cloud Monitoring. Key feedback from the review includes addressing the use of local temporary directories which may fail on distributed runners, fixing a filtering bug that incorrectly drops valid vocabulary indices, and reconciling documentation that mentions TF-IDF transformations not present in the current implementation.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a new batch-only vocabulary generation example using MLTransform, accompanied by comprehensive tests, a cost benchmark, and performance documentation. It also updates the DataflowCostBenchmark utility to correctly parse various numeric value types from Cloud Monitoring. Key feedback points out that the default local artifact storage will fail on distributed runners like Dataflow and identifies a naming mismatch in the benchmark's PCollection monitoring that would prevent accurate throughput tracking.
|
there are failing checks |
aIbrahiim
force-pushed
the
ml-transform-generate-vocab
branch
from
a3582cd to
15293a9
Compare
| @@ -400,7 +400,6 @@ def expand( | |||
|
|
|||
| ret = ( | |||
| input | |||
| | beam.Reshuffle() | |||
There was a problem hiding this comment.
Why are we removing this?
There was a problem hiding this comment.
Btw, if you make functional changes (not just benchmark changes) please clearly call that out in a comment or the description. I don't think we want this change though
There was a problem hiding this comment.
Also, please avoid rebasing when possible. This breaks GitHub's ability to show an incremental diff, so I have to re-review everything (this is particularly problematic when we're close to merging and just need a small change). We can rely on the final merge to squash the commits
There was a problem hiding this comment.
Btw, if you make functional changes (not just benchmark changes) please clearly call that out in a comment or the description. I don't think we want this change though
I remmoved reshffule as test_one_detector_1 was failing on the portable runner in py 311 ml and tje ZScore learns in input order (score, then update) and I found out that Reshuffle() can change that order so i saw wrong scores (e.g. nan instead of 8.0 for the outlier row) that's why I removed it to keep order stable for online detectors but if this is not the right approach i can add reshuffle back and try to fix that test
There was a problem hiding this comment.
Also, please avoid rebasing when possible. This breaks GitHub's ability to show an incremental diff, so I have to re-review everything (this is particularly problematic when we're close to merging and just need a small change). We can rely on the final merge to squash the commits
yeah sorry about that. I will avoid rebasing it was a conflict i had to resolve
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a batch-only vocabulary generation pipeline using MLTransform, complete with documentation, unit/integration tests, and a Dataflow cost benchmark. It also includes updates to the performance monitoring infrastructure and a modification to metric extraction logic. Feedback highlights potential issues with local temporary directories when running on Dataflow, non-deterministic artifact path resolution, and a discrepancy in a test verifying token order. Reviewers also questioned the removal of a Reshuffle transform in the anomaly detection module and suggested a minor code cleanup in the benchmark utility.
| ret = ( | ||
| input | ||
| | beam.Reshuffle() | ||
| | f"Score and Learn ({model_uuid})" >> RunScoreAndLearn(self._detector)) |
There was a problem hiding this comment.
The beam.Reshuffle() transform was removed here. Reshuffle is often used to prevent fusion and ensure that work is distributed across workers, especially before heavy processing steps like RunScoreAndLearn. Removing it might impact the scalability and performance of the anomaly detection pipeline by allowing the runner to fuse the scoring step with preceding transforms, potentially limiting parallelism. Could you provide the rationale for this change?
| return 0.0 | ||
| if active_field == 'money_value': | ||
| money = value.money_value | ||
| nanos = getattr(money, 'nanos', 0) or 0 |
There was a problem hiding this comment.
The or 0 at the end of this line is redundant because getattr(money, 'nanos', 0) already provides a default value of 0 if the attribute is missing. Additionally, if the attribute exists but is 0, the or 0 still results in 0.
| nanos = getattr(money, 'nanos', 0) or 0 | |
| nanos = getattr(money, 'nanos', 0) |
3 participants
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.