-
Notifications
You must be signed in to change notification settings - Fork 17.4k
Revert "Suspend Apache Beam Provider due to grpcio limitation (#61926)" #66952
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| de29604dc4d7031b9a26a1e0350d7806 | ||
| c4788f285994a2bcc6bdf4c26b100921 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 4b44b7c52911adf1c1bdfb87e10e873b | ||
| 2fe6904ce0c36330b00bd16b643da23d |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 57466e5cb9470b0d4398724fda2a1825 | ||
| 8a738bf76c1dd778b135492aa86f40ab |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| e2d5118e3339b53ce6e87dc6be6ae053 | ||
| 0837f7e405c27375919e1c33782d8b4d |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 3581abc65c55d4b8a096005c312df783 | ||
| e4ed57cfecc4b27fdad8bcf0b0c2e862 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 0a45b898de7677eed8aa19c7e4f07345 | ||
| fc31bc3a8190f254c85ecbcc71596117 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 30b0621c66436e215e0d34ec9bfd407b | ||
| 5693f4a2ce03094c58480b2cdbc82080 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 01f2b301d8965951b49e2c8cd114700a | ||
| a70a0f1ed2b858ed43443aa3b024e8e5 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -61,7 +61,11 @@ requires-python = ">=3.10" | |
| dependencies = [ | ||
| "apache-airflow>=2.11.0", | ||
| "apache-airflow-providers-common-compat>=1.12.0", | ||
| "apache-beam>=2.69.0", | ||
| "apache-beam>=2.72.0", | ||
| # For Python>=3.11, the apache-beam package depends on envoy-data-plane. | ||
| # Currently, there is only one compatible version(1.0.3) of envoy-data-plane available for these | ||
| # Python versions, and it explicitly requires a prerelease version of betterproto. | ||
| "betterproto==2.0.0b6; python_version >= '3.11'", | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This exact-prerelease pin is a workaround for the |
||
| "pyarrow>=16.1.0; python_version < '3.14'", | ||
| "pyarrow>=22.0.0; python_version >= '3.14'", | ||
| "numpy>=1.22.4; python_version<'3.11'", | ||
|
|
@@ -74,7 +78,7 @@ dependencies = [ | |
| # Any change in the dependencies is preserved when the file is regenerated | ||
| [project.optional-dependencies] | ||
| "google" = [ | ||
| "apache-beam[gcp]>=2.69.0", | ||
| "apache-beam[gcp]>=2.72.0", | ||
| ] | ||
|
|
||
| [dependency-groups] | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not a fan of this.
this is a beta version from 2023. This is not a case where in a few days/weeks things will change and a stable version would be released, is it?
I am not in the details of this so I will leave @potiuk the call here but from my point of view we should not deliever stable release with depenedency on beta packages.
Who gurentees that tommorow the author won't remove them from PyPi with a thought that there won't be any production impact on anyone?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a big fan either.
But I think thee newer version of envoy-data-plane has been questioned by others for example here: apache/beam#37854
And I see that this is what latest beam has:
And the 2.1.0 version of envoy-data-plane uses already betterproto2 - which is a maintained replacement of betterproto.
So maybe we can ask
apache-beamto upgrade to theenvoy-data-plane>=2?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I commented in the issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not a big fan of this idea either.
The reason why I decided to add it was in that this beta version exists from 2023 and I am not sure that it will be release in the future. One more point for me to added it was that I noticed that Apache Beam SDK use this beta version in there generated requirements for containers. And I had idea that if they already had hardcoded it in
apache-beamrequirements, from which our provider already depends, maybe it makes sense temporary hardcoded it in provider as well for fixing constrains problem and unsuspend provider.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can't tell beam what to do from thier side but Airflow should not do this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created this PR in beam to try and tackle this issue:
apache/beam#39213