Adds support for reading at a given Delta Lake version or timestamp - #39758
Conversation
|
Assigning reviewers: R: @Abacn for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
chamikaramj
changed the title
| ExperimentalOptions.addExperiment(options, "use_runner_v2"); | ||
|
|
||
| Map<String, String> hadoopConfig = new HashMap<>(); | ||
| hadoopConfig.put("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); |
There was a problem hiding this comment.
There are duplicated codes. Can we move the creation of hadoop Configuration into a helper function?
| Engine engine = DefaultEngine.create(conf); | ||
|
|
||
| // Wait briefly to ensure timestamp is after version 0 commit | ||
| Thread.sleep(1000); |
There was a problem hiding this comment.
sleep is generally unreliable especially on CI. Consider using a reliable way to detect initial setup is finished, e.g. poll the file system?
| readPipeline.run().waitUntilFinish(); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
Is DeltaIOIT exercised by any GHA tests? Checking https://github.com/apache/beam/blob/master/.github/workflows/beam_PreCommit_Java_Delta_IO_Direct.yml it only runs :sdks:java:io:delta:build not :sdks:java:io:delta:integrationTest
There was a problem hiding this comment.
It's included in https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Java_Delta_IO_Dataflow.yml.
Executed for the current PR here: https://github.com/apache/beam/actions/runs/31854212817/job/94935775426?pr=39758
| File tableDir = tempFolder.newFolder("delta-table-read-version"); | ||
| Engine engine = DefaultEngine.create(new org.apache.hadoop.conf.Configuration()); | ||
|
|
||
| Schema schema = Schema.builder().addField("name", Schema.FieldType.STRING).build(); |
There was a problem hiding this comment.
Same here, consider move repeated test data setup fixtures into a helper function
| readPipeline.run().waitUntilFinish(); | ||
| } | ||
|
|
||
| @Test |
There was a problem hiding this comment.
It's included in https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Java_Delta_IO_Dataflow.yml.
Executed for the current PR here: https://github.com/apache/beam/actions/runs/31854212817/job/94935775426?pr=39758
| Engine engine = DefaultEngine.create(conf); | ||
|
|
||
| // Wait briefly to ensure timestamp is after version 0 commit | ||
| Thread.sleep(1000); |
| ExperimentalOptions.addExperiment(options, "use_runner_v2"); | ||
|
|
||
| Map<String, String> hadoopConfig = new HashMap<>(); | ||
| hadoopConfig.put("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem"); |
| File tableDir = tempFolder.newFolder("delta-table-read-version"); | ||
| Engine engine = DefaultEngine.create(new org.apache.hadoop.conf.Configuration()); | ||
|
|
||
| Schema schema = Schema.builder().addField("name", Schema.FieldType.STRING).build(); |
| } | ||
|
|
||
| /** | ||
| * Specifies the version of the Delta Lake table to read. |
There was a problem hiding this comment.
Are these Javadoc removal intended?
There was a problem hiding this comment.
Yeah (just removed from the commit not the repo). We expect users to use the Managed I/O API. Documentation there is generated from what's in DeltaReadSchemaTransformProvider.
|
Thanks! |
2 participants
Please add a meaningful description for your change here
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.