dataset write api

[dbelozerovx1]

why is dataset is designed to write a single parquet file, i mean it throws an error if write to non empty directory into s3, but usually tables consists of many files, am i did something wrong? but i do not see an option to write multiple parquet files into a directory using dataset

[amoeba]

Hi @dbelozerovx1, arrow-java's Dataset implementation supports writing a dataset out to multiple files like you expect, see https://github.com/apache/arrow-java/blob/main/dataset/src/test/java/org/apache/arrow/dataset/file/TestDatasetFileWriter.java#L66 for an example. If you want to share the code you have that isn't working, I can take a look.

[amoeba]

I see now, thanks.

I think there are two options,

1. The JNI bindings need to be extended to support setting custom DatasetWriteOptions. This might let you do what you want though I'm not totally sure.
2. (Probably easier) Create a new reader for each file you want to write and call into DatasetFileWriter.write once for each reader

[dbelozerovx1]

first one seems better since its generally supports whats already done on cpp side, i can provide a basic contribution with existing_data_behaviour param and compression codec(the second thing i need the most) and try to contribute, but theres many other options maybe you have plans to support em all?

[dbelozerovx1]

my suggestion is

options.existing_data_behavior =
      arrow::dataset::ExistingDataBehavior::kOverwriteOrIgnore;

and compressionCodec allow as a param to be passed from java

[amoeba]

PR review capacity is pretty limited at the moment but PRs are always welcome. Supporting all options at once would probably be the better approach rather than picking and choosing.