Append method [in memory vs disk size]

[Vasampa23]

Apache Iceberg version

0.8.1

Please describe the bug 🐞

During my initial exploration of append() method.
The flow of execution is roughly as follows append() -> _dataframe_to_data_files() -> bin_pack_arrow_table()

1. In the bin_pack_arrow_table, we consider its in memory size using .nbytes
2. we use write.target-file-size-bytes this seems to consider in memory size rather than disk memory.
3. As I loaded a 300MB parquet file as pyarrow table it occupied 21637095875 bytes in memory. after ingestion this had 50 parquet files in MinIO. [20GB//512 roughly 50 files]

It would be better if we consider disk memory of files to be written rather than in memory as considering in memory could actually result in many small files

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[ifxchris]

Hi all,

we are experiencing the same issue but a bit more severe:

    avg_row_size_bytes = tbl.nbytes / tbl.num_rows
    target_rows_per_file = target_file_size // avg_row_size_bytes
    batches = tbl.to_batches(max_chunksize=target_rows_per_file)

Our data is loaded from a parquet file with the following metadata:
Row group 0: count: 1 4.163 MB records start: 4 total(compressed): 4.163 MB total(uncompressed):40.739 MB
So we only have one record in the table.

According to tbl.nbytes this is around 600MB in memory.
Since this one record is bigger than 512Mb in memory, target_rows_per_file is calculated to be zero.
As a result the max_chunksize is set to 0 and pyiceberg will crash due to that.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'

[kevinjqliu]

Done by #2844

[kevinjqliu]

It would be better if we consider disk memory of files to be written rather than in memory as considering in memory could actually result in many small files

its hard to know what the file size is from the given data frame. I dont think pyarrow exposes the file size information during writing. Otherwise, I agree, we should use the actual file size