diff --git a/docs/docs/concepts/spec/fileformat.md b/docs/docs/concepts/spec/fileformat.md index 65a4d5af0c44..f5a4c03550d2 100644 --- a/docs/docs/concepts/spec/fileformat.md +++ b/docs/docs/concepts/spec/fileformat.md @@ -930,4 +930,47 @@ Limitations: 2. BLOB format does not support predicate pushdown. 3. Statistics collection is not supported for BLOB columns. +### Video + +Video is an independent, versioned format with the `.video` extension. It packs one or more +complete encoded-video payloads and logical frame runs. The payloads are raw byte ranges without +the ordinary BLOB entry header, length trailer, or per-entry CRC: + +``` ++----------------------------+ +| Encoded Video Payload 1 | Raw complete video bytes ++----------------------------+ +| Encoded Video Payload 2 | ++----------------------------+ +| ... | ++----------------------------+ +| Physical Length Index | Delta-Varint video lengths ++----------------------------+ +| Run Length Index | Delta-Varint logical row counts ++----------------------------+ +| Run Reference Index | Delta-Varint physical video ordinals ++----------------------------+ +| Run First-Frame Index | Delta-Varint frame ordinals ++----------------------------+ +| Physical Index Length | 4 bytes (Little Endian) +| Run-Length Index Length | 4 bytes (Little Endian) +| Run-Reference Index Length | 4 bytes (Little Endian) +| First-Frame Index Length | 4 bytes (Little Endian) +| Magic Number | 4 bytes (0x4F454449, Little Endian) +| Version | 1 byte ++----------------------------+ +``` + +The run arrays have equal element counts. A non-negative run reference is an ordinal in the +physical length index. For logical row `r` in a run beginning at logical row `s`, the returned +`VideoFrameDescriptor` identifies the referenced raw video range and frame ordinal +`run_first_frame + (r - s)`. `-1` is a NULL run and `-2` is a data-evolution placeholder run. +Non-negative runs have fixed frame stride one in version 1; a discontinuity starts another run. + +Readers validate footer and index bounds, positive physical lengths, full coverage of the payload +region, equal run-index counts, positive run lengths, physical ordinals, and non-negative first +frames. The format currently supports one scalar BLOB field per file. Physical video reuse uses +exact input payload `BlobDescriptor` identity and is file-local; there are no cross-file payload +references. Ordinary `.blob` files keep their existing version, wrappers, checksums, and layout. + For usage details, configuration options, and examples, see [Blob Type](../../multimodal-table/blob). diff --git a/docs/docs/multimodal-table/blob.mdx b/docs/docs/multimodal-table/blob.mdx index ed92e7a832b1..048a0ee2d561 100644 --- a/docs/docs/multimodal-table/blob.mdx +++ b/docs/docs/multimodal-table/blob.mdx @@ -49,6 +49,11 @@ When you define a table with a Blob column, Paimon automatically separates the s 1. **Normal Data Files** (e.g., `.parquet`, `.orc`): Store regular columns (INT, STRING, etc.) 2. **Blob Data Files** (`.blob`): Store the actual blob data +For append-only video-frame workloads, one scalar BLOB field can instead use the independent +**video pack format** (`.video`). A video pack concatenates several complete encoded videos and +embeds a compact frame-run index. The frame ordinal lives in `VideoFrameDescriptor`; it does not +need a `frame_index` or `frame_timestamp` column in the normal data file. + For example, given a table with schema `(id INT, name STRING, picture BLOB)`: ``` @@ -104,6 +109,13 @@ Flink, Spark, and Python. + +
video-frame-field
+ No + - + String + Names one scalar BLOB field whose logical values are frames in complete encoded videos packed into .video files. This first version supports append-only tables and exact VideoFrameDescriptor-backed input only. +
blob-as-descriptor
No @@ -267,6 +279,61 @@ schema = Schema.from_pyarrow_schema( +## Video Frame Storage + +Use `video-frame-field` when the logical table has one row per frame while the physical storage +and decoding unit is a complete encoded video. The option also marks a SQL `BYTES` or `BINARY` +column as a BLOB column. It selects an independent `.video` format; the ordinary `.blob` version +and layout are not changed. + +```sql +CREATE TABLE video_frames ( + episode_id BIGINT, + video BYTES +) WITH ( + 'row-tracking.enabled' = 'true', + 'data-evolution.enabled' = 'true', + 'video-frame-field' = 'video' +); +``` + +Each logical value is a `VideoFrameDescriptor`: its URI range identifies one complete encoded +video and its frame ordinal selects a frame inside that video. On write, a `.video` data region +concatenates raw video payloads without ordinary BLOB entry wrappers. Four embedded delta-varint +indexes record physical video lengths, logical run lengths, run-to-video references, and the first +frame ordinal of each run. Consecutive frames therefore need one run entry rather than one index +entry per row. NULL and data-evolution placeholders use negative run references. + +Physical reuse uses exact payload descriptor identity (URI, offset, and length), not a content +hash. It is local to each `.video` file: every pack is self-contained and never points at payloads +owned by another Paimon data file. A pack can contain multiple MP4 payloads. After a size or row +target is reached, rolling waits for the current physical video group to end, making the target +soft for large videos. A later commit or an earlier roll can store another physical copy of the +same source video. + +Compaction byte-copies the complete encoded-video ranges into a new self-contained `.video` pack +and rebuilds the embedded indexes; it does not decode or re-encode frames. The aligned normal data +file contains only application columns such as `episode_id`, state, and action. Paimon stores the +frame mapping in the `.video` descriptor/index path. + +Current restrictions: + +- Append-only tables only; primary-key tables continue to use managed BLOB storage. +- At most one `video-frame-field` per table. +- The field must be a scalar `BLOB`; `ARRAY` and `MAP` continue to use `.blob`. +- Non-null writes must be exact descriptor-backed `BlobRef` values containing a + `VideoFrameDescriptor`. Inline bytes and ordinary `BlobDescriptor` values are rejected. +- Version 1 addresses frames by zero-based presentation-order ordinal with stride one. It does not + store PTS values or parse codec/container metadata. +- A `BlobConsumer` callback is not supported for the video field. + +Reads of a `.video` field return serialized `VideoFrameDescriptor` values even when +`blob-as-descriptor` is false. Materializing the complete MP4 once per logical frame would defeat +the format's purpose. + +For Python ingestion and PyTorch `DataLoader` usage, including decoder-session reuse in workers, +see [PyPaimon Multimodal API: Video Frame Storage](../pypaimon/multimodal-api#video-frame-storage). + The comment directive format is `__DIRECTIVE; optional user comment`. Paimon converts `BYTES`/`BINARY` to `BLOB`, `ARRAY`/`ARRAY` to `ARRAY`, and `MAP`/`MAP` to `MAP`. It registers the field in the corresponding option and stores the text after `;` as the column's real comment. Supported directives: diff --git a/docs/docs/primary-key-table/blob-storage.md b/docs/docs/primary-key-table/blob-storage.md index 44ccd638d3a9..98696d4517a5 100644 --- a/docs/docs/primary-key-table/blob-storage.md +++ b/docs/docs/primary-key-table/blob-storage.md @@ -37,6 +37,11 @@ This mode stores: For general BLOB concepts and read options, see [BLOB Storage](../multimodal-table/blob). +The append-only `video-frame-field` mode is deliberately separate from primary-key managed BLOB +storage. It writes self-contained `.video` packs containing complete encoded videos and embedded +frame-run indexes, and is not supported on primary-key tables. Enabling it does not change +`.managed.blob` packs, `.blobref` ownership, or their garbage-collection behavior. + ## Create a Table Use `blob-field` to mark scalar, array, or map fields whose payloads should be stored in managed BLOB files. diff --git a/docs/docs/pypaimon/blob.md b/docs/docs/pypaimon/blob.md index dddc32bf2f3f..ec07ef177c97 100644 --- a/docs/docs/pypaimon/blob.md +++ b/docs/docs/pypaimon/blob.md @@ -76,6 +76,14 @@ write_builder.new_commit().commit(writer.prepare_commit()) writer.close() ``` +For frame tables, configure `video-frame-field`. The high-level multimodal +API creates `VideoFrameDescriptor` values, packs multiple complete videos in +`.video` files, keeps frame ordinals out of the normal data file, and provides +`add_video` / `add_videos` / `replace_video` for physical video writes. +Ordinary frame-column updates and all reads continue to use the existing table +and BLOB APIs. See +[Video Frame Storage](./multimodal-api#video-frame-storage). + ## Reading Blob Data ### Batch reading (recommended) @@ -160,8 +168,9 @@ blob = Blob.from_bytes(descriptor_bytes, file_io) data = blob.to_data() ``` -The factory auto-dispatches based on the bytes content (BLOBDESC magic -header). This mirrors Java's `Blob.fromBytes(...)`. +The factory auto-dispatches based on the bytes content (`BLOBDESC`, +`VIDEOFRM`, or blob-view magic header). This mirrors Java's +`Blob.fromBytes(...)`. ## See Also @@ -169,3 +178,5 @@ header). This mirrors Java's `Blob.fromBytes(...)`. SQL/Java API - [Data Evolution](./data-evolution) — required for blob tables +- [Multimodal video frames](./multimodal-api#video-frame-storage) — + `.video` pack writing and PyTorch DataLoader decoding diff --git a/docs/docs/pypaimon/multimodal-api.mdx b/docs/docs/pypaimon/multimodal-api.mdx index 6ca8d85634b3..7b4d3f8794ff 100644 --- a/docs/docs/pypaimon/multimodal-api.mdx +++ b/docs/docs/pypaimon/multimodal-api.mdx @@ -179,7 +179,10 @@ vector or blob types. `add` accepts `pyarrow.Table`, `pyarrow.RecordBatch`, a list of dictionaries, a dictionary of arrays, or a pandas DataFrame. Input columns are aligned and cast -to the Paimon table schema before writing. +to the Paimon table schema before writing. For a BLOB column, list, dictionary, +and pandas inputs may also contain `Blob` or `BlobDescriptor` objects. A +descriptor-backed `Blob` is copied through its stream; it is not first loaded +into Python memory. ```python docs.add([ @@ -192,6 +195,203 @@ docs.add([ ]) ``` +## Video Frame Storage + +For a frame table, set `video-frame-field` on one scalar BLOB column. Paimon +stores the frame ordinal in `VideoFrameDescriptor` and the compact `.video` +index, so the normal data file does not need a `frame_index` or +`frame_timestamp` column: + +```python +import pyarrow as pa +import pypaimon.multimodal as pm + +frames = conn.create_table( + "video_frames", + schema=pa.schema([ + pa.field("episode_id", pa.int64()), + pa.field("state", pa.list_(pa.float32())), + pa.field("action", pa.list_(pa.float32())), + pa.field("video", pa.large_binary()), + ]), + options={ + "video-frame-field": "video", + }, +) +``` + +`add_video` accepts a complete encoded-video source plus the ordinary columns +for its logical frame rows. `Blob.from_local` is descriptor-backed and streams +the source; the MP4 is not first loaded into Python memory: + +```python +video = pm.Blob.from_local("/data/episode-42.mp4") +frames.add_video(video, frame_rows, first_frame=0) +``` + +Use `add_videos` to keep one writer and one commit open across several source +videos. This is the path that lets one `.video` object pack multiple complete +MP4 payloads and reduces object/manifest count: + +```python +frames.add_videos([ + (pm.Blob.from_local("/data/episode-42.mp4"), episode_42_rows), + (pm.Blob.from_local("/data/episode-43.mp4"), episode_43_rows), +]) +``` + +Each item may also be `(video, frame_rows, first_frame)`. Frame ordinals are +generated consecutively from `first_frame`. If application semantics require +PTS, wall-clock time, or a non-unit sampling map, retain that value in a normal +column; `.video` version 1 addresses presentation-order frames with stride one. + +Remote sources can use an explicit descriptor. Its URI must be readable with +the table's configured `FileIO` credentials: + +```python +video = pm.BlobDescriptor( + "oss://source-bucket/episode-43.mp4", + offset=0, + length=video_size, +) +frames.add_video(video, episode_43_rows) +``` + +The writer deduplicates exact payload descriptor identity inside one `.video` +file and delays a pending roll until the current video group ends. A later +call, checkpoint, or earlier file roll can create another physical copy of the +same source video. The normal `.blob` format is unchanged. + +### Update frame rows and replace a video + +Frame rows use the ordinary table update API for non-video columns. This writes +only the changed data-evolution columns and keeps the existing `.video` object +and frame descriptors unchanged: + +```python +frames.update( + where="episode_id = 42", + values={"state": corrected_states}, +) +``` + +Writing a complete encoded video uses the specialized API. `replace_video` +selects the matching logical rows in `_ROW_ID` order, assigns consecutive frame +ordinals, and writes a video-column delta without rewriting the normal data +file: + +```python +frames.replace_video( + where="episode_id = 42", + video=pm.Blob.from_local("/data/episode-42-corrected.mp4"), + first_frame=0, +) +``` + +The generic `update()` API rejects assignments to the configured +`video-frame-field`; use `replace_video()` so Paimon can preserve the complete +video payload and its embedded row-to-frame mapping. + +### Read with PyTorch DataLoader + +Install the PyTorch extra, then convert the multimodal scan directly. Unlike a +regular BLOB materialization, `ScanQuery.to_torch` always returns serialized +descriptors for BLOB columns. DataLoader workers receive Paimon splits and open +the selected video ranges themselves: + +```shell +pip install 'pypaimon[torch]' +``` + +```python +from torch.utils.data import DataLoader + +dataset = ( + frames.scan() + .select(["episode_id", "state", "action", "video"]) + .to_torch(streaming=True) +) + +loader = DataLoader( + dataset, + batch_size=32, + num_workers=4, + shuffle=False, +) +``` + +The `video` value is serialized `VideoFrameDescriptor` bytes, not repeated MP4 +bytes. `VideoFrameCollator` owns a bounded, process-local LRU cache keyed by the +physical video range, so different frame descriptors reuse one decoder session +in each DataLoader worker. `decoder_factory(stream)` opens any codec library; +`decode_fn(decoder, frame_index, row)` receives the embedded frame ordinal. + +```python +import av +import torch +import pypaimon.multimodal as pm +from torch.utils.data import DataLoader + + +class SequentialPyAvDecoder: + def __init__(self, stream): + self.container = av.open(stream) + self._reset() + + def _reset(self): + self.frames = iter(self.container.decode(video=0)) + self.next_index = 0 + + def frame(self, index): + if index < self.next_index: + self.container.seek(0) + self._reset() + while self.next_index <= index: + frame = next(self.frames) + self.next_index += 1 + array = frame.to_ndarray(format="rgb24") + return torch.from_numpy(array).permute(2, 0, 1) + + def close(self): + self.container.close() + + +def decode_frame(decoder, frame_index, row): + return decoder.frame(frame_index) + + +collator = pm.VideoFrameCollator( + frames, + video_column="video", + decoder_factory=SequentialPyAvDecoder, + decode_fn=decode_frame, + output_column="frame", + max_open_videos=4, +) + +loader = DataLoader( + dataset, + batch_size=32, + num_workers=4, + shuffle=False, + collate_fn=collator, + persistent_workers=True, +) + +for batch in loader: + # batch["frame"] is [N, C, H, W]. + train(batch["frame"], batch["episode_id"]) +``` + +The example decoder additionally requires `pip install av`; PyAV is not a +PyPaimon dependency. + +Define the decoder class and function at module scope when DataLoader uses the +`spawn` multiprocessing method. For sequential video decoding, keep +`shuffle=False`; row-buffer shuffle can turn monotonic frame access into seeks +and require more open decoder sessions. Paimon's worker sharding is split-level, +so it does not duplicate a split across workers. + ## Load HDF5 `MultimodalConnection.load_from_hdf5` streams one or more local or remote HDF5 @@ -365,6 +565,10 @@ docs.update( ) ``` +On a frame table, updating ordinary columns leaves the configured +`video-frame-field` untouched. Replacing that field requires the specialized +`replace_video()` API described in [Video Frame Storage](#update-frame-rows-and-replace-a-video). + ## Delete `delete` removes rows matched by a SQL-like predicate. It uses the same diff --git a/docs/docs/pypaimon/pytorch.md b/docs/docs/pypaimon/pytorch.md index 9488b20fda1e..ac9f6b8d64eb 100644 --- a/docs/docs/pypaimon/pytorch.md +++ b/docs/docs/pypaimon/pytorch.md @@ -88,6 +88,25 @@ disable a second batching step. Batch streaming does not support `shuffle=True`. Numeric tensors may share read-only Arrow buffers; clone them before in-place mutation. Batch formats currently require `prefetch_concurrency=1`. +## Video frame descriptors + +For a multimodal frame table, use the higher-level scan API: + +```python +dataset = ( + frames.scan() + .select(["episode_id", "state", "action", "video"]) + .to_torch(streaming=True) +) +``` + +The `.video` column yields serialized `VideoFrameDescriptor` values whose +embedded frame ordinals keep frame mapping out of the normal data file. Use +`pypaimon.multimodal.VideoFrameCollator` as the DataLoader `collate_fn` to open +physical video ranges and cache decoder sessions per worker. See +[Multimodal API: Video Frame Storage](multimodal-api#video-frame-storage) +for the write path and a complete decoder example. + ## File Format Metadata Cache Reusable PyArrow Dataset metadata is cached across reads. Configure its estimated diff --git a/docs/generated/core_configuration.html b/docs/generated/core_configuration.html index 4dca9eb1d6b7..472624ae1ba3 100644 --- a/docs/generated/core_configuration.html +++ b/docs/generated/core_configuration.html @@ -84,7 +84,7 @@
blob-field
(none) String - Specifies column names that should be stored as blob type. This is used when you want to treat a BYTES column as a BLOB. Fields listed in blob-descriptor-field or blob-view-field are also treated as BLOB fields. + Specifies column names that should be stored as blob type. This is used when you want to treat a BYTES column as a BLOB. Fields listed in blob-descriptor-field, blob-view-field, or video-frame-field are also treated as BLOB fields.
blob-view-field
@@ -1296,6 +1296,12 @@ Boolean Enables clustering by non-primary key fields. When set to true, the physical sort order of data files is determined by the configured 'clustering.columns' instead of the primary key, optimizing query performance for non-PK columns. + +
pk-fm.index.columns
+ (none) + String + Comma-separated character columns indexed by primary-key FM indexes. +
pk-full-text.index.columns
(none) @@ -1885,6 +1891,12 @@ MemorySize Target size of a vector-store file. Default is the same as TARGET_FILE_SIZE. + +
video-frame-field
+ (none) + String + Specifies one scalar BLOB field whose logical rows are video frames. Complete encoded videos and embedded frame-run indexes are packed into '.video' files. The first version supports append-only data-evolution tables and exact VideoFrameDescriptor input. +
visibility-callback.check-interval
10 s diff --git a/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java b/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java index 0494d9d4cdb8..fc8375d5e37e 100644 --- a/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java +++ b/paimon-api/src/main/java/org/apache/paimon/CoreOptions.java @@ -50,6 +50,7 @@ import java.util.HashMap; import java.util.HashSet; import java.util.LinkedHashMap; +import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; import java.util.Map; @@ -2685,8 +2686,20 @@ public String toString() { .withDescription( "Specifies column names that should be stored as blob type. " + "This is used when you want to treat a BYTES column as a BLOB. " - + "Fields listed in blob-descriptor-field or blob-view-field " - + "are also treated as BLOB fields."); + + "Fields listed in blob-descriptor-field, blob-view-field, " + + "or video-frame-field are also treated as BLOB fields."); + + @Immutable + public static final ConfigOption VIDEO_FRAME_FIELD = + key("video-frame-field") + .stringType() + .noDefaultValue() + .withDescription( + "Specifies one scalar BLOB field whose logical rows are video frames. " + + "Complete encoded videos and embedded frame-run indexes are " + + "packed into '.video' files. The first version supports " + + "append-only data-evolution tables and exact " + + "VideoFrameDescriptor input."); @Immutable public static final ConfigOption BLOB_DESCRIPTOR_FIELD = @@ -3585,6 +3598,11 @@ public Set blobDescriptorField() { return parseCommaSeparatedSet(BLOB_DESCRIPTOR_FIELD); } + /** Resolve the scalar BLOB field stored as frame runs in video pack files. */ + public Set videoFrameField() { + return parseCommaSeparatedSet(VIDEO_FRAME_FIELD); + } + /** * Resolve blob fields that should be stored as serialized view metadata in data files. * @@ -3953,12 +3971,20 @@ public List sequenceField() { } public static List blobField(Map options) { - String string = options.get(BLOB_FIELD.key()); - if (string == null) { - return Collections.emptyList(); - } + Set fields = new LinkedHashSet<>(); + addCommaSeparatedFields(fields, options.get(BLOB_FIELD.key())); + addCommaSeparatedFields(fields, options.get(VIDEO_FRAME_FIELD.key())); + return new ArrayList<>(fields); + } - return Arrays.stream(string.split(",")).map(String::trim).collect(Collectors.toList()); + private static void addCommaSeparatedFields(Set fields, @Nullable String configured) { + if (configured == null) { + return; + } + Arrays.stream(configured.split(",")) + .map(String::trim) + .filter(field -> !field.isEmpty()) + .forEach(fields::add); } public static List blobViewField(Map options) { diff --git a/paimon-common/src/main/java/org/apache/paimon/data/Blob.java b/paimon-common/src/main/java/org/apache/paimon/data/Blob.java index da3821a8996a..049854962e07 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/Blob.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/Blob.java @@ -97,8 +97,12 @@ static Blob fromBytes( return fromView(BlobViewStruct.deserialize(bytes)); } - if (BlobDescriptor.isBlobDescriptor(bytes) || !allowBlobData) { - BlobDescriptor descriptor = BlobDescriptor.deserialize(bytes); + boolean videoFrameDescriptor = VideoFrameDescriptor.isVideoFrameDescriptor(bytes); + if (videoFrameDescriptor || BlobDescriptor.isBlobDescriptor(bytes) || !allowBlobData) { + BlobDescriptor descriptor = + videoFrameDescriptor + ? VideoFrameDescriptor.deserialize(bytes) + : BlobDescriptor.deserialize(bytes); UriReader reader = uriReaderFactory != null ? uriReaderFactory.create(descriptor.uri()) @@ -127,8 +131,12 @@ static Blob fromBytesWithReader( return fromView(BlobViewStruct.deserialize(bytes)); } - if (BlobDescriptor.isBlobDescriptor(bytes) || !allowBlobData) { - BlobDescriptor descriptor = BlobDescriptor.deserialize(bytes); + boolean videoFrameDescriptor = VideoFrameDescriptor.isVideoFrameDescriptor(bytes); + if (videoFrameDescriptor || BlobDescriptor.isBlobDescriptor(bytes) || !allowBlobData) { + BlobDescriptor descriptor = + videoFrameDescriptor + ? VideoFrameDescriptor.deserialize(bytes) + : BlobDescriptor.deserialize(bytes); UriReader reader = uriReader == null ? UriReader.fromFile(fileIO) : uriReader; return fromDescriptor(reader, descriptor); } diff --git a/paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java b/paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java index 241eca120a6a..f802a88470db 100644 --- a/paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java +++ b/paimon-common/src/main/java/org/apache/paimon/data/BlobDescriptor.java @@ -132,6 +132,9 @@ public static BlobDescriptor deserialize(byte[] bytes) { if (bytes == null || bytes.length < Byte.BYTES) { throw invalidPayload("too short"); } + if (VideoFrameDescriptor.isVideoFrameDescriptor(bytes)) { + return VideoFrameDescriptor.deserialize(bytes); + } ByteBuffer buffer = ByteBuffer.wrap(bytes); buffer.order(ByteOrder.LITTLE_ENDIAN); diff --git a/paimon-common/src/main/java/org/apache/paimon/data/VideoFrameDescriptor.java b/paimon-common/src/main/java/org/apache/paimon/data/VideoFrameDescriptor.java new file mode 100644 index 000000000000..198e19629849 --- /dev/null +++ b/paimon-common/src/main/java/org/apache/paimon/data/VideoFrameDescriptor.java @@ -0,0 +1,151 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.data; + +import org.apache.paimon.annotation.Public; + +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.util.Objects; + +import static org.apache.paimon.utils.Preconditions.checkArgument; + +/** A descriptor for one logical frame in a descriptor-backed encoded video payload. */ +@Public +public class VideoFrameDescriptor extends BlobDescriptor { + + private static final long serialVersionUID = 1L; + private static final long MAGIC = 0x564944454F46524DL; // "VIDEOFRM" + private static final byte CURRENT_VERSION = 1; + private static final int FIXED_LENGTH = + Byte.BYTES + Long.BYTES + Integer.BYTES + 3 * Long.BYTES; + + private final long frameIndex; + + public VideoFrameDescriptor(String uri, long offset, long length, long frameIndex) { + super(uri, offset, length); + checkArgument( + frameIndex >= 0, "Video frame index must be non-negative, but was %s.", frameIndex); + this.frameIndex = frameIndex; + } + + public long frameIndex() { + return frameIndex; + } + + /** Returns the physical video identity without the logical frame locator. */ + public BlobDescriptor payloadDescriptor() { + return new BlobDescriptor(uri(), offset(), length()); + } + + @Override + public byte[] serialize() { + byte[] uriBytes = uri().getBytes(StandardCharsets.UTF_8); + ByteBuffer buffer = + ByteBuffer.allocate(FIXED_LENGTH + uriBytes.length).order(ByteOrder.LITTLE_ENDIAN); + buffer.put(CURRENT_VERSION); + buffer.putLong(MAGIC); + buffer.putInt(uriBytes.length); + buffer.put(uriBytes); + buffer.putLong(offset()); + buffer.putLong(length()); + buffer.putLong(frameIndex); + return buffer.array(); + } + + public static VideoFrameDescriptor deserialize(byte[] bytes) { + if (bytes == null || bytes.length < FIXED_LENGTH) { + throw invalidPayload("too short"); + } + + ByteBuffer buffer = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN); + byte version = buffer.get(); + if (version != CURRENT_VERSION) { + throw new UnsupportedOperationException( + "Expecting VideoFrameDescriptor version to be " + + CURRENT_VERSION + + ", but found " + + version + + "."); + } + long magic = buffer.getLong(); + if (magic != MAGIC) { + throw invalidPayload("missing magic header"); + } + int uriLength = buffer.getInt(); + if (uriLength < 0 || buffer.remaining() < uriLength + 3 * Long.BYTES) { + throw invalidPayload("invalid URI length: " + uriLength); + } + + byte[] uriBytes = new byte[uriLength]; + buffer.get(uriBytes); + String uri = new String(uriBytes, StandardCharsets.UTF_8); + long offset = buffer.getLong(); + long length = buffer.getLong(); + long frameIndex = buffer.getLong(); + if (buffer.hasRemaining()) { + throw invalidPayload("trailing bytes"); + } + if (frameIndex < 0) { + throw invalidPayload("negative frame index: " + frameIndex); + } + return new VideoFrameDescriptor(uri, offset, length, frameIndex); + } + + public static boolean isVideoFrameDescriptor(byte[] bytes) { + if (bytes == null || bytes.length < Byte.BYTES + Long.BYTES) { + return false; + } + ByteBuffer buffer = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN); + return buffer.get() == CURRENT_VERSION && buffer.getLong() == MAGIC; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof VideoFrameDescriptor)) { + return false; + } + VideoFrameDescriptor that = (VideoFrameDescriptor) o; + return frameIndex == that.frameIndex + && payloadDescriptor().equals(that.payloadDescriptor()); + } + + @Override + public int hashCode() { + return Objects.hash(payloadDescriptor(), frameIndex); + } + + @Override + public String toString() { + return "VideoFrameDescriptor{" + + "payload=" + + payloadDescriptor() + + ", frameIndex=" + + frameIndex + + '}'; + } + + private static IllegalArgumentException invalidPayload(String message) { + return new IllegalArgumentException("Invalid VideoFrameDescriptor data: " + message); + } +} diff --git a/paimon-common/src/test/java/org/apache/paimon/data/VideoFrameDescriptorTest.java b/paimon-common/src/test/java/org/apache/paimon/data/VideoFrameDescriptorTest.java new file mode 100644 index 000000000000..938987fad0d3 --- /dev/null +++ b/paimon-common/src/test/java/org/apache/paimon/data/VideoFrameDescriptorTest.java @@ -0,0 +1,70 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.data; + +import org.junit.jupiter.api.Test; + +import java.util.Arrays; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Tests for {@link VideoFrameDescriptor}. */ +public class VideoFrameDescriptorTest { + + @Test + public void testRoundTripAndPayloadIdentity() { + VideoFrameDescriptor frame = + new VideoFrameDescriptor("oss://bucket/source.mp4", 17, 103, 42); + + assertThat(VideoFrameDescriptor.isVideoFrameDescriptor(frame.serialize())).isTrue(); + assertThat(BlobDescriptor.isBlobDescriptor(frame.serialize())).isFalse(); + assertThat(BlobDescriptor.deserialize(frame.serialize())).isEqualTo(frame); + assertThat(VideoFrameDescriptor.deserialize(frame.serialize())).isEqualTo(frame); + assertThat(frame.payloadDescriptor()) + .isEqualTo(new BlobDescriptor("oss://bucket/source.mp4", 17, 103)); + + VideoFrameDescriptor next = + new VideoFrameDescriptor("oss://bucket/source.mp4", 17, 103, 43); + assertThat(next).isNotEqualTo(frame); + assertThat(next.payloadDescriptor()).isEqualTo(frame.payloadDescriptor()); + } + + @Test + public void testBlobFromBytesPreservesFrameDescriptor() { + VideoFrameDescriptor expected = new VideoFrameDescriptor("file:/video.mp4", 0, 9, 7); + Blob blob = Blob.fromBytes(expected.serialize(), null, null); + + assertThat(blob).isInstanceOf(BlobRef.class); + assertThat(blob.toDescriptor()).isEqualTo(expected); + } + + @Test + public void testRejectInvalidPayload() { + VideoFrameDescriptor descriptor = new VideoFrameDescriptor("file:/video.mp4", 0, 9, 7); + byte[] trailing = Arrays.copyOf(descriptor.serialize(), descriptor.serialize().length + 1); + + assertThatThrownBy(() -> VideoFrameDescriptor.deserialize(trailing)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("trailing bytes"); + assertThatThrownBy(() -> new VideoFrameDescriptor("file:/video.mp4", 0, 9, -1)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("non-negative"); + } +} diff --git a/paimon-core/src/main/java/org/apache/paimon/append/DedicatedFormatRollingFileWriter.java b/paimon-core/src/main/java/org/apache/paimon/append/DedicatedFormatRollingFileWriter.java index 19c5875fd491..0c9fe566cb7d 100644 --- a/paimon-core/src/main/java/org/apache/paimon/append/DedicatedFormatRollingFileWriter.java +++ b/paimon-core/src/main/java/org/apache/paimon/append/DedicatedFormatRollingFileWriter.java @@ -18,7 +18,11 @@ package org.apache.paimon.append; +import org.apache.paimon.data.Blob; +import org.apache.paimon.data.BlobDescriptor; +import org.apache.paimon.data.BlobRef; import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.VideoFrameDescriptor; import org.apache.paimon.fileindex.FileIndexOptions; import org.apache.paimon.format.FileFormat; import org.apache.paimon.fs.FileIO; @@ -50,6 +54,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Objects; import java.util.Set; import java.util.function.Supplier; import java.util.stream.Collectors; @@ -110,6 +115,7 @@ public class DedicatedFormatRollingFileWriter vectorStoreWriterFactory; private final long targetFileSize; private final long targetFileRowNum; + private final int videoFrameFieldIndex; // State management private final List closedWriters; @@ -123,6 +129,8 @@ public class DedicatedFormatRollingFileWriter vectorStoreWriter; private long recordCount = 0; private long currentFileRecordCount = 0; + private @Nullable BlobDescriptor currentVideoGroup; + private boolean pendingGroupAwareRoll; private boolean closed = false; public DedicatedFormatRollingFileWriter( @@ -150,6 +158,7 @@ public DedicatedFormatRollingFileWriter( targetFileRowNum); this.targetFileSize = targetFileSize; this.targetFileRowNum = targetFileRowNum; + this.videoFrameFieldIndex = videoFrameFieldIndex(writeSchema, context); this.results = new ArrayList<>(); this.closedWriters = new ArrayList<>(); @@ -207,6 +216,7 @@ public DedicatedFormatRollingFileWriter( blobTargetFileSize, context.blobConsumer(), context.blobInlineFields(), + context.videoFrameFields(), context.writeNullOnMissingFile(), context.writeNullOnFetchFailure(), context.blobFetchMetricReporter(), @@ -346,6 +356,10 @@ public DedicatedFormatRollingFileWriter( @Override public void write(InternalRow row) throws IOException { try { + BlobDescriptor nextVideoGroup = videoPayloadDescriptor(row); + if (pendingGroupAwareRoll && !Objects.equals(currentVideoGroup, nextVideoGroup)) { + closeCurrentWriter(); + } if (writerFactory != null && currentWriter == null) { currentWriter = writerFactory.get(); } @@ -366,9 +380,14 @@ public void write(InternalRow row) throws IOException { } recordCount++; currentFileRecordCount++; + currentVideoGroup = nextVideoGroup; if (rollingFile()) { - closeCurrentWriter(); + if (nextVideoGroup == null) { + closeCurrentWriter(); + } else { + pendingGroupAwareRoll = true; + } } } catch (Throwable e) { handleWriteException(e); @@ -478,6 +497,31 @@ private void closeCurrentWriter() throws IOException { // Reset current writer currentWriter = null; currentFileRecordCount = 0; + currentVideoGroup = null; + pendingGroupAwareRoll = false; + } + + private static int videoFrameFieldIndex( + RowType writeSchema, @Nullable BlobFileContext context) { + if (context == null || context.videoFrameFields().isEmpty()) { + return -1; + } + String field = context.videoFrameFields().iterator().next(); + return writeSchema.containsField(field) ? writeSchema.getFieldIndex(field) : -1; + } + + private @Nullable BlobDescriptor videoPayloadDescriptor(InternalRow row) { + if (videoFrameFieldIndex < 0 || row.isNullAt(videoFrameFieldIndex)) { + return null; + } + Blob blob = row.getBlob(videoFrameFieldIndex); + if (blob == null || blob.getClass() != BlobRef.class) { + return null; + } + BlobDescriptor descriptor = blob.toDescriptor(); + return descriptor instanceof VideoFrameDescriptor + ? ((VideoFrameDescriptor) descriptor).payloadDescriptor() + : null; } /** Closes the main writer and returns its metadata. */ diff --git a/paimon-core/src/main/java/org/apache/paimon/append/MultipleBlobFileWriter.java b/paimon-core/src/main/java/org/apache/paimon/append/MultipleBlobFileWriter.java index 27f40f2ecd1b..582b417911b2 100644 --- a/paimon-core/src/main/java/org/apache/paimon/append/MultipleBlobFileWriter.java +++ b/paimon-core/src/main/java/org/apache/paimon/append/MultipleBlobFileWriter.java @@ -22,7 +22,9 @@ import org.apache.paimon.data.BlobFetchMetricReporter; import org.apache.paimon.data.InternalRow; import org.apache.paimon.fileindex.FileIndexOptions; +import org.apache.paimon.format.FileFormat; import org.apache.paimon.format.blob.BlobFileFormat; +import org.apache.paimon.format.blob.VideoFileFormat; import org.apache.paimon.fs.FileIO; import org.apache.paimon.io.DataFileMeta; import org.apache.paimon.io.DataFilePathFactory; @@ -30,7 +32,6 @@ import org.apache.paimon.io.RollingFileWriter; import org.apache.paimon.io.RollingFileWriterImpl; import org.apache.paimon.io.RowDataFileWriter; -import org.apache.paimon.io.SingleFileWriter; import org.apache.paimon.manifest.FileSource; import org.apache.paimon.statistics.NoneSimpleColStatsCollector; import org.apache.paimon.statistics.SimpleColStatsCollector; @@ -66,6 +67,7 @@ public MultipleBlobFileWriter( long targetFileSize, @Nullable BlobConsumer blobConsumer, Set blobInlineFields, + Set videoFrameFields, boolean writeNullOnMissingFile, boolean writeNullOnFetchFailure, BlobFetchMetricReporter blobFetchMetricReporter, @@ -73,36 +75,60 @@ public MultipleBlobFileWriter( RowType blobRowType = new RowType(fieldsInBlobFile(writeSchema, blobInlineFields)); this.blobWriters = new ArrayList<>(); for (String blobFieldName : blobRowType.getFieldNames()) { - BlobFileFormat blobFileFormat = new BlobFileFormat(false, copyBufferSize); - blobFileFormat.setWriteConsumer(blobConsumer); - blobFileFormat.setWriteNullOnMissingFile(writeNullOnMissingFile); - blobFileFormat.setWriteNullOnFetchFailure(writeNullOnFetchFailure); - blobFileFormat.setBlobFetchMetricReporter(blobFetchMetricReporter); + boolean video = videoFrameFields.contains(blobFieldName); + FileFormat blobFileFormat; + if (video) { + if (blobConsumer != null) { + throw new IllegalArgumentException( + "BlobConsumer is not supported for video frame field '" + + blobFieldName + + "'."); + } + VideoFileFormat format = new VideoFileFormat(copyBufferSize); + format.setWriteNullOnMissingFile(writeNullOnMissingFile); + format.setWriteNullOnFetchFailure(writeNullOnFetchFailure); + format.setBlobFetchMetricReporter(blobFetchMetricReporter); + blobFileFormat = format; + } else { + BlobFileFormat format = new BlobFileFormat(false, copyBufferSize); + format.setWriteConsumer(blobConsumer); + format.setWriteNullOnMissingFile(writeNullOnMissingFile); + format.setWriteNullOnFetchFailure(writeNullOnFetchFailure); + format.setBlobFetchMetricReporter(blobFetchMetricReporter); + blobFileFormat = format; + } + RowType fieldType = writeSchema.project(blobFieldName); + Supplier writerFactory = + () -> + new RowDataFileWriter( + fileIO, + RollingFileWriter.createFileWriterContext( + blobFileFormat, + fieldType, + new SimpleColStatsCollector.Factory[] { + NoneSimpleColStatsCollector::new + }, + "none"), + video ? pathFactory.newVideoPath() : pathFactory.newBlobPath(), + fieldType, + schemaId, + seqNumCounterSupplier, + new FileIndexOptions(), + fileSource, + asyncFileWrite, + statsDenseStore, + pathFactory.isExternalPath(), + singletonList(blobFieldName), + null, + null); + RollingFileWriter rollingWriter = + video + ? new VideoRollingFileWriter<>(writerFactory, targetFileSize) + : new RollingFileWriterImpl<>( + writerFactory, targetFileSize, Long.MAX_VALUE); blobWriters.add( new BlobProjectedFileWriter( - () -> - new RowDataFileWriter( - fileIO, - RollingFileWriter.createFileWriterContext( - blobFileFormat, - writeSchema.project(blobFieldName), - new SimpleColStatsCollector.Factory[] { - NoneSimpleColStatsCollector::new - }, - "none"), - pathFactory.newBlobPath(), - writeSchema.project(blobFieldName), - schemaId, - seqNumCounterSupplier, - new FileIndexOptions(), - fileSource, - asyncFileWrite, - statsDenseStore, - pathFactory.isExternalPath(), - singletonList(blobFieldName), - null, - null), - targetFileSize, + rollingWriter, writeSchema.projectIndexes(singletonList(blobFieldName)))); } } @@ -137,21 +163,27 @@ public List result() throws IOException { List drainAbortExecutors() { List abortExecutors = new ArrayList<>(); for (BlobProjectedFileWriter blobWriter : blobWriters) { - abortExecutors.addAll(blobWriter.writer().drainAbortExecutors()); + abortExecutors.addAll(blobWriter.drainAbortExecutors()); } return abortExecutors; } private static class BlobProjectedFileWriter extends ProjectedFileWriter< - RollingFileWriterImpl, List> { + RollingFileWriter, List> { public BlobProjectedFileWriter( - Supplier> writerFactory, - long targetFileSize, - int[] projection) { - super( - new RollingFileWriterImpl<>(writerFactory, targetFileSize, Long.MAX_VALUE), - projection); + RollingFileWriter writer, int[] projection) { + super(writer, projection); + } + + @SuppressWarnings("unchecked") + private List drainAbortExecutors() { + RollingFileWriter writer = writer(); + if (writer instanceof RollingFileWriterImpl) { + return ((RollingFileWriterImpl) writer) + .drainAbortExecutors(); + } + return ((VideoRollingFileWriter) writer).drainAbortExecutors(); } } } diff --git a/paimon-core/src/main/java/org/apache/paimon/append/VideoRollingFileWriter.java b/paimon-core/src/main/java/org/apache/paimon/append/VideoRollingFileWriter.java new file mode 100644 index 000000000000..3c00f34ef919 --- /dev/null +++ b/paimon-core/src/main/java/org/apache/paimon/append/VideoRollingFileWriter.java @@ -0,0 +1,176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.append; + +import org.apache.paimon.data.Blob; +import org.apache.paimon.data.BlobDescriptor; +import org.apache.paimon.data.BlobRef; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.VideoFrameDescriptor; +import org.apache.paimon.io.BundleRecords; +import org.apache.paimon.io.FileWriterAbortExecutor; +import org.apache.paimon.io.RollingFileWriter; +import org.apache.paimon.io.SingleFileWriter; +import org.apache.paimon.utils.Preconditions; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.annotation.Nullable; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.function.Supplier; + +/** + * Rolls video pack files only between complete physical-video groups. + * + *

The target size is soft: after it is reached, immediately following frames backed by the same + * encoded video remain in the current file. A different payload, NULL, or placeholder starts a new + * file. + */ +class VideoRollingFileWriter implements RollingFileWriter { + + private static final Logger LOG = LoggerFactory.getLogger(VideoRollingFileWriter.class); + + private final Supplier> writerFactory; + private final long targetFileSize; + private final List closedWriters = new ArrayList<>(); + private final List results = new ArrayList<>(); + + private @Nullable SingleFileWriter currentWriter; + private @Nullable BlobDescriptor currentVideo; + private long recordCount; + private boolean pendingRoll; + private boolean closed; + + VideoRollingFileWriter( + Supplier> writerFactory, + long targetFileSize) { + this.writerFactory = writerFactory; + this.targetFileSize = targetFileSize; + } + + @Override + public void write(InternalRow row) throws IOException { + try { + BlobDescriptor nextVideo = payloadDescriptor(row); + if (currentWriter != null && pendingRoll && !Objects.equals(currentVideo, nextVideo)) { + closeCurrentWriter(); + } + if (currentWriter == null) { + currentWriter = writerFactory.get(); + } + + currentWriter.write(row); + recordCount++; + currentVideo = nextVideo; + if (currentWriter.reachTargetSize( + recordCount % CHECK_ROLLING_RECORD_CNT == 0, targetFileSize)) { + pendingRoll = true; + } + } catch (Throwable e) { + LOG.warn( + "Exception occurs when writing video file {}. Cleaning up.", + currentWriter == null ? null : currentWriter.path(), + e); + abort(); + throw e; + } + } + + @Override + public void writeBundle(BundleRecords records) throws IOException { + for (InternalRow row : records) { + write(row); + } + } + + @Override + public long recordCount() { + return recordCount; + } + + @Override + public void abort() { + if (currentWriter != null) { + currentWriter.abort(); + currentWriter = null; + } + for (FileWriterAbortExecutor abortExecutor : closedWriters) { + abortExecutor.abort(); + } + } + + @Override + public List result() { + Preconditions.checkState(closed, "Cannot access the results unless close all writers."); + return results; + } + + List drainAbortExecutors() { + Preconditions.checkState(closed, "Cannot drain abort executors unless close all writers."); + List result = new ArrayList<>(closedWriters); + closedWriters.clear(); + return result; + } + + @Override + public void close() throws IOException { + if (closed) { + return; + } + try { + closeCurrentWriter(); + } catch (IOException e) { + abort(); + throw e; + } finally { + closed = true; + } + } + + private void closeCurrentWriter() throws IOException { + if (currentWriter == null) { + return; + } + currentWriter.close(); + currentWriter.abortExecutor().ifPresent(closedWriters::add); + results.add(currentWriter.result()); + currentWriter = null; + currentVideo = null; + pendingRoll = false; + } + + private static @Nullable BlobDescriptor payloadDescriptor(InternalRow row) { + if (row.isNullAt(0)) { + return null; + } + Blob blob = row.getBlob(0); + if (blob == null || blob.getClass() != BlobRef.class) { + return null; + } + BlobDescriptor descriptor = blob.toDescriptor(); + return descriptor instanceof VideoFrameDescriptor + ? ((VideoFrameDescriptor) descriptor).payloadDescriptor() + : null; + } +} diff --git a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionBlobCompactTask.java b/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionBlobCompactTask.java index 9056c7b83088..9dbedabe3474 100644 --- a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionBlobCompactTask.java +++ b/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionBlobCompactTask.java @@ -23,7 +23,9 @@ import org.apache.paimon.data.BinaryRow; import org.apache.paimon.data.InternalRow; import org.apache.paimon.fileindex.FileIndexOptions; +import org.apache.paimon.format.FileFormat; import org.apache.paimon.format.blob.BlobFileFormat; +import org.apache.paimon.format.blob.VideoFileFormat; import org.apache.paimon.io.DataFileMeta; import org.apache.paimon.io.DataFilePathFactory; import org.apache.paimon.io.FileWriter; @@ -128,7 +130,11 @@ private FileWriter createBlobFileWriter( RowType blobWriteType, String blobFieldName, DataFilePathFactory pathFactory) { - BlobFileFormat blobFileFormat = new BlobFileFormat(false, options.blobCopyBufferSize()); + boolean video = options.videoFrameField().contains(blobFieldName); + FileFormat blobFileFormat = + video + ? new VideoFileFormat(options.blobCopyBufferSize()) + : new BlobFileFormat(false, options.blobCopyBufferSize()); return new RowDataFileWriter( table.fileIO(), RollingFileWriter.createFileWriterContext( @@ -136,7 +142,7 @@ private FileWriter createBlobFileWriter( blobWriteType, new SimpleColStatsCollector.Factory[] {NoneSimpleColStatsCollector::new}, "none"), - pathFactory.newBlobPath(), + video ? pathFactory.newVideoPath() : pathFactory.newBlobPath(), blobWriteType, table.schema().id(), () -> new LongCounter(0), diff --git a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionRowIdAssignmentPlanner.java b/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionRowIdAssignmentPlanner.java index 939daa6ff652..71e447c1d820 100644 --- a/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionRowIdAssignmentPlanner.java +++ b/paimon-core/src/main/java/org/apache/paimon/append/dataevolution/DataEvolutionRowIdAssignmentPlanner.java @@ -77,6 +77,7 @@ final class DataEvolutionRowIdAssignmentPlanner { private static final BinaryString ROW_ID_FIELD = BinaryString.fromString(SpecialFields.ROW_ID.name()); private static final BinaryString BLOB_FILE_SUFFIX = BinaryString.fromString(".blob"); + private static final BinaryString VIDEO_FILE_SUFFIX = BinaryString.fromString(".video"); private static final BinaryString VECTOR_FILE_MARKER = BinaryString.fromString(".vector."); private static final Projection ADD_IDENTIFIER_PROJECTION = manifestProjection( @@ -455,7 +456,7 @@ private static void readRowRange( } private static int fileOrder(BinaryString fileName) { - if (fileName.endsWith(BLOB_FILE_SUFFIX)) { + if (fileName.endsWith(BLOB_FILE_SUFFIX) || fileName.endsWith(VIDEO_FILE_SUFFIX)) { return 1; } if (fileName.contains(VECTOR_FILE_MARKER)) { diff --git a/paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java b/paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java index 4b8fc56e65a6..33068859dc39 100644 --- a/paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java +++ b/paimon-core/src/main/java/org/apache/paimon/io/DataFilePathFactory.java @@ -86,6 +86,10 @@ public Path newBlobPath() { return newPathFromName(newFileName(dataFilePrefix, ".blob")); } + public Path newVideoPath() { + return newPathFromName(newFileName(dataFilePrefix, ".video")); + } + public Path newChangelogPath() { return newPath(changelogFilePrefix); } diff --git a/paimon-core/src/main/java/org/apache/paimon/operation/BlobFileContext.java b/paimon-core/src/main/java/org/apache/paimon/operation/BlobFileContext.java index b1b50e157b82..b0a443e3e6cc 100644 --- a/paimon-core/src/main/java/org/apache/paimon/operation/BlobFileContext.java +++ b/paimon-core/src/main/java/org/apache/paimon/operation/BlobFileContext.java @@ -34,6 +34,7 @@ public class BlobFileContext { private final Set blobDescriptorFields; private final Set blobInlineFields; + private final Set videoFrameFields; private final boolean writeNullOnMissingFile; private final boolean writeNullOnFetchFailure; private final int copyBufferSize; @@ -44,11 +45,13 @@ public class BlobFileContext { private BlobFileContext( Set blobDescriptorFields, Set blobInlineFields, + Set videoFrameFields, boolean writeNullOnMissingFile, boolean writeNullOnFetchFailure, int copyBufferSize) { this.blobDescriptorFields = blobDescriptorFields; this.blobInlineFields = blobInlineFields; + this.videoFrameFields = videoFrameFields; this.writeNullOnMissingFile = writeNullOnMissingFile; this.writeNullOnFetchFailure = writeNullOnFetchFailure; this.copyBufferSize = copyBufferSize; @@ -74,6 +77,7 @@ public static BlobFileContext create(RowType rowType, CoreOptions options) { return new BlobFileContext( descriptorFields, inlineFields, + options.videoFrameField(), options.blobWriteNullOnMissingFile(), options.blobWriteNullOnFetchFailure(), options.blobCopyBufferSize()); @@ -105,6 +109,10 @@ public Set blobInlineFields() { return blobInlineFields; } + public Set videoFrameFields() { + return videoFrameFields; + } + @Nullable public BlobConsumer blobConsumer() { return blobConsumer; diff --git a/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java b/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java index cc0ee88ad702..4b3db415bb7e 100644 --- a/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java +++ b/paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java @@ -238,6 +238,8 @@ public static void validateTableSchema(TableSchema schema, Set dynamicOp Set blobDescriptorFields = validateBlobDescriptorFields(tableRowType, options); Set blobViewFields = validateBlobViewFields(tableRowType, options, blobDescriptorFields); + validateVideoFrameFields( + schema, tableRowType, options, blobDescriptorFields, blobViewFields); validatePrimaryKeyBlobKeyConfiguration(schema, options); validatePrimaryKeyBlobConfiguration(schema, options); Set blobInlineFields = new HashSet<>(blobDescriptorFields); @@ -1599,6 +1601,45 @@ private static Set validateBlobViewFields( return configured; } + private static void validateVideoFrameFields( + TableSchema schema, + RowType rowType, + CoreOptions options, + Set blobDescriptorFields, + Set blobViewFields) { + Set configured = options.videoFrameField(); + checkArgument( + configured.size() <= 1, + "'%s' currently supports exactly one field, but found %s.", + CoreOptions.VIDEO_FRAME_FIELD.key(), + configured); + for (String field : configured) { + checkArgument( + rowType.containsField(field) + && rowType.getTypeAt(rowType.getFieldIndex(field)).getTypeRoot() + == DataTypeRoot.BLOB, + "Field '%s' in '%s' must be a scalar BLOB field in table schema.", + field, + CoreOptions.VIDEO_FRAME_FIELD.key()); + checkArgument( + !blobDescriptorFields.contains(field), + "Field '%s' in '%s' can not also be in '%s'.", + field, + CoreOptions.VIDEO_FRAME_FIELD.key(), + CoreOptions.BLOB_DESCRIPTOR_FIELD.key()); + checkArgument( + !blobViewFields.contains(field), + "Field '%s' in '%s' can not also be in '%s'.", + field, + CoreOptions.VIDEO_FRAME_FIELD.key(), + CoreOptions.BLOB_VIEW_FIELD.key()); + } + checkArgument( + configured.isEmpty() || schema.primaryKeys().isEmpty(), + "'%s' only supports append-only tables.", + CoreOptions.VIDEO_FRAME_FIELD.key()); + } + private static void validatePrimaryKeyBlobConfiguration( TableSchema schema, CoreOptions options) { if (schema.primaryKeys().isEmpty()) { diff --git a/paimon-core/src/test/java/org/apache/paimon/CoreOptionsTest.java b/paimon-core/src/test/java/org/apache/paimon/CoreOptionsTest.java index 93bde01a2f1b..32ab4155e730 100644 --- a/paimon-core/src/test/java/org/apache/paimon/CoreOptionsTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/CoreOptionsTest.java @@ -270,6 +270,16 @@ public void testBlobCopyBufferSize() { .hasMessageContaining("blob.copy-buffer-size"); } + @Test + public void testVideoFrameFieldIsRecognizedAsBlobField() { + Options options = new Options(); + options.set(CoreOptions.BLOB_FIELD, "image, video"); + options.set(CoreOptions.VIDEO_FRAME_FIELD, "video"); + + assertThat(CoreOptions.blobField(options.toMap())).containsExactly("image", "video"); + assertThat(new CoreOptions(options).videoFrameField()).containsExactly("video"); + } + @Test public void testLocalKvDbBlockSize() { Options conf = new Options(); diff --git a/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java b/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java index caa268e0ca82..b910de8369bd 100644 --- a/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/append/BlobTableTest.java @@ -22,6 +22,7 @@ import org.apache.paimon.append.dataevolution.DataEvolutionCompactCoordinator; import org.apache.paimon.append.dataevolution.DataEvolutionCompactTask; import org.apache.paimon.catalog.Identifier; +import org.apache.paimon.data.BinaryRow; import org.apache.paimon.data.BinaryString; import org.apache.paimon.data.Blob; import org.apache.paimon.data.BlobArrayPlaceholder; @@ -37,10 +38,13 @@ import org.apache.paimon.data.InternalArray; import org.apache.paimon.data.InternalMap; import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.VideoFrameDescriptor; import org.apache.paimon.data.serializer.InternalRowSerializer; +import org.apache.paimon.format.blob.VideoFileMeta; import org.apache.paimon.fs.FileIO; import org.apache.paimon.fs.Path; import org.apache.paimon.fs.SeekableInputStream; +import org.apache.paimon.fs.local.LocalFileIO; import org.apache.paimon.io.DataFileMeta; import org.apache.paimon.manifest.ManifestEntry; import org.apache.paimon.operation.DataEvolutionSplitRead; @@ -1437,6 +1441,160 @@ public void testBlobCompactionSingleField() throws Exception { assertThat(tasks2.stream().anyMatch(task -> task.type() == BLOB)).isFalse(); } + @Test + public void testVideoRollingAndCompaction() throws Exception { + Schema.Builder schemaBuilder = Schema.newBuilder(); + schemaBuilder.column("id", DataTypes.INT()); + schemaBuilder.column("video", DataTypes.BLOB()); + schemaBuilder.option(CoreOptions.TARGET_FILE_SIZE.key(), "1 GB"); + schemaBuilder.option(CoreOptions.BLOB_TARGET_FILE_SIZE.key(), "1 GB"); + schemaBuilder.option(CoreOptions.TARGET_FILE_ROW_NUM.key(), "1"); + schemaBuilder.option(CoreOptions.COMPACTION_MIN_FILE_NUM.key(), "2"); + schemaBuilder.option(CoreOptions.ROW_TRACKING_ENABLED.key(), "true"); + schemaBuilder.option(CoreOptions.DATA_EVOLUTION_ENABLED.key(), "true"); + schemaBuilder.option(CoreOptions.VIDEO_FRAME_FIELD.key(), "video"); + catalog.createTable(identifier(), schemaBuilder.build(), true); + + byte[] firstBytes = "first-video".getBytes(); + byte[] secondBytes = "second-video".getBytes(); + java.nio.file.Path firstSource = tempPath.resolve("first-source.mp4"); + java.nio.file.Path secondSource = tempPath.resolve("second-source.mp4"); + java.nio.file.Files.write(firstSource, firstBytes); + java.nio.file.Files.write(secondSource, secondBytes); + UriReader sourceReader = UriReader.fromFile(LocalFileIO.create()); + String firstUri = new Path(firstSource.toUri()).toString(); + String secondUri = new Path(secondSource.toUri()).toString(); + + writeRows( + getTableDefault(), + Arrays.asList( + GenericRow.of( + 0, + Blob.fromDescriptor( + sourceReader, + new VideoFrameDescriptor( + firstUri, 0, firstBytes.length, 0))), + GenericRow.of( + 1, + Blob.fromDescriptor( + sourceReader, + new VideoFrameDescriptor( + firstUri, 0, firstBytes.length, 1))), + GenericRow.of( + 2, + Blob.fromDescriptor( + sourceReader, + new VideoFrameDescriptor( + firstUri, 0, firstBytes.length, 2))), + GenericRow.of( + 3, + Blob.fromDescriptor( + sourceReader, + new VideoFrameDescriptor( + secondUri, 0, secondBytes.length, 0))), + GenericRow.of( + 4, + Blob.fromDescriptor( + sourceReader, + new VideoFrameDescriptor( + secondUri, 0, secondBytes.length, 1))))); + + FileStoreTable table = getTableDefault(); + List videoFiles = liveVideoFiles(table); + assertThat(videoFiles.size()).isEqualTo(2); + assertThat( + videoFiles.stream() + .map(DataFileMeta::rowCount) + .sorted() + .collect(Collectors.toList())) + .isEqualTo(Arrays.asList(2L, 3L)); + for (DataFileMeta videoFile : videoFiles) { + Path path = + table.store() + .pathFactory() + .createDataFilePathFactory(BinaryRow.EMPTY_ROW, 0) + .toPath(videoFile); + try (SeekableInputStream in = table.fileIO().newInputStream(path)) { + VideoFileMeta meta = new VideoFileMeta(in, table.fileIO().getFileSize(path), null); + assertThat(meta.physicalVideoNumber()).isOne(); + assertThat(meta.runNumber()).isOne(); + assertThat(meta.recordNumber()).isEqualTo(videoFile.rowCount()); + } + } + assertVideoRows(table, firstBytes, secondBytes); + + DataEvolutionCompactCoordinator coordinator = + new DataEvolutionCompactCoordinator( + table, true, false, table.latestSnapshot().get()); + List tasks = coordinator.plan(); + List compactMessages = new ArrayList<>(); + int blobTaskCount = 0; + for (DataEvolutionCompactTask task : tasks) { + if (task.type() == BLOB) { + blobTaskCount++; + } + compactMessages.add(task.doCompact(table, commitUser)); + } + assertThat(blobTaskCount).isEqualTo(1); + commitDefault(compactMessages); + + table = getTableDefault(); + videoFiles = liveVideoFiles(table); + assertThat(videoFiles.size()).isEqualTo(1); + DataFileMeta compacted = videoFiles.get(0); + Path compactedPath = + table.store() + .pathFactory() + .createDataFilePathFactory(BinaryRow.EMPTY_ROW, 0) + .toPath(compacted); + try (SeekableInputStream in = table.fileIO().newInputStream(compactedPath)) { + VideoFileMeta meta = + new VideoFileMeta(in, table.fileIO().getFileSize(compactedPath), null); + assertThat(meta.recordNumber()).isEqualTo(5); + assertThat(meta.physicalVideoNumber()).isEqualTo(2); + assertThat(meta.runNumber()).isEqualTo(2); + } + assertVideoRows(table, firstBytes, secondBytes); + } + + private List liveVideoFiles(FileStoreTable table) { + return table.store().newScan().plan().files().stream() + .map(ManifestEntry::file) + .filter(file -> file.fileName().endsWith(".video")) + .collect(Collectors.toList()); + } + + private void assertVideoRows(FileStoreTable table, byte[] firstBytes, byte[] secondBytes) + throws Exception { + Map readOptions = new HashMap<>(); + readOptions.put(CoreOptions.BLOB_AS_DESCRIPTOR.key(), "true"); + Table descriptorTable = table.copy(readOptions); + ReadBuilder readBuilder = descriptorTable.newReadBuilder(); + List rows = new ArrayList<>(); + InternalRowSerializer serializer = new InternalRowSerializer(descriptorTable.rowType()); + try (RecordReader reader = + readBuilder.newRead().createReader(readBuilder.newScan().plan())) { + reader.forEachRemaining(row -> rows.add(serializer.copy(row))); + } + rows.sort((left, right) -> Integer.compare(left.getInt(0), right.getInt(0))); + assertThat(rows.size()).isEqualTo(5); + assertThat(rows.get(0).getBlob(1).toData()).isEqualTo(firstBytes); + VideoFrameDescriptor frame0 = (VideoFrameDescriptor) rows.get(0).getBlob(1).toDescriptor(); + VideoFrameDescriptor frame1 = (VideoFrameDescriptor) rows.get(1).getBlob(1).toDescriptor(); + VideoFrameDescriptor frame2 = (VideoFrameDescriptor) rows.get(2).getBlob(1).toDescriptor(); + assertThat(frame0.frameIndex()).isZero(); + assertThat(frame1.frameIndex()).isOne(); + assertThat(frame2.frameIndex()).isEqualTo(2); + assertThat(frame1.payloadDescriptor()).isEqualTo(frame0.payloadDescriptor()); + assertThat(frame2.payloadDescriptor()).isEqualTo(frame0.payloadDescriptor()); + assertThat(rows.get(3).getBlob(1).toData()).isEqualTo(secondBytes); + VideoFrameDescriptor frame3 = (VideoFrameDescriptor) rows.get(3).getBlob(1).toDescriptor(); + VideoFrameDescriptor frame4 = (VideoFrameDescriptor) rows.get(4).getBlob(1).toDescriptor(); + assertThat(frame3.frameIndex()).isZero(); + assertThat(frame4.frameIndex()).isOne(); + assertThat(frame4.payloadDescriptor()).isEqualTo(frame3.payloadDescriptor()); + } + @Test public void testPartitionedTableWithBlob() throws Exception { Schema.Builder schemaBuilder = Schema.newBuilder(); diff --git a/paimon-core/src/test/java/org/apache/paimon/io/DataFilePathFactoryTest.java b/paimon-core/src/test/java/org/apache/paimon/io/DataFilePathFactoryTest.java index 266e6b4dc900..d2d0eb8f366e 100644 --- a/paimon-core/src/test/java/org/apache/paimon/io/DataFilePathFactoryTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/io/DataFilePathFactoryTest.java @@ -96,6 +96,23 @@ public void testWithPartition() { .toString()); } + @Test + public void testVideoPathAndFormatIdentifier() { + DataFilePathFactory pathFactory = + new DataFilePathFactory( + new Path(tempDir + "/bucket-123"), + CoreOptions.FILE_FORMAT.defaultValue(), + CoreOptions.DATA_FILE_PREFIX.defaultValue(), + CoreOptions.CHANGELOG_FILE_PREFIX.defaultValue(), + CoreOptions.FILE_SUFFIX_INCLUDE_COMPRESSION.defaultValue(), + CoreOptions.FILE_COMPRESSION.defaultValue(), + null); + + Path video = pathFactory.newVideoPath(); + assertThat(video.getName()).endsWith(".video"); + assertThat(DataFilePathFactory.formatIdentifier(video.getName())).isEqualTo("video"); + } + @Test public void testEntropyInjectWithNoPartition() { EntropyInjectExternalPathProvider externalPathProvider = diff --git a/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java b/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java index 5f5aa2fce0eb..471d53670d5f 100644 --- a/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/schema/SchemaValidationTest.java @@ -459,6 +459,60 @@ public void testPrimaryKeyBlobFileField() { assertThatCode(() -> validateTableSchema(schema)).doesNotThrowAnyException(); } + @Test + public void testVideoFrameFieldValidation() { + List fields = + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField(1, "video", DataTypes.BLOB()), + new DataField(2, "other_video", DataTypes.BLOB())); + Map options = new HashMap<>(); + options.put(BUCKET.key(), "-1"); + options.put(CoreOptions.ROW_TRACKING_ENABLED.key(), "true"); + options.put(CoreOptions.DATA_EVOLUTION_ENABLED.key(), "true"); + options.put(CoreOptions.VIDEO_FRAME_FIELD.key(), "video"); + + TableSchema schema = new TableSchema(1, fields, 10, emptyList(), emptyList(), options, ""); + assertThatCode(() -> validateTableSchema(schema)).doesNotThrowAnyException(); + + options.put(CoreOptions.VIDEO_FRAME_FIELD.key(), "video,other_video"); + assertThatThrownBy(() -> validateTableSchema(schema)) + .hasMessageContaining("currently supports exactly one field"); + + options.put(CoreOptions.VIDEO_FRAME_FIELD.key(), "video"); + options.put(CoreOptions.BLOB_DESCRIPTOR_FIELD.key(), "video"); + assertThatThrownBy(() -> validateTableSchema(schema)) + .hasMessageContaining("video-frame-field") + .hasMessageContaining("blob-descriptor-field"); + } + + @Test + public void testVideoFrameRejectsNestedAndPrimaryKeyFields() { + Map options = new HashMap<>(); + options.put(BUCKET.key(), "-1"); + options.put(CoreOptions.ROW_TRACKING_ENABLED.key(), "true"); + options.put(CoreOptions.DATA_EVOLUTION_ENABLED.key(), "true"); + options.put(CoreOptions.VIDEO_FRAME_FIELD.key(), "video"); + List nestedFields = + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField(1, "video", DataTypes.ARRAY(DataTypes.BLOB()))); + TableSchema nested = + new TableSchema(1, nestedFields, 10, emptyList(), emptyList(), options, ""); + assertThatThrownBy(() -> validateTableSchema(nested)) + .hasMessageContaining("must be a scalar BLOB field"); + + options.put(BUCKET.key(), "1"); + List scalarFields = + Arrays.asList( + new DataField(0, "id", DataTypes.INT()), + new DataField(1, "video", DataTypes.BLOB())); + TableSchema primaryKey = + new TableSchema(1, scalarFields, 10, emptyList(), singletonList("id"), options, ""); + assertThatThrownBy(() -> validateTableSchema(primaryKey)) + .hasMessageContaining("only supports append-only tables"); + } + @Test public void testPrimaryKeyInlineBlobDoesNotTriggerManagedRestrictions() { // Partial-update supports scalar blob-descriptor-field, managed blob-field, and diff --git a/paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFileFormat.java b/paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFileFormat.java index f00877d3494f..902b30c4478b 100644 --- a/paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFileFormat.java +++ b/paimon-format/src/main/java/org/apache/paimon/format/blob/BlobFileFormat.java @@ -65,7 +65,8 @@ public BlobFileFormat(boolean blobAsDescriptor, int copyBufferSize) { } public static boolean isBlobFile(String fileName) { - return fileName.endsWith("." + BlobFileFormatFactory.IDENTIFIER); + return fileName.endsWith("." + BlobFileFormatFactory.IDENTIFIER) + || fileName.endsWith("." + VideoFileFormatFactory.IDENTIFIER); } public void setWriteNullOnMissingFile(boolean writeNullOnMissingFile) { diff --git a/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileFormat.java b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileFormat.java new file mode 100644 index 000000000000..226ee4b14d52 --- /dev/null +++ b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileFormat.java @@ -0,0 +1,158 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.format.blob; + +import org.apache.paimon.data.BlobFetchMetricReporter; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.format.EmptyStatsExtractor; +import org.apache.paimon.format.FileFormat; +import org.apache.paimon.format.FormatReaderFactory; +import org.apache.paimon.format.FormatWriter; +import org.apache.paimon.format.FormatWriterFactory; +import org.apache.paimon.format.SimpleStatsExtractor; +import org.apache.paimon.fs.FileIO; +import org.apache.paimon.fs.Path; +import org.apache.paimon.fs.PositionOutputStream; +import org.apache.paimon.fs.SeekableInputStream; +import org.apache.paimon.predicate.Predicate; +import org.apache.paimon.reader.FileRecordReader; +import org.apache.paimon.statistics.SimpleColStatsCollector; +import org.apache.paimon.types.DataTypeRoot; +import org.apache.paimon.types.RowType; +import org.apache.paimon.utils.IOUtils; +import org.apache.paimon.utils.Preconditions; + +import javax.annotation.Nullable; + +import java.io.IOException; +import java.util.List; +import java.util.Optional; + +import static org.apache.paimon.utils.Preconditions.checkArgument; + +/** File format that packs complete encoded videos and maps logical rows to frame ordinals. */ +public class VideoFileFormat extends FileFormat { + + private final int copyBufferSize; + private boolean writeNullOnMissingFile; + private boolean writeNullOnFetchFailure; + private BlobFetchMetricReporter blobFetchMetricReporter = BlobFetchMetricReporter.NOOP; + + public VideoFileFormat(int copyBufferSize) { + super(VideoFileFormatFactory.IDENTIFIER); + this.copyBufferSize = copyBufferSize; + } + + public void setWriteNullOnMissingFile(boolean writeNullOnMissingFile) { + this.writeNullOnMissingFile = writeNullOnMissingFile; + } + + public void setWriteNullOnFetchFailure(boolean writeNullOnFetchFailure) { + this.writeNullOnFetchFailure = writeNullOnFetchFailure; + } + + public void setBlobFetchMetricReporter(BlobFetchMetricReporter blobFetchMetricReporter) { + this.blobFetchMetricReporter = blobFetchMetricReporter; + } + + @Override + public FormatReaderFactory createReaderFactory( + RowType dataSchemaRowType, + RowType projectedRowType, + @Nullable List filters) { + return new VideoFormatReaderFactory(projectedRowType); + } + + @Override + public FormatWriterFactory createWriterFactory(RowType type) { + validateDataFields(type); + return new VideoFormatWriterFactory(type); + } + + @Override + public void validateDataFields(RowType rowType) { + checkArgument( + rowType.getFieldCount() == 1 + && rowType.getTypeAt(0).getTypeRoot() == DataTypeRoot.BLOB, + "VideoFileFormat only supports one scalar BLOB field."); + } + + @Override + public Optional createStatsExtractor( + RowType type, SimpleColStatsCollector.Factory[] statsCollectors) { + return Optional.of(new EmptyStatsExtractor()); + } + + private class VideoFormatWriterFactory implements FormatWriterFactory { + + private final RowType type; + + private VideoFormatWriterFactory(RowType type) { + this.type = type; + } + + @Override + public FormatWriter create(PositionOutputStream out, String compression) { + return new VideoFormatWriter( + out, + type, + writeNullOnMissingFile, + writeNullOnFetchFailure, + blobFetchMetricReporter, + copyBufferSize); + } + } + + private static class VideoFormatReaderFactory implements FormatReaderFactory { + + private final int fieldCount; + private final int blobIndex; + + private VideoFormatReaderFactory(RowType projectedRowType) { + this.fieldCount = projectedRowType.getFieldCount(); + this.blobIndex = findBlobFieldIndex(projectedRowType); + Preconditions.checkState( + blobIndex >= 0, + "Read type of a video format does not contain a scalar BLOB field."); + } + + @Override + public FileRecordReader createReader(Context context) throws IOException { + FileIO fileIO = context.fileIO(); + Path filePath = context.filePath(); + SeekableInputStream in = fileIO.newInputStream(filePath); + VideoFileMeta fileMeta; + try { + fileMeta = new VideoFileMeta(in, context.fileSize(), context.selection()); + } finally { + IOUtils.closeQuietly(in); + } + return new VideoFormatReader(fileIO, filePath, fileMeta, fieldCount, blobIndex); + } + + private static int findBlobFieldIndex(RowType rowType) { + for (int i = 0; i < rowType.getFieldCount(); i++) { + if (rowType.getTypeAt(i).getTypeRoot() == DataTypeRoot.BLOB) { + return i; + } + } + return -1; + } + } +} diff --git a/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileFormatFactory.java b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileFormatFactory.java new file mode 100644 index 000000000000..beb591c953be --- /dev/null +++ b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileFormatFactory.java @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.format.blob; + +import org.apache.paimon.CoreOptions; +import org.apache.paimon.format.FileFormat; +import org.apache.paimon.format.FileFormatFactory; + +/** Factory for {@link VideoFileFormat}. */ +public class VideoFileFormatFactory implements FileFormatFactory { + + public static final String IDENTIFIER = "video"; + + @Override + public String identifier() { + return IDENTIFIER; + } + + @Override + public FileFormat create(FormatContext formatContext) { + int copyBufferSize = + CoreOptions.checkedBlobCopyBufferSize( + formatContext.options().get(CoreOptions.BLOB_COPY_BUFFER_SIZE).getBytes()); + return new VideoFileFormat(copyBufferSize); + } +} diff --git a/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileMeta.java b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileMeta.java new file mode 100644 index 000000000000..4aca43304290 --- /dev/null +++ b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFileMeta.java @@ -0,0 +1,248 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.format.blob; + +import org.apache.paimon.fs.SeekableInputStream; +import org.apache.paimon.memory.BytesUtils; +import org.apache.paimon.utils.DeltaVarintCompressor; +import org.apache.paimon.utils.IOUtils; +import org.apache.paimon.utils.RoaringBitmap32; + +import javax.annotation.Nullable; + +import java.io.IOException; +import java.util.Arrays; +import java.util.Iterator; + +/** Embedded physical-video and logical-frame-run metadata of a {@code .video} file. */ +public class VideoFileMeta { + + private final long[] physicalVideoLengths; + private final long[] physicalVideoOffsets; + private final long[] runEnds; + private final long[] runReferences; + private final long[] runFirstFrames; + private final int rowCount; + private final @Nullable int[] selectedPositions; + + public VideoFileMeta(SeekableInputStream in, long fileSize, @Nullable RoaringBitmap32 selection) + throws IOException { + if (fileSize < VideoFormatWriter.FILE_FOOTER_LENGTH) { + throw corrupt( + "file size %s is smaller than footer size %s.", + fileSize, VideoFormatWriter.FILE_FOOTER_LENGTH); + } + + long footerStart = fileSize - VideoFormatWriter.FILE_FOOTER_LENGTH; + in.seek(footerStart); + byte[] footer = new byte[VideoFormatWriter.FILE_FOOTER_LENGTH]; + IOUtils.readFully(in, footer); + int physicalIndexLength = BytesUtils.getInt(footer, 0); + int runLengthIndexLength = BytesUtils.getInt(footer, Integer.BYTES); + int runReferenceIndexLength = BytesUtils.getInt(footer, Integer.BYTES * 2); + int firstFrameIndexLength = BytesUtils.getInt(footer, Integer.BYTES * 3); + int magic = BytesUtils.getInt(footer, Integer.BYTES * 4); + byte version = footer[Integer.BYTES * 5]; + if (magic != VideoFormatWriter.MAGIC_NUMBER) { + throw corrupt("invalid footer magic %s.", magic); + } + if (version != VideoFormatWriter.VERSION) { + throw new IOException("Unsupported video format version: " + version); + } + + int[] indexLengths = { + physicalIndexLength, + runLengthIndexLength, + runReferenceIndexLength, + firstFrameIndexLength + }; + long totalIndexLength = 0; + for (int length : indexLengths) { + if (length < 0) { + throw corrupt("negative index length %s.", length); + } + totalIndexLength += length; + } + if (totalIndexLength > footerStart) { + throw corrupt("index length %s exceeds file size %s.", totalIndexLength, fileSize); + } + + long indexStart = footerStart - totalIndexLength; + long offset = indexStart; + long[] physicalVideoLengths = readIndex(in, offset, physicalIndexLength, "physical video"); + offset += physicalIndexLength; + long[] runLengths = readIndex(in, offset, runLengthIndexLength, "run length"); + offset += runLengthIndexLength; + long[] runReferences = readIndex(in, offset, runReferenceIndexLength, "run reference"); + offset += runReferenceIndexLength; + long[] runFirstFrames = readIndex(in, offset, firstFrameIndexLength, "run first-frame"); + + long[] physicalVideoOffsets = new long[physicalVideoLengths.length]; + long payloadOffset = 0; + for (int i = 0; i < physicalVideoLengths.length; i++) { + long length = physicalVideoLengths[i]; + if (length <= 0 || length > indexStart - payloadOffset) { + throw corrupt("invalid physical video length %s at ordinal %s.", length, i); + } + physicalVideoOffsets[i] = payloadOffset; + payloadOffset += length; + } + if (payloadOffset != indexStart) { + throw corrupt( + "indexed videos use %s bytes, but payload region contains %s bytes.", + payloadOffset, indexStart); + } + + if (runLengths.length != runReferences.length + || runLengths.length != runFirstFrames.length) { + throw corrupt( + "run indexes have different counts: %s, %s, and %s.", + runLengths.length, runReferences.length, runFirstFrames.length); + } + long[] runEnds = new long[runLengths.length]; + long rows = 0; + for (int i = 0; i < runLengths.length; i++) { + long length = runLengths[i]; + if (length <= 0 || rows > Integer.MAX_VALUE - length) { + throw corrupt("invalid run length %s at run %s.", length, i); + } + long reference = runReferences[i]; + if (reference != VideoFormatWriter.NULL_REFERENCE + && reference != VideoFormatWriter.PLACEHOLDER_REFERENCE + && (reference < 0 || reference >= physicalVideoLengths.length)) { + throw corrupt( + "run %s references physical video %s, but physical video count is %s.", + i, reference, physicalVideoLengths.length); + } + if (reference >= 0 && runFirstFrames[i] < 0) { + throw corrupt("run %s has negative first frame %s.", i, runFirstFrames[i]); + } + rows += length; + runEnds[i] = rows; + } + + int[] selectedPositions = null; + if (selection != null) { + long cardinality = selection.getCardinality(); + if (cardinality > rows) { + throw new IOException( + String.format( + "Invalid video selection: cardinality %s exceeds row count %s.", + cardinality, rows)); + } + selectedPositions = new int[(int) cardinality]; + Iterator iterator = selection.iterator(); + for (int i = 0; i < selectedPositions.length; i++) { + int position = iterator.next(); + if (position < 0 || position >= rows) { + throw new IOException( + String.format( + "Invalid video selection: position %s is outside row count %s.", + position, rows)); + } + selectedPositions[i] = position; + } + } + + this.physicalVideoLengths = physicalVideoLengths; + this.physicalVideoOffsets = physicalVideoOffsets; + this.runEnds = runEnds; + this.runReferences = runReferences; + this.runFirstFrames = runFirstFrames; + this.rowCount = (int) rows; + this.selectedPositions = selectedPositions; + } + + public boolean isNull(int returnedRow) { + return runReference(returnedRow) == VideoFormatWriter.NULL_REFERENCE; + } + + public boolean isPlaceHolder(int returnedRow) { + return runReference(returnedRow) == VideoFormatWriter.PLACEHOLDER_REFERENCE; + } + + public long videoOffset(int returnedRow) { + return physicalVideoOffsets[physicalOrdinal(returnedRow)]; + } + + public long videoLength(int returnedRow) { + return physicalVideoLengths[physicalOrdinal(returnedRow)]; + } + + public long frameIndex(int returnedRow) { + int row = logicalPosition(returnedRow); + int run = run(row); + long runStart = run == 0 ? 0 : runEnds[run - 1]; + return runFirstFrames[run] + row - runStart; + } + + public int returnedPosition(int currentPosition) { + return logicalPosition(currentPosition - 1); + } + + public int recordNumber() { + return selectedPositions == null ? rowCount : selectedPositions.length; + } + + public int physicalVideoNumber() { + return physicalVideoLengths.length; + } + + public int runNumber() { + return runEnds.length; + } + + private int physicalOrdinal(int returnedRow) { + long reference = runReference(returnedRow); + if (reference < 0 || reference > Integer.MAX_VALUE) { + throw new IllegalStateException( + "Row " + logicalPosition(returnedRow) + " does not reference a video."); + } + return (int) reference; + } + + private long runReference(int returnedRow) { + return runReferences[run(logicalPosition(returnedRow))]; + } + + private int logicalPosition(int returnedRow) { + return selectedPositions == null ? returnedRow : selectedPositions[returnedRow]; + } + + private int run(int logicalRow) { + int run = Arrays.binarySearch(runEnds, logicalRow + 1L); + return run >= 0 ? run : -run - 1; + } + + private static long[] readIndex(SeekableInputStream in, long start, int length, String name) + throws IOException { + in.seek(start); + byte[] bytes = new byte[length]; + IOUtils.readFully(in, bytes); + try { + return DeltaVarintCompressor.decompress(bytes); + } catch (RuntimeException e) { + throw new IOException("Corrupt video file: invalid " + name + " index.", e); + } + } + + private static IOException corrupt(String message, Object... args) { + return new IOException("Corrupt video file: " + String.format(message, args)); + } +} diff --git a/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFormatReader.java b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFormatReader.java new file mode 100644 index 000000000000..91c3511ef08d --- /dev/null +++ b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFormatReader.java @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.format.blob; + +import org.apache.paimon.data.Blob; +import org.apache.paimon.data.BlobPlaceholder; +import org.apache.paimon.data.GenericRow; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.VideoFrameDescriptor; +import org.apache.paimon.fs.FileIO; +import org.apache.paimon.fs.Path; +import org.apache.paimon.reader.FileRecordIterator; +import org.apache.paimon.reader.FileRecordReader; +import org.apache.paimon.utils.UriReader; + +import javax.annotation.Nullable; + +/** Reader that exposes each logical row as a descriptor for one frame in a packed video. */ +public class VideoFormatReader implements FileRecordReader { + + private final Path filePath; + private final VideoFileMeta fileMeta; + private final int fieldCount; + private final int blobIndex; + private final UriReader uriReader; + private boolean returned; + + public VideoFormatReader( + FileIO fileIO, Path filePath, VideoFileMeta fileMeta, int fieldCount, int blobIndex) { + this.filePath = filePath; + this.fileMeta = fileMeta; + this.fieldCount = fieldCount; + this.blobIndex = blobIndex; + this.uriReader = UriReader.fromFile(fileIO); + } + + @Nullable + @Override + public FileRecordIterator readBatch() { + if (returned) { + return null; + } + returned = true; + return new FileRecordIterator() { + + private int currentPosition; + + @Override + public long returnedPosition() { + return fileMeta.returnedPosition(currentPosition); + } + + @Override + public Path filePath() { + return filePath; + } + + @Nullable + @Override + public InternalRow next() { + if (currentPosition >= fileMeta.recordNumber()) { + return null; + } + + Object field; + if (fileMeta.isNull(currentPosition)) { + field = null; + } else if (fileMeta.isPlaceHolder(currentPosition)) { + field = BlobPlaceholder.INSTANCE; + } else { + VideoFrameDescriptor descriptor = + new VideoFrameDescriptor( + filePath.toString(), + fileMeta.videoOffset(currentPosition), + fileMeta.videoLength(currentPosition), + fileMeta.frameIndex(currentPosition)); + field = Blob.fromDescriptor(uriReader, descriptor); + } + currentPosition++; + GenericRow row = new GenericRow(fieldCount); + row.setField(blobIndex, field); + return row; + } + + @Override + public boolean skip() { + if (currentPosition >= fileMeta.recordNumber()) { + return false; + } + currentPosition++; + return true; + } + + @Override + public void releaseBatch() {} + }; + } + + @Override + public void close() {} +} diff --git a/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFormatWriter.java b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFormatWriter.java new file mode 100644 index 000000000000..1e3c747f0a35 --- /dev/null +++ b/paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFormatWriter.java @@ -0,0 +1,246 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.format.blob; + +import org.apache.paimon.data.Blob; +import org.apache.paimon.data.BlobDescriptor; +import org.apache.paimon.data.BlobFetchMetricReporter; +import org.apache.paimon.data.BlobPlaceholder; +import org.apache.paimon.data.BlobRef; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.VideoFrameDescriptor; +import org.apache.paimon.format.FileAwareFormatWriter; +import org.apache.paimon.format.FormatWriter; +import org.apache.paimon.fs.Path; +import org.apache.paimon.fs.PositionOutputStream; +import org.apache.paimon.types.RowType; +import org.apache.paimon.utils.DeltaVarintCompressor; +import org.apache.paimon.utils.LongArrayList; + +import java.io.IOException; +import java.util.HashMap; +import java.util.Map; + +import static org.apache.paimon.utils.Preconditions.checkArgument; +import static org.apache.paimon.utils.StreamUtils.intToLittleEndian; + +/** + * {@link FormatWriter} for a Paimon video pack. + * + *

The data region concatenates complete encoded-video payloads without per-payload wrappers. + * Logical frame rows are represented by compact contiguous runs. A run points to one physical video + * and stores its first frame; subsequent rows increment the frame ordinal by one. + */ +public class VideoFormatWriter implements FileAwareFormatWriter { + + public static final byte VERSION = 1; + public static final int MAGIC_NUMBER = 0x4F454449; // "IDEO" in little endian + public static final long NULL_REFERENCE = -1L; + public static final long PLACEHOLDER_REFERENCE = -2L; + public static final int FILE_FOOTER_LENGTH = Integer.BYTES * 5 + Byte.BYTES; + + private final PositionOutputStream out; + private final RawVideoPayloadWriter payloadWriter; + private final LongArrayList physicalVideoLengths; + private final LongArrayList runLengths; + private final LongArrayList runReferences; + private final LongArrayList runFirstFrames; + private final Map physicalVideos; + + private long currentRunLength; + private long currentRunReference; + private long currentRunFirstFrame; + private long currentRunLastFrame; + private boolean closed; + + public VideoFormatWriter( + PositionOutputStream out, + RowType type, + boolean writeNullOnMissingFile, + boolean writeNullOnFetchFailure, + BlobFetchMetricReporter blobFetchMetricReporter, + int copyBufferSize) { + checkArgument(type.getFieldCount() == 1, "VideoFormatWriter only supports one field."); + this.out = out; + this.payloadWriter = + new RawVideoPayloadWriter( + out, + type.getFieldNames().get(0), + writeNullOnMissingFile, + writeNullOnFetchFailure, + blobFetchMetricReporter, + copyBufferSize); + this.physicalVideoLengths = new LongArrayList(16); + this.runLengths = new LongArrayList(16); + this.runReferences = new LongArrayList(16); + this.runFirstFrames = new LongArrayList(16); + this.physicalVideos = new HashMap<>(); + } + + @Override + public void setFile(Path file) { + payloadWriter.setFile(file); + } + + @Override + public boolean deleteFileUponAbort() { + return true; + } + + @Override + public void addElement(InternalRow element) throws IOException { + checkArgument(element.getFieldCount() == 1, "VideoFormatWriter only supports one field."); + if (element.isNullAt(0)) { + append(NULL_REFERENCE, 0); + return; + } + + Blob blob = element.getBlob(0); + if (blob == BlobPlaceholder.INSTANCE) { + append(PLACEHOLDER_REFERENCE, 0); + return; + } + checkArgument( + blob != null + && blob.getClass() == BlobRef.class + && blob.toDescriptor() instanceof VideoFrameDescriptor, + "Video fields require an exact BlobRef containing a VideoFrameDescriptor."); + + VideoFrameDescriptor frame = (VideoFrameDescriptor) blob.toDescriptor(); + BlobDescriptor payload = frame.payloadDescriptor(); + Integer ordinal = physicalVideos.get(payload); + if (ordinal == null) { + long length = payloadWriter.write(element); + if (length == BlobFormatWriter.NULL_LENGTH) { + append(NULL_REFERENCE, 0); + return; + } + ordinal = physicalVideoLengths.size(); + physicalVideoLengths.add(length); + physicalVideos.put(payload, ordinal); + } + append(ordinal, frame.frameIndex()); + } + + @Override + public boolean reachTargetSize(boolean suggestedCheck, long targetSize) throws IOException { + return out.getPos() >= targetSize; + } + + @Override + public void close() throws IOException { + if (closed) { + return; + } + flushRun(); + payloadWriter.close(); + + byte[] physicalIndex = DeltaVarintCompressor.compressLongArrayList(physicalVideoLengths); + byte[] runLengthIndex = DeltaVarintCompressor.compressLongArrayList(runLengths); + byte[] runReferenceIndex = DeltaVarintCompressor.compressLongArrayList(runReferences); + byte[] firstFrameIndex = DeltaVarintCompressor.compressLongArrayList(runFirstFrames); + out.write(physicalIndex); + out.write(runLengthIndex); + out.write(runReferenceIndex); + out.write(firstFrameIndex); + out.write(intToLittleEndian(physicalIndex.length)); + out.write(intToLittleEndian(runLengthIndex.length)); + out.write(intToLittleEndian(runReferenceIndex.length)); + out.write(intToLittleEndian(firstFrameIndex.length)); + out.write(intToLittleEndian(MAGIC_NUMBER)); + out.write(VERSION); + closed = true; + } + + int physicalVideoCount() { + return physicalVideoLengths.size(); + } + + int runCount() { + return runLengths.size() + (currentRunLength == 0 ? 0 : 1); + } + + private void append(long reference, long frameIndex) { + if (canExtend(reference, frameIndex)) { + currentRunLength++; + currentRunLastFrame = frameIndex; + return; + } + flushRun(); + currentRunReference = reference; + currentRunFirstFrame = frameIndex; + currentRunLastFrame = frameIndex; + currentRunLength = 1; + } + + private boolean canExtend(long reference, long frameIndex) { + if (currentRunLength == 0 || currentRunReference != reference) { + return false; + } + return reference < 0 || frameIndex == currentRunLastFrame + 1; + } + + private void flushRun() { + if (currentRunLength == 0) { + return; + } + runLengths.add(currentRunLength); + runReferences.add(currentRunReference); + runFirstFrames.add(currentRunFirstFrame); + currentRunLength = 0; + } + + /** Copies raw video bytes without the ordinary BLOB record header and trailer. */ + private static class RawVideoPayloadWriter extends AbstractBlobElementWriter { + + private RawVideoPayloadWriter( + PositionOutputStream out, + String fieldName, + boolean writeNullOnMissingFile, + boolean writeNullOnFetchFailure, + BlobFetchMetricReporter blobFetchMetricReporter, + int copyBufferSize) { + super( + out, + fieldName, + null, + writeNullOnMissingFile, + writeNullOnFetchFailure, + blobFetchMetricReporter, + copyBufferSize); + } + + @Override + public long write(InternalRow row) throws IOException { + BlobFetchResult fetchResult = getBlob(() -> row.getBlob(0)); + if (fetchResult.fetchFailure()) { + return BlobFormatWriter.NULL_LENGTH; + } + Blob blob = fetchResult.blob(); + BlobCopySource source = prepareBlobSource(blob); + if (source == null) { + return BlobFormatWriter.NULL_LENGTH; + } + BlobDescriptor written = writeBlobData(source); + checkArgument(written.length() > 0, "Encoded video payload must not be empty."); + recordSuccess(written.length()); + return written.length(); + } + } +} diff --git a/paimon-format/src/main/resources/META-INF/services/org.apache.paimon.format.FileFormatFactory b/paimon-format/src/main/resources/META-INF/services/org.apache.paimon.format.FileFormatFactory index f34a5af57e48..8c9319967fa2 100644 --- a/paimon-format/src/main/resources/META-INF/services/org.apache.paimon.format.FileFormatFactory +++ b/paimon-format/src/main/resources/META-INF/services/org.apache.paimon.format.FileFormatFactory @@ -20,4 +20,5 @@ org.apache.paimon.format.csv.CsvFileFormatFactory org.apache.paimon.format.text.TextFileFormatFactory org.apache.paimon.format.json.JsonFileFormatFactory org.apache.paimon.format.blob.BlobFileFormatFactory +org.apache.paimon.format.blob.VideoFileFormatFactory org.apache.paimon.format.row.RowFileFormatFactory diff --git a/paimon-format/src/test/java/org/apache/paimon/format/blob/VideoFileFormatTest.java b/paimon-format/src/test/java/org/apache/paimon/format/blob/VideoFileFormatTest.java new file mode 100644 index 000000000000..38b48ecd44e7 --- /dev/null +++ b/paimon-format/src/test/java/org/apache/paimon/format/blob/VideoFileFormatTest.java @@ -0,0 +1,272 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.format.blob; + +import org.apache.paimon.data.Blob; +import org.apache.paimon.data.BlobData; +import org.apache.paimon.data.BlobPlaceholder; +import org.apache.paimon.data.GenericRow; +import org.apache.paimon.data.InternalRow; +import org.apache.paimon.data.VideoFrameDescriptor; +import org.apache.paimon.format.FileAwareFormatWriter; +import org.apache.paimon.format.FileFormat; +import org.apache.paimon.format.FormatReaderContext; +import org.apache.paimon.format.FormatReaderFactory; +import org.apache.paimon.format.FormatWriter; +import org.apache.paimon.fs.FileIO; +import org.apache.paimon.fs.Path; +import org.apache.paimon.fs.PositionOutputStream; +import org.apache.paimon.fs.SeekableInputStream; +import org.apache.paimon.fs.local.LocalFileIO; +import org.apache.paimon.options.Options; +import org.apache.paimon.reader.FileRecordIterator; +import org.apache.paimon.reader.FileRecordReader; +import org.apache.paimon.types.DataTypes; +import org.apache.paimon.types.RowType; +import org.apache.paimon.utils.DeltaVarintCompressor; +import org.apache.paimon.utils.RoaringBitmap32; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.List; + +import static org.apache.paimon.utils.StreamUtils.intToLittleEndian; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** Tests for {@link VideoFileFormat}. */ +public class VideoFileFormatTest { + + @TempDir java.nio.file.Path tempPath; + + private FileIO fileIO; + private Path file; + private RowType rowType; + + @BeforeEach + public void beforeEach() { + fileIO = LocalFileIO.create(); + file = new Path(tempPath.resolve("data.video").toUri()); + rowType = RowType.of(DataTypes.BLOB()); + } + + @Test + public void testPackRawVideosAndMapFrameRuns() throws IOException { + byte[] firstBytes = "first-mp4".getBytes(); + byte[] secondBytes = "second-mp4".getBytes(); + Blob first0 = sourceFrame("first.mp4", firstBytes, 0); + Blob first1 = sourceFrame("first.mp4", firstBytes, 1); + Blob second7 = sourceFrame("second.mp4", secondBytes, 7); + Blob first4 = sourceFrame("first.mp4", firstBytes, 4); + + write(first0, first1, second7, first4, null, BlobPlaceholder.INSTANCE); + + try (SeekableInputStream in = fileIO.newInputStream(file)) { + VideoFileMeta meta = new VideoFileMeta(in, fileIO.getFileSize(file), null); + assertThat(meta.recordNumber()).isEqualTo(6); + assertThat(meta.physicalVideoNumber()).isEqualTo(2); + assertThat(meta.runNumber()).isEqualTo(5); + assertThat(meta.videoOffset(0)).isZero(); + assertThat(meta.videoLength(0)).isEqualTo(firstBytes.length); + assertThat(meta.frameIndex(0)).isZero(); + assertThat(meta.frameIndex(1)).isOne(); + assertThat(meta.frameIndex(2)).isEqualTo(7); + assertThat(meta.frameIndex(3)).isEqualTo(4); + assertThat(meta.isNull(4)).isTrue(); + assertThat(meta.isPlaceHolder(5)).isTrue(); + } + + byte[] stored = Files.readAllBytes(java.nio.file.Paths.get(file.toUri())); + assertThat(stored).startsWith(firstBytes); + assertThat(stored).containsSubsequence(secondBytes); + + List rows = read(null); + assertThat(rows).hasSize(6); + VideoFrameDescriptor frame0 = descriptor(rows.get(0)); + VideoFrameDescriptor frame1 = descriptor(rows.get(1)); + VideoFrameDescriptor frame2 = descriptor(rows.get(2)); + VideoFrameDescriptor frame3 = descriptor(rows.get(3)); + assertThat(frame0.frameIndex()).isZero(); + assertThat(frame1.frameIndex()).isOne(); + assertThat(frame0.payloadDescriptor()).isEqualTo(frame1.payloadDescriptor()); + assertThat(frame2.frameIndex()).isEqualTo(7); + assertThat(frame2.payloadDescriptor()).isNotEqualTo(frame0.payloadDescriptor()); + assertThat(frame3.frameIndex()).isEqualTo(4); + assertThat(frame3.payloadDescriptor()).isEqualTo(frame0.payloadDescriptor()); + assertThat(rows.get(4).isNullAt(0)).isTrue(); + assertThat(rows.get(5).getBlob(0)).isSameAs(BlobPlaceholder.INSTANCE); + } + + @Test + public void testSelectionKeepsLogicalRowPositions() throws IOException { + byte[] bytes = "first-mp4".getBytes(); + write( + sourceFrame("first.mp4", bytes, 0), + sourceFrame("first.mp4", bytes, 1), + sourceFrame("first.mp4", bytes, 2), + sourceFrame("first.mp4", bytes, 3)); + + RoaringBitmap32 selection = new RoaringBitmap32(); + selection.add(1); + selection.add(3); + + VideoFileFormat format = new VideoFileFormat(BlobFormatWriter.DEFAULT_COPY_BUFFER_SIZE); + FormatReaderFactory readerFactory = format.createReaderFactory(null, rowType, null); + FormatReaderContext context = + new FormatReaderContext(fileIO, file, fileIO.getFileSize(file), selection, null); + try (FileRecordReader reader = readerFactory.createReader(context)) { + FileRecordIterator iterator = reader.readBatch(); + assertThat(descriptor(iterator.next()).frameIndex()).isOne(); + assertThat(iterator.returnedPosition()).isOne(); + assertThat(descriptor(iterator.next()).frameIndex()).isEqualTo(3); + assertThat(iterator.returnedPosition()).isEqualTo(3L); + assertThat(iterator.next()).isNull(); + } + } + + @Test + public void testRejectNonVideoFrameInputsAndNestedBlobTypes() throws IOException { + VideoFileFormat format = new VideoFileFormat(BlobFormatWriter.DEFAULT_COPY_BUFFER_SIZE); + assertThatThrownBy( + () -> + format.validateDataFields( + RowType.of(DataTypes.ARRAY(DataTypes.BLOB())))) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("scalar BLOB"); + + try (PositionOutputStream out = fileIO.newOutputStream(file, false)) { + FormatWriter writer = format.createWriterFactory(rowType).create(out, null); + assertThatThrownBy( + () -> + writer.addElement( + GenericRow.of(new BlobData("inline".getBytes())))) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("VideoFrameDescriptor"); + writer.close(); + } + } + + @Test + public void testRejectEmptyVideoPayload() throws IOException { + Blob empty = sourceFrame("empty.mp4", new byte[0], 0); + + assertThatThrownBy(() -> write(empty)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("Encoded video payload must not be empty"); + } + + @Test + public void testIndependentFormatRegistrationAndClassification() { + assertThat(FileFormat.fromIdentifier("video", new Options())) + .isInstanceOf(VideoFileFormat.class); + assertThat(BlobFileFormat.isBlobFile("a.blob")).isTrue(); + assertThat(BlobFileFormat.isBlobFile("a.video")).isTrue(); + assertThat(BlobFileFormat.isBlobFile("a.parquet")).isFalse(); + } + + @Test + public void testRejectCorruptRunReference() throws IOException { + byte[] physicalIndex = DeltaVarintCompressor.compress(new long[0]); + byte[] runLengthIndex = DeltaVarintCompressor.compress(new long[] {1}); + byte[] runReferenceIndex = DeltaVarintCompressor.compress(new long[] {0}); + byte[] firstFrameIndex = DeltaVarintCompressor.compress(new long[] {0}); + byte[] bytes = + new byte + [physicalIndex.length + + runLengthIndex.length + + runReferenceIndex.length + + firstFrameIndex.length + + VideoFormatWriter.FILE_FOOTER_LENGTH]; + int position = 0; + position = put(bytes, position, physicalIndex); + position = put(bytes, position, runLengthIndex); + position = put(bytes, position, runReferenceIndex); + position = put(bytes, position, firstFrameIndex); + position = putInt(bytes, position, physicalIndex.length); + position = putInt(bytes, position, runLengthIndex.length); + position = putInt(bytes, position, runReferenceIndex.length); + position = putInt(bytes, position, firstFrameIndex.length); + position = putInt(bytes, position, VideoFormatWriter.MAGIC_NUMBER); + bytes[position] = VideoFormatWriter.VERSION; + Files.write(java.nio.file.Paths.get(file.toUri()), bytes); + + assertThatThrownBy( + () -> { + try (SeekableInputStream in = fileIO.newInputStream(file)) { + new VideoFileMeta(in, fileIO.getFileSize(file), null); + } + }) + .isInstanceOf(IOException.class) + .hasMessageContaining( + "run 0 references physical video 0, but physical video count is 0"); + } + + private Blob sourceFrame(String name, byte[] bytes, long frameIndex) throws IOException { + java.nio.file.Path source = tempPath.resolve(name); + if (!Files.exists(source)) { + Files.write(source, bytes); + } + VideoFrameDescriptor descriptor = + new VideoFrameDescriptor( + new Path(source.toUri()).toString(), 0, bytes.length, frameIndex); + return Blob.fromDescriptor(org.apache.paimon.utils.UriReader.fromFile(fileIO), descriptor); + } + + private void write(Object... frames) throws IOException { + VideoFileFormat format = new VideoFileFormat(BlobFormatWriter.DEFAULT_COPY_BUFFER_SIZE); + try (PositionOutputStream out = fileIO.newOutputStream(file, false)) { + FormatWriter writer = format.createWriterFactory(rowType).create(out, null); + ((FileAwareFormatWriter) writer).setFile(file); + for (Object frame : frames) { + writer.addElement(GenericRow.of(frame)); + } + writer.close(); + } + } + + private List read(RoaringBitmap32 selection) throws IOException { + VideoFileFormat format = new VideoFileFormat(BlobFormatWriter.DEFAULT_COPY_BUFFER_SIZE); + FormatReaderFactory readerFactory = format.createReaderFactory(null, rowType, null); + FormatReaderContext context = + new FormatReaderContext(fileIO, file, fileIO.getFileSize(file), selection, null); + List rows = new ArrayList<>(); + try (FileRecordReader reader = readerFactory.createReader(context)) { + reader.forEachRemaining(rows::add); + } + return rows; + } + + private static VideoFrameDescriptor descriptor(InternalRow row) { + return (VideoFrameDescriptor) row.getBlob(0).toDescriptor(); + } + + private static int put(byte[] target, int position, byte[] value) { + System.arraycopy(value, 0, target, position, value.length); + return position + value.length; + } + + private static int putInt(byte[] target, int position, int value) { + return put(target, position, intToLittleEndian(value)); + } +} diff --git a/paimon-python/pypaimon/common/options/core_options.py b/paimon-python/pypaimon/common/options/core_options.py index e76db7887000..2f063b738971 100644 --- a/paimon-python/pypaimon/common/options/core_options.py +++ b/paimon-python/pypaimon/common/options/core_options.py @@ -132,6 +132,7 @@ class CoreOptions: "data-evolution.enabled", "index-file-in-data-file-dir", "blob-field", + "video-frame-field", "blob-descriptor-field", "blob-view-field", "pk-clustering-override", @@ -142,6 +143,7 @@ class CoreOptions: FILE_FORMAT_AVRO: str = "avro" FILE_FORMAT_PARQUET: str = "parquet" FILE_FORMAT_BLOB: str = "blob" + FILE_FORMAT_VIDEO: str = "video" FILE_FORMAT_LANCE: str = "lance" FILE_FORMAT_VORTEX: str = "vortex" FILE_FORMAT_ROW: str = "row" @@ -399,6 +401,16 @@ class CoreOptions: .with_description("Comma-separated column names that should be stored as blob type.") ) + VIDEO_FRAME_FIELD: ConfigOption[str] = ( + ConfigOptions.key("video-frame-field") + .string_type() + .no_default_value() + .with_description( + "One scalar BLOB field whose logical values are frames in encoded " + "videos packed into '.video' files." + ) + ) + BLOB_DESCRIPTOR_FIELD: ConfigOption[str] = ( ConfigOptions.key("blob-descriptor-field") .string_type() @@ -1235,6 +1247,10 @@ def blob_field(self, default=None): value = self.options.get(CoreOptions.BLOB_FIELD, default) return CoreOptions._parse_field_set(value) + def video_frame_fields(self, default=None): + value = self.options.get(CoreOptions.VIDEO_FRAME_FIELD, default) + return CoreOptions._parse_field_set(value) + def blob_view_resolve_enabled(self, default=True): return self.options.get(CoreOptions.BLOB_VIEW_RESOLVE_ENABLED, default) diff --git a/paimon-python/pypaimon/daft/daft_datasource.py b/paimon-python/pypaimon/daft/daft_datasource.py index cc1dde88d397..663a39435234 100644 --- a/paimon-python/pypaimon/daft/daft_datasource.py +++ b/paimon-python/pypaimon/daft/daft_datasource.py @@ -493,7 +493,8 @@ def _blob_native_covering_files( native reader, or ``None`` if the split must use the pypaimon fallback. A blob table stores each column bunch in its own file: scalar columns in - parquet, BLOB / ARRAY / MAP columns in ``.blob`` files, vector columns in + parquet, BLOB / ARRAY / MAP columns in ``.blob`` or + ``.video`` files, vector columns in ``.vector`` files, aligned by row id. Reading the base parquet files natively is only correct when every projected data column lives in parquet files that each fully cover the projection over disjoint row-id ranges -- @@ -512,7 +513,7 @@ def _blob_native_covering_files( name = f.file_name write_cols = set(f.write_cols or []) carried = write_cols & projected - if name.endswith(".blob") or ".vector." in name: + if name.endswith((".blob", ".video")) or ".vector." in name: if carried: return None # a projected column lives in a blob/vector bunch continue diff --git a/paimon-python/pypaimon/manifest/schema/data_file_meta.py b/paimon-python/pypaimon/manifest/schema/data_file_meta.py index 4cc9a3a93519..5a5a70b3e730 100644 --- a/paimon-python/pypaimon/manifest/schema/data_file_meta.py +++ b/paimon-python/pypaimon/manifest/schema/data_file_meta.py @@ -175,7 +175,7 @@ def copy_without_stats(self) -> 'DataFileMeta': @staticmethod def is_blob_file(file_name: str) -> bool: - return file_name.endswith(".blob") + return file_name.endswith(".blob") or file_name.endswith(".video") @staticmethod def is_vector_file(file_name: str) -> bool: diff --git a/paimon-python/pypaimon/multimodal/__init__.py b/paimon-python/pypaimon/multimodal/__init__.py index 96afcbbea392..b669267b41da 100644 --- a/paimon-python/pypaimon/multimodal/__init__.py +++ b/paimon-python/pypaimon/multimodal/__init__.py @@ -36,6 +36,8 @@ text_route, vector_route, ) +from pypaimon.multimodal.video import VideoFrameCollator +from pypaimon.table.row.blob import Blob, BlobDescriptor, VideoFrameDescriptor from pypaimon.table.data_evolution_merge_into import ( lit, source_col, @@ -43,6 +45,8 @@ ) __all__ = [ + "Blob", + "BlobDescriptor", "BlobObject", "BlobStore", "Hdf5File", @@ -54,6 +58,8 @@ "PutObjectResult", "TextRoute", "VectorRoute", + "VideoFrameCollator", + "VideoFrameDescriptor", "connect", "lit", "source_col", diff --git a/paimon-python/pypaimon/multimodal/blob_read.py b/paimon-python/pypaimon/multimodal/blob_read.py index ec6930453c0e..675eeeb08ce9 100644 --- a/paimon-python/pypaimon/multimodal/blob_read.py +++ b/paimon-python/pypaimon/multimodal/blob_read.py @@ -27,7 +27,11 @@ def fetch_blob_bodies( ``None``, or a MAP represented by key-value pairs. Returned values preserve row and MAP entry order and are grouped per column. """ - from pypaimon.table.row.blob import BlobDescriptor, BlobViewStruct + from pypaimon.table.row.blob import ( + BlobDescriptor, + BlobViewStruct, + VideoFrameDescriptor, + ) ranges = [] inline = {} @@ -46,7 +50,10 @@ def queue_blob_fetch(value): raise ValueError( "read_blobs does not support unresolved blob-view columns; " "read such a column on its own, or enable blob-view resolution.") - if BlobDescriptor.is_blob_descriptor(raw): + if ( + VideoFrameDescriptor.is_video_frame_descriptor(raw) + or BlobDescriptor.is_blob_descriptor(raw) + ): descriptor = BlobDescriptor.deserialize(raw) ranges.append( (descriptor.uri, descriptor.offset, descriptor.length) diff --git a/paimon-python/pypaimon/multimodal/query.py b/paimon-python/pypaimon/multimodal/query.py index 39d24fd88d48..9046498f7cf8 100644 --- a/paimon-python/pypaimon/multimodal/query.py +++ b/paimon-python/pypaimon/multimodal/query.py @@ -72,8 +72,10 @@ def to_arrow(self): plan = scan.plan() return read_builder.new_read().to_arrow(plan.splits()) - def _configured_read_builder(self): - read_builder = self._table.new_read_builder() + def _configured_read_builder(self, table=None): + read_builder = ( + self._table if table is None else table + ).new_read_builder() if self._predicate is not None: read_builder = read_builder.with_filter(self._predicate) projection = self._effective_projection() @@ -106,6 +108,49 @@ def to_pandas(self): def to_list(self) -> List[dict]: return self.to_arrow().to_pylist() + def to_torch( + self, + streaming: bool = True, + prefetch_concurrency: int = 1, + *, + batch_format: str = "row", + batch_size: Optional[int] = None, + to_tensor_fn: Optional[Callable] = None, + shuffle: bool = False, + seed: int = 0, + buffer_size: int = 1000, + max_buffer_input_splits: int = 10): + """Read this scan as a PyTorch Dataset. + + BLOB columns stay as serialized descriptors so DataLoader workers can + open and decode the referenced payload without materialising it in the + planning process. Use :class:`VideoFrameCollator` as ``collate_fn`` to + reuse one decoder session across rows that reference the same video. + """ + if self._result_factory is not None: + raise TypeError( + "to_torch is only supported on scan(), not search queries." + ) + + from pypaimon.common.options.core_options import CoreOptions + read_table = self._table.copy({ + CoreOptions.BLOB_AS_DESCRIPTOR.key(): "true" + }) + read_builder = self._configured_read_builder(read_table) + splits = read_builder.new_scan().plan().splits() + return read_builder.new_read().to_torch( + splits, + streaming=streaming, + prefetch_concurrency=prefetch_concurrency, + batch_format=batch_format, + batch_size=batch_size, + to_tensor_fn=to_tensor_fn, + shuffle=shuffle, + seed=seed, + buffer_size=buffer_size, + max_buffer_input_splits=max_buffer_input_splits, + ) + def to_ray( self, *, diff --git a/paimon-python/pypaimon/multimodal/table.py b/paimon-python/pypaimon/multimodal/table.py index d6ef7faa4183..70cb3dd73173 100644 --- a/paimon-python/pypaimon/multimodal/table.py +++ b/paimon-python/pypaimon/multimodal/table.py @@ -113,6 +113,129 @@ def add(self, data): table_commit.close() return self + def add_video( + self, + video, + frames, + *, + video_column=None, + first_frame=0): + """Append logical frame rows backed by one complete encoded video. + + ``frames`` supplies every table column except the configured + ``video-frame-field``. Frame ordinals are generated from + ``first_frame`` and stored in the video descriptor, not in the normal + data file. + """ + return self.add_videos( + [(video, frames, first_frame)], video_column=video_column + ) + + def add_videos(self, videos, *, video_column=None): + """Append several encoded videos with one writer and one commit. + + Each item is ``(video, frames)`` or ``(video, frames, first_frame)``. + Keeping one writer open lets a single ``.video`` object pack multiple + complete encoded videos up to the configured rolling target. + """ + column = self._resolve_video_frame_column(video_column) + target_schema = _target_schema(self.raw_table) + + def frame_batches(): + for item in videos: + try: + item = tuple(item) + except TypeError as error: + raise ValueError( + "Each videos item must be (video, frames) or " + "(video, frames, first_frame)." + ) from error + if len(item) == 2: + video, frames = item + first_frame = 0 + elif len(item) == 3: + video, frames, first_frame = item + else: + raise ValueError( + "Each videos item must be (video, frames) or " + "(video, frames, first_frame)." + ) + yield _video_frame_table( + video, + frames, + column, + first_frame, + target_schema, + ) + + return self.add_batches(frame_batches()) + + def _resolve_video_frame_column(self, requested): + configured = self.raw_table.options.video_frame_fields() + if not configured: + raise ValueError( + "add_video requires table option 'video-frame-field'." + ) + column = requested or next(iter(configured)) + if column not in configured: + raise ValueError( + "Video column %r is not configured by 'video-frame-field'." + % column + ) + return column + + def add_batches(self, batches): + """Append an iterable of batches with one writer and one commit. + + Keeping the writer open across batches also preserves video payload + groups and lets one ``.video`` file pack several encoded videos. + """ + try: + iterator = iter(batches) + except TypeError as error: + raise ValueError("batches must be an iterable of input batches.") from error + + target_schema = _target_schema(self.raw_table) + table_write = None + table_commit = None + commit_started = False + try: + for data in iterator: + arrow_table = _to_arrow_table(data, target_schema) + if arrow_table.num_rows == 0: + continue + if table_write is None: + write_builder = self.raw_table.new_batch_write_builder() + table_write = write_builder.new_write() + table_commit = write_builder.new_commit() + table_write.write_arrow(arrow_table) + + close_iterator = getattr(iterator, "close", None) + iterator = None + if close_iterator is not None: + close_iterator() + if table_write is None: + return self + commit_messages = table_write.prepare_commit() + commit_started = True + table_commit.commit(commit_messages) + return self + except BaseException: + if table_write is not None and not commit_started: + table_write.abort() + raise + finally: + if iterator is not None: + close_iterator = getattr(iterator, "close", None) + if close_iterator is not None: + close_iterator() + try: + if table_write is not None: + table_write.close() + finally: + if table_commit is not None: + table_commit.close() + def overwrite(self, data, partition: Optional[Mapping[str, object]] = None): arrow_table = _to_arrow_table(data, _target_schema(self.raw_table)) overwrite_partition = dict(partition) if partition is not None else None @@ -131,6 +254,15 @@ def overwrite(self, data, partition: Optional[Mapping[str, object]] = None): return self def update(self, where, values): + video_columns = self.raw_table.options.video_frame_fields() + if isinstance(values, Mapping): + updated_video_columns = video_columns.intersection(values) + if updated_video_columns: + raise ValueError( + "update() cannot write video-frame-field %r; use " + "replace_video() with a complete encoded video." + % sorted(updated_video_columns) + ) query = self.scan().where(where) predicate = query._predicate write_builder = self.raw_table.new_batch_write_builder() @@ -143,6 +275,56 @@ def update(self, where, values): table_commit.close() return self + def replace_video( + self, + where, + video, + *, + video_column=None, + first_frame=0): + """Replace the video backing the logical frame rows matching ``where``. + + Matching rows are ordered by ``_ROW_ID`` and assigned consecutive + frame ordinals starting at ``first_frame``. Only the configured video + column is updated; ordinary columns and the normal data files remain + untouched. + """ + column = self._resolve_video_frame_column(video_column) + target_schema = _target_schema(self.raw_table) + payload, first_frame = _video_payload(video, first_frame) + + row_ids = ( + self.scan() + .where(where) + .select([]) + .with_row_id() + .to_arrow()[SpecialFields.ROW_ID.name] + .to_pylist() + ) + row_ids.sort() + if not row_ids: + return self + + descriptors = _video_frame_descriptors( + payload, len(row_ids), first_frame) + update_data = pa.Table.from_arrays( + [ + pa.array(row_ids, type=pa.int64()), + pa.array(descriptors, type=target_schema.field(column).type), + ], + names=[SpecialFields.ROW_ID.name, column], + ) + + write_builder = self.raw_table.new_batch_write_builder() + table_update = write_builder.new_update().with_update_type([column]) + table_commit = write_builder.new_commit() + try: + messages = table_update.update_by_arrow_with_row_id(update_data) + table_commit.commit(messages) + finally: + table_commit.close() + return self + def delete(self, where): query = self.scan().where(where) predicate = query._predicate @@ -380,6 +562,8 @@ def _blob_columns(table): def _to_arrow_table(data, target_schema=None): + if target_schema is not None: + data = _serialize_blob_values(data, target_schema) if isinstance(data, pa.Table): table = data elif isinstance(data, pa.RecordBatch): @@ -398,6 +582,91 @@ def _to_arrow_table(data, target_schema=None): return _align_to_schema(table, target_schema) +def _serialize_blob_values(data, target_schema): + if isinstance(data, (pa.Table, pa.RecordBatch)): + return data + binary_fields = { + field.name: field.type + for field in target_schema + if _contains_binary(field.type) + } + if not binary_fields: + return data + + if isinstance(data, list): + return [ + { + name: _serialize_blob_value(value, binary_fields.get(name)) + for name, value in row.items() + } + if isinstance(row, Mapping) + else row + for row in data + ] + if isinstance(data, dict): + converted = dict(data) + for name, arrow_type in binary_fields.items(): + if name not in converted: + continue + column = converted[name] + if isinstance(column, (pa.Array, pa.ChunkedArray)): + column = column.to_pylist() + converted[name] = [ + _serialize_blob_value(value, arrow_type) + for value in column + ] + return converted + if ( + hasattr(data, "__dataframe__") + or data.__class__.__module__.startswith("pandas") + ): + converted = data.copy() + for name, arrow_type in binary_fields.items(): + if name in converted.columns: + converted[name] = converted[name].map( + lambda value: _serialize_blob_value(value, arrow_type) + ) + return converted + return data + + +def _contains_binary(arrow_type): + if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): + return True + if pa.types.is_list(arrow_type) or pa.types.is_large_list(arrow_type): + return _contains_binary(arrow_type.value_type) + if pa.types.is_map(arrow_type): + return _contains_binary(arrow_type.item_type) + return False + + +def _serialize_blob_value(value, arrow_type): + if value is None or arrow_type is None: + return value + if pa.types.is_binary(arrow_type) or pa.types.is_large_binary(arrow_type): + from pypaimon.table.row.blob import Blob, BlobDescriptor + if isinstance(value, BlobDescriptor): + return value.serialize() + if isinstance(value, Blob): + try: + return value.to_descriptor().serialize() + except RuntimeError: + return value.to_data() + return value + if pa.types.is_list(arrow_type) or pa.types.is_large_list(arrow_type): + return [ + _serialize_blob_value(element, arrow_type.value_type) + for element in value + ] + if pa.types.is_map(arrow_type): + entries = value.items() if isinstance(value, Mapping) else value + return [ + (key, _serialize_blob_value(element, arrow_type.item_type)) + for key, element in entries + ] + return value + + def _coerce_row_ids(row_ids): if row_ids is None or isinstance(row_ids, (str, bytes)): raise ValueError("row_ids must be an iterable of row id integers.") @@ -433,6 +702,73 @@ def _target_schema(table): return PyarrowFieldParser.from_paimon_schema(table.table_schema.fields) +def _video_payload(video, first_frame): + from pypaimon.table.row.blob import ( + Blob, + BlobDescriptor, + VideoFrameDescriptor, + ) + + if isinstance(first_frame, bool) or not isinstance(first_frame, int): + raise ValueError("first_frame must be a non-negative int.") + if first_frame < 0: + raise ValueError("first_frame must be a non-negative int.") + + if isinstance(video, str): + video = Blob.from_local(video) + if isinstance(video, Blob): + try: + payload = video.to_descriptor() + except RuntimeError as error: + raise ValueError( + "video must be descriptor-backed; inline video bytes are not " + "accepted by video write APIs." + ) from error + elif isinstance(video, BlobDescriptor): + payload = video + else: + raise ValueError( + "video must be a path, Blob, or BlobDescriptor, got %r." + % type(video) + ) + if isinstance(payload, VideoFrameDescriptor): + payload = payload.payload_descriptor + + return payload, first_frame + + +def _video_frame_descriptors(payload, count, first_frame): + from pypaimon.table.row.blob import VideoFrameDescriptor + + return [ + VideoFrameDescriptor( + payload.uri, + payload.offset, + payload.length, + first_frame + index, + ).serialize() + for index in range(count) + ] + + +def _video_frame_table(video, frames, video_column, first_frame, target_schema): + payload, first_frame = _video_payload(video, first_frame) + + non_video_schema = pa.schema([ + field for field in target_schema if field.name != video_column + ]) + frame_table = _to_arrow_table(frames, non_video_schema) + descriptor_values = _video_frame_descriptors( + payload, frame_table.num_rows, first_frame) + arrays = [] + for field in target_schema: + if field.name == video_column: + arrays.append(pa.array(descriptor_values, type=field.type)) + else: + arrays.append(frame_table[field.name]) + return pa.Table.from_arrays(arrays, schema=target_schema) + + def _align_to_schema( table: pa.Table, schema: pa.Schema, diff --git a/paimon-python/pypaimon/multimodal/video.py b/paimon-python/pypaimon/multimodal/video.py new file mode 100644 index 000000000000..bbc1f6b2def3 --- /dev/null +++ b/paimon-python/pypaimon/multimodal/video.py @@ -0,0 +1,194 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""PyTorch DataLoader helpers for descriptor-backed video frame rows.""" + +import os +from collections import OrderedDict +from collections.abc import Mapping + +from pypaimon.table.row.blob import Blob, VideoFrameDescriptor + + +class VideoFrameCollator: + """Decode frame rows in a DataLoader worker while reusing video sessions. + + ``decoder_factory`` receives a seekable stream containing exactly one + descriptor-backed video. ``decode_fn`` receives the cached decoder, the + frame ordinal embedded in the descriptor, and one row dictionary. This + keeps Paimon independent of a particular video codec library while allowing + PyAV, TorchCodec, or an application decoder to be plugged in. + + The cache is process-local and keyed by physical video payload identity. + ``collate_fn`` defaults to PyTorch's ``default_collate`` and may be replaced + for decoders that already return batched objects. + """ + + def __init__( + self, + table, + *, + video_column, + decoder_factory, + decode_fn, + output_column="frame", + max_open_videos=8, + collate_fn=None): + if not video_column: + raise ValueError("video_column is required.") + if not callable(decoder_factory): + raise ValueError("decoder_factory must be callable.") + if not callable(decode_fn): + raise ValueError("decode_fn must be callable.") + if ( + isinstance(max_open_videos, bool) + or not isinstance(max_open_videos, int) + or max_open_videos <= 0 + ): + raise ValueError("max_open_videos must be a positive int.") + if collate_fn is not None and not callable(collate_fn): + raise ValueError("collate_fn must be callable or None.") + + raw_table = getattr(table, "raw_table", table) + file_io = getattr(raw_table, "file_io", None) + if file_io is None: + raise ValueError("table must provide raw_table.file_io or file_io.") + + self.file_io = file_io + self.video_column = video_column + self.decoder_factory = decoder_factory + self.decode_fn = decode_fn + self.output_column = output_column + self.max_open_videos = max_open_videos + self.collate_fn = collate_fn + self._decoders = OrderedDict() + self._owner_pid = os.getpid() + + def __call__(self, rows): + self._ensure_process_local_cache() + single_row = isinstance(rows, Mapping) + input_rows = [rows] if single_row else list(rows) + decoded_rows = [self._decode_row(row) for row in input_rows] + if single_row: + return decoded_rows[0] + return self._collate(decoded_rows) + + def close(self): + while self._decoders: + _, resource = self._decoders.popitem(last=False) + self._close_resource(resource) + + def __getstate__(self): + # DataLoader spawn workers must never inherit non-picklable decoder or + # stream state opened in the parent process. + state = self.__dict__.copy() + state["_decoders"] = OrderedDict() + state["_owner_pid"] = None + return state + + def _decode_row(self, row): + if not isinstance(row, Mapping): + raise ValueError("VideoFrameCollator expects row dictionaries.") + if self.video_column not in row: + raise ValueError( + "Video column %r is missing from the row." % self.video_column + ) + + raw = row[self.video_column] + output = dict(row) + if raw is None: + output[self.output_column] = None + return output + if hasattr(raw, "as_py"): + raw = raw.as_py() + if not VideoFrameDescriptor.is_video_frame_descriptor(raw): + raise ValueError( + "Video column %r must contain serialized " + "VideoFrameDescriptor bytes from a .video file." + % self.video_column + ) + + serialized = bytes(raw) + descriptor = VideoFrameDescriptor.deserialize(serialized) + if descriptor.serialize() != serialized: + raise ValueError( + "Video column %r must contain one exact serialized " + "VideoFrameDescriptor without trailing bytes." + % self.video_column + ) + decoder = self._decoder(descriptor.payload_descriptor) + output[self.output_column] = self.decode_fn( + decoder, descriptor.frame_index, output + ) + return output + + def _decoder(self, descriptor): + resource = self._decoders.pop(descriptor, None) + if resource is not None: + self._decoders[descriptor] = resource + return resource[0] + + # Reuse the table's resolved FileIO. Rebuilding a reader from raw URI + # options can drop merged REST/DLF storage credentials in workers. + stream = Blob.from_file( + self.file_io, + descriptor.uri, + descriptor.offset, + descriptor.length, + ).new_input_stream() + try: + decoder = self.decoder_factory(stream) + except Exception: + stream.close() + raise + resource = (decoder, stream) + self._decoders[descriptor] = resource + if len(self._decoders) > self.max_open_videos: + _, evicted = self._decoders.popitem(last=False) + self._close_resource(evicted) + return decoder + + def _collate(self, rows): + if self.collate_fn is not None: + return self.collate_fn(rows) + try: + from torch.utils.data import default_collate + except ImportError as error: + raise ImportError( + "VideoFrameCollator requires PyTorch for its default collate " + "function; install pypaimon[torch] or pass collate_fn=." + ) from error + return default_collate(rows) + + def _ensure_process_local_cache(self): + pid = os.getpid() + if self._owner_pid == pid: + return + # A forked worker owns duplicate descriptors for any inherited file + # handles. Closing them here affects only the worker's copies. + self.close() + self._owner_pid = pid + + @staticmethod + def _close_resource(resource): + decoder, stream = resource + close = getattr(decoder, "close", None) + try: + if close is not None: + close() + finally: + stream.close() diff --git a/paimon-python/pypaimon/read/reader/concat_batch_reader.py b/paimon-python/pypaimon/read/reader/concat_batch_reader.py index 39b7dac1d607..843f194031c9 100644 --- a/paimon-python/pypaimon/read/reader/concat_batch_reader.py +++ b/paimon-python/pypaimon/read/reader/concat_batch_reader.py @@ -23,7 +23,10 @@ from pyarrow import RecordBatch from pypaimon.manifest.schema.data_file_meta import DataFileMeta -from pypaimon.read.reader.format_blob_reader import BlobRecordIterator +from pypaimon.read.reader.format_blob_reader import ( + BlobRecordIterator, + VideoFrameRecordIterator, +) from pypaimon.read.reader.iface.record_batch_reader import RecordBatchReader from pypaimon.schema.data_types import DataField, PyarrowFieldParser from pypaimon.table.row.blob import Blob @@ -588,23 +591,39 @@ def _read_blob_values( return {} try: - blob_lengths = [reader.blob_lengths[pos] for pos, _ in positions_and_row_ids] - blob_offsets = [reader.blob_offsets[pos] for pos, _ in positions_and_row_ids] - iterator = BlobRecordIterator( - reader._file_io, - reader.file_path, - blob_lengths, - blob_offsets, - self._data_field, - reader._input_stream, - blob_as_descriptor=( - self._blob_as_descriptor or self._blob_parallelism > 1 - ), - ) - - blobs = [] - for row in iterator: - blobs.append(row.values[0]) + if getattr(reader, "_is_video", False): + iterator = VideoFrameRecordIterator( + reader._file_io, + reader.file_path, + reader._video_meta, + self._data_field, + ) + blobs = [] + for position, _ in positions_and_row_ids: + iterator.current_position = position + blobs.append(next(iterator).values[0]) + else: + blob_lengths = [ + reader.blob_lengths[pos] + for pos, _ in positions_and_row_ids + ] + blob_offsets = [ + reader.blob_offsets[pos] + for pos, _ in positions_and_row_ids + ] + iterator = BlobRecordIterator( + reader._file_io, + reader.file_path, + blob_lengths, + blob_offsets, + self._data_field, + reader._input_stream, + blob_as_descriptor=( + self._blob_as_descriptor + or self._blob_parallelism > 1 + ), + ) + blobs = [row.values[0] for row in iterator] return { row_id: blob for (_, row_id), blob in zip(positions_and_row_ids, blobs) @@ -647,7 +666,11 @@ def _reader_for_state(self, state: _BlobFileState): if reader is None: state.reader_initialized = True return None - actual_rows = len(reader.blob_lengths) + actual_rows = ( + reader.record_count + if hasattr(reader, "record_count") + else len(reader.blob_lengths) + ) expected_rows = state.selected_count if actual_rows != expected_rows: reader.close() diff --git a/paimon-python/pypaimon/read/reader/format_blob_reader.py b/paimon-python/pypaimon/read/reader/format_blob_reader.py index 1e403f53a0ce..d12b0ebf878b 100644 --- a/paimon-python/pypaimon/read/reader/format_blob_reader.py +++ b/paimon-python/pypaimon/read/reader/format_blob_reader.py @@ -15,6 +15,7 @@ # specific language governing permissions and limitations # under the License. +import bisect import struct from typing import List, Optional, Any, Iterator, BinaryIO @@ -25,6 +26,7 @@ from pypaimon.common.delta_varint_compressor import DeltaVarintCompressor from pypaimon.common.file_io import FileIO from pypaimon.common.map_blob_key_serializer import create_map_blob_key_serializer +from pypaimon.common.uri_reader import UriReader from pypaimon.read.reader.iface.record_batch_reader import RecordBatchReader from pypaimon.schema.data_types import ( DataField, @@ -34,7 +36,7 @@ is_array_blob_type, is_map_blob_type, ) -from pypaimon.table.row.blob import Blob +from pypaimon.table.row.blob import Blob, VideoFrameDescriptor from pypaimon.table.row.generic_row import GenericRow from pypaimon.table.row.row_kind import RowKind @@ -53,6 +55,8 @@ def __init__(self, file_io: FileIO, file_path: str, read_fields: List[str], self._blob_as_descriptor = blob_as_descriptor self._batch_size = batch_size self._blob_parallelism = blob_parallelism + self._is_video = file_path.endswith('.video') + self._video_meta = None # Initialize the low-level blob format reader self.file_path = file_path @@ -96,7 +100,11 @@ def __init__(self, file_io: FileIO, file_path: str, read_fields: List[str], if ( not self._is_array_blob and not self._is_map_blob - and (self._blob_as_descriptor or self._blob_parallelism > 1) + and ( + self._is_video + or self._blob_as_descriptor + or self._blob_parallelism > 1 + ) ): self._input_stream.close() self._input_stream = None @@ -113,13 +121,21 @@ def read_arrow_batch(self, start_idx=None, end_idx=None) -> Optional[RecordBatch if self.returned: return None self.returned = True - batch_iterator = BlobRecordIterator( - self._file_io, self.file_path, self.blob_lengths, - self.blob_offsets, self._data_field, self._input_stream, - blob_as_descriptor=( - self._blob_as_descriptor or self._blob_parallelism > 1 + if self._is_video: + batch_iterator = VideoFrameRecordIterator( + self._file_io, + self.file_path, + self._video_meta, + self._data_field, + ) + else: + batch_iterator = BlobRecordIterator( + self._file_io, self.file_path, self.blob_lengths, + self.blob_offsets, self._data_field, self._input_stream, + blob_as_descriptor=( + self._blob_as_descriptor or self._blob_parallelism > 1 + ) ) - ) self._blob_iterator = iter(batch_iterator) read_size = self._batch_size if start_idx is not None and end_idx is not None: @@ -167,7 +183,7 @@ def read_arrow_batch(self, start_idx=None, end_idx=None) -> Optional[RecordBatch raise RuntimeError( "Blob placeholder is not supported by FormatBlobReader yet." ) - elif self._blob_as_descriptor: + elif self._is_video or self._blob_as_descriptor: pydict_data[field_name].append(blob.to_descriptor().serialize()) elif self._blob_parallelism > 1: idx = len(pydict_data[field_name]) @@ -289,7 +305,19 @@ def close(self): self._input_stream.close() self._input_stream = None + @property + def record_count(self) -> int: + if self._is_video: + return self._video_meta.record_count + return len(self.blob_lengths) + def _read_index(self) -> None: + if self._is_video: + self._video_meta = VideoFileMeta( + self._input_stream, self._file_size + ) + return + f = self._input_stream # Seek to header: last 5 bytes @@ -330,6 +358,10 @@ def _apply_row_indices(self, row_indices: Optional[Any]) -> None: if row_indices is None: return + if self._is_video: + self._video_meta.select(row_indices) + return + selected_lengths = [] selected_offsets = [] record_count = len(self.blob_lengths) @@ -347,6 +379,178 @@ def _apply_row_indices(self, row_indices: Optional[Any]) -> None: self.blob_offsets = selected_offsets +class VideoFileMeta: + """Validated embedded index of a ``.video`` file.""" + + VERSION = 1 + MAGIC_NUMBER = 0x4F454449 + FOOTER_SIZE = 21 + NULL_REFERENCE = -1 + PLACE_HOLDER_REFERENCE = -2 + + def __init__(self, stream, file_size: int): + if file_size < self.FOOTER_SIZE: + raise IOError( + "Corrupt video file: file is smaller than its footer." + ) + footer_start = file_size - self.FOOTER_SIZE + stream.seek(footer_start) + footer = stream.read(self.FOOTER_SIZE) + if len(footer) != self.FOOTER_SIZE: + raise IOError("Corrupt video file: cannot read footer.") + lengths = struct.unpack(' footer_start: + raise IOError("Corrupt video file: indexes exceed the file size.") + index_start = footer_start - total_index_length + indexes = [] + offset = index_start + for name, length in zip( + ("physical video", "run length", "run reference", "first frame"), + index_lengths): + stream.seek(offset) + raw = stream.read(length) + if len(raw) != length: + raise IOError( + "Corrupt video file: cannot read %s index." % name + ) + indexes.append(DeltaVarintCompressor.decompress(raw)) + offset += length + + physical_lengths, run_lengths, references, first_frames = indexes + physical_offsets = [] + payload_offset = 0 + for ordinal, length in enumerate(physical_lengths): + if length <= 0 or length > index_start - payload_offset: + raise IOError( + "Corrupt video file: invalid physical video length %s " + "at ordinal %s." % (length, ordinal) + ) + physical_offsets.append(payload_offset) + payload_offset += length + if payload_offset != index_start: + raise IOError( + "Corrupt video file: indexed videos use %s bytes, but payload " + "region contains %s bytes." % (payload_offset, index_start) + ) + + if not (len(run_lengths) == len(references) == len(first_frames)): + raise IOError( + "Corrupt video file: run indexes have different counts." + ) + run_ends = [] + row_count = 0 + for run, (length, reference, first_frame) in enumerate(zip( + run_lengths, references, first_frames)): + if length <= 0: + raise IOError( + "Corrupt video file: invalid run length %s at run %s." + % (length, run) + ) + if ( + reference not in ( + self.NULL_REFERENCE, self.PLACE_HOLDER_REFERENCE + ) + and (reference < 0 or reference >= len(physical_lengths)) + ): + raise IOError( + "Corrupt video file: run %s references physical video %s, " + "but physical video count is %s." + % (run, reference, len(physical_lengths)) + ) + if reference >= 0 and first_frame < 0: + raise IOError( + "Corrupt video file: run %s has negative first frame %s." + % (run, first_frame) + ) + row_count += length + run_ends.append(row_count) + + self.physical_lengths = physical_lengths + self.physical_offsets = physical_offsets + self.run_ends = run_ends + self.references = references + self.first_frames = first_frames + self.row_count = row_count + self.selected_positions = None + + @property + def record_count(self) -> int: + return ( + self.row_count + if self.selected_positions is None + else len(self.selected_positions) + ) + + def select(self, row_indices) -> None: + selected = [] + for value in row_indices: + position = int(value) + if position < 0 or position >= self.row_count: + raise IndexError( + "Video row index %s is out of range, record count: %s." + % (position, self.row_count) + ) + selected.append(position) + self.selected_positions = selected + + def logical_position(self, returned_row: int) -> int: + if self.selected_positions is None: + return returned_row + return self.selected_positions[returned_row] + + def frame(self, returned_row: int): + logical = self.logical_position(returned_row) + run = bisect.bisect_left(self.run_ends, logical + 1) + reference = self.references[run] + if reference == self.NULL_REFERENCE: + return None + if reference == self.PLACE_HOLDER_REFERENCE: + return Blob.PLACE_HOLDER + run_start = 0 if run == 0 else self.run_ends[run - 1] + return ( + self.physical_offsets[reference], + self.physical_lengths[reference], + self.first_frames[run] + logical - run_start, + ) + + +class VideoFrameRecordIterator: + + def __init__(self, file_io, file_path, meta, field): + self.file_io = file_io + self.file_path = file_path + self.meta = meta + self.field = field + self.current_position = 0 + self._uri_reader = UriReader.from_file(file_io) + + def __iter__(self): + return self + + def __next__(self): + if self.current_position >= self.meta.record_count: + raise StopIteration + value = self.meta.frame(self.current_position) + if isinstance(value, tuple): + offset, length, frame_index = value + descriptor = VideoFrameDescriptor( + self.file_path, offset, length, frame_index + ) + value = Blob.from_descriptor(self._uri_reader, descriptor) + self.current_position += 1 + return GenericRow([value], [self.field], RowKind.INSERT) + + class BlobRecordIterator: MAGIC_NUMBER_SIZE = 4 METADATA_OVERHEAD = 16 diff --git a/paimon-python/pypaimon/read/split_read.py b/paimon-python/pypaimon/read/split_read.py index cc0cca45c955..01cd1aaf4aed 100644 --- a/paimon-python/pypaimon/read/split_read.py +++ b/paimon-python/pypaimon/read/split_read.py @@ -264,6 +264,7 @@ def file_reader_supplier(self, file: DataFileMeta, for_merge_read: bool, parquet_row_ranges = None if effective_row_ranges is not None: row_index_formats = (CoreOptions.FILE_FORMAT_BLOB, + CoreOptions.FILE_FORMAT_VIDEO, CoreOptions.FILE_FORMAT_VORTEX, CoreOptions.FILE_FORMAT_LANCE, CoreOptions.FILE_FORMAT_ROW) @@ -329,7 +330,9 @@ def file_reader_supplier(self, file: DataFileMeta, for_merge_read: bool, list(name_to_field.values()), read_arrow_predicate, batch_size=batch_size, nested_name_paths=avro_nested_paths) - elif file_format == CoreOptions.FILE_FORMAT_BLOB: + elif file_format in ( + CoreOptions.FILE_FORMAT_BLOB, + CoreOptions.FILE_FORMAT_VIDEO): if has_nested: raise NotImplementedError( "Nested-field projection is not supported on BLOB files") @@ -482,6 +485,9 @@ def file_reader_supplier(self, file: DataFileMeta, for_merge_read: bool, def _read_blob_as_descriptor(self, field_names: List[str]) -> bool: if CoreOptions.blob_as_descriptor(self.table.options): return True + video_fields = CoreOptions.video_frame_fields(self.table.options) + if any(field_name in video_fields for field_name in field_names): + return True deferred_fields = getattr(self, '_deferred_blob_fields', set()) return any(field_name in deferred_fields for field_name in field_names) diff --git a/paimon-python/pypaimon/schema/schema_manager.py b/paimon-python/pypaimon/schema/schema_manager.py index 928e177a8b8b..1e6c8d8a5bfe 100644 --- a/paimon-python/pypaimon/schema/schema_manager.py +++ b/paimon-python/pypaimon/schema/schema_manager.py @@ -393,6 +393,19 @@ def _validate_blob_fields( descriptor_fields = core_options.blob_descriptor_fields() view_fields = core_options.blob_view_fields() + video_fields = core_options.video_frame_fields() + + if len(video_fields) > 1: + raise ValueError( + "'video-frame-field' currently supports exactly one field, but found " + f"{sorted(video_fields)}." + ) + non_scalar_video_fields = video_fields.difference(scalar_blob_field_names) + if non_scalar_video_fields: + raise ValueError( + "Fields in 'video-frame-field' must be scalar BLOB fields in schema. " + f"Invalid fields: {sorted(non_scalar_video_fields)}" + ) all_inline_fields = descriptor_fields.union(view_fields) non_blob_inline_fields = all_inline_fields.difference(scalar_blob_field_names) @@ -421,6 +434,15 @@ def _validate_blob_fields( "Overlapping fields: {}".format(sorted(overlapping_inline_fields)) ) + overlapping_video_fields = video_fields.intersection(all_inline_fields) + if overlapping_video_fields: + raise ValueError( + "Fields in 'video-frame-field' must not also use descriptor-only or " + "blob-view storage. Overlapping fields: {}".format( + sorted(overlapping_video_fields) + ) + ) + if blob_field_names: required_options = { CoreOptions.ROW_TRACKING_ENABLED.key(): 'true', diff --git a/paimon-python/pypaimon/table/row/blob.py b/paimon-python/pypaimon/table/row/blob.py index 5a0195c78a16..9af8f4327550 100644 --- a/paimon-python/pypaimon/table/row/blob.py +++ b/paimon-python/pypaimon/table/row/blob.py @@ -65,6 +65,13 @@ def serialize(self) -> bytes: @classmethod def deserialize(cls, data: bytes) -> 'BlobDescriptor': + video_type = globals().get('VideoFrameDescriptor') + if ( + cls is BlobDescriptor + and video_type is not None + and video_type.is_video_frame_descriptor(data) + ): + return video_type.deserialize(data) if len(data) < 5: raise ValueError("Invalid BlobDescriptor data: too short") @@ -163,6 +170,117 @@ def __repr__(self) -> str: return self.__str__() +class VideoFrameDescriptor(BlobDescriptor): + """Descriptor for one logical frame in an encoded video payload. + + The URI range identifies the complete encoded video. ``frame_index`` is a + zero-based presentation-order frame ordinal interpreted by the decoder. + """ + + CURRENT_VERSION = 1 + MAGIC = 0x564944454F46524D # "VIDEOFRM" + _FIXED_LENGTH = 1 + 8 + 4 + 8 + 8 + 8 + + def __init__(self, uri: str, offset: int, length: int, frame_index: int): + if isinstance(frame_index, bool) or not isinstance(frame_index, int): + raise TypeError("Video frame index must be an int.") + if frame_index < 0: + raise ValueError( + "Video frame index must be non-negative, but was %s." + % frame_index + ) + super().__init__(uri, offset, length) + self._frame_index = frame_index + + @property + def frame_index(self) -> int: + return self._frame_index + + @property + def payload_descriptor(self) -> BlobDescriptor: + """Physical video identity without the logical frame locator.""" + return BlobDescriptor(self.uri, self.offset, self.length) + + def serialize(self) -> bytes: + uri_bytes = self.uri.encode('utf-8') + return ( + struct.pack(' 'VideoFrameDescriptor': + if not isinstance(data, (bytes, bytearray)): + raise TypeError( + "VideoFrameDescriptor.deserialize expects bytes, got %s" + % type(data) + ) + raw = bytes(data) + if len(raw) < cls._FIXED_LENGTH: + raise ValueError("Invalid VideoFrameDescriptor data: too short") + + version, magic, uri_length = struct.unpack(' expected_length + else "invalid URI length: %s" % uri_length + ) + raise ValueError("Invalid VideoFrameDescriptor data: " + message) + + uri_end = 13 + uri_length + uri = raw[13:uri_end].decode('utf-8') + offset, length, frame_index = struct.unpack( + ' bool: + if not isinstance(data, (bytes, bytearray)) or len(data) < 9: + return False + raw = bytes(data) + return ( + raw[0] == cls.CURRENT_VERSION + and struct.unpack(' bool: + return ( + isinstance(other, VideoFrameDescriptor) + and self.payload_descriptor == other.payload_descriptor + and self.frame_index == other.frame_index + ) + + def __hash__(self) -> int: + return hash((self.payload_descriptor, self.frame_index)) + + def __str__(self) -> str: + return ( + "VideoFrameDescriptor(payload=%s, frame_index=%s)" + % (self.payload_descriptor, self.frame_index) + ) + + class BlobViewStruct: CURRENT_VERSION = 1 MAGIC = 0x424C4F4256494557 # "BLOBVIEW" @@ -401,13 +519,18 @@ def from_bytes( data = bytes(data) if BlobViewStruct.is_blob_view_struct(data): return Blob.from_view(BlobViewStruct.deserialize(data)) - is_descriptor = BlobDescriptor.is_blob_descriptor(data) + is_video_frame = VideoFrameDescriptor.is_video_frame_descriptor(data) + is_descriptor = is_video_frame or BlobDescriptor.is_blob_descriptor(data) if not allow_blob_data and not is_descriptor: raise ValueError( "Expected BlobDescriptor bytes, got raw bytes (allow_blob_data=False)" ) if is_descriptor: - descriptor = BlobDescriptor.deserialize(data) + descriptor = ( + VideoFrameDescriptor.deserialize(data) + if is_video_frame + else BlobDescriptor.deserialize(data) + ) if uri_reader_factory is None: if file_io is None: raise ValueError("file_io is required to resolve BlobDescriptor bytes") diff --git a/paimon-python/pypaimon/tests/data_evolution_row_rolling_test.py b/paimon-python/pypaimon/tests/data_evolution_row_rolling_test.py index 63bebc717310..5e74114460d6 100644 --- a/paimon-python/pypaimon/tests/data_evolution_row_rolling_test.py +++ b/paimon-python/pypaimon/tests/data_evolution_row_rolling_test.py @@ -25,7 +25,7 @@ from pypaimon import CatalogFactory, Schema from pypaimon.common.uri_reader import FileUriReader -from pypaimon.table.row.blob import Blob +from pypaimon.table.row.blob import Blob, BlobDescriptor, VideoFrameDescriptor class DataEvolutionRowRollingTest(unittest.TestCase): @@ -196,6 +196,60 @@ def test_blob_writer_supports_target_file_row_num(self): self.assertEqual([1, 3, 3], blob_rows) self.assertEqual(list(range(7)), self._read_ids(table)) + def test_video_writer_rolls_between_payload_groups(self): + first = os.path.join(self.tempdir, 'first.mp4') + second = os.path.join(self.tempdir, 'second.mp4') + with open(first, 'wb') as output: + output.write(b'first-video') + with open(second, 'wb') as output: + output.write(b'second-video') + first_descriptor = BlobDescriptor(first, 0, len(b'first-video')) + second_descriptor = BlobDescriptor(second, 0, len(b'second-video')) + + table = self._create_with_schema( + self.blob_schema, + { + **self.de_options, + 'target-file-row-num': '1', + 'video-frame-field': 'payload', + }, + ) + data = pa.Table.from_pydict( + { + 'id': list(range(5)), + 'payload': [ + VideoFrameDescriptor( + first_descriptor.uri, + first_descriptor.offset, + first_descriptor.length, + frame, + ).serialize() + for frame in range(3) + ] + [ + VideoFrameDescriptor( + second_descriptor.uri, + second_descriptor.offset, + second_descriptor.length, + frame, + ).serialize() + for frame in range(2) + ], + }, + schema=self.blob_schema, + ) + + files = self._write_files(table, data) + + video_rows = sorted( + f.row_count for f in files if f.file_name.endswith('.video') + ) + normal_rows = sorted( + f.row_count for f in files if not f.file_name.endswith('.video') + ) + self.assertEqual([2, 3], video_rows) + self.assertEqual([2, 3], normal_rows) + self.assertEqual(list(range(5)), self._read_ids(table)) + def test_blob_consumer_descriptors_survive_abort_after_rolling(self): table = self._create_with_schema( self.blob_schema, diff --git a/paimon-python/pypaimon/tests/multimodal_table_test.py b/paimon-python/pypaimon/tests/multimodal_table_test.py index 4a826626c839..85794d232c9b 100644 --- a/paimon-python/pypaimon/tests/multimodal_table_test.py +++ b/paimon-python/pypaimon/tests/multimodal_table_test.py @@ -21,6 +21,7 @@ import shutil import tempfile import unittest +from unittest.mock import patch import pyarrow as pa import pypaimon.multimodal as pmm @@ -117,6 +118,298 @@ def test_create_table_defaults_data_evolution_options(self): self.assertEqual(["id", "content", "embedding", "payload"], [field.name for field in table.raw_table.fields]) + def test_add_video_embeds_frame_ordinals_outside_normal_data(self): + table = self.conn.create_table( + "video_frames", + schema=_schema({ + "episode_id": pa.int64(), + "video": pa.large_binary(), + }), + options=dict(_PARQUET_OPTIONS, **{ + "video-frame-field": "video", + "blob-as-descriptor": "true", + }), + ) + video_path = os.path.join(self.temp_dir, "episode-42.mp4") + video_bytes = b"fake-mp4-payload" + with open(video_path, "wb") as output: + output.write(video_bytes) + video = pmm.Blob.from_local(video_path) + + table.add_video( + video, + [{"episode_id": 42} for _ in range(3)], + first_frame=7, + ) + + rows = table.scan().select(["episode_id", "video"]).to_list() + descriptors = [ + pmm.VideoFrameDescriptor.deserialize(row["video"]) + for row in rows + ] + self.assertEqual([7, 8, 9], [value.frame_index for value in descriptors]) + self.assertTrue(all( + value.payload_descriptor == descriptors[0].payload_descriptor + for value in descriptors + )) + self.assertTrue(descriptors[0].uri.endswith(".video")) + self.assertEqual(len(video_bytes), descriptors[0].length) + + _, bodies = table.scan().read_blobs("video", parallelism=2) + self.assertEqual([video_bytes] * 3, bodies["video"]) + + def test_add_videos_packs_multiple_videos_in_one_commit(self): + from pypaimon.table.row.blob import Blob, VideoFrameDescriptor + + table = self.conn.create_table( + "batched_video_frames", + schema=_schema({ + "episode_id": pa.int64(), + "video": pa.large_binary(), + }), + options=dict(_PARQUET_OPTIONS, **{ + "video-frame-field": "video", + "blob-as-descriptor": "true", + }), + ) + paths = [os.path.join(self.temp_dir, "episode-%s.mp4" % index) + for index in (1, 2)] + payloads = [b"first-video", b"second-video"] + for path, payload in zip(paths, payloads): + with open(path, "wb") as output: + output.write(payload) + + table.add_videos([ + (Blob.from_local(paths[0]), [{"episode_id": 1}] * 2, 3), + (Blob.from_local(paths[1]), [{"episode_id": 2}] * 3, 10), + ]) + + snapshot = table.raw_table.snapshot_manager().get_latest_snapshot() + self.assertEqual(1, snapshot.id) + rows = table.scan().select(["episode_id", "video"]).to_list() + rows.sort(key=lambda row: (row["episode_id"], row["video"])) + descriptors = [ + VideoFrameDescriptor.deserialize(row["video"]) + for row in rows + ] + self.assertEqual([3, 4, 10, 11, 12], [d.frame_index for d in descriptors]) + self.assertEqual(1, len({d.uri for d in descriptors})) + self.assertTrue(descriptors[0].uri.endswith(".video")) + self.assertEqual(2, len({d.payload_descriptor for d in descriptors})) + + def test_normal_update_preserves_video_descriptors(self): + from pypaimon.table.row.blob import Blob + + table = self.conn.create_table( + "update_video_frame_metadata", + schema=_schema({ + "episode_id": pa.int64(), + "frame_id": pa.int32(), + "label": pa.string(), + "video": pa.large_binary(), + }), + options=dict(_PARQUET_OPTIONS, **{ + "video-frame-field": "video", + "blob-as-descriptor": "false", + }), + ) + video_path = os.path.join(self.temp_dir, "update-metadata.mp4") + with open(video_path, "wb") as output: + output.write(b"video-kept-by-normal-update") + table.add_video( + Blob.from_local(video_path), + [ + {"episode_id": 42, "frame_id": index, "label": "old"} + for index in range(3) + ], + ) + before = sorted( + table.scan().select(["frame_id", "video"]).to_list(), + key=lambda row: row["frame_id"], + ) + + table.update("episode_id = 42", {"label": "new"}) + + after = sorted( + table.scan().select(["frame_id", "label", "video"]).to_list(), + key=lambda row: row["frame_id"], + ) + self.assertEqual(["new"] * 3, [row["label"] for row in after]) + self.assertEqual( + [row["video"] for row in before], + [row["video"] for row in after], + ) + + def test_update_rejects_video_field(self): + table = self.conn.create_table( + "reject_generic_video_update", + schema=_schema({ + "frame_id": pa.int32(), + "video": pa.large_binary(), + }), + options=dict(_PARQUET_OPTIONS, **{ + "video-frame-field": "video", + }), + ) + + with self.assertRaisesRegex(ValueError, "replace_video"): + table.update("frame_id = 0", {"video": b"not-an-mp4"}) + + self.assertIsNone( + table.raw_table.snapshot_manager().get_latest_snapshot() + ) + + def test_replace_video_updates_only_matching_frame_rows(self): + from pypaimon.table.row.blob import Blob, VideoFrameDescriptor + + table = self.conn.create_table( + "replace_video_frames", + schema=_schema({ + "episode_id": pa.int64(), + "frame_id": pa.int32(), + "video": pa.large_binary(), + }), + options=dict(_PARQUET_OPTIONS, **{ + "video-frame-field": "video", + "blob-as-descriptor": "false", + }), + ) + old_path = os.path.join(self.temp_dir, "old-episode.mp4") + new_path = os.path.join(self.temp_dir, "new-episode.mp4") + old_payload = b"old-complete-video" + new_payload = b"new-complete-video" + with open(old_path, "wb") as output: + output.write(old_payload) + with open(new_path, "wb") as output: + output.write(new_payload) + table.add_video( + Blob.from_local(old_path), + [ + {"episode_id": 42, "frame_id": index} + for index in range(3) + ], + ) + + table.replace_video( + "frame_id >= 1", + Blob.from_local(new_path), + first_frame=20, + ) + + rows = sorted( + table.scan().select(["frame_id", "video"]).to_list(), + key=lambda row: row["frame_id"], + ) + descriptors = [ + VideoFrameDescriptor.deserialize(row["video"]) + for row in rows + ] + self.assertEqual([0, 20, 21], [value.frame_index for value in descriptors]) + self.assertNotEqual(descriptors[0].uri, descriptors[1].uri) + self.assertEqual(descriptors[1].payload_descriptor, + descriptors[2].payload_descriptor) + self.assertTrue(descriptors[1].uri.endswith(".video")) + + scalar, bodies = table.scan().read_blobs("video", parallelism=2) + body_by_frame = dict(zip( + scalar["frame_id"].to_pylist(), bodies["video"] + )) + self.assertEqual({ + 0: old_payload, + 1: new_payload, + 2: new_payload, + }, body_by_frame) + + def test_add_batches_aborts_before_commit_on_invalid_video_frame(self): + from pypaimon.table.row.blob import Blob, VideoFrameDescriptor + + table = self.conn.create_table( + "failed_batched_video_frames", + schema=_schema({ + "episode_id": pa.int32(), + "video": pa.large_binary(), + }), + options=dict(_PARQUET_OPTIONS, **{ + "video-frame-field": "video", + }), + ) + video_path = os.path.join(self.temp_dir, "valid-before-failure.mp4") + with open(video_path, "wb") as output: + output.write(b"valid-video") + + payload = Blob.from_local(video_path).to_descriptor() + valid = VideoFrameDescriptor( + payload.uri, payload.offset, payload.length, 0 + ).serialize() + with self.assertRaisesRegex(ValueError, "VideoFrameDescriptor"): + table.add_batches([ + [{"episode_id": 1, "video": valid}], + [{"episode_id": 1, "video": b"inline-is-invalid"}], + ]) + + self.assertIsNone( + table.raw_table.snapshot_manager().get_latest_snapshot() + ) + + def test_add_batches_empty_iterable_is_noop(self): + table = self.conn.create_table( + "empty_batched_frames", + schema=_schema({"frame_index": pa.int32()}), + options=_PARQUET_OPTIONS, + ) + + self.assertIs(table, table.add_batches(iter(()))) + self.assertIsNone( + table.raw_table.snapshot_manager().get_latest_snapshot() + ) + + def test_scan_to_torch_keeps_blob_descriptors(self): + table = self.conn.create_table( + "torch_video_frames", + schema=_schema({ + "episode_id": pa.int32(), + "video": pa.large_binary(), + }), + options=dict(_PARQUET_OPTIONS, **{ + "video-frame-field": "video", + "blob-as-descriptor": "false", + }), + ) + from pypaimon.table.row.blob import Blob + video_path = os.path.join(self.temp_dir, "torch-episode.mp4") + with open(video_path, "wb") as output: + output.write(b"fake-video") + table.add_video( + Blob.from_local(video_path), [{"episode_id": 1}] + ) + sentinel = object() + captured = {} + + def fake_to_torch(table_read, splits, **kwargs): + captured["descriptor"] = table_read.table.options.blob_as_descriptor() + captured["splits"] = splits + captured["kwargs"] = kwargs + return sentinel + + with patch( + "pypaimon.read.table_read.TableRead.to_torch", + autospec=True, + side_effect=fake_to_torch, + ): + result = table.scan().select( + ["episode_id", "video"] + ).to_torch( + streaming=True, + prefetch_concurrency=2, + shuffle=False, + ) + + self.assertIs(sentinel, result) + self.assertTrue(captured["descriptor"]) + self.assertTrue(captured["splits"]) + self.assertEqual(True, captured["kwargs"]["streaming"]) + self.assertEqual(2, captured["kwargs"]["prefetch_concurrency"]) + def test_create_table_uses_options_and_partitioned(self): table = self.conn.create_table( "docs", diff --git a/paimon-python/pypaimon/tests/multimodal_video_test.py b/paimon-python/pypaimon/tests/multimodal_video_test.py new file mode 100644 index 000000000000..438e7f2556df --- /dev/null +++ b/paimon-python/pypaimon/tests/multimodal_video_test.py @@ -0,0 +1,180 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import io +import os +import tempfile +import unittest +from types import SimpleNamespace + +from pypaimon.common.file_io import FileIO +from pypaimon.multimodal import VideoFrameCollator +from pypaimon.table.row.blob import VideoFrameDescriptor + + +class _Decoder: + + def __init__(self, stream, calls): + self._stream = stream + self._calls = calls + self.closed = False + + def decode(self, frame_index): + self._stream.seek(0) + return self._stream.read(), frame_index + + def close(self): + self.closed = True + self._calls.append("close") + + +class VideoFrameCollatorTest(unittest.TestCase): + + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory() + self.file_io = FileIO.get("file://" + self.temp_dir.name, {}) + self.table = SimpleNamespace( + raw_table=SimpleNamespace(file_io=self.file_io) + ) + + def tearDown(self): + self.temp_dir.cleanup() + + def test_reuses_decoder_for_rows_with_same_descriptor(self): + descriptors = [ + self._descriptor("episode-1.mp4", b"video-one", frame) + for frame in (0, 1) + ] + factory_calls = [] + + def factory(stream): + factory_calls.append("open") + return _Decoder(stream, factory_calls) + + collator = VideoFrameCollator( + self.table, + video_column="video", + decoder_factory=factory, + decode_fn=lambda decoder, frame, row: decoder.decode(frame), + collate_fn=lambda rows: rows, + ) + try: + result = collator([ + {"episode_id": 1, "video": descriptors[0]}, + {"episode_id": 1, "video": descriptors[1]}, + ]) + finally: + collator.close() + + self.assertEqual(["open", "close"], factory_calls) + self.assertEqual( + [(b"video-one", 0), (b"video-one", 1)], + [row["frame"] for row in result], + ) + self.assertEqual(descriptors[0], result[0]["video"]) + + def test_evicts_least_recently_used_decoder(self): + descriptors = [ + self._descriptor("episode-%d.mp4" % index, bytes([index]), index) + for index in range(3) + ] + factory_calls = [] + + def factory(stream): + factory_calls.append("open") + return _Decoder(stream, factory_calls) + + collator = VideoFrameCollator( + self.table, + video_column="video", + decoder_factory=factory, + decode_fn=lambda decoder, frame, row: decoder.decode(frame), + max_open_videos=2, + collate_fn=lambda rows: rows, + ) + try: + collator([ + {"episode_id": index, "video": descriptor} + for index, descriptor in enumerate(descriptors) + ]) + self.assertEqual(3, factory_calls.count("open")) + self.assertEqual(1, factory_calls.count("close")) + + collator([{"episode_id": 3, "video": descriptors[0]}]) + self.assertEqual(4, factory_calls.count("open")) + self.assertEqual(2, factory_calls.count("close")) + finally: + collator.close() + + self.assertEqual(4, factory_calls.count("close")) + + def test_rejects_non_descriptor_video_cell(self): + collator = VideoFrameCollator( + self.table, + video_column="video", + decoder_factory=lambda stream: _Decoder(stream, []), + decode_fn=lambda decoder, frame, row: decoder.decode(frame), + collate_fn=lambda rows: rows, + ) + valid = self._descriptor("valid.mp4", b"video", 0) + for value in (b"inline-mp4", valid + b"trailing"): + with self.subTest(value=value): + with self.assertRaisesRegex(ValueError, "VideoFrameDescriptor"): + collator([{"episode_id": 0, "video": value}]) + + def test_reuses_resolved_table_file_io(self): + class ResolvedFileIO: + + @property + def uri_reader_factory(self): + raise AssertionError("must not rebuild a URI reader") + + def new_input_stream(self, path): + self.path = path + return io.BytesIO(b"resolved-video") + + file_io = ResolvedFileIO() + table = SimpleNamespace(raw_table=SimpleNamespace(file_io=file_io)) + descriptor = VideoFrameDescriptor( + "oss://bucket/internal.video", 0, 14, 2 + ).serialize() + collator = VideoFrameCollator( + table, + video_column="video", + decoder_factory=lambda stream: _Decoder(stream, []), + decode_fn=lambda decoder, frame, row: decoder.decode(frame), + collate_fn=lambda rows: rows, + ) + try: + result = collator([{"episode_id": 1, "video": descriptor}]) + finally: + collator.close() + + self.assertEqual("oss://bucket/internal.video", file_io.path) + self.assertEqual((b"resolved-video", 2), result[0]["frame"]) + + def _descriptor(self, name, data, frame_index): + path = os.path.join(self.temp_dir.name, name) + with open(path, "wb") as output: + output.write(data) + return VideoFrameDescriptor( + path, 0, len(data), frame_index + ).serialize() + + +if __name__ == "__main__": + unittest.main() diff --git a/paimon-python/pypaimon/tests/torch_read_test.py b/paimon-python/pypaimon/tests/torch_read_test.py index 2b3b126b6c2f..8d0abca713f8 100644 --- a/paimon-python/pypaimon/tests/torch_read_test.py +++ b/paimon-python/pypaimon/tests/torch_read_test.py @@ -28,6 +28,7 @@ from torch.utils.data import DataLoader from pypaimon import CatalogFactory, Schema +from pypaimon.multimodal.table import MultimodalTable from pypaimon.table.file_store_table import FileStoreTable @@ -579,6 +580,59 @@ def test_blob_torch_read(self): print(f"✓ Blob torch read test passed: Successfully read and verified {len(blob_data)} bytes of blob data") + def test_video_frame_rows_through_streaming_dataloader(self): + from pypaimon.table.row.blob import Blob, VideoFrameDescriptor + + pa_schema = pa.schema([ + ('episode_id', pa.int64()), + ('video', pa.large_binary()), + ]) + schema = Schema.from_pyarrow_schema( + pa_schema, + options={ + 'row-tracking.enabled': 'true', + 'data-evolution.enabled': 'true', + 'video-frame-field': 'video', + # ScanQuery.to_torch must override this read setting. + 'blob-as-descriptor': 'false', + }, + ) + identifier = 'default.test_shared_video_torch_read' + self.catalog.create_table(identifier, schema, False) + raw_table = self.catalog.get_table(identifier) + table = MultimodalTable(self.catalog, identifier, raw_table) + + video_path = os.path.join(self.tempdir, 'shared-video.mp4') + video_bytes = b'one-physical-video' + with open(video_path, 'wb') as output: + output.write(video_bytes) + video = Blob.from_local(video_path) + table.add_video(video, [{'episode_id': 7} for _ in range(8)]) + + dataset = table.scan().select([ + 'episode_id', 'video' + ]).to_torch(streaming=True) + rows = [] + for batch in DataLoader( + dataset, batch_size=2, num_workers=2, shuffle=False): + rows.extend(zip( + batch['episode_id'].tolist(), + batch['video'], + )) + + descriptors = [ + VideoFrameDescriptor.deserialize(row[1]) + for row in rows + ] + descriptors.sort(key=lambda value: value.frame_index) + self.assertEqual(list(range(8)), [d.frame_index for d in descriptors]) + self.assertTrue(all( + value.payload_descriptor == descriptors[0].payload_descriptor + for value in descriptors + )) + self.assertTrue(descriptors[0].uri.endswith('.video')) + self.assertEqual(len(video_bytes), descriptors[0].length) + def test_torch_read_pk_table(self): """Test torch read with primary key table.""" # Create PK table with user_id as primary key and behavior as partition key diff --git a/paimon-python/pypaimon/tests/video_format_test.py b/paimon-python/pypaimon/tests/video_format_test.py new file mode 100644 index 000000000000..71ed6c94ae25 --- /dev/null +++ b/paimon-python/pypaimon/tests/video_format_test.py @@ -0,0 +1,232 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import struct +import tempfile +import unittest +from pathlib import Path + +from pypaimon.common.delta_varint_compressor import DeltaVarintCompressor +from pypaimon.common.options import Options +from pypaimon.filesystem.local_file_io import LocalFileIO +from pypaimon.read.reader.format_blob_reader import FormatBlobReader, VideoFileMeta +from pypaimon.schema.data_types import AtomicType, DataField +from pypaimon.table.row.blob import ( + Blob, + BlobData, + BlobDescriptor, + VideoFrameDescriptor, +) +from pypaimon.table.row.generic_row import GenericRow +from pypaimon.table.row.row_kind import RowKind +from pypaimon.write.video_format_writer import VideoFormatWriter + + +class VideoFormatTest(unittest.TestCase): + + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory() + self.root = Path(self.temp_dir.name) + self.file_io = LocalFileIO(str(self.root), Options({})) + self.field = DataField(0, "video", AtomicType("BLOB")) + + def tearDown(self): + self.temp_dir.cleanup() + + def test_descriptor_round_trip_preserves_payload_and_frame(self): + descriptor = VideoFrameDescriptor("s3://bucket/a.video", 7, 99, 42) + serialized = descriptor.serialize() + + self.assertTrue( + VideoFrameDescriptor.is_video_frame_descriptor(serialized) + ) + self.assertFalse(BlobDescriptor.is_blob_descriptor(serialized)) + self.assertEqual(descriptor, BlobDescriptor.deserialize(serialized)) + self.assertEqual(descriptor, VideoFrameDescriptor.deserialize(serialized)) + self.assertEqual( + BlobDescriptor("s3://bucket/a.video", 7, 99), + descriptor.payload_descriptor, + ) + restored = Blob.from_bytes(serialized, file_io=self.file_io) + self.assertIsInstance(restored.to_descriptor(), VideoFrameDescriptor) + self.assertEqual(descriptor, restored.to_descriptor()) + with self.assertRaisesRegex(ValueError, "trailing bytes"): + VideoFrameDescriptor.deserialize(serialized + b"x") + with self.assertRaisesRegex(ValueError, "non-negative"): + VideoFrameDescriptor("x", 0, 1, -1) + + def test_pack_raw_videos_and_map_frame_runs(self): + first_bytes = b"first-mp4" + second_bytes = b"second-mp4" + first0 = self._source_frame("first.mp4", first_bytes, 0) + first1 = self._source_frame("first.mp4", first_bytes, 1) + second7 = self._source_frame("second.mp4", second_bytes, 7) + first4 = self._source_frame("first.mp4", first_bytes, 4) + target = (self.root / "data.video").as_uri() + + writer = VideoFormatWriter( + self.file_io.new_output_stream(target), file_path=target + ) + for value in (first0, first1, second7, first4, None, Blob.PLACE_HOLDER): + writer.add_element( + GenericRow([value], [self.field], RowKind.INSERT) + ) + self.assertEqual(2, writer.physical_video_count) + self.assertEqual(5, writer.run_count) + writer.close() + + stored = (self.root / "data.video").read_bytes() + self.assertTrue(stored.startswith(first_bytes + second_bytes)) + with self.file_io.new_input_stream(target) as stream: + meta = VideoFileMeta(stream, len(stored)) + self.assertEqual(6, meta.record_count) + self.assertEqual((0, len(first_bytes), 0), meta.frame(0)) + self.assertEqual((0, len(first_bytes), 1), meta.frame(1)) + self.assertEqual( + (len(first_bytes), len(second_bytes), 7), meta.frame(2) + ) + self.assertEqual((0, len(first_bytes), 4), meta.frame(3)) + self.assertIsNone(meta.frame(4)) + self.assertIs(Blob.PLACE_HOLDER, meta.frame(5)) + + values = self._read(target, row_indices=range(5)) + frames = [VideoFrameDescriptor.deserialize(value) for value in values[:4]] + self.assertEqual([0, 1, 7, 4], [frame.frame_index for frame in frames]) + self.assertEqual(frames[0].payload_descriptor, frames[1].payload_descriptor) + self.assertEqual(frames[0].payload_descriptor, frames[3].payload_descriptor) + self.assertNotEqual(frames[0].payload_descriptor, frames[2].payload_descriptor) + self.assertIsNone(values[4]) + + def test_selection_keeps_logical_frame_positions(self): + target = (self.root / "selection.video").as_uri() + writer = VideoFormatWriter(self.file_io.new_output_stream(target)) + source = [ + self._source_frame("selection.mp4", b"video", frame) + for frame in range(4) + ] + for value in source: + writer.add_element(GenericRow([value], [self.field], RowKind.INSERT)) + writer.close() + + values = self._read(target, row_indices=[1, 3]) + frames = [VideoFrameDescriptor.deserialize(value) for value in values] + self.assertEqual([1, 3], [frame.frame_index for frame in frames]) + + def test_reader_exposes_record_count_without_blob_indexes(self): + target = (self.root / "record-count.video").as_uri() + writer = VideoFormatWriter(self.file_io.new_output_stream(target)) + for frame_index in range(4): + frame = self._source_frame( + "record-count.mp4", b"video", frame_index + ) + writer.add_element( + GenericRow([frame], [self.field], RowKind.INSERT) + ) + writer.close() + + reader = FormatBlobReader( + file_io=self.file_io, + file_path=target, + read_fields=["video"], + full_fields=[self.field], + push_down_predicate=None, + blob_as_descriptor=True, + row_indices=[1, 3], + ) + try: + self.assertEqual(2, reader.record_count) + self.assertEqual([], reader.blob_lengths) + self.assertEqual([], reader.blob_offsets) + finally: + reader.close() + + def test_rejects_non_video_frame_input(self): + target = (self.root / "reject.video").as_uri() + writer = VideoFormatWriter(self.file_io.new_output_stream(target)) + for value in ( + BlobData(b"inline"), + self._source_blob("ordinary.mp4", b"ordinary"), + ): + with self.subTest(value=value): + with self.assertRaisesRegex(ValueError, "VideoFrameDescriptor"): + writer.add_element( + GenericRow([value], [self.field], RowKind.INSERT) + ) + writer.close() + + def test_rejects_out_of_range_run_reference(self): + target_path = self.root / "corrupt.video" + indexes = [ + DeltaVarintCompressor.compress([]), + DeltaVarintCompressor.compress([1]), + DeltaVarintCompressor.compress([0]), + DeltaVarintCompressor.compress([0]), + ] + target_path.write_bytes( + b"".join(indexes) + + struct.pack( + ' Blob: if isinstance(col_data, Blob): return col_data if isinstance(col_data, bytes): + if VideoFrameDescriptor.is_video_frame_descriptor(col_data): + if uri_reader_factory is None: + raise RuntimeError("uri_reader_factory is required for descriptor bytes.") + descriptor = VideoFrameDescriptor.deserialize(col_data) + uri_reader = uri_reader_factory.create(descriptor.uri) + return Blob.from_descriptor(uri_reader, descriptor) if BlobDescriptor.is_blob_descriptor(col_data): if uri_reader_factory is None: raise RuntimeError("uri_reader_factory is required for BlobDescriptor bytes.") diff --git a/paimon-python/pypaimon/write/table_update_by_row_id.py b/paimon-python/pypaimon/write/table_update_by_row_id.py index 943ca29b7920..783336d1d04d 100644 --- a/paimon-python/pypaimon/write/table_update_by_row_id.py +++ b/paimon-python/pypaimon/write/table_update_by_row_id.py @@ -793,6 +793,10 @@ def _write_group( 0, column_name, self.table.options, + video=( + column_name + in self.table.options.video_frame_fields() + ), ) blob_writers.append(blob_writer) arrow_type = original_data.schema.field(column_name).type diff --git a/paimon-python/pypaimon/write/video_format_writer.py b/paimon-python/pypaimon/write/video_format_writer.py new file mode 100644 index 000000000000..ffce9417f6ad --- /dev/null +++ b/paimon-python/pypaimon/write/video_format_writer.py @@ -0,0 +1,184 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import struct + +from pypaimon.common.delta_varint_compressor import DeltaVarintCompressor +from pypaimon.schema.data_types import is_blob_type +from pypaimon.table.row.blob import ( + Blob, + BlobRef, + VideoFrameDescriptor, +) +from pypaimon.write.blob_format_writer import BlobFormatWriter + + +class VideoFormatWriter(BlobFormatWriter): + """Pack complete encoded videos and an embedded logical frame-run index.""" + + VERSION = 1 + FOOTER_MAGIC_NUMBER = 0x4F454449 + FOOTER_SIZE = 21 + NULL_REFERENCE = -1 + PLACE_HOLDER_REFERENCE = -2 + + def __init__( + self, + output_stream, + file_path=None, + copy_buffer_size=BlobFormatWriter.BUFFER_SIZE): + super().__init__( + output_stream, + blob_consumer=None, + file_path=file_path, + copy_buffer_size=copy_buffer_size, + ) + self._physical_lengths = [] + self._physical_videos = {} + self._run_lengths = [] + self._run_references = [] + self._run_first_frames = [] + self._current_run_length = 0 + self._current_run_reference = None + self._current_run_first_frame = 0 + self._current_run_last_frame = 0 + self._closed = False + + def add_element(self, row) -> None: + if not hasattr(row, 'values') or len(row.values) != 1: + raise ValueError("VideoFormatWriter only supports one field") + if not is_blob_type(row.fields[0].type): + raise ValueError( + "VideoFormatWriter only supports one scalar BLOB field" + ) + + value = row.values[0] + if value is None: + self._append(self.NULL_REFERENCE, 0) + return + if value is Blob.PLACE_HOLDER: + self._append(self.PLACE_HOLDER_REFERENCE, 0) + return + if type(value) is not BlobRef: + raise ValueError( + "Video fields require an exact BlobRef containing a " + "VideoFrameDescriptor." + ) + + frame = value.to_descriptor() + if not isinstance(frame, VideoFrameDescriptor): + raise ValueError( + "Video fields require an exact BlobRef containing a " + "VideoFrameDescriptor." + ) + payload = frame.payload_descriptor + ordinal = self._physical_videos.get(payload) + if ordinal is None: + length = self._write_video_payload(value) + ordinal = len(self._physical_lengths) + self._physical_lengths.append(length) + self._physical_videos[payload] = ordinal + self._append(ordinal, frame.frame_index) + + @property + def physical_video_count(self) -> int: + return len(self._physical_lengths) + + @property + def run_count(self) -> int: + return len(self._run_lengths) + (1 if self._current_run_length else 0) + + def close(self) -> None: + if self._closed: + return + self._flush_run() + physical_index = DeltaVarintCompressor.compress(self._physical_lengths) + run_length_index = DeltaVarintCompressor.compress(self._run_lengths) + run_reference_index = DeltaVarintCompressor.compress( + self._run_references + ) + first_frame_index = DeltaVarintCompressor.compress( + self._run_first_frames + ) + for index in ( + physical_index, + run_length_index, + run_reference_index, + first_frame_index, + ): + self.output_stream.write(index) + self.output_stream.write(struct.pack( + ' int: + start = self.position + stream = blob.new_input_stream() + try: + while True: + chunk = stream.read(self.copy_buffer_size) + if not chunk: + break + self.output_stream.write(chunk) + self.position += len(chunk) + finally: + stream.close() + length = self.position - start + if length <= 0: + raise ValueError("Encoded video payload must not be empty.") + return length + + def _append(self, reference: int, frame_index: int) -> None: + if self._can_extend(reference, frame_index): + self._current_run_length += 1 + self._current_run_last_frame = frame_index + return + self._flush_run() + self._current_run_reference = reference + self._current_run_first_frame = frame_index + self._current_run_last_frame = frame_index + self._current_run_length = 1 + + def _can_extend(self, reference: int, frame_index: int) -> bool: + if ( + self._current_run_length == 0 + or self._current_run_reference != reference + ): + return False + return ( + reference < 0 + or frame_index == self._current_run_last_frame + 1 + ) + + def _flush_run(self) -> None: + if self._current_run_length == 0: + return + self._run_lengths.append(self._current_run_length) + self._run_references.append(self._current_run_reference) + self._run_first_frames.append(self._current_run_first_frame) + self._current_run_length = 0 diff --git a/paimon-python/pypaimon/write/writer/blob_file_writer.py b/paimon-python/pypaimon/write/writer/blob_file_writer.py index b30fbabb348b..0d214b2afef3 100644 --- a/paimon-python/pypaimon/write/writer/blob_file_writer.py +++ b/paimon-python/pypaimon/write/writer/blob_file_writer.py @@ -21,8 +21,15 @@ import pyarrow as pa from pypaimon.write.blob_format_writer import BlobFormatWriter +from pypaimon.write.video_format_writer import VideoFormatWriter from pypaimon.table.row.generic_row import GenericRow, RowKind -from pypaimon.table.row.blob import Blob, BlobConsumer, BlobData, BlobDescriptor +from pypaimon.table.row.blob import ( + Blob, + BlobConsumer, + BlobData, + BlobDescriptor, + VideoFrameDescriptor, +) from pypaimon.schema.data_types import ( DataField, PyarrowFieldParser, @@ -38,17 +45,28 @@ class BlobFileWriter: """ def __init__(self, file_io, file_path: Path, blob_consumer: Optional[BlobConsumer] = None, - copy_buffer_size: int = BlobFormatWriter.BUFFER_SIZE): + copy_buffer_size: int = BlobFormatWriter.BUFFER_SIZE, + video: bool = False): self.file_io = file_io self.file_path = file_path self._blob_consumer = blob_consumer + if video: + if blob_consumer is not None: + raise ValueError("BlobConsumer is not supported for video frame fields.") self.output_stream = file_io.new_output_stream(file_path) - self.writer = BlobFormatWriter( - self.output_stream, - blob_consumer=blob_consumer, - file_path=str(file_path), - copy_buffer_size=copy_buffer_size, - ) + if video: + self.writer = VideoFormatWriter( + self.output_stream, + file_path=str(file_path), + copy_buffer_size=copy_buffer_size, + ) + else: + self.writer = BlobFormatWriter( + self.output_stream, + blob_consumer=blob_consumer, + file_path=str(file_path), + copy_buffer_size=copy_buffer_size, + ) self.row_count = 0 self.closed = False @@ -103,7 +121,11 @@ def _to_blob(self, col_data) -> Optional[Blob]: return col_data if isinstance(col_data, bytes): - if BlobDescriptor.is_blob_descriptor(col_data): + if VideoFrameDescriptor.is_video_frame_descriptor(col_data): + descriptor = VideoFrameDescriptor.deserialize(col_data) + uri_reader = self.file_io.uri_reader_factory.create(descriptor.uri) + return Blob.from_descriptor(uri_reader, descriptor) + elif BlobDescriptor.is_blob_descriptor(col_data): descriptor = BlobDescriptor.deserialize(col_data) uri_reader = self.file_io.uri_reader_factory.create(descriptor.uri) return Blob.from_descriptor(uri_reader, descriptor) diff --git a/paimon-python/pypaimon/write/writer/blob_writer.py b/paimon-python/pypaimon/write/writer/blob_writer.py index 276968eee2e0..2c4be5927553 100644 --- a/paimon-python/pypaimon/write/writer/blob_writer.py +++ b/paimon-python/pypaimon/write/writer/blob_writer.py @@ -22,7 +22,12 @@ from pypaimon.common.options.core_options import CoreOptions from pypaimon.data.timestamp import Timestamp -from pypaimon.table.row.blob import BlobConsumer +from pypaimon.table.row.blob import ( + Blob, + BlobConsumer, + BlobRef, + VideoFrameDescriptor, +) from pypaimon.write.writer.append_only_data_writer import AppendOnlyDataWriter from pypaimon.write.writer.blob_file_writer import BlobFileWriter @@ -32,12 +37,21 @@ class BlobWriter(AppendOnlyDataWriter): def __init__(self, table, partition: Tuple, bucket: int, max_seq_number: int, blob_column: str, - options: Dict[str, str] = None, blob_consumer: Optional[BlobConsumer] = None): + options: Dict[str, str] = None, blob_consumer: Optional[BlobConsumer] = None, + video: bool = False): super().__init__(table, partition, bucket, max_seq_number, options, write_cols=[blob_column]) - # Override file format to "blob" - self.file_format = CoreOptions.FILE_FORMAT_BLOB + self.video = video + if video and blob_consumer is not None: + raise ValueError( + f"BlobConsumer is not supported for video frame field '{blob_column}'." + ) + self.file_format = ( + CoreOptions.FILE_FORMAT_VIDEO + if video + else CoreOptions.FILE_FORMAT_BLOB + ) # Store blob column name for use in metadata creation self.blob_column = blob_column @@ -54,6 +68,8 @@ def __init__(self, table, partition: Tuple, bucket: int, max_seq_number: int, bl self.file_uuid = str(uuid.uuid4()) self.file_count = 0 + self._current_video_group = None + self._pending_video_roll = False logger.info(f"Initialized BlobWriter with blob file format, blob_target_file_size={self.blob_target_file_size}") @@ -69,11 +85,17 @@ def _check_and_roll_if_needed(self): # in-memory serialized descriptor size. for i in range(pending.num_rows): row_data = pending.slice(i, 1) + next_group = self._video_payload_descriptor(row_data.column(0)[0]) + self._roll_before_video_group(next_group) self._write_row_to_file(row_data) self.record_count += 1 + self._current_video_group = next_group if self.rolling_file(): - self.close_current_writer() + if self.video and next_group is not None: + self._pending_video_roll = True + else: + self.close_current_writer() def _write_row_to_file(self, row_data: pa.Table): """Write a single row to the current blob file. Opens a new file if needed.""" @@ -88,15 +110,21 @@ def _write_row_to_file(self, row_data: pa.Table): self.sequence_generator.next() def write_blob(self, value, arrow_type=pa.large_binary()): + next_group = self._video_payload_descriptor(value) + self._roll_before_video_group(next_group) if self.current_writer is None: self.open_current_writer() self.current_writer.write_blob(self.blob_column, arrow_type, value) self.sequence_generator.next() self.record_count += 1 + self._current_video_group = next_group if self.rolling_file(): - self.close_current_writer() + if self.video and next_group is not None: + self._pending_video_roll = True + else: + self.close_current_writer() def open_current_writer(self): file_name = (f"{CoreOptions.data_file_prefix(self.options)}" @@ -109,6 +137,7 @@ def open_current_writer(self): file_path, blob_consumer=self._blob_consumer, copy_buffer_size=self.blob_copy_buffer_size, + video=self.video, ) def rolling_file(self) -> bool: @@ -137,10 +166,38 @@ def close_current_writer(self): self.current_writer = None self.current_file_path = None + self._current_video_group = None + self._pending_video_roll = False + + def _roll_before_video_group(self, next_group): + if ( + self.video + and self.current_writer is not None + and self._pending_video_roll + and self._current_video_group != next_group + ): + self.close_current_writer() + + @staticmethod + def _video_payload_descriptor(value): + if hasattr(value, 'as_py'): + value = value.as_py() + if value is None or value is Blob.PLACE_HOLDER: + return None + if type(value) is BlobRef: + descriptor = value.to_descriptor() + if isinstance(descriptor, VideoFrameDescriptor): + return descriptor.payload_descriptor + return None + if isinstance(value, (bytes, bytearray)): + raw = bytes(value) + if VideoFrameDescriptor.is_video_frame_descriptor(raw): + return VideoFrameDescriptor.deserialize(raw).payload_descriptor + return None def _write_data_to_file(self, data): """ - Keep a fallback path for direct blob table writes while preserving the shared uuid+counter + Keep a fallback path for direct blob table writes while preserving the writer uuid+counter naming behavior. """ if data.num_rows == 0: @@ -253,6 +310,8 @@ def abort(self): logger.warning(f"Error aborting blob writer: {e}", exc_info=e) self.current_writer = None self.current_file_path = None + self._current_video_group = None + self._pending_video_roll = False if not self.delete_file_upon_abort(): self._buffer.reset() self.committed_files.clear() diff --git a/paimon-python/pypaimon/write/writer/dedicated_format_writer.py b/paimon-python/pypaimon/write/writer/dedicated_format_writer.py index 001b0f9f692f..e6303541ee8b 100644 --- a/paimon-python/pypaimon/write/writer/dedicated_format_writer.py +++ b/paimon-python/pypaimon/write/writer/dedicated_format_writer.py @@ -31,7 +31,12 @@ is_blob_file_field, is_blob_type, ) -from pypaimon.table.row.blob import BlobConsumer +from pypaimon.table.row.blob import ( + Blob, + BlobConsumer, + BlobRef, + VideoFrameDescriptor, +) from pypaimon.table.row.generic_row import GenericRow from pypaimon.write.row_utils import ( require_columns, @@ -49,7 +54,7 @@ class DedicatedFormatWriter(DataWriter): Splits incoming data three ways: - Normal columns → standard data files (.parquet / .orc / .vortex / …) - - Blob columns (large_binary) → .blob files + - Blob columns (large_binary) → .blob or .video files - Vector columns (when vector.file.format is configured) → .vector. files This mirrors Java's DedicatedFormatRollingFileWriter. @@ -71,8 +76,15 @@ def __init__(self, table, partition: Tuple, bucket: int, max_seq_number: int, op self.blob_column_names = self._get_blob_columns_from_schema() self.blob_descriptor_fields = CoreOptions.blob_descriptor_fields(self.options) self.blob_view_fields = CoreOptions.blob_view_fields(self.options) + self.video_frame_fields = CoreOptions.video_frame_fields(self.options) self.blob_inline_fields = self.blob_descriptor_fields.union(self.blob_view_fields) + if len(self.video_frame_fields) > 1: + raise ValueError("'video-frame-field' currently supports exactly one field.") + self.video_frame_column = ( + next(iter(self.video_frame_fields)) if self.video_frame_fields else None + ) + unknown_descriptor_fields = self.blob_descriptor_fields.difference( set(self.blob_column_names) ) @@ -92,7 +104,7 @@ def __init__(self, table, partition: Tuple, bucket: int, max_seq_number: int, op f"Invalid inline blob fields: {sorted(inline_nested_blob_fields)}" ) - # Blob fields that should still be written to `.blob` files. + # Blob fields that should still be written to dedicated BLOB files. self.blob_file_column_names = [ col for col in self.blob_column_names if col not in self.blob_inline_fields ] @@ -125,6 +137,8 @@ def __init__(self, table, partition: Tuple, bucket: int, max_seq_number: int, op self.normal_column_names = [ col for col in all_column_names if col not in dedicated_set ] + if self.video_frame_column not in self.blob_file_column_names: + self.video_frame_column = None normal_name_set = set(self.normal_column_names) self.normal_columns = [ field for field in self.table.table_schema.fields if field.name in normal_name_set @@ -134,6 +148,8 @@ def __init__(self, table, partition: Tuple, bucket: int, max_seq_number: int, op # State management for blob writer self.record_count = 0 self.closed = False + self._current_video_group = None + self._pending_video_group_roll = False # Normal columns are buffered separately from the blob and vector # columns, which their own writers own. @@ -156,6 +172,7 @@ def __init__(self, table, partition: Tuple, bucket: int, max_seq_number: int, op blob_column=blob_column, options=options, blob_consumer=blob_consumer, + video=blob_column in self.video_frame_fields, ) # Initialize vector writer when vector.file.format is configured. @@ -210,9 +227,18 @@ def write(self, data: pa.RecordBatch): # writer, or the unfinished flush would lose its chance to be retried. self._require_finished_flush() try: + if self.video_frame_column is not None: + for index in range(data.num_rows): + row = data.slice(index, 1) + next_group = self._video_payload_descriptor_from_batch(row) + self._roll_before_video_group(next_group) + self._current_video_group = next_group + self._write_batch(row) + return + offset = 0 # _write_batch keeps normal/blob/vector pending rows in lockstep - # and closes all writers when the shared row limit is reached. + # and closes all writers when the common row limit is reached. while offset < data.num_rows: capacity = self.target_file_row_num - self.pending_row_count if capacity <= 0: @@ -253,8 +279,7 @@ def _write_batch(self, data: pa.RecordBatch): # Check if normal data rolling is needed if self._should_roll_normal(): - # When normal data rolls, close both writers and fetch blob metadata - self._close_current_writers() + self._roll_or_defer_for_video_group() def write_row(self, row): self._require_finished_flush() @@ -268,6 +293,13 @@ def write_row(self, row): ) require_columns(values_by_name, required_columns, "write_row") + if self.video_frame_column is not None: + next_group = self._video_payload_descriptor( + values_by_name[self.video_frame_column] + ) + self._roll_before_video_group(next_group) + self._current_video_group = next_group + if self.normal_column_names: normal_values = dict(values_by_name) for field_name in self.normal_column_names: @@ -299,7 +331,7 @@ def write_row(self, row): self.record_count += 1 if self._should_roll_normal(): - self._close_current_writers() + self._roll_or_defer_for_video_group() except Exception as e: logger.error("Exception occurs when writing row. Cleaning up.", exc_info=e) @@ -485,6 +517,47 @@ def _should_roll_normal(self) -> bool: # Check if normal data exceeds target size return self._normal_buffer.nbytes > self.target_file_size + def _roll_or_defer_for_video_group(self): + if ( + self.video_frame_column is not None + and self._current_video_group is not None + ): + self._pending_video_group_roll = True + else: + self._close_current_writers() + + def _roll_before_video_group(self, next_group): + if ( + self._pending_video_group_roll + and self._current_video_group != next_group + ): + self._close_current_writers() + + def _video_payload_descriptor_from_batch(self, data: pa.RecordBatch): + column_index = data.schema.get_field_index(self.video_frame_column) + if column_index < 0: + raise KeyError( + f"Column '{self.video_frame_column}' was not found in the record batch." + ) + return self._video_payload_descriptor(data.column(column_index)[0]) + + @staticmethod + def _video_payload_descriptor(value): + if hasattr(value, 'as_py'): + value = value.as_py() + if value is None or value is Blob.PLACE_HOLDER: + return None + if type(value) is BlobRef: + descriptor = value.to_descriptor() + if isinstance(descriptor, VideoFrameDescriptor): + return descriptor.payload_descriptor + return None + if isinstance(value, (bytes, bytearray)): + raw = bytes(value) + if VideoFrameDescriptor.is_video_frame_descriptor(raw): + return VideoFrameDescriptor.deserialize(raw).payload_descriptor + return None + @property def pending_row_count(self) -> int: # Overrides the base property, which reads a buffer this writer never @@ -555,6 +628,8 @@ def _close_current_writers(self): self._pending_normal_meta = None self.record_count = 0 + self._current_video_group = None + self._pending_video_group_roll = False if normal_meta is not None or blob_metas or vector_metas: normal_name = normal_meta.file_name if normal_meta is not None else ''