Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions docs/docs/append-table/bucketed.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,11 @@ set of buckets through this optimization.

## Bucketed Join

Spark can use compatible bucket distributions to avoid a shuffle in a batch join. Enable V2 bucketing and join on the
distribution keys:
Spark can use compatible bucket distributions to avoid a shuffle in a batch join. Enable
`scan.preserve-data-grouping` and V2 bucketing, then join on the distribution keys:

```sql
SET spark.paimon.scan.preserve-data-grouping = true;
SET spark.sql.sources.v2.bucketing.enabled = true;

CREATE TABLE fact_table (order_id INT, f1 STRING) USING paimon
Expand Down
4 changes: 3 additions & 1 deletion docs/docs/flink/action-jars.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,9 @@ The following command is used to compact a table.
<FLINK_HOME>/bin/flink run \
/path/to/paimon-flink-action-@@VERSION@@.jar \
compact \
--path <TABLE_PATH>
--warehouse <warehouse-path> \
--database <database-name> \
--table <table-name>
```

Use `<action> --help` to inspect its arguments. Partition specifications use comma-separated
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/flink/sql-ddl.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ You can set location in the properties of table/database by the config of `locat

By using the Paimon JDBC catalog, changes to the catalog will be directly stored in relational databases such as SQLite, MySQL, postgres, etc.

Currently, lock configuration is only supported for MySQL and SQLite. If you are using a different type of database for catalog storage, please do not configure `lock.enabled`.
Currently, lock configuration is only supported for MySQL, PostgreSQL, and SQLite. If you are using a different type of database for catalog storage, please do not configure `lock.enabled`.

Paimon JDBC Catalog in Flink needs to correctly add the corresponding jar package for connecting to the database. You should first download JDBC connector bundled jar and add it to classpath. such as MySQL, postgres

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/learn-paimon/scenario-guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ An equality or `IN` predicate on the complete bucket key can prune unrelated buc
not guarantee reading exactly one eighth of the bytes: bucket sizes can be skewed, and matching
buckets still contain other keys. A composite bucket key requires constraints on every key column.

Spark can exploit compatible bucket distributions in joins with V2 bucketing enabled. Confirm
Spark can exploit compatible bucket distributions in joins with `scan.preserve-data-grouping` and V2 bucketing enabled. Confirm
shuffle removal with `EXPLAIN`; see [Bucketed Join](../append-table/bucketed#bucketed-join).

### Ordered Streaming {#scenario-7-queue-like-ordered-streaming-bucketed-append}
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/maintenance/filesystems.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,7 @@ SELECT COUNT(1) FROM test_table;

From version 0.8, paimon-trino uses trino filesystem as basic file read and write system. We strongly recommend you to use jindo-sdk in trino.

You can find [How to config jindo sdk on trino](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto) here.
You can find [How to config jindo sdk on trino](https://github.com/aliyun/alibabacloud-jindodata/blob/master/docs/user/4.x/4.6.x/4.6.12/oss/presto/jindosdk_on_presto.md) here.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Link to the Trino setup guide

The new URL opens a Presto guide that installs the jars under $PRESTO_HOME/plugin/hive-hadoop2/. Readers following this Trino section still have to translate Presto-specific paths and setup steps. Please link to the Trino guide and identify its OSS-HDFS scope. Alibaba Cloud Trino guide

Please note that:
* Use paimon to replace hive-hadoop2 when you decompress the plugin jar and find location to put in.
* You can specify the `core-site.xml` in `paimon.properties` on configuration [hive.config.resources](https://trino.io/docs/current/connector/hive.html#hdfs-configuration).
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/maintenance/manage-snapshots.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -280,9 +280,9 @@ Run the following command:
/path/to/paimon-flink-action-@@VERSION@@.jar \
expire_snapshots \
--warehouse <warehouse-path> \
--identifier <identifier> \
--database <database-name> \
--table <table-name> \
--older_than <timestamp> \
--version <snapshot-id> \
--max_deletes <max-deletes> \
--retain_max <retain-max> \
--retain_min <retain-min> \
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/multimodal-table/variant.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ suitable for event attributes, model output, tool parameters, and other data tha
frequently to model every field as a table column.

Paimon stores `VARIANT` with the
[Parquet Variant binary encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding),
[Parquet Variant binary encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md),
instead of storing its JSON text. The data is parsed when it is converted to `VARIANT`, and queries
can extract a path as a requested SQL type.

Expand Down
4 changes: 3 additions & 1 deletion docs/docs/primary-key-table/query-performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ coverage rules, especially for Vector and Full Text search.
Spark can use compatible fixed-bucket layouts to avoid a join shuffle. For example:

```sql
SET spark.paimon.scan.preserve-data-grouping = true;
SET spark.sql.sources.v2.bucketing.enabled = true;

CREATE TABLE fact_table (order_id INT, f1 STRING) USING paimon
Expand All @@ -115,6 +116,7 @@ SELECT * FROM fact_table AS fact
JOIN dim_table AS dim ON fact.order_id = dim.order_id;
```

The setting enables Spark to use partitioning reported by V2 data sources. The join keys, bucket
These settings let Paimon report its bucket layout to Spark (see
[Scan Layout and Storage Partition Joins](../spark/sql-query#scan-layout-and-storage-partition-joins)). The join keys, bucket
keys, and bucket counts must be compatible; the same bucket count alone is not sufficient.
Check `EXPLAIN` to confirm that Spark avoids the shuffle for the actual query.
2 changes: 1 addition & 1 deletion docs/docs/pypaimon/sql.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ batches = ctx.sql("SELECT * FROM paimon.default.my_temp")

## Supported SQL Syntax

The SQL engine is powered by Apache DataFusion, which supports a rich set of SQL syntax. For the full SQL reference, see the [paimon-rust SQL documentation](https://paimon-rust.apache.org/sql.html) which covers:
The SQL engine is powered by Apache DataFusion, which supports a rich set of SQL syntax. For the full SQL reference, see the [paimon-rust SQL documentation](https://paimon.apache.org/docs/rust/sql/) which covers:

- **DDL**: `CREATE SCHEMA`, `CREATE TABLE` (with `PARTITIONED BY`, `PRIMARY KEY`, `WITH` options), `DROP TABLE`, `ALTER TABLE`, `CREATE TEMPORARY TABLE/VIEW`
- **DML**: `INSERT INTO`, `INSERT OVERWRITE` (dynamic/static partitions), `UPDATE`, `DELETE`, `MERGE INTO`, `TRUNCATE TABLE`
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/spark/catalogs.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ If you want to see a partitioned table in Hive and also synchronize newly create

The JDBC catalog stores catalog metadata in a relational database such as SQLite, MySQL, or PostgreSQL.

Currently, lock configuration is only supported for MySQL and SQLite. If you are using a different type of database for catalog storage, please do not configure `lock.enabled`.
Currently, lock configuration is only supported for MySQL, PostgreSQL, and SQLite. If you are using a different type of database for catalog storage, please do not configure `lock.enabled`.

Add the JDBC driver for your database to Spark's classpath along with the Paimon connector.

Expand Down
Loading