Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.datafusion</groupId>
<artifactId>comet-parent-spark${spark.version.short}_${scala.binary.version}</artifactId>
<version>0.14.0-SNAPSHOT</version>
<version>0.14.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/3.4.3.diff
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ index d3544881af1..9c16099090c 100644
<ivy.version>2.5.1</ivy.version>
<oro.version>2.0.8</oro.version>
+ <spark.version.short>3.4</spark.version.short>
+ <comet.version>0.14.0-SNAPSHOT</comet.version>
+ <comet.version>0.14.0</comet.version>
<!--
If you changes codahale.metrics.version, you also need to change
the link to metrics.dropwizard.io in docs/monitoring.md.
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/3.5.8.diff
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ index edd2ad57880..77a975ea48f 100644
<ivy.version>2.5.1</ivy.version>
<oro.version>2.0.8</oro.version>
+ <spark.version.short>3.5</spark.version.short>
+ <comet.version>0.14.0-SNAPSHOT</comet.version>
+ <comet.version>0.14.0</comet.version>
<!--
If you changes codahale.metrics.version, you also need to change
the link to metrics.dropwizard.io in docs/monitoring.md.
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/4.0.1.diff
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ index 22922143fc3..7c56e5e8641 100644
<ivy.version>2.5.3</ivy.version>
<oro.version>2.0.8</oro.version>
+ <spark.version.short>4.0</spark.version.short>
+ <comet.version>0.14.0-SNAPSHOT</comet.version>
+ <comet.version>0.14.0</comet.version>
<!--
If you change codahale.metrics.version, you also need to change
the link to metrics.dropwizard.io in docs/monitoring.md.
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/iceberg-rust/1.10.0.diff
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ index eeabe54f5..867018058 100644
caffeine = "2.9.3"
calcite = "1.40.0"
-comet = "0.8.1"
+comet = "0.14.0-SNAPSHOT"
+comet = "0.14.0"
datasketches = "6.2.0"
delta-standalone = "3.3.2"
delta-spark = "3.3.2"
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/iceberg-rust/1.8.1.diff
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ index 04ffa8f4e..3a57af315 100644
awssdk-s3accessgrants = "2.3.0"
caffeine = "2.9.3"
calcite = "1.10.0"
+comet = "0.14.0-SNAPSHOT"
+comet = "0.14.0"
datasketches = "6.2.0"
delta-standalone = "3.3.0"
delta-spark = "3.3.0"
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/iceberg-rust/1.9.1.diff
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ index c50991c5f..3acb395a6 100644
bson-ver = "4.11.5"
caffeine = "2.9.3"
calcite = "1.39.0"
+comet = "0.14.0-SNAPSHOT"
+comet = "0.14.0"
datasketches = "6.2.0"
delta-standalone = "3.3.1"
delta-spark = "3.3.1"
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/iceberg/1.10.0.diff
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ index eeabe54f5f0..867018058ee 100644
caffeine = "2.9.3"
calcite = "1.40.0"
-comet = "0.8.1"
+comet = "0.14.0-SNAPSHOT"
+comet = "0.14.0"
datasketches = "6.2.0"
delta-standalone = "3.3.2"
delta-spark = "3.3.2"
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/iceberg/1.8.1.diff
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ index 04ffa8f4edc..3a57af3156a 100644
awssdk-s3accessgrants = "2.3.0"
caffeine = "2.9.3"
calcite = "1.10.0"
+comet = "0.14.0-SNAPSHOT"
+comet = "0.14.0"
datasketches = "6.2.0"
delta-standalone = "3.3.0"
delta-spark = "3.3.0"
Expand Down
2 changes: 1 addition & 1 deletion dev/diffs/iceberg/1.9.1.diff
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ index c50991c5fc6..f7ad00f0b78 100644
bson-ver = "4.11.5"
caffeine = "2.9.3"
calcite = "1.39.0"
+comet = "0.14.0-SNAPSHOT"
+comet = "0.14.0"
datasketches = "6.2.0"
delta-standalone = "3.3.1"
delta-spark = "3.3.1"
Expand Down
108 changes: 57 additions & 51 deletions docs/source/user-guide/latest/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,96 +157,102 @@ Cast operations in Comet fall into three levels of support:

<!--BEGIN:CAST_LEGACY_TABLE-->
<!-- prettier-ignore-start -->
| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A |
| boolean | N/A | - | C | N/A | C | C | C | C | C | C | C | C |
| byte | C | C | - | N/A | C | C | C | C | C | C | C | C |
| date | N/A | C | C | - | C | C | C | C | C | C | C | C |
| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C |
| double | N/A | C | C | N/A | I | - | C | C | C | C | C | C |
| float | N/A | C | C | N/A | I | C | - | C | C | C | C | C |
| integer | C | C | C | N/A | C | C | C | - | C | C | C | C |
| long | C | C | C | N/A | C | C | C | C | - | C | C | C |
| short | C | C | C | N/A | C | C | C | C | C | - | C | C |
| string | C | C | C | C | I | C | C | C | C | C | - | I |
| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - |

| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp |
| --------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- |
| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A |
| boolean | N/A | - | C | N/A | C | C | C | C | C | C | C | C |
| byte | C | C | - | N/A | C | C | C | C | C | C | C | C |
| date | N/A | C | C | - | C | C | C | C | C | C | C | C |
| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C |
| double | N/A | C | C | N/A | I | - | C | C | C | C | C | C |
| float | N/A | C | C | N/A | I | C | - | C | C | C | C | C |
| integer | C | C | C | N/A | C | C | C | - | C | C | C | C |
| long | C | C | C | N/A | C | C | C | C | - | C | C | C |
| short | C | C | C | N/A | C | C | C | C | C | - | C | C |
| string | C | C | C | C | I | C | C | C | C | C | - | I |
| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - |

**Notes:**

- **decimal -> string**: There can be formatting differences in some case due to Spark using scientific notation where Comet does not
- **double -> decimal**: There can be rounding differences
- **double -> string**: There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45
- **float -> decimal**: There can be rounding differences
- **float -> string**: There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45
- **string -> date**: Only supports years between 262143 BC and 262142 AD
- **string -> decimal**: Does not support fullwidth unicode digits (e.g \\uFF10)
or strings containing null bytes (e.g \\u0000)
or strings containing null bytes (e.g \\u0000)
- **string -> timestamp**: Not all valid formats are supported
<!-- prettier-ignore-end -->
<!--END:CAST_LEGACY_TABLE-->
<!-- prettier-ignore-end -->
<!--END:CAST_LEGACY_TABLE-->

### Try Mode

<!--BEGIN:CAST_TRY_TABLE-->
<!-- prettier-ignore-start -->
| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A |
| boolean | N/A | - | C | N/A | C | C | C | C | C | C | C | U |
| byte | U | C | - | N/A | C | C | C | C | C | C | C | C |
| date | N/A | U | U | - | U | U | U | U | U | U | C | C |
| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C |
| double | N/A | C | C | N/A | I | - | C | C | C | C | C | C |
| float | N/A | C | C | N/A | I | C | - | C | C | C | C | C |
| integer | U | C | C | N/A | C | C | C | - | C | C | C | C |
| long | U | C | C | N/A | C | C | C | C | - | C | C | C |
| short | U | C | C | N/A | C | C | C | C | C | - | C | C |
| string | C | C | C | C | I | C | C | C | C | C | - | I |
| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - |

| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp |
| --------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- |
| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A |
| boolean | N/A | - | C | N/A | C | C | C | C | C | C | C | U |
| byte | U | C | - | N/A | C | C | C | C | C | C | C | C |
| date | N/A | U | U | - | U | U | U | U | U | U | C | C |
| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C |
| double | N/A | C | C | N/A | I | - | C | C | C | C | C | C |
| float | N/A | C | C | N/A | I | C | - | C | C | C | C | C |
| integer | U | C | C | N/A | C | C | C | - | C | C | C | C |
| long | U | C | C | N/A | C | C | C | C | - | C | C | C |
| short | U | C | C | N/A | C | C | C | C | C | - | C | C |
| string | C | C | C | C | I | C | C | C | C | C | - | I |
| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - |

**Notes:**

- **decimal -> string**: There can be formatting differences in some case due to Spark using scientific notation where Comet does not
- **double -> decimal**: There can be rounding differences
- **double -> string**: There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45
- **float -> decimal**: There can be rounding differences
- **float -> string**: There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45
- **string -> date**: Only supports years between 262143 BC and 262142 AD
- **string -> decimal**: Does not support fullwidth unicode digits (e.g \\uFF10)
or strings containing null bytes (e.g \\u0000)
or strings containing null bytes (e.g \\u0000)
- **string -> timestamp**: Not all valid formats are supported
<!-- prettier-ignore-end -->
<!--END:CAST_TRY_TABLE-->
<!-- prettier-ignore-end -->
<!--END:CAST_TRY_TABLE-->

### ANSI Mode

<!--BEGIN:CAST_ANSI_TABLE-->
<!-- prettier-ignore-start -->
| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A |
| boolean | N/A | - | C | N/A | C | C | C | C | C | C | C | U |
| byte | U | C | - | N/A | C | C | C | C | C | C | C | C |
| date | N/A | U | U | - | U | U | U | U | U | U | C | C |
| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C |
| double | N/A | C | C | N/A | I | - | C | C | C | C | C | C |
| float | N/A | C | C | N/A | I | C | - | C | C | C | C | C |
| integer | U | C | C | N/A | C | C | C | - | C | C | C | C |
| long | U | C | C | N/A | C | C | C | C | - | C | C | C |
| short | U | C | C | N/A | C | C | C | C | C | - | C | C |
| string | C | C | C | C | I | C | C | C | C | C | - | I |
| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - |

| | binary | boolean | byte | date | decimal | double | float | integer | long | short | string | timestamp |
| --------- | ------ | ------- | ---- | ---- | ------- | ------ | ----- | ------- | ---- | ----- | ------ | --------- |
| binary | - | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | C | N/A |
| boolean | N/A | - | C | N/A | C | C | C | C | C | C | C | U |
| byte | U | C | - | N/A | C | C | C | C | C | C | C | C |
| date | N/A | U | U | - | U | U | U | U | U | U | C | C |
| decimal | N/A | C | C | N/A | - | C | C | C | C | C | C | C |
| double | N/A | C | C | N/A | I | - | C | C | C | C | C | C |
| float | N/A | C | C | N/A | I | C | - | C | C | C | C | C |
| integer | U | C | C | N/A | C | C | C | - | C | C | C | C |
| long | U | C | C | N/A | C | C | C | C | - | C | C | C |
| short | U | C | C | N/A | C | C | C | C | C | - | C | C |
| string | C | C | C | C | I | C | C | C | C | C | - | I |
| timestamp | N/A | U | U | C | U | U | U | U | C | U | C | - |

**Notes:**

- **decimal -> string**: There can be formatting differences in some case due to Spark using scientific notation where Comet does not
- **double -> decimal**: There can be rounding differences
- **double -> string**: There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45
- **float -> decimal**: There can be rounding differences
- **float -> string**: There can be differences in precision. For example, the input "1.4E-45" will produce 1.0E-45 instead of 1.4E-45
- **string -> date**: Only supports years between 262143 BC and 262142 AD
- **string -> decimal**: Does not support fullwidth unicode digits (e.g \\uFF10)
or strings containing null bytes (e.g \\u0000)
or strings containing null bytes (e.g \\u0000)
- **string -> timestamp**: ANSI mode not supported
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->
<!-- prettier-ignore-end -->
<!--END:CAST_ANSI_TABLE-->

See the [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details.
2 changes: 1 addition & 1 deletion fuzz-testing/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ under the License.
<parent>
<groupId>org.apache.datafusion</groupId>
<artifactId>comet-parent-spark${spark.version.short}_${scala.binary.version}</artifactId>
<version>0.14.0-SNAPSHOT</version>
<version>0.14.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ under the License.
</parent>
<groupId>org.apache.datafusion</groupId>
<artifactId>comet-parent-spark${spark.version.short}_${scala.binary.version}</artifactId>
<version>0.14.0-SNAPSHOT</version>
<version>0.14.0</version>
<packaging>pom</packaging>
<name>Comet Project Parent POM</name>

Expand Down
2 changes: 1 addition & 1 deletion spark-integration/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.datafusion</groupId>
<artifactId>comet-parent-spark${spark.version.short}_${scala.binary.version}</artifactId>
<version>0.14.0-SNAPSHOT</version>
<version>0.14.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion spark/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ under the License.
<parent>
<groupId>org.apache.datafusion</groupId>
<artifactId>comet-parent-spark${spark.version.short}_${scala.binary.version}</artifactId>
<version>0.14.0-SNAPSHOT</version>
<version>0.14.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
Loading