RFC-0025 Derived column

[ScrapCodes]

What is a derived column?

A column created by applying a SQL expression or a UDF to an existing column in a table.

Why do we need that, since we can always apply a UDF to a column during project, filter or join?

Indeed, a derived column consumes O(N) storage, where N is the number of rows in the table. We still need them because, the performance benefits outweigh the disadvantage of extra storage it consumes. Let us understand with the following use case example:

A compute engine like Presto can easily push down a filter predicate e.g. SELECT col1, col2, FROM table T1 WHERE col1='constant_value' , this allows for pruning the number of rows required for TableScan by applying the filtering WHERE col1=’constant_value’. This is not true of when a UDF is involved in the filter predicate, let us take an example SELECT col1, col2, FROM table T1 WHERE lower(col1)='constant_value'. While optimizers can easily push down the filter predicate, however, it can not be used in filtering using the lower and upper bound metrics, for example Iceberg manifest statics and Parquet row group statistics. As a result, we end up scanning a large number of rows.

So, to support push down of certain predicates (with UDFs in them) and reduce the amount of data scanned, derived column bring massive performance improvements. Derived columns have already been proven in RDBMS system e.g. DB2 [1], and now we intend to bring them to Presto.

[jja725]

Agree that how write work would be the main concern here with compatibility with all the engine

[ScrapCodes]

@tdcmeehan has volunteered to be a co-author ! Yay!

[aditi-pandit]

Can you give more info about the SQL dialect of this expression ? Seems like you want atleast Presto and Spark to understand it.

[tdcmeehan]

To be clear, deriving a common subset of expressions that are interpretable by both Spark and Presto is hard and likely outside of the scope of this RFC. I think the most straightforward thing is to treat them like views, which defer on cross-platform interpretability and force any consumer of the view SQL to understand Presto's dialect. Cross platform expressions can be considered an orthogonal yet important task.

[ScrapCodes]

Hi Folks @tdcmeehan , @aditi-pandit, @aaneja and @agrawalreetika Can you please give me a review feedback !

[ScrapCodes]

prestodb/presto#28123 is the PR with just parser and storing table properties in iceberg connector.

POC PR: prestodb/presto#27832

[hantangwangd]

Hi @ScrapCodes, thanks for this RFC. Regarding our eventual perspective on derived columns, I have some high-level questions and thoughts for discussion:

1. When adding a derived column to an existing table that already contains data, how are the values for existing rows handled?
2. When inserting rows, is the derived column: (a) not specifiable, (b) specifiable but auto-computed if omitted, or (c) mandatory (NULL when absent)?
3. For the tracking and detection of sync states of derived columns:

• Is the engine responsible for tracking and detection? If so, how would that be implemented? (Considering that other unaware engines might perform writes, source columns could be updated, or even the function logic itself could change, etc.)
• Or, the engine wouldn't manage the sync state—users are responsible for their own data. The engine would simply provide sync capability for users to call on demand, similar to REFRESH MV or REWRITE_DATA_FILES.

4. Filter and projection rewrite optimizations are indeed a huge performance advantage that derived columns can bring. But VIRTUAL-mode derived columns don't seem to offer that benefit—so what's the value of having them?

Ultimately, the core question is: in our final perspective, is a derived column a special type of column, or is it essentially more like a normal column with an optimization hint?

Looking at the overall design and the POC examples, it seems we're mainly considering the latter—where derived columns are essentially more like normal columns with an optimization hint. However, many of the concerns outlined in the "Considerations" and "Challenges" sections appear to stem from the former.

In my view, the gap between these two modes is quite significant. I think it would be helpful to first clearly define the boundaries—specifically, what is feasible and on our roadmap versus what is out of scope or not advisable.

Just some initial thoughts on my side—happy to be corrected if I've misunderstood anything, and looking forward to further discussion.

[hantangwangd]

Just curious: how do we track and detect that a derived column is out of sync? And can we simply drop and re-add the derived column to sync it?

[ScrapCodes]

Yes, a drop and re-add the derived column will trigger the expression to be applied on existing rows and new column updated.

In the absence of INSERT support for derived column we will have to do UPDATE e.g. UPDATE test2 SET c2_derived = lower(c2).

[ScrapCodes]

Hi @ScrapCodes, thanks for this RFC. Regarding our eventual perspective on derived columns, I have some high-level questions and thoughts for discussion:

Hi Dong Wang (@hantangwangd), thanks for the thorough review and your comments are all valid. Here is my attempt to answer, I could be missing some things and hope @tdcmeehan can cover up and correct! @aditi-pandit and @aaneja have also glanced at it and can help !

Some background:

Derived columns is a standard feature in RDBMS and has a SQL standard spec, the key challenge here is RDBMS is like a walled garden and they do not have the problem of data may be updated from outside their system in anyway possible. In Presto, this cannot be guaranteed and data may be updated in various ways from outside our system.

While dealing with table formats e.g. Iceberg this is specially a concern. And only ultimate solution is a Spec update from Iceberg which enables the derived column as a first class feature. I started working on it, but haven't finished. (Link).

In the absence of a first class feature support from Iceberg, we are using table properties to store derived column metadata. Which appears as if derived columns are regular columns with just a optimization hint. However, this is not the intention. Once this feature is fully complete - this will be clearly spelled.

The work for derived column is divided into parts,

a) The parser and related connector changes to store the derived column metadata and basic validation for checking derived column may be out of sync.

b) The query rewrite rule to benefit from derived columns.

c) Implementation of Insert/Merge/Update i.e. for Derived columns we provide the option of a column could be GENERATED ALWAYS i.e. user inserts are rejected and the values are always generated. This way derived column is not a normal column with a optimization hint, it is a special column. It is also possible to have derived column without this restriction and user insert is permitted and also auto-generate when requested - the syntax in MariaDb for example looks like INSERT INTO TABLE_NAME VALUES ('col1', col2, default, coln), where the keyword default instructs the DB to generate the value for that particular row and column via derived column definition. This detail will be added to RFC, once we transition into Implementation of Insert part of work.

d) Once we control the insert, we can track last updated snapshot ID by presto and if this ID does not match with the latest Snapshot ID, we can say there may be a update to the table outside of presto and derived column's data "may have gone" out of sync.

e) While we develop this with table properties, we are also planning to move ahead with Iceberg Spec change and presto already embracing derived columns will be a strong case for Iceberg's spec change acceptance. Iceberg spec change can take much longer time.

The current RFC only covers a) and b), leaves the details of c, d, e in future work section. I took this route, so that we can evolve the RFC (hoping that RFCs can evolve), i.e. develop in stages and we do not have to have everything flushed out at once.

I am open to be corrected on my approach overall or even the part that we should have all the details of fully developed Feature before we can begin.

1. When adding a derived column to an existing table that already contains data, how are the values for existing rows handled?

With INSERT support not ready, this is done manually using UPDATE followed by adding a derived column. Same is true of adding rows to the table, INSERT to derived column are done manually.

Once the INSERT support is added, this step will be automated. i.e. Derived column will run an update with the expression for all the existing rows in the table. Therefore dropping and recreating the column will have the effect of syncing the derived column.

2. When inserting rows, is the derived column: (a) not specifiable, (b) specifiable but auto-computed if omitted, or (c) mandatory (NULL when absent)?

Specifiable,

1. For derived column of type GENERATED ALWAYS, the specified value is always rejected and the column value is computed using the derived column expression.
2. For derived column of type just PERSISTED, the value is both computable for example using following syntax :

INSERT INTO table1 VALUES (1, 'some text',default,default);
Query OK, 1 row affected (0.00 sec)

INSERT INTO table1 VALUES (2, 'more text',5,default);
Query OK, 1 row affected, 1 warning (0.00 sec)

Warning (Code 1645): The value specified for computed column 'c' in table 'table1' has been ignored.

[above excerpt is taken from MariaDb docs, link in RFC]

3. We can even keep GENERATED ALWAYS in parser to be compliant with SQL Standard, but always enforce it. Other SQL Engines do this. They also allow NULL to be set instead of default e.g. INSERT INTO table1 VALUES (1, 'some text',NULL,NULL); is a valid. I do not understand, why that would be needed.

These details are not updated in the current RFC as they are future work.

3. For the tracking and detection of sync states of derived columns:


* Is the engine responsible for tracking and detection? If so, how would that be implemented? (Considering that other unaware engines might perform writes, source columns could be updated, or even the function logic itself could change, etc.)

True for detecting function logic change we do not have anything, because built-in functions in Presto are not versioned and a plugin can add built-in functions and change without a way for us to know.

For Tableformats like Iceberg, engine is responsible for tracking and detection. And the way would be to track snapshot ID for any updates made to the table. This way we can tell "may be there was a change" and table is out of sync due to an external update. But we cannot tell for sure, there is a need to sync the derived column.

* Or, the engine wouldn't manage the sync state—users are responsible for their own data. The engine would simply provide sync capability for users to call on demand, similar to `REFRESH MV` or `REWRITE_DATA_FILES`.

That is correct !

4. Filter and projection rewrite optimizations are indeed a huge performance advantage that derived columns can bring. But VIRTUAL-mode derived columns don't seem to offer that benefit—so what's the value of having them?

From Iceberg connector perspective, VIRTUAL mode in derived column will need a support from ICEBERG spec. They will help us in improving data-file pruning only ! For example they will maintain only column metadata/metrics i.e. lower/upper bounds or bloom filters in Iceberg manifests/puffin files. When they are used in filters, this metadata will help us prune the data files. The performance boast will come when those expressions cannot be pushed down, however the virtual column representing them can be pushed and data files pruned.

Ultimately, the core question is: in our final perspective, is a derived column a special type of column, or is it essentially more like a normal column with an optimization hint?

Looking at the overall design and the POC examples, it seems we're mainly considering the latter—where derived columns are essentially more like normal columns with an optimization hint. However, many of the concerns outlined in the "Considerations" and "Challenges" sections appear to stem from the former.

In my view, the gap between these two modes is quite significant. I think it would be helpful to first clearly define the boundaries—specifically, what is feasible and on our roadmap versus what is out of scope or not advisable.

You are right, the gap between two modes is quite significant, and derived column is a special column all RDBMS system treat it as a special column. For ICEBERG table format, we have gone the table properties route which is what I think you alluding to as normal column with a optimizer hint. But, that is not what we want ultimately, we want to have derived column as a special column - with support defined in Iceberg Spec.

Just some initial thoughts on my side—happy to be corrected if I've misunderstood anything, and looking forward to further discussion.

Thanks again for these questions!

[ScrapCodes]

Hi @hantangwangd ,

1. How about we drop VIRTUAL columns for now even from the Parser

2. We can make the derived column insert not specifiable and always GENERATED. We can still allow GENERATED ALWAYS in create/alter table queries to stay compliant with SQL spec.

WDYT ?

cc @tdcmeehan / @aditi-pandit / @aaneja

[hantangwangd]

@ScrapCodes, thanks for your detailed answer. A few thoughts below—happy to discuss.

When we consider only (a) and (b), or in other words, when we treat a derived column more like a normal column plus optimization hint, then:

• Presto and the underlying table format are completely aligned in their view of how a derived column exists in the table schema—both treat it as a normal column.
• The derived column feature is fully compatible with other engines and older Presto versions that are unaware of it. There will be no inconsistency in the table schema interpretation that would require adjustments to user SQL statements.
• Engines aware of derived column fields can use them for additional rewrite optimizations to improve performance, whereas unaware engines simply treat them as regular columns.

Based on the above, the derived column feature at this stage appears to be fully compatible with the underlying table format and across engines, and there seem to be no blockers to supporting and implementing it at this stage.

Regarding the latter points (c), (d), (e), I have given them some thought. Please correct me if I have misunderstood anything.

When the Presto engine begins to support derived columns that do not allow user specification and can only be computed automatically—as discussed earlier—such a column appears to the engine as an entirely new type of column: one whose representation in the table schema is inconsistent between reads and writes. For example, consider the following statements:

create table tbl1 (a int, b varchar, c varchar GENERATED ALWAYS AS lower(b) PERSISTENT);
select * from tbl1;    -- Read schema: (a int, b varchar, c varchar)?
insert into tbl1 values(...);    -- Write schema: (a int, b varchar)?
create table tbl2 as select * from tbl1 with data;    -- What should tbl2's schema be? What is the expected behavior?
insert into table tbl2 select * from tbl1;    -- If tbl2 has the exact same schema as tbl1, should this work?
......

One idea is to treat derived columns as metadata columns, much like _pos or _file. That might alleviate internal conflicts between read and write schemas within the Presto engine. But in reality, those conflicts are just shifted elsewhere—to the engine–table format boundary (e.g., with Iceberg) and to interoperability across different engines. And as you mentioned above, the RDBMSs that support derived columns generally don't have these problems.

I fully agree with the steps you are taking: first, support features that are compatible with the table format specification and do not break cross-engine interoperability; in parallel, drive the proposal-level change to the Iceberg specification; and finally, once the spec change is adopted and finalized, revisit the Presto implementation to support the full-featured, spec-aligned version of the feature. One thing that still puzzles me, however, is how exactly this derived column should be positioned—what type of column it should be, even within the Iceberg specification. That is, even if Iceberg spec were to support derived columns, it would still need to address similar issues to those discussed above. I would be very interested to hear your thoughts on this.