Skip to content

Basic Integration with Datafusion#324

Merged
liurenjie1024 merged 37 commits intoapache:mainfrom
marvinlanhenke:draft_datafusion
May 2, 2024
Merged

Basic Integration with Datafusion#324
liurenjie1024 merged 37 commits intoapache:mainfrom
marvinlanhenke:draft_datafusion

Conversation

@marvinlanhenke
Copy link
Copy Markdown
Contributor

I had some fun, working with Datafusion and I want to share the (rough) design with you.

As outlined in #242 I went ahead and:

  • created a new crate integrations
  • a subfolder datafusion with its respective modules (catalog.rs, schema.rs, etc.)
  • implemented some of the traits e.g. CatalogProvider
  • basic test infra for integration test with HiveMetastore

The overall structure should be extensible, meaning - if we want to support an integration with e.g. polars we simply create a new subfolder and implement the traits.

The test infra is supposed to be used by every integration.

The IcebergCatalogProvider, for example, should be usable by any Catalog (since it Arc for the client (dynamic dispatch))

I really would appreciate your thoughts on this approach - so we can discuss how to proceed, whats missing upstream (like #277), etc.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants