From 797c2c77e3f05228df147ce68e33244b480b54f1 Mon Sep 17 00:00:00 2001 From: Alston Dsouza Date: Mon, 22 Jun 2026 00:41:05 -0700 Subject: [PATCH 1/3] split integrations into plugins and cookbook --- docs/code-comparisons/dagster.rst | 3 +- docs/code-comparisons/kedro.rst | 2 +- docs/{integrations => cookbook}/fastapi.md | 2 +- .../fastapi_docs.png | Bin .../fastapi_summarization.png | Bin docs/cookbook/index.rst | 26 +++++++++++++++++ docs/{integrations => cookbook}/streamlit.md | 0 .../streamlit_logic.png | Bin docs/ecosystem/index.md | 10 +++---- docs/index.md | 3 +- docs/{integrations => plugins}/dbt.rst | 0 docs/{integrations => plugins}/dlt/index.md | 0 .../dlt/materialization.png | Bin .../dlt/transform.png | Bin .../ibis/ibis_column.png | Bin .../ibis/ibis_config.png | Bin .../ibis/ibis_plan.png | Bin .../ibis/ibis_table.png | Bin .../ibis/ibis_udf.png | Bin docs/{integrations => plugins}/ibis/index.md | 0 docs/{integrations => plugins}/index.rst | 27 ++++++++++++++---- 21 files changed, 57 insertions(+), 16 deletions(-) rename docs/{integrations => cookbook}/fastapi.md (98%) rename docs/{integrations => cookbook}/fastapi_docs.png (100%) rename docs/{integrations => cookbook}/fastapi_summarization.png (100%) create mode 100644 docs/cookbook/index.rst rename docs/{integrations => cookbook}/streamlit.md (100%) rename docs/{integrations => cookbook}/streamlit_logic.png (100%) rename docs/{integrations => plugins}/dbt.rst (100%) rename docs/{integrations => plugins}/dlt/index.md (100%) rename docs/{integrations => plugins}/dlt/materialization.png (100%) rename docs/{integrations => plugins}/dlt/transform.png (100%) rename docs/{integrations => plugins}/ibis/ibis_column.png (100%) rename docs/{integrations => plugins}/ibis/ibis_config.png (100%) rename docs/{integrations => plugins}/ibis/ibis_plan.png (100%) rename docs/{integrations => plugins}/ibis/ibis_table.png (100%) rename docs/{integrations => plugins}/ibis/ibis_udf.png (100%) rename docs/{integrations => plugins}/ibis/index.md (100%) rename docs/{integrations => plugins}/index.rst (60%) diff --git a/docs/code-comparisons/dagster.rst b/docs/code-comparisons/dagster.rst index 1e33d5ba8..cc8e8eafa 100644 --- a/docs/code-comparisons/dagster.rst +++ b/docs/code-comparisons/dagster.rst @@ -28,7 +28,7 @@ TL;DR - DIY or in tandem with Dagster, Airflow, Prefect, Metaflow, etc. - Includes: manual, schedules, sensors, conditional execution * - Micro orchestration (i.e., ``dbt``, ``LangChain``) - - Can run anywhere (locally, notebook, macro orchestrator, `FastAPI `_, `Streamlit `_, pyodide, etc.) + - Can run anywhere (locally, notebook, macro orchestrator, :doc:`FastAPI <../cookbook/fastapi>`, :doc:`Streamlit <../cookbook/streamlit>`, pyodide, etc.) - ❌ * - Code structure - Since it's micro, there are no restrictions. @@ -159,4 +159,3 @@ More information For a full side-by-side example of Dagster and Apache Hamilton, visit `this GitHub repository `_ For more questions, join our `Slack Channel `_! - \ No newline at end of file diff --git a/docs/code-comparisons/kedro.rst b/docs/code-comparisons/kedro.rst index ad238014a..8650cc1ed 100644 --- a/docs/code-comparisons/kedro.rst +++ b/docs/code-comparisons/kedro.rst @@ -73,7 +73,7 @@ These benefits of ``Apache Hamilton`` encourage developers to write smaller func The primary way to execute ``Kedro`` pipelines is to use the command line tool with ``kedro run --pipeline=my_pipeline``. Pipelines are typically designed for all nodes to be executed while reading data and writing results while going through nodes. It is closer to macro-orchestration frameworks like Airflow in spirit. -On the opposite, ``Apache Hamilton`` dataflows are primarily meant to be executed programmatically (i.e., via Python code) and return results in-memory. This makes it easy to use ``Apache Hamilton`` within a :doc:`FastAPI service <../integrations/fastapi>` service or to power an LLM application. +On the opposite, ``Apache Hamilton`` dataflows are primarily meant to be executed programmatically (i.e., via Python code) and return results in-memory. This makes it easy to use ``Apache Hamilton`` within a :doc:`FastAPI service <../cookbook/fastapi>` service or to power an LLM application. For comparable side-by-side code, we can dig into ``Kedro`` and use the ``SequentialRunner`` programmatically. To return pipeline results in-memory we would need to hack further with ``kedro.io.MemoryDataset``. diff --git a/docs/integrations/fastapi.md b/docs/cookbook/fastapi.md similarity index 98% rename from docs/integrations/fastapi.md rename to docs/cookbook/fastapi.md index c56561f84..f274b5efd 100644 --- a/docs/integrations/fastapi.md +++ b/docs/cookbook/fastapi.md @@ -173,5 +173,5 @@ app.routes[-1].description += f""" **PySpark** | Distributed data processing with Spark | [Examples](https://github.com/apache/hamilton/tree/main/examples/spark) \| [GraphAdapter](../reference/graph-adapters/index.rst) | | **Dask** | Parallel computing and distributed arrays | [Examples](https://github.com/apache/hamilton/tree/main/examples/dask) \| [GraphAdapter](../reference/graph-adapters/DaskGraphAdapter.rst) | | **Ray** | Distributed computing framework | [Examples](https://github.com/apache/hamilton/tree/main/examples/ray) \| [GraphAdapter](../reference/graph-adapters/RayGraphAdapter.rst) | -| **Ibis** | Portable DataFrame API across backends | [Integration Guide](../integrations/ibis/index.md) | +| **Ibis** | Portable DataFrame API across backends | [Plugin Guide](../plugins/ibis/index.md) | | **Vaex** | Out-of-core DataFrame library | [Examples](https://github.com/apache/hamilton/tree/main/examples/vaex) | | **Narwhals** | DataFrame-agnostic interface | [Examples](https://github.com/apache/hamilton/tree/main/examples/narwhals) \| [Lifecycle Hook](../reference/lifecycle-hooks/Narwhals.rst) | | **NumPy** | Numerical computing arrays | [ResultBuilder](../reference/result-builders/Numpy.rst) | @@ -75,7 +75,7 @@ Use Apache Hamilton within your existing orchestration infrastructure: | **Prefect** | Workflow orchestration | [Examples](https://github.com/apache/hamilton/tree/main/examples/prefect) | | **Kedro** | Data science pipelines | [Examples](https://github.com/apache/hamilton/tree/main/examples/kedro) | | **Metaflow** | ML infrastructure | [Integration](https://github.com/outerbounds/hamilton-metaflow) | -| **dbt** | Data transformation tool | [Integration Guide](../integrations/dbt.rst) | +| **dbt** | Data transformation tool | [Plugin Guide](../plugins/dbt.rst) | ### Data Engineering & ETL @@ -83,10 +83,10 @@ Tools for building robust data pipelines: | Integration | Description | Documentation | |------------|-------------|---------------| -| **dlt** | Data loading and transformation | [Integration Guide](../integrations/dlt/index.md) | +| **dlt** | Data loading and transformation | [Plugin Guide](../plugins/dlt/index.md) | | **Feast** | Feature store | [Examples](https://github.com/apache/hamilton/tree/main/examples/feast) | -| **FastAPI** | Web service framework | [Integration Guide](../integrations/fastapi.md) | -| **Streamlit** | Interactive web applications | [Integration Guide](../integrations/streamlit.md) | +| **FastAPI** | Web service framework | [Cookbook](../cookbook/fastapi.md) | +| **Streamlit** | Interactive web applications | [Cookbook](../cookbook/streamlit.md) | ### Observability & Monitoring diff --git a/docs/index.md b/docs/index.md index 2c2c25de7..4336aeeed 100644 --- a/docs/index.md +++ b/docs/index.md @@ -35,7 +35,8 @@ concepts/index how-tos/index hamilton-ui/index hamilton-vscode/index -integrations/index +plugins/index +cookbook/index code-comparisons/index ``` diff --git a/docs/integrations/dbt.rst b/docs/plugins/dbt.rst similarity index 100% rename from docs/integrations/dbt.rst rename to docs/plugins/dbt.rst diff --git a/docs/integrations/dlt/index.md b/docs/plugins/dlt/index.md similarity index 100% rename from docs/integrations/dlt/index.md rename to docs/plugins/dlt/index.md diff --git a/docs/integrations/dlt/materialization.png b/docs/plugins/dlt/materialization.png similarity index 100% rename from docs/integrations/dlt/materialization.png rename to docs/plugins/dlt/materialization.png diff --git a/docs/integrations/dlt/transform.png b/docs/plugins/dlt/transform.png similarity index 100% rename from docs/integrations/dlt/transform.png rename to docs/plugins/dlt/transform.png diff --git a/docs/integrations/ibis/ibis_column.png b/docs/plugins/ibis/ibis_column.png similarity index 100% rename from docs/integrations/ibis/ibis_column.png rename to docs/plugins/ibis/ibis_column.png diff --git a/docs/integrations/ibis/ibis_config.png b/docs/plugins/ibis/ibis_config.png similarity index 100% rename from docs/integrations/ibis/ibis_config.png rename to docs/plugins/ibis/ibis_config.png diff --git a/docs/integrations/ibis/ibis_plan.png b/docs/plugins/ibis/ibis_plan.png similarity index 100% rename from docs/integrations/ibis/ibis_plan.png rename to docs/plugins/ibis/ibis_plan.png diff --git a/docs/integrations/ibis/ibis_table.png b/docs/plugins/ibis/ibis_table.png similarity index 100% rename from docs/integrations/ibis/ibis_table.png rename to docs/plugins/ibis/ibis_table.png diff --git a/docs/integrations/ibis/ibis_udf.png b/docs/plugins/ibis/ibis_udf.png similarity index 100% rename from docs/integrations/ibis/ibis_udf.png rename to docs/plugins/ibis/ibis_udf.png diff --git a/docs/integrations/ibis/index.md b/docs/plugins/ibis/index.md similarity index 100% rename from docs/integrations/ibis/index.md rename to docs/plugins/ibis/index.md diff --git a/docs/integrations/index.rst b/docs/plugins/index.rst similarity index 60% rename from docs/integrations/index.rst rename to docs/plugins/index.rst index e131dd8bf..b11766936 100644 --- a/docs/integrations/index.rst +++ b/docs/plugins/index.rst @@ -1,14 +1,29 @@ -============ -Integrations -============ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at -This section showcases how Apache Hamilton integrates with popular frameworks. + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Plugins +------- + +Apache Hamilton provides plugins for working with popular data libraries, +execution frameworks, and developer tools. .. toctree:: dlt/index - fastapi ibis/index - streamlit dbt MLFlow Airflow From 5402ef19a198186c32ee439ee5a3f1ce68ab0b94 Mon Sep 17 00:00:00 2001 From: Alston Dsouza Date: Mon, 22 Jun 2026 00:42:49 -0700 Subject: [PATCH 2/3] separate lifecycle api from plugin adapters --- docs/code-comparisons/dagster.rst | 2 +- docs/ecosystem/index.md | 14 ++--- docs/index.md | 1 + docs/reference/lifecycle-hooks/index.rst | 52 +++---------------- .../DDOGTracer.rst | 0 .../FunctionInputOutputTypeChecker.rst | 0 .../GracefulErrorAdapter.rst | 0 .../MLFlowTracker.rst | 0 .../{lifecycle-hooks => plugins}/Narwhals.rst | 0 .../NoEdgeAndInputTypeChecking.rst | 0 .../OpenLineageAdapter.rst | 0 .../PDBDebugger.rst | 0 .../{lifecycle-hooks => plugins}/PrintLn.rst | 0 .../ProgressBar.rst | 0 .../RichProgressBar.rst | 0 .../SlackNotifier.rst} | 0 .../SparkInputValidator.rst | 0 docs/reference/plugins/index.rst | 52 +++++++++++++++++++ 18 files changed, 67 insertions(+), 54 deletions(-) rename docs/reference/{lifecycle-hooks => plugins}/DDOGTracer.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/FunctionInputOutputTypeChecker.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/GracefulErrorAdapter.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/MLFlowTracker.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/Narwhals.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/NoEdgeAndInputTypeChecking.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/OpenLineageAdapter.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/PDBDebugger.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/PrintLn.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/ProgressBar.rst (100%) rename docs/reference/{lifecycle-hooks => plugins}/RichProgressBar.rst (100%) rename docs/reference/{lifecycle-hooks/SlackNotifierHook.rst => plugins/SlackNotifier.rst} (100%) rename docs/reference/{lifecycle-hooks => plugins}/SparkInputValidator.rst (100%) create mode 100644 docs/reference/plugins/index.rst diff --git a/docs/code-comparisons/dagster.rst b/docs/code-comparisons/dagster.rst index cc8e8eafa..87b230b52 100644 --- a/docs/code-comparisons/dagster.rst +++ b/docs/code-comparisons/dagster.rst @@ -73,7 +73,7 @@ TL;DR - `Lifecycle hooks `_ (easier to extend) - `Op Hooks `_ * - Plugins - - `Spark `_, Dask, Ray, `Datadog `_, polars, pandera, and more (Apache Hamilton is less restrictive and easier to extend) + - `Spark `_, Dask, Ray, :doc:`Datadog <../reference/plugins/DDOGTracer>`, polars, pandera, and more (Apache Hamilton is less restrictive and easier to extend) - `Spark, Dask, polars, pandera, Databricks, Snowflake, Great Expections, and more `_ (Dagster integrations are more involved to develop) * - Interactive Development - `Jupyter Magic `_, `VSCode extension `_ diff --git a/docs/ecosystem/index.md b/docs/ecosystem/index.md index c7d715127..329f58503 100644 --- a/docs/ecosystem/index.md +++ b/docs/ecosystem/index.md @@ -46,7 +46,7 @@ Apache Hamilton integrates seamlessly with popular data manipulation libraries: | **Ray** | Distributed computing framework | [Examples](https://github.com/apache/hamilton/tree/main/examples/ray) \| [GraphAdapter](../reference/graph-adapters/RayGraphAdapter.rst) | | **Ibis** | Portable DataFrame API across backends | [Plugin Guide](../plugins/ibis/index.md) | | **Vaex** | Out-of-core DataFrame library | [Examples](https://github.com/apache/hamilton/tree/main/examples/vaex) | -| **Narwhals** | DataFrame-agnostic interface | [Examples](https://github.com/apache/hamilton/tree/main/examples/narwhals) \| [Lifecycle Hook](../reference/lifecycle-hooks/Narwhals.rst) | +| **Narwhals** | DataFrame-agnostic interface | [Examples](https://github.com/apache/hamilton/tree/main/examples/narwhals) \| [Plugin API](../reference/plugins/Narwhals.rst) | | **NumPy** | Numerical computing arrays | [ResultBuilder](../reference/result-builders/Numpy.rst) | | **PyArrow** | Columnar in-memory data | [ResultBuilder](../reference/result-builders/PyArrow.rst) | @@ -56,7 +56,7 @@ Build and deploy ML workflows with Apache Hamilton: | Integration | Description | Documentation | |------------|-------------|---------------| -| **MLflow** | Experiment tracking and model registry | [Examples](https://github.com/apache/hamilton/tree/main/examples/mlflow) \| [Lifecycle Hook](../reference/lifecycle-hooks/MLFlowTracker.rst) | +| **MLflow** | Experiment tracking and model registry | [Examples](https://github.com/apache/hamilton/tree/main/examples/mlflow) \| [Plugin API](../reference/plugins/MLFlowTracker.rst) | | **scikit-learn** | Machine learning algorithms | [Examples](https://github.com/apache/hamilton/tree/main/examples/scikit-learn) | | **XGBoost** | Gradient boosting framework | [IO Adapters](../reference/io/available-data-adapters.rst) | | **LightGBM** | Gradient boosting framework | [IO Adapters](../reference/io/available-data-adapters.rst) | @@ -94,9 +94,9 @@ Track and monitor your Apache Hamilton dataflows: | Integration | Description | Documentation | |------------|-------------|---------------| -| **Datadog** | Monitoring and analytics | [Lifecycle Hook](../reference/lifecycle-hooks/DDOGTracer.rst) | +| **Datadog** | Monitoring and analytics | [Plugin API](../reference/plugins/DDOGTracer.rst) | | **OpenTelemetry** | Observability framework | [Examples](https://github.com/apache/hamilton/tree/main/examples/opentelemetry) | -| **OpenLineage** | Data lineage tracking | [Examples](https://github.com/apache/hamilton/tree/main/examples/openlineage) \| [Lifecycle Hook](../reference/lifecycle-hooks/OpenLineageAdapter.rst) | +| **OpenLineage** | Data lineage tracking | [Examples](https://github.com/apache/hamilton/tree/main/examples/openlineage) \| [Plugin API](../reference/plugins/OpenLineageAdapter.rst) | | **Hamilton UI** | Built-in execution tracking | [UI Guide](../hamilton-ui/index.rst) | | **Experiment Manager** | Lightweight experiment tracking | [Examples](https://github.com/apache/hamilton/tree/main/examples/experiment_management) | @@ -108,7 +108,7 @@ Create visualizations from your dataflows: |------------|-------------|---------------| | **Plotly** | Interactive plotting | [Examples](https://github.com/apache/hamilton/tree/main/examples/plotly) | | **Matplotlib** | Static plotting | [IO Adapters](../reference/io/available-data-adapters.rst) | -| **Rich** | Terminal formatting and progress | [Lifecycle Hook](../reference/lifecycle-hooks/RichProgressBar.rst) | +| **Rich** | Terminal formatting and progress | [Plugin API](../reference/plugins/RichProgressBar.rst) | ### Developer Tools @@ -120,7 +120,7 @@ Improve your development workflow: | **VS Code** | Language server and extension | [VS Code Guide](../hamilton-vscode/index.rst) | | **Claude Code** | AI assistant plugin for Hamilton development | [Plugin Guide](claude-code-plugin.md) | | **MCP Server** | LLM tool server for interactive DAG development | [MCP Guide](mcp-server.md) | -| **tqdm** | Progress bars | [Lifecycle Hook](../reference/lifecycle-hooks/ProgressBar.rst) | +| **tqdm** | Progress bars | [Plugin API](../reference/plugins/ProgressBar.rst) | ### Cloud Providers & Infrastructure @@ -145,7 +145,7 @@ Persist and cache your data: | Integration | Description | Documentation | |------------|-------------|---------------| -| **Slack** | Notifications and integrations | [Examples](https://github.com/apache/hamilton/tree/main/examples/slack) \| [Lifecycle Hook](../reference/lifecycle-hooks/SlackNotifierHook.rst) | +| **Slack** | Notifications and integrations | [Examples](https://github.com/apache/hamilton/tree/main/examples/slack) \| [Plugin API](../reference/plugins/SlackNotifier.rst) | | **GeoPandas** | Geospatial data analysis | [Type extension](https://github.com/apache/hamilton/blob/main/hamilton/plugins/geopandas_extensions.py) for GeoDataFrame support | | **YAML** | Configuration management | [IO Adapters](../reference/io/available-data-adapters.rst) | | **Neo4j** | Knowledge graph RAG | [Examples](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/neo4j_graph_rag) | diff --git a/docs/index.md b/docs/index.md index 4336aeeed..c4c3adeb8 100644 --- a/docs/index.md +++ b/docs/index.md @@ -74,6 +74,7 @@ reference/drivers/index reference/caching/index reference/graph-adapters/index reference/lifecycle-hooks/index +reference/plugins/index reference/result-builders/index reference/io/index reference/dataflows/index diff --git a/docs/reference/lifecycle-hooks/index.rst b/docs/reference/lifecycle-hooks/index.rst index 24b1eaaa2..524bb57a8 100644 --- a/docs/reference/lifecycle-hooks/index.rst +++ b/docs/reference/lifecycle-hooks/index.rst @@ -1,19 +1,15 @@ -================== -Lifecycle Adapters -================== - -Currently a few of the API extensions are still experimental. Note this doesn't mean they're not well-tested or thought out -- rather that we're actively looking for feedback. More docs upcoming, but for now fish around the `experimental package `_, and give the extensions a try! - -The other extensions live within `plugins `_. These are fully supported and will be backwards compatible across major versions. - -Customization -------------- +============= +Lifecycle API +============= The subsequent documents contain public-facing APIs for customizing Apache Hamilton's execution. Note that the public-facing APIs are still a work in progress -- we will be improving the documentation. We plan for the APIs, however, to be stable looking forward. +For ready-to-use implementations of these interfaces, see the +:doc:`Plugin API Reference <../plugins/index>`. + .. toctree:: ResultBuilder LegacyResultMixin @@ -28,39 +24,3 @@ looking forward. TaskReturnHook TaskExecutionHook TaskGroupingHook - -Available Adapters -------------------- - -In addition to the base classes for lifecycle adapters, we have a few adapters implemented and available for use. -Note that some of these are plugins, meaning they require installing additional (external) libraries. - -Recall to add lifecycle adapters, you just need to call the ``with_adapters`` method of the driver: - -.. code-block:: python - - dr = ( - driver - .Builder() - .with_modules(...) - .with_adapters( - Adapter1(...), - Adapter2(...), - *more_adapters) - ...build() - ) - -.. toctree:: - PDBDebugger - PrintLn - ProgressBar - RichProgressBar - DDOGTracer - FunctionInputOutputTypeChecker - SlackNotifierHook - GracefulErrorAdapter - SparkInputValidator - Narwhals - MLFlowTracker - NoEdgeAndInputTypeChecking - OpenLineageAdapter diff --git a/docs/reference/lifecycle-hooks/DDOGTracer.rst b/docs/reference/plugins/DDOGTracer.rst similarity index 100% rename from docs/reference/lifecycle-hooks/DDOGTracer.rst rename to docs/reference/plugins/DDOGTracer.rst diff --git a/docs/reference/lifecycle-hooks/FunctionInputOutputTypeChecker.rst b/docs/reference/plugins/FunctionInputOutputTypeChecker.rst similarity index 100% rename from docs/reference/lifecycle-hooks/FunctionInputOutputTypeChecker.rst rename to docs/reference/plugins/FunctionInputOutputTypeChecker.rst diff --git a/docs/reference/lifecycle-hooks/GracefulErrorAdapter.rst b/docs/reference/plugins/GracefulErrorAdapter.rst similarity index 100% rename from docs/reference/lifecycle-hooks/GracefulErrorAdapter.rst rename to docs/reference/plugins/GracefulErrorAdapter.rst diff --git a/docs/reference/lifecycle-hooks/MLFlowTracker.rst b/docs/reference/plugins/MLFlowTracker.rst similarity index 100% rename from docs/reference/lifecycle-hooks/MLFlowTracker.rst rename to docs/reference/plugins/MLFlowTracker.rst diff --git a/docs/reference/lifecycle-hooks/Narwhals.rst b/docs/reference/plugins/Narwhals.rst similarity index 100% rename from docs/reference/lifecycle-hooks/Narwhals.rst rename to docs/reference/plugins/Narwhals.rst diff --git a/docs/reference/lifecycle-hooks/NoEdgeAndInputTypeChecking.rst b/docs/reference/plugins/NoEdgeAndInputTypeChecking.rst similarity index 100% rename from docs/reference/lifecycle-hooks/NoEdgeAndInputTypeChecking.rst rename to docs/reference/plugins/NoEdgeAndInputTypeChecking.rst diff --git a/docs/reference/lifecycle-hooks/OpenLineageAdapter.rst b/docs/reference/plugins/OpenLineageAdapter.rst similarity index 100% rename from docs/reference/lifecycle-hooks/OpenLineageAdapter.rst rename to docs/reference/plugins/OpenLineageAdapter.rst diff --git a/docs/reference/lifecycle-hooks/PDBDebugger.rst b/docs/reference/plugins/PDBDebugger.rst similarity index 100% rename from docs/reference/lifecycle-hooks/PDBDebugger.rst rename to docs/reference/plugins/PDBDebugger.rst diff --git a/docs/reference/lifecycle-hooks/PrintLn.rst b/docs/reference/plugins/PrintLn.rst similarity index 100% rename from docs/reference/lifecycle-hooks/PrintLn.rst rename to docs/reference/plugins/PrintLn.rst diff --git a/docs/reference/lifecycle-hooks/ProgressBar.rst b/docs/reference/plugins/ProgressBar.rst similarity index 100% rename from docs/reference/lifecycle-hooks/ProgressBar.rst rename to docs/reference/plugins/ProgressBar.rst diff --git a/docs/reference/lifecycle-hooks/RichProgressBar.rst b/docs/reference/plugins/RichProgressBar.rst similarity index 100% rename from docs/reference/lifecycle-hooks/RichProgressBar.rst rename to docs/reference/plugins/RichProgressBar.rst diff --git a/docs/reference/lifecycle-hooks/SlackNotifierHook.rst b/docs/reference/plugins/SlackNotifier.rst similarity index 100% rename from docs/reference/lifecycle-hooks/SlackNotifierHook.rst rename to docs/reference/plugins/SlackNotifier.rst diff --git a/docs/reference/lifecycle-hooks/SparkInputValidator.rst b/docs/reference/plugins/SparkInputValidator.rst similarity index 100% rename from docs/reference/lifecycle-hooks/SparkInputValidator.rst rename to docs/reference/plugins/SparkInputValidator.rst diff --git a/docs/reference/plugins/index.rst b/docs/reference/plugins/index.rst new file mode 100644 index 000000000..bbac05735 --- /dev/null +++ b/docs/reference/plugins/index.rst @@ -0,0 +1,52 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Plugin API Reference +-------------------- + +Apache Hamilton includes ready-to-use adapters built on the +:doc:`Lifecycle API <../lifecycle-hooks/index>`. Some plugins require an +additional third-party library to be installed. + +Add one or more adapters when building the driver: + +.. code-block:: python + + dr = ( + driver.Builder() + .with_modules(...) + .with_adapters( + Adapter1(...), + Adapter2(...), + ) + .build() + ) + +.. toctree:: + PDBDebugger + PrintLn + ProgressBar + RichProgressBar + DDOGTracer + FunctionInputOutputTypeChecker + SlackNotifier + GracefulErrorAdapter + SparkInputValidator + Narwhals + MLFlowTracker + NoEdgeAndInputTypeChecking + OpenLineageAdapter From e5d80e64e64ee97bd11f4c32a1d081e17eb9e532 Mon Sep 17 00:00:00 2001 From: Alston Dsouza Date: Mon, 22 Jun 2026 00:43:28 -0700 Subject: [PATCH 3/3] fix example links after moving the docs --- examples/dlt/README.md | 2 +- examples/ibis/feature_engineering/README.md | 2 +- examples/ibis/jaffle_shop/README.md | 2 +- examples/streamlit/README.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/examples/dlt/README.md b/examples/dlt/README.md index 0017c5b3b..d6611b3b3 100644 --- a/examples/dlt/README.md +++ b/examples/dlt/README.md @@ -19,7 +19,7 @@ under the License. # dlt -This example shows dlt + Apache Hamilton can help you cover the full ELT cycle using portable Python code. It is a companion to this [documentation page](https://hamilton.apache.org/integrations/dlt/). +This example shows dlt + Apache Hamilton can help you cover the full ELT cycle using portable Python code. It is a companion to this [documentation page](https://hamilton.apache.org/plugins/dlt/). # Content It includes a pipeline to ingest messages from Slack channels and generate threads summaries. diff --git a/examples/ibis/feature_engineering/README.md b/examples/ibis/feature_engineering/README.md index 94bef2acf..06033dfe6 100644 --- a/examples/ibis/feature_engineering/README.md +++ b/examples/ibis/feature_engineering/README.md @@ -53,5 +53,5 @@ Follow these steps to get the example working: # Resources -- [Learn more about Apache Hamilton + Ibis](https://hamilton.apache.org/integrations/ibis/) +- [Learn more about Apache Hamilton + Ibis](https://hamilton.apache.org/plugins/ibis/) - [Ibis documentation](https://ibis-project.org/) diff --git a/examples/ibis/jaffle_shop/README.md b/examples/ibis/jaffle_shop/README.md index 9106156e7..24768b148 100644 --- a/examples/ibis/jaffle_shop/README.md +++ b/examples/ibis/jaffle_shop/README.md @@ -21,7 +21,7 @@ under the License. This repository is a reimplementation of the canonical [dbt example jaffle_shop](https://github.com/dbt-labs/jaffle_shop). It illustrates data transformations for an ecommerce store. -Data transformations are implemented using the Python library [Ibis](https://ibis-project.org/) which allows to define SQL operations that works across backends. By default, it uses [duckdb](https://duckdb.org/) for local development. Apache Hamilton + Ibis provides a Python-centric alternative to dbt ([Learn more](https://hamilton.apache.org/integrations/ibis/)). +Data transformations are implemented using the Python library [Ibis](https://ibis-project.org/) which allows to define SQL operations that works across backends. By default, it uses [duckdb](https://duckdb.org/) for local development. Apache Hamilton + Ibis provides a Python-centric alternative to dbt ([Learn more](https://hamilton.apache.org/plugins/ibis/)). ## Content The content and structure aims to match the [original dbt `jaffle_shop`](https://github.com/dbt-labs/jaffle_shop/tree/main) example. On the other hand, Apache Hamilton is just a Python library and is flexible regarding project structure. diff --git a/examples/streamlit/README.md b/examples/streamlit/README.md index 2120a323e..63212aaf2 100644 --- a/examples/streamlit/README.md +++ b/examples/streamlit/README.md @@ -19,7 +19,7 @@ under the License. # Streamlit + Apache Hamilton -> This example accompanies the documentation page for [Streamlit](https://hamilton.apache.org/integrations/streamlit/) integration. +> This example accompanies the [Streamlit cookbook page](https://hamilton.apache.org/cookbook/streamlit/). Streamlit is an open-source Python library to create web applications with minimal effort. It's an effective solution to create simple dashboards, interactive data visualizations, and proof-of-concepts for data science, machine learning, and LLM applications.