Improve documentation structure for plugins and cookbook

docs/code-comparisons/dagster.rst

@@ -28,7 +28,7 @@ TL;DR
       - DIY or in tandem with Dagster, Airflow, Prefect, Metaflow, etc.
       - Includes: manual, schedules, sensors, conditional execution
     * - Micro orchestration (i.e., ``dbt``, ``LangChain``)
-      - Can run anywhere (locally, notebook, macro orchestrator, `FastAPI <https://hamilton.apache.org/integrations/fastapi/>`_, `Streamlit <https://hamilton.apache.org/integrations/streamlit/>`_, pyodide, etc.)
+      - Can run anywhere (locally, notebook, macro orchestrator, :doc:`FastAPI <../cookbook/fastapi>`, :doc:`Streamlit <../cookbook/streamlit>`, pyodide, etc.)
       - ❌
     * - Code structure
       - Since it's micro, there are no restrictions.
@@ -73,7 +73,7 @@ TL;DR
       - `Lifecycle hooks <https://hamilton.apache.org/reference/lifecycle-hooks/>`_ (easier to extend)
       - `Op Hooks <https://docs.dagster.io/concepts/ops-jobs-graphs/op-hooks#op-hooks>`_
     * - Plugins
-      - `Spark <https://blog.dagworks.io/p/expressing-pyspark-transformations>`_, Dask, Ray, `Datadog <https://hamilton.apache.org/reference/lifecycle-hooks/DDOGTracer/>`_, polars, pandera, and more (Apache Hamilton is less restrictive and easier to extend)
+      - `Spark <https://blog.dagworks.io/p/expressing-pyspark-transformations>`_, Dask, Ray, :doc:`Datadog <../reference/plugins/DDOGTracer>`, polars, pandera, and more (Apache Hamilton is less restrictive and easier to extend)
       - `Spark, Dask, polars, pandera, Databricks, Snowflake, Great Expections, and more <https://docs.dagster.io/integrations>`_  (Dagster integrations are more involved to develop)
     * - Interactive Development
       - `Jupyter Magic <https://hamilton.apache.org/how-tos/use-in-jupyter-notebook/#use-hamilton-jupyter-magic>`_, `VSCode extension <https://marketplace.visualstudio.com/items?itemName=ThierryJean.hamilton>`_
@@ -159,4 +159,3 @@ More information
 For a full side-by-side example of Dagster and Apache Hamilton, visit `this GitHub repository <https://github.com/apache/hamilton/tree/main/examples/dagster>`_
 
 For more questions, join our `Slack Channel <https://join.slack.com/t/hamilton-opensource/shared_invite/zt-2niepkra8-DGKGf_tTYhXuJWBTXtIs4g>`_!
-

docs/code-comparisons/kedro.rst

@@ -73,7 +73,7 @@ These benefits of ``Apache Hamilton`` encourage developers to write smaller func
 
 The primary way to execute ``Kedro`` pipelines is to use the command line tool with ``kedro run --pipeline=my_pipeline``. Pipelines are typically designed for all nodes to be executed while reading data and writing results while going through nodes. It is closer to macro-orchestration frameworks like Airflow in spirit.
 
-On the opposite, ``Apache Hamilton`` dataflows are primarily meant to be executed programmatically (i.e., via Python code) and return results in-memory. This makes it easy to use ``Apache Hamilton`` within a :doc:`FastAPI service <../integrations/fastapi>` service or to power an LLM application.
+On the opposite, ``Apache Hamilton`` dataflows are primarily meant to be executed programmatically (i.e., via Python code) and return results in-memory. This makes it easy to use ``Apache Hamilton`` within a :doc:`FastAPI service <../cookbook/fastapi>` service or to power an LLM application.
 
 For comparable side-by-side code, we can dig into ``Kedro`` and use the ``SequentialRunner`` programmatically. To return pipeline results in-memory we would need to hack further with ``kedro.io.MemoryDataset``.
 

docs/integrations/fastapi.md → docs/cookbook/fastapi.md

@@ -173,5 +173,5 @@ app.routes[-1].description += f"""<img src="data:image/png;base64,{base64_viz}"/
 
 ### Benefits
 - **Separation of concerns**: the decoupling between `server.py` and `summarization.py` makes it easier to extend and test the server separately from the data transformations.
-- **Reusable code**: the module `summarization.py` can be reused elsewhere with Apache Hamilton. For instance, if you first started by building a proof-of-concept with [Streamlit + Apache Hamilton](https://hamilton.apache.org/integrations/streamlit), the logic you produced could be reused to power your FastAPI server.
+- **Reusable code**: the module `summarization.py` can be reused elsewhere with Apache Hamilton. For instance, if you first started by building a proof-of-concept with [Streamlit + Apache Hamilton](streamlit.md), the logic you produced could be reused to power your FastAPI server.
 - **Richer documentation**: Apache Hamilton allows to view and better understand the dataflow of an operation. This helps onboard new API users and greatly facilitates transferring the ownership of the API to other engineers.

docs/cookbook/index.rst

@@ -0,0 +1,26 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Cookbook
+--------
+
+Practical examples for using Apache Hamilton inside applications and web
+frameworks.
+
+.. toctree::
+    fastapi
+    streamlit

docs/ecosystem/index.md

@@ -44,9 +44,9 @@ Apache Hamilton integrates seamlessly with popular data manipulation libraries:
 | <img src="../_static/logos/pyspark.svg" width="20" height="20" style="vertical-align: middle;"> **PySpark** | Distributed data processing with Spark | [Examples](https://github.com/apache/hamilton/tree/main/examples/spark) \| [GraphAdapter](../reference/graph-adapters/index.rst) |
 | <img src="../_static/logos/dask.svg" width="20" height="20" style="vertical-align: middle;"> **Dask** | Parallel computing and distributed arrays | [Examples](https://github.com/apache/hamilton/tree/main/examples/dask) \| [GraphAdapter](../reference/graph-adapters/DaskGraphAdapter.rst) |
 | <img src="../_static/logos/ray.svg" width="20" height="20" style="vertical-align: middle;"> **Ray** | Distributed computing framework | [Examples](https://github.com/apache/hamilton/tree/main/examples/ray) \| [GraphAdapter](../reference/graph-adapters/RayGraphAdapter.rst) |
-| <img src="../_static/logos/ibis.png" width="20" height="20" style="vertical-align: middle;"> **Ibis** | Portable DataFrame API across backends | [Integration Guide](../integrations/ibis/index.md) |
+| <img src="../_static/logos/ibis.png" width="20" height="20" style="vertical-align: middle;"> **Ibis** | Portable DataFrame API across backends | [Plugin Guide](../plugins/ibis/index.md) |
 | <img src="../_static/logos/vaex.png" width="20" height="20" style="vertical-align: middle;"> **Vaex** | Out-of-core DataFrame library | [Examples](https://github.com/apache/hamilton/tree/main/examples/vaex) |
-| <img src="../_static/logos/narwhals.png" width="20" height="20" style="vertical-align: middle;"> **Narwhals** | DataFrame-agnostic interface | [Examples](https://github.com/apache/hamilton/tree/main/examples/narwhals) \| [Lifecycle Hook](../reference/lifecycle-hooks/Narwhals.rst) |
+| <img src="../_static/logos/narwhals.png" width="20" height="20" style="vertical-align: middle;"> **Narwhals** | DataFrame-agnostic interface | [Examples](https://github.com/apache/hamilton/tree/main/examples/narwhals) \| [Plugin API](../reference/plugins/Narwhals.rst) |
 | <img src="../_static/logos/numpy.svg" width="20" height="20" style="vertical-align: middle;"> **NumPy** | Numerical computing arrays | [ResultBuilder](../reference/result-builders/Numpy.rst) |
 | <img src="../_static/logos/pyarrow.png" width="20" height="20" style="vertical-align: middle;"> **PyArrow** | Columnar in-memory data | [ResultBuilder](../reference/result-builders/PyArrow.rst) |
 
@@ -56,7 +56,7 @@ Build and deploy ML workflows with Apache Hamilton:
 
 | Integration | Description | Documentation |
 |------------|-------------|---------------|
-| <img src="../_static/logos/mlflow.png" width="20" height="20" style="vertical-align: middle;"> **MLflow** | Experiment tracking and model registry | [Examples](https://github.com/apache/hamilton/tree/main/examples/mlflow) \| [Lifecycle Hook](../reference/lifecycle-hooks/MLFlowTracker.rst) |
+| <img src="../_static/logos/mlflow.png" width="20" height="20" style="vertical-align: middle;"> **MLflow** | Experiment tracking and model registry | [Examples](https://github.com/apache/hamilton/tree/main/examples/mlflow) \| [Plugin API](../reference/plugins/MLFlowTracker.rst) |
 | <img src="../_static/logos/scikit-learn.png" width="20" height="20" style="vertical-align: middle;"> **scikit-learn** | Machine learning algorithms | [Examples](https://github.com/apache/hamilton/tree/main/examples/scikit-learn) |
 | <img src="../_static/logos/xgboost.png" width="20" height="20" style="vertical-align: middle;"> **XGBoost** | Gradient boosting framework | [IO Adapters](../reference/io/available-data-adapters.rst) |
 | <img src="../_static/logos/lightgbm.svg" width="20" height="20" style="vertical-align: middle;"> **LightGBM** | Gradient boosting framework | [IO Adapters](../reference/io/available-data-adapters.rst) |
@@ -75,28 +75,28 @@ Use Apache Hamilton within your existing orchestration infrastructure:
 | <img src="../_static/logos/prefect.png" width="20" height="20" style="vertical-align: middle;"> **Prefect** | Workflow orchestration | [Examples](https://github.com/apache/hamilton/tree/main/examples/prefect) |
 | <img src="../_static/logos/kedro.png" width="20" height="20" style="vertical-align: middle;"> **Kedro** | Data science pipelines | [Examples](https://github.com/apache/hamilton/tree/main/examples/kedro) |
 | <img src="../_static/logos/metaflow.png" width="20" height="20" style="vertical-align: middle;"> **Metaflow** | ML infrastructure | [Integration](https://github.com/outerbounds/hamilton-metaflow) |
-| <img src="../_static/logos/dbt.png" width="20" height="20" style="vertical-align: middle;"> **dbt** | Data transformation tool | [Integration Guide](../integrations/dbt.rst) |
+| <img src="../_static/logos/dbt.png" width="20" height="20" style="vertical-align: middle;"> **dbt** | Data transformation tool | [Plugin Guide](../plugins/dbt.rst) |
 
 ### Data Engineering & ETL
 
 Tools for building robust data pipelines:
 
 | Integration | Description | Documentation |
 |------------|-------------|---------------|
-| <img src="../_static/logos/dlt.svg" width="20" height="20" style="vertical-align: middle;"> **dlt** | Data loading and transformation | [Integration Guide](../integrations/dlt/index.md) |
+| <img src="../_static/logos/dlt.svg" width="20" height="20" style="vertical-align: middle;"> **dlt** | Data loading and transformation | [Plugin Guide](../plugins/dlt/index.md) |
 | <img src="../_static/logos/feast.png" width="20" height="20" style="vertical-align: middle;"> **Feast** | Feature store | [Examples](https://github.com/apache/hamilton/tree/main/examples/feast) |
-| <img src="../_static/logos/fastapi.svg" width="20" height="20" style="vertical-align: middle;"> **FastAPI** | Web service framework | [Integration Guide](../integrations/fastapi.md) |
-| <img src="../_static/logos/streamlit.png" width="20" height="20" style="vertical-align: middle;"> **Streamlit** | Interactive web applications | [Integration Guide](../integrations/streamlit.md) |
+| <img src="../_static/logos/fastapi.svg" width="20" height="20" style="vertical-align: middle;"> **FastAPI** | Web service framework | [Cookbook](../cookbook/fastapi.md) |
+| <img src="../_static/logos/streamlit.png" width="20" height="20" style="vertical-align: middle;"> **Streamlit** | Interactive web applications | [Cookbook](../cookbook/streamlit.md) |
 
 ### Observability & Monitoring
 
 Track and monitor your Apache Hamilton dataflows:
 
 | Integration | Description | Documentation |
 |------------|-------------|---------------|
-| <img src="../_static/logos/datadog.png" width="20" height="20" style="vertical-align: middle;"> **Datadog** | Monitoring and analytics | [Lifecycle Hook](../reference/lifecycle-hooks/DDOGTracer.rst) |
+| <img src="../_static/logos/datadog.png" width="20" height="20" style="vertical-align: middle;"> **Datadog** | Monitoring and analytics | [Plugin API](../reference/plugins/DDOGTracer.rst) |
 | <img src="../_static/logos/opentelemetry.png" width="20" height="20" style="vertical-align: middle;"> **OpenTelemetry** | Observability framework | [Examples](https://github.com/apache/hamilton/tree/main/examples/opentelemetry) |
-| <img src="../_static/logos/openlineage.svg" width="20" height="20" style="vertical-align: middle;"> **OpenLineage** | Data lineage tracking | [Examples](https://github.com/apache/hamilton/tree/main/examples/openlineage) \| [Lifecycle Hook](../reference/lifecycle-hooks/OpenLineageAdapter.rst) |
+| <img src="../_static/logos/openlineage.svg" width="20" height="20" style="vertical-align: middle;"> **OpenLineage** | Data lineage tracking | [Examples](https://github.com/apache/hamilton/tree/main/examples/openlineage) \| [Plugin API](../reference/plugins/OpenLineageAdapter.rst) |
 | **Hamilton UI** | Built-in execution tracking | [UI Guide](../hamilton-ui/index.rst) |
 | **Experiment Manager** | Lightweight experiment tracking | [Examples](https://github.com/apache/hamilton/tree/main/examples/experiment_management) |
 
@@ -108,7 +108,7 @@ Create visualizations from your dataflows:
 |------------|-------------|---------------|
 | <img src="../_static/logos/plotly.png" width="20" height="20" style="vertical-align: middle;"> **Plotly** | Interactive plotting | [Examples](https://github.com/apache/hamilton/tree/main/examples/plotly) |
 | <img src="../_static/logos/matplotlib.png" width="20" height="20" style="vertical-align: middle;"> **Matplotlib** | Static plotting | [IO Adapters](../reference/io/available-data-adapters.rst) |
-| <img src="../_static/logos/rich.svg" width="20" height="20" style="vertical-align: middle;"> **Rich** | Terminal formatting and progress | [Lifecycle Hook](../reference/lifecycle-hooks/RichProgressBar.rst) |
+| <img src="../_static/logos/rich.svg" width="20" height="20" style="vertical-align: middle;"> **Rich** | Terminal formatting and progress | [Plugin API](../reference/plugins/RichProgressBar.rst) |
 
 ### Developer Tools
 
@@ -120,7 +120,7 @@ Improve your development workflow:
 | <img src="../_static/logos/vscode.png" width="20" height="20" style="vertical-align: middle;"> **VS Code** | Language server and extension | [VS Code Guide](../hamilton-vscode/index.rst) |
 | **Claude Code** | AI assistant plugin for Hamilton development | [Plugin Guide](claude-code-plugin.md) |
 | **MCP Server** | LLM tool server for interactive DAG development | [MCP Guide](mcp-server.md) |
-| <img src="../_static/logos/tqdm.png" width="20" height="20" style="vertical-align: middle;"> **tqdm** | Progress bars | [Lifecycle Hook](../reference/lifecycle-hooks/ProgressBar.rst) |
+| <img src="../_static/logos/tqdm.png" width="20" height="20" style="vertical-align: middle;"> **tqdm** | Progress bars | [Plugin API](../reference/plugins/ProgressBar.rst) |
 
 ### Cloud Providers & Infrastructure
 
@@ -145,7 +145,7 @@ Persist and cache your data:
 
 | Integration | Description | Documentation |
 |------------|-------------|---------------|
-| <img src="../_static/logos/slack.svg" width="20" height="20" style="vertical-align: middle;"> **Slack** | Notifications and integrations | [Examples](https://github.com/apache/hamilton/tree/main/examples/slack) \| [Lifecycle Hook](../reference/lifecycle-hooks/SlackNotifierHook.rst) |
+| <img src="../_static/logos/slack.svg" width="20" height="20" style="vertical-align: middle;"> **Slack** | Notifications and integrations | [Examples](https://github.com/apache/hamilton/tree/main/examples/slack) \| [Plugin API](../reference/plugins/SlackNotifier.rst) |
 | <img src="../_static/logos/geopandas.png" width="20" height="20" style="vertical-align: middle;"> **GeoPandas** | Geospatial data analysis | [Type extension](https://github.com/apache/hamilton/blob/main/hamilton/plugins/geopandas_extensions.py) for GeoDataFrame support |
 | <img src="../_static/logos/yaml.svg" width="20" height="20" style="vertical-align: middle;"> **YAML** | Configuration management | [IO Adapters](../reference/io/available-data-adapters.rst) |
 | **Neo4j** | Knowledge graph RAG | [Examples](https://github.com/apache/hamilton/tree/main/examples/LLM_Workflows/neo4j_graph_rag) |

docs/index.md

@@ -35,7 +35,8 @@ concepts/index
 how-tos/index
 hamilton-ui/index
 hamilton-vscode/index
-integrations/index
+plugins/index
+cookbook/index
 code-comparisons/index
 ```
 
@@ -73,6 +74,7 @@ reference/drivers/index
 reference/caching/index
 reference/graph-adapters/index
 reference/lifecycle-hooks/index
+reference/plugins/index
 reference/result-builders/index
 reference/io/index
 reference/dataflows/index

docs/integrations/index.rst → docs/plugins/index.rst

@@ -1,14 +1,29 @@
-============
-Integrations
-============
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
 
-This section showcases how Apache Hamilton integrates with popular frameworks.
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+Plugins
+-------
+
+Apache Hamilton provides plugins for working with popular data libraries,
+execution frameworks, and developer tools.
 
 .. toctree::
     dlt/index
-    fastapi
     ibis/index
-    streamlit
     dbt
     MLFlow <https://github.com/apache/hamilton/tree/main/examples/mlflow>
     Airflow <https://github.com/apache/hamilton/tree/main/examples/airflow>