Haystack instrumentation part1 by srinjoy356 · Pull Request #373 · open-telemetry/opentelemetry-python-genai

srinjoy356 · 2026-08-10T05:58:08Z

Description

This PR adds core OpenTelemetry instrumentation for Haystack 3.0.0.

Context: To ease the maintainer review burden, we have decided to split the original implementation (which included Agent, Tool, Retrieval, and Workflow spans) into two parts. This PR contains Part 1, which establishes the core instrumentation infrastructure and basic Pipeline tracing.

Once this core setup is reviewed and merged, we will open a follow-up PR to introduce the remaining components (Agent, Tool, Retrieval, and Workflow) that were stripped from this branch.

Related to #12279

Type of change

New feature (non-breaking change which adds functionality)

How has this been tested?

Verified locally that the core Pipeline execution correctly emits spans matching the GenAI semantic conventions, including capturing messages and model parameters.

Ran uv run tox -e py312-test-instrumentation-genai-haystack-oldest locally
Ran uv run tox -e py312-test-instrumentation-genai-haystack-latest locally
Verified VCR cassettes matched the expected network and span output for Pipeline traces

Checklist

See CONTRIBUTING.md
for the style guide, changelog guidance, and more.

Followed the style guidelines of this project
Changelog updated if the change requires an entry
Unit tests added
Documentation updated

linux-foundation-easycla · 2026-08-10T05:58:15Z

The committers listed above are authorized under a signed CLA.

✅ login: srinjoy356 / name: srinjoy356 (10ed7fb)

opentelemetry-pr-dashboard · 2026-08-10T05:58:49Z

Pull request dashboard status

Waiting on the author · refreshed 2026-08-13 21:02 UTC

Two things need attention:

Required checks are failing — investigate the failures.
4 review items — respond to each (e.g. link a commit, explain why not, ask a follow-up):
- Inline threads: 1, 2, 3, 4

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Copilot

Pull request overview

This PR introduces a new opentelemetry-instrumentation-genai-haystack package to the monorepo, wiring it into the workspace and tox matrix, and adding initial tests + VCR cassettes for Haystack 3.0.0 generator and embedder components.

Changes:

Add a new Haystack GenAI instrumentation package (instrumentor, patching logic, README, examples, licensing, towncrier config).
Add unit tests + conformance scaffolding and VCR cassettes for chat inference and embeddings.
Wire the new package into uv.lock and tox.ini, and relax Sphinx dependency pinning in dev/docs requirements.

Reviewed changes

Copilot reviewed 35 out of 39 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
uv.lock	Adds the new Haystack instrumentation package + haystack-ai dependency into the workspace lock.
tox.ini	Adds Haystack test/lint/conformance environments and includes it in typecheck deps.
docs-requirements.txt	Changes Sphinx-related dependency constraints for docs builds.
dev-requirements.txt	Changes Sphinx-related dependency constraints for dev tooling/lint.
.gitignore	Ignores local `.env*` files.
.gitattributes	Enforces LF endings for `*.sh`.
instrumentation/opentelemetry-instrumentation-genai-haystack/pyproject.toml	Defines the new package metadata, deps, extras, and entry point.
instrumentation/opentelemetry-instrumentation-genai-haystack/src/opentelemetry/instrumentation/genai/haystack/init.py	Defines `HaystackInstrumentor` and component registration wrapping.
instrumentation/opentelemetry-instrumentation-genai-haystack/src/opentelemetry/instrumentation/genai/haystack/patch.py	Implements component `run`/`run_async` wrappers building util-genai invocations.
instrumentation/opentelemetry-instrumentation-genai-haystack/src/opentelemetry/instrumentation/genai/haystack/component_types.py	Classifies components into generator/embedder/unknown based on heuristics.
instrumentation/opentelemetry-instrumentation-genai-haystack/src/opentelemetry/instrumentation/genai/haystack/message_utils.py	Converts Haystack message/tool shapes into util-genai message models.
instrumentation/opentelemetry-instrumentation-genai-haystack/src/opentelemetry/instrumentation/genai/haystack/provider.py	Infers `gen_ai.provider.name` from component class names.
instrumentation/opentelemetry-instrumentation-genai-haystack/src/opentelemetry/instrumentation/genai/haystack/package.py	Declares `haystack-ai >= 3.0.0` as the instrumented dependency.
instrumentation/opentelemetry-instrumentation-genai-haystack/src/opentelemetry/instrumentation/genai/haystack/version.py	Sets the package dev version.
instrumentation/opentelemetry-instrumentation-genai-haystack/README.rst	Documents installation, supported instrumentation, limitations, and content capture/hook config.
instrumentation/opentelemetry-instrumentation-genai-haystack/LICENSE	Adds the Apache 2.0 license file for the new package.
instrumentation/opentelemetry-instrumentation-genai-haystack/examples/manual/main.py	Adds a manual usage example.
instrumentation/opentelemetry-instrumentation-genai-haystack/.changelog/318.added	Adds a towncrier fragment describing the package addition.
instrumentation/opentelemetry-instrumentation-genai-haystack/.changelog/.gitignore	Keeps the changelog dir tracked.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/conftest.py	Adds shared fixtures, VCR config, and instrumentation helpers.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/test_instrumentor.py	Verifies entry point loading and wrap/unwrap behavior.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/test_utils.py	Adds shared assertion helpers for span attributes and message parsing.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/test_inference.py	Tests chat generator sync/async paths and error behavior (plus a skipped tool-calling test).
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/test_embedding.py	Tests document embedding span attributes.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/test_conformance.py	Adds conformance test runner (currently references missing scenarios).
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/requirements.oldest.txt	Oldest-factor test requirements placeholder (no pins).
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/requirements.latest.txt	Latest-factor test requirements including editable installs.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/init.py	Marks test package.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/conformance/init.py	Marks conformance scenario package.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/conformance/_known_gaps.py	Declares reusable conformance expected violations.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/conformance/inference.py	Adds chat inference conformance scenario.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/conformance/embedding.py	Adds embedding conformance scenario.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/cassettes/test_chat_generator_sync.yaml	VCR cassette for sync chat generator test.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/cassettes/test_chat_generator_async.yaml	VCR cassette for async chat generator test.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/cassettes/test_chat_generator_no_content_capture.yaml	VCR cassette for no-content-capture chat test.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/cassettes/test_chat_generator_error.yaml	VCR cassette for invalid API key error test.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/cassettes/test_document_embedder.yaml	VCR cassette for embedding test.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/cassettes/inference_conformance.yaml	VCR cassette for inference conformance scenario.
instrumentation/opentelemetry-instrumentation-genai-haystack/tests/cassettes/embedding_conformance.yaml	VCR cassette for embedding conformance scenario.

Suppressed comments (1)

instrumentation/opentelemetry-instrumentation-genai-haystack/README.rst:61

The README repeats "embedder" twice again in the limitations section ("generator, embedder, or embedder"), which should be corrected for clarity.

- Only components classified as a generator, embedder, or embedder are wrapped -- there's no ``opentelemetry-util-genai`` invocation
  type for a generic pipeline step (prompt builders, routers, converters,

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+from .conformance.embedding import EmbeddingScenario  # noqa: E402
+from .conformance.inference import InferenceScenario  # noqa: E402
+from .conformance.invoke_agent import InvokeAgentScenario  # noqa: E402
+from .conformance.invoke_workflow import WorkflowScenario  # noqa: E402
+from .conformance.retrieval import RetrievalScenario  # noqa: E402
+from .conformance.tool_calling import ToolCallingScenario  # noqa: E402
+
+
+@pytest.mark.parametrize(
+    "scenario",
+    [
+        InferenceScenario(),
+        EmbeddingScenario(),
+        RetrievalScenario(),
+        ToolCallingScenario(),
+        WorkflowScenario(),
+        InvokeAgentScenario(),
+    ],
+    ids=lambda s: type(s).__name__,
+)


+- ``Pipeline.run`` / ``Pipeline.run_async`` — one ``invoke_workflow`` span per
+  pipeline execution.
+- Components classified as a generator (``chat`` / ``text_completion``), an
+  embedder (``embeddings``), a retriever/ranker (``retrieval``), or an
+  ``Agent`` (``invoke_agent``) — one span per component ``run`` /
+  ``run_async`` call, classified by class name and ``run`` method type
+  hints (Haystack has no static component-kind marker). Components that
+  don't fall into one of these (prompt builders, routers, converters, ...)
+  are not wrapped — there is no corresponding ``opentelemetry-util-genai``
+  invocation type for a generic pipeline step.
+- ``haystack.tools.Tool.invoke`` / ``invoke_async`` — one ``execute_tool``
+  span per tool call.


+Builds ``opentelemetry-util-genai`` invocations around:
+
+- ``haystack.Pipeline.run`` / ``run_async`` -> ``WorkflowInvocation``
+- classified component ``run`` / ``run_async`` methods -> ``InferenceInvocation``
+  (generators), ``EmbeddingInvocation`` (embedders), ``RetrievalInvocation``
+  (retrievers/rankers), or ``AgentInvocation`` (``Agent``)
+- ``haystack.tools.tool.Tool.invoke`` / ``invoke_async`` -> ``ToolInvocation``
+


+What gets instrumented
+***********************
+
+- Components classified as a generator, embedder, or embedder -- one span per component ``run`` / ``run_async`` call.


@@ -0,0 +1 @@
+Add ``opentelemetry-instrumentation-genai-haystack``, migrated from ``openinference-instrumentation-haystack``, with support for pipeline, chat/embedding/retrieval, agent, and tool invocations built on ``opentelemetry-util-genai``.


+# 2. Instrument Haystack
+# Set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=True to capture message content
+os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "True"
+HaystackInstrumentor().instrument()


+@pytest.mark.skip(reason="Missing VCR cassette from upstream PR")
+@pytest.mark.vcr
+def test_tool_calling_captures_tool_call_on_output_message(


 pyright==1.1.411
-sphinx==9.1.0
-sphinx-rtd-theme==3.1.0
-sphinx-autodoc-typehints==3.12.1
+sphinx<9.0.0
+sphinx-rtd-theme
+sphinx-autodoc-typehints


+sphinx<9.0.0
+sphinx-rtd-theme
+sphinx-autodoc-typehints


+Uploading prompts and completions
+***********************************
+
+Instead of recording message content inline, prompts and completions can be uploaded to external
+storage via a completion hook. To enable the built-in upload hook, set:
+
+- ``OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload``
+- ``OTEL_INSTRUMENTATION_GENAI_UPLOAD_BASE_PATH`` to an ``fsspec``-compatible URI/path
+  (e.g. ``/path/to/prompts`` or ``gs://my_bucket``), and install the ``upload`` extra
+  (``pip install opentelemetry-util-genai[upload]``).
+
+A custom ``CompletionHook`` can also be passed programmatically, taking precedence over the
+environment variable::
+
+    HaystackInstrumentor().instrument(completion_hook=my_hook)
+


lmolkova · 2026-08-11T20:32:17Z

+            content_capture="SPAN_ONLY",
+        ):
+            with vcr.use_cassette("embedding_conformance.yaml"):
+                OpenAIDocumentEmbedder(model="text-embedding-3-small").run(


does it happen through openai clients? assuming openai is also instrumented, would we end up with duplicate spans for inference and embeddings?

I just added some guidance on what to instrument #375

could you please check if we need to cover inference and embeddings for haystack according to this ? thanks!

does it happen through openai clients? assuming openai is also instrumented, would we end up with duplicate spans for inference and embeddings?

I just added some guidance on what to instrument #375

could you please check if we need to cover inference and embeddings for haystack according to this ? thanks!

I am doing it. it is taking a bit longer but I am on it.

Copilot AI lite review requested due to automatic review settings August 10, 2026 05:58

srinjoy356 requested a review from a team as a code owner August 10, 2026 05:58

github-project-automation Bot added this to GenAI Semantic Conventions and Instrumentation libraries Aug 10, 2026

Copilot started reviewing on behalf of srinjoy356 August 10, 2026 05:58 View session

opentelemetry-pr-dashboard Bot mentioned this pull request Aug 10, 2026

Pull Request Dashboard #171

Open

Copilot AI reviewed Aug 10, 2026

View reviewed changes

Add core OpenTelemetry instrumentation for Haystack 3.0

10ed7fb

srinjoy356 force-pushed the haystack-instrumentation-part1 branch from 79c2963 to 10ed7fb Compare August 10, 2026 06:05

srinjoy356 mentioned this pull request Aug 10, 2026

feat(haystack): migrate openinference-instrumentation-haystack #318

Open

5 tasks

lmolkova reviewed Aug 11, 2026

View reviewed changes

Navigation Menu

Uh oh!

Haystack instrumentation part1 - #373

Haystack instrumentation part1#373
srinjoy356 wants to merge 1 commit into
open-telemetry:mainfrom
srinjoy356:haystack-instrumentation-part1

srinjoy356 commented Aug 10, 2026

Uh oh!

linux-foundation-easycla Bot commented Aug 10, 2026 •

edited

Loading

Uh oh!

opentelemetry-pr-dashboard Bot commented Aug 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

lmolkova Aug 11, 2026

Uh oh!

srinjoy356 Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1 @@
		Add ``opentelemetry-instrumentation-genai-haystack``, migrated from ``openinference-instrumentation-haystack``, with support for pipeline, chat/embedding/retrieval, agent, and tool invocations built on ``opentelemetry-util-genai``.

Uh oh!

Conversation

srinjoy356 commented Aug 10, 2026

Description

Type of change

How has this been tested?

Checklist

Uh oh!

linux-foundation-easycla Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opentelemetry-pr-dashboard Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

lmolkova Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

srinjoy356 Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

linux-foundation-easycla Bot commented Aug 10, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Aug 10, 2026 •

edited

Loading