Skip to content

docs: add Factory incident resolution recipe - #899

Open
warp-agent-staging[bot] wants to merge 6 commits into
factory/factory-use-cases-structurefrom
factory/factory-use-case-incident-resolution
Open

warp-agent-staging[bot] wants to merge 6 commits into
factory/factory-use-cases-structurefrom
factory/factory-use-case-incident-resolution

Conversation

@warp-agent-staging

@warp-agent-staging warp-agent-staging Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

What this feature does

Configure incident investigation in an existing factory. Authenticated production events go to the foreman for read-only diagnosis, and a person decides whether to request a reviewed code fix. Paging, incident command, production changes, deployment, and merging remain under the team's existing controls.

Summary

Replaces the production-incident scaffold with an investigation-first recipe and a signed Sentry issue-webhook example.

Review status: Revisions committed and pushed at 3fba6b5eb5af0912a95fd534c0087e21186b4b3f. Validation below covers the revised article; live-provider checks remain unverified as listed below.

Changes

src/content/docs/factories/use-cases/production-incident-resolution.mdx

  • Uses existing-factory framing, concise prerequisites, and task-based headings.
  • Separates intake selection, automation reuse, read-only diagnostics, runbook preparation, webhook application, sender configuration, and disabled automation setup.
  • Scopes the Sentry example to new fatal issues in a production-only project. Distinguishes issue / created / data.issue from event_alert / triggered / data.event. Does not imply a project filter separates mixed environments.
  • Explains the Client Secret versus API-token distinction and the two-phase webhook UID setup. Secret values never go into definition files.
  • Stops initial automation after investigation. Code fixes need an explicit human request, regression tests, repository checks, and independent review.
  • Tests matching and nonmatching deliveries before enabling the automation, then checks authentication failures, duplicate deliveries, investigation-only behavior, and missing evidence.
  • Clarifies that webhook intake has no reply channel, different events about one incident are not automatically combined, and prompt restrictions do not enforce credential boundaries.
  • Removes repetitive best-practice and product-boundary sections by placing safeguards beside the relevant actions.

Content design plan

Audience and JTBD: An incident owner or service maintainer who wants agents to gather production evidence and prepare a code fix without granting production mutation authority.

Problem: Webhook and integration references describe configuration primitives but not the decisions and validation needed for an incident workflow.

Goals: Route a signed, scoped production event to an existing factory; investigate using read-only diagnostics; return evidence and uncertainty; require a separate human decision before a code fix.

Purpose and value: Connect webhook configuration, agent roles, and credential controls into a task-oriented recipe without duplicating the full reference documentation.

Content type: Procedural recipe with a configuration summary and a bounded example workflow.

Skill and template: draft_procedural / .agents/templates/procedural.md; reviewed against repository style and documentation-quality guidance.

High-impact scenarios: Covers Sentry issue-created events from a production-only project, investigation reports, filter tests, and human-requested fixes. Excludes paging, incident command, production mutation, deployment, rollback automation, and a universal payload format for every alert provider.

Source files consulted

  • warpdotdev/warp-server at 084e0e425af93139c6e95a78e2981dce55557c31: router/handlers/external/webhook_signature_schemes.go, router/handlers/external/warp_webhook.go, logic/factory_webhooks.go, logic/ai/ambient_agents/automations/provider_webhook.go, model/types/triggers/pattern.go, logic/factoryfile/resolution/webhook.go, and logic/factoryfile/projector/webhook.go. Checked signing-secret verification, acceptance versus dispatch, deduplication, dry-run matching, nested payload filters, managed-secret references, and generated UID application.
  • Default foreman and triage instructions in warpdotdev/warp-factory-examples at 84e7c952f1506c4bd2784d96fd40a6f4d6398cad. The default foreman can continue to implementation, so this automation explicitly requests investigation only.
  • Sentry webhook authentication, issue webhook payload, and issue-alert payload checked October 22, 2026.
  • Canonical factory webhook, automation, agent, definition, and infrastructure/security documentation.

Validation and rollout evidence

  • Factory MCP parser validation and the production app.warp.dev factory-files validation endpoint both accept the example tree with no diagnostics. A placeholder webhook UID was replaced with a sample UUID for syntax validation. State-dependent apply checks were not run.
  • Production schemas retrieved October 22, 2026: webhook.schema.json SHA256 b108204a53bf55c3b5e9d55ce55ed7761e54d6e59592327a6b740152f5169428; automation.schema.json SHA256 61c15fe6e32dd9e3f918739499b2f2720488609e510cd7d455a7016fd0427fce. Schema/parser acceptance is not end-to-end delivery evidence.
  • Targeted rollout inspection at the server revision above found all three custom-webhook admission gates enabled in checked-in production configuration. This is configuration evidence, not proof of deployment or live delivery. Unrelated broad-search matches were excluded from the verdict. No private flag names or raw rollout output are published.
  • Typecheck: 0 errors. Heading-variable tests: 5/5. Build, heading-TOC validation, homepage JSON-LD, compression contract, and whitespace checks pass. Style lint reports one glossary warning for the shipped Webhook UI label.
  • Rendered structure checked: eight setup steps, five test steps, and two code-fix steps. All 17 internal links and anchors resolve.
  • Six local fixture checks exercise the example's documented nested-equality conditions: matching production fatal issue, wrong project, lower severity, missing project, different action, and issue-alert payload shape. These are local example checks, not execution of Warp's Go matcher or a live provider flow.
  • The original October 7 stable-release assertion was removed: no release-date evidence was established for this recipe.

Unverified claims

  • No live Sentry integration was configured and no signed delivery was sent. Verify Client Secret setup, sender subscription, ingress URL, signature acceptance/rejection, stored-delivery filter testing, retry deduplication, and activation in a connected test factory.
  • No live diagnostic credentials or permission scopes were exercised. Confirm read-only access and inherited credential boundaries with the team's observability provider.
  • No full investigation, external notification, human-requested implementation, or independent-review handoff was exercised. Verify those with synthetic evidence before connecting production events.
  • The example depends on a production-only Sentry project. A mixed-environment setup needs a payload and explicit environment filter verified against that sender.

Documentation risk

Risk: engineering-review-required
Rationale: Documents signed webhook intake, payload filtering, production credential boundaries, and a multi-agent workflow.
Docs override: none

PR stack

Dependency: #895 targets main. This PR targets factory/factory-use-cases-structure; remaining recipe PRs are reviewed independently.

Human review focus

  • Confirm the production-only Sentry project scope and the issue-webhook versus issue-alert distinction.
  • Review the disabled-first setup and separate human request for implementation.
  • Validate live provider authentication and diagnostic permissions before publication.

Agent review conversation

Co-Authored-By: Oz oz-agent@warp.dev
Co-Authored-By: Warp agent@warp.dev

@cla-bot cla-bot Bot added the cla-signed label Oct 10, 2026
@vercel

vercel Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Oct 22, 2026 10:09pm UTC

Request Review

@warp-agent-staging warp-agent-staging Bot added the warpy-factory Opened by the Warp factory agents label Oct 10, 2026
@warp-agent-staging

Copy link
Copy Markdown
Contributor Author

This PR was generated with Warp.

Comment @warp-staging-factory on this PR to send it follow-up work.

View run View conversation View on Slack

Preserve the reviewed incident recipe and its investigation-first summary.

Co-Authored-By: Warp <agent@warp.dev>
@rachaelrenk
rachaelrenk marked this pull request as ready for review October 22, 2026 21:50
@warp-for-oss

warp-for-oss Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

@rachaelrenk

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

The independent agent completed its review for this commit.

Findings

  • src/content/docs/factories/use-cases/production-incident-resolution.mdx:93 — [SUGGESTION] The instruction to send signed synthetic deliveries provides no runnable sender for the matching and nonmatching payloads, and the linked Sentry authentication reference demonstrates verification rather than sending. Requested resolution: add or directly link a runnable signing-and-POST example that reads the Client Secret from an environment variable without printing it, sends the four fixtures, and preserves the disabled-first filter checks and subsequent enabled-run check.

Verdict

Approve with nits

@warp-for-oss warp-for-oss Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

This PR replaces the incident-resolution scaffold with a procedure for signed Sentry webhook intake, read-only triage, validation, and human-gated code fixes. The overall direction matches the stated investigation-first scope, and the security-sensitive prompt and credential boundaries are called out. One changed link sends factory readers to the legacy standalone secrets workflow, so that should be fixed before merge.

Concerns

  • src/content/docs/factories/use-cases/production-incident-resolution.mdx:39 links factory credential setup to /platform/secrets/, whose page tells Warp Factories users to follow /factories/secrets/ instead.
  • src/content/docs/factories/use-cases/production-incident-resolution.mdx:2 still frames the page as resolving production incidents, while the article now focuses on investigation and keeps production mitigation/deployment separate.

Verdict

Found: 0 critical, 1 important, 1 suggestions

Request changes

Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz


If your Sentry project mixes environments, use an event type that includes environment information and filter that field explicitly. The example's project filter alone can't distinguish environments within one project. Other senders, such as PagerDuty or Grafana, need their own authentication mode and payload filter.
2. Check whether an existing webhook and automation already handle those events. Reuse them where appropriate, or narrow overlapping triggers before adding another automation. One delivery can match multiple automations. See [automation matching rules](/factories/automations/#how-matching-works).
3. Configure the triage agent's [MCP servers](/platform/mcp/) or [managed secrets](/platform/secrets/) for read-only diagnostic access. Scope permissions at the observability provider, and remove production write credentials from any shared factory configuration the agents inherit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [IMPORTANT] This sends factory readers to the legacy standalone cloud-agent secrets page. Link this step to the current factory secrets workflow so they create and grant the credential on the right surface.

Suggested change
3. Configure the triage agent's [MCP servers](/platform/mcp/) or [managed secrets](/platform/secrets/) for read-only diagnostic access. Scope permissions at the observability provider, and remove production write credentials from any shared factory configuration the agents inherit.
3. Configure the triage agent's [MCP servers](/platform/mcp/) or [factory secrets](/factories/secrets/) for read-only diagnostic access. Scope permissions at the observability provider, and remove production write credentials from any shared factory configuration the agents inherit.

@@ -1,8 +1,119 @@
---
title: Resolving production incidents with your factory

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 [SUGGESTION] The title still promises incident resolution, but the article teaches investigation-first setup and leaves mitigation, deployment, and merging outside the automation. Rename it so the H1 and search result match the page scope.

Suggested change
title: Resolving production incidents with your factory
title: Investigating production incidents with your factory

This branch was successfully deployed

1 active deployment
Preview — 4e7eb50d Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed warpy-factory Opened by the Warp factory agents

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant