Skip to content

feat: add create-kafka-topic skill#157

Open
evanh wants to merge 6 commits into
mainfrom
evanh/feat/create-kafka-topic-skill
Open

feat: add create-kafka-topic skill#157
evanh wants to merge 6 commits into
mainfrom
evanh/feat/create-kafka-topic-skill

Conversation

@evanh

@evanh evanh commented Jun 22, 2026

Copy link
Copy Markdown
Member

Adds a create-kafka-topic skill that provisions a new Kafka topic across the four repos that must all agree for a Sentry topic to exist and deploy:

  • sentry-kafka-schemas — schema definition + CODEOWNERS entry
  • ops — default partitions, per-region overrides, all_topics.yaml, schemas requirement bump
  • sentryTopic enum + KAFKA_TOPIC_TO_CLUSTER, schemas dependency bump
  • getsentry — cell-silo topic→cluster mapping

It opens one PR per repo and reports the four PR links annotated with their merge-order dependencies (the schemas PR must merge + release before the ops/sentry version bumps).

The steps and file formats were reconciled against the real taskworker-ingest-push topic PRs (sentry-kafka-schemas#484, ops#20917, sentry#117135, getsentry#20495). Per how ops works, CI regenerates the materialized/k8s config (make materialize), so the skill only edits source-of-truth files.

Includes the required SPEC.md and registration in README.md, .claude/settings.json, and the claude-settings-audit allowlist.

🤖 Generated with Claude Code

Adds a skill that provisions a new Kafka topic across the four repos that
must agree for a Sentry topic to exist: sentry-kafka-schemas (schema +
CODEOWNERS), ops (partitions, per-region overrides, all_topics), sentry
(Topic enum + cluster mapping), and getsentry (cellsilo cluster mapping).
Opens one PR per repo and reports the four PR links with their merge-order
dependencies.

Reconciled against the taskworker-ingest-push topic PRs (sentry-kafka-schemas#484,
ops#20917, sentry#117135, getsentry#20495). CI handles ops materialized config,
so the skill only edits source-of-truth files.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread skills/create-kafka-topic/SKILL.md
Comment thread skills/create-kafka-topic/SKILL.md Outdated
Testing the skill surfaced two issues:

- getsentry no longer needs a per-topic change. It loads KAFKA_TOPIC_TO_CLUSTER
  at runtime from the topicctl-generated YAML mounted by ops (getsentry#20512,
  #20661), so the cellsilo.py edit is obsolete. The skill is now 3 repos, not 4.
- Make the "which regions to enable" prompt an explicit blocking gate so it is
  never inferred from a reference topic.

Updates SKILL.md, SPEC.md, and the README description accordingly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread skills/create-kafka-topic/SKILL.md
Comment thread skills/create-kafka-topic/SKILL.md Outdated
The team used for the sentry-kafka-schemas CODEOWNERS entry is now also added
as a reviewer (--reviewer) on all three PRs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f8a349d. Configure here.

Comment thread skills/create-kafka-topic/SKILL.md Outdated
evanh and others added 2 commits June 22, 2026 15:40
New ops regional override files must have a counterpart in the
cookiecutter-region template (disabled: true) so newly-created regions get the
topic too; Warden's cookiecutter-region-backport check flags the omission.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Instead of deferring the sentry-kafka-schemas pin bump, the ops and sentry PRs
now bump to the anticipated next version (latest released patch + 1) so the
required change is visible in the diff. The PR body flags the version as
anticipated, and for sentry notes that uv.lock's resolved entry must be
regenerated with `uv lock` once the release publishes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@linear-code

linear-code Bot commented Jun 23, 2026

Copy link
Copy Markdown

STREAM-1224

- Enabled regions outside the sibling-derived candidate set are now always
  given an override file (file set = candidate ∪ user-enabled regions).
- Use the full shared_config/kafka/topics/regional_overrides/... path for
  per-region override files so they're staged by git add.
- Clarify the Topic enum member is SCREAMING_SNAKE_CASE while the value is the
  kebab-case topic name, with a concrete example.
- Note the KAFKA_TOPIC_TO_CLUSTER value is "default" today, but to match a
  sibling family if that ever differs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@evanh

evanh commented Jun 23, 2026

Copy link
Copy Markdown
Member Author

Addressed the review comments in 6daf901:

  • Enabled regions omit override files (Bugbot) — fixed. The "file set" is now explicitly the sibling-derived candidate set ∪ every region the user enabled, so an enabled region always gets an override file even if the sibling topic didn't define it.
  • Ambiguous / missing regional-override path (Bugbot + Sentry) — fixed. Step 3.2 now uses the full shared_config/kafka/topics/regional_overrides/<region>/<topic_name>.yaml path, matching git add shared_config/kafka/topics/.
  • Enum template conflates names (Bugbot) — fixed. Clarified the Topic member is SCREAMING_SNAKE_CASE (hyphens→underscores, uppercased) while the value is the kebab-case name, with a concrete TASKWORKER_LAUNCHPAD_PUSH = "taskworker-launchpad-push" example.
  • Hardcoded default cluster mapping (Bugbot) — not a current bug: all 97 KAFKA_TOPIC_TO_CLUSTER entries map to "default". Softened the wording anyway to say match a sibling's value if a family ever uses a non-default cluster.

@untitaker untitaker left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think the skill should ask a few clarifying questions about whether this really needs a schema change. if this is just another taskbroker or outcomes topic then a few steps can be skipped

@evanh

evanh commented Jun 23, 2026

Copy link
Copy Markdown
Member Author

i think the skill should ask a few clarifying questions about whether this really needs a schema change. if this is just another taskbroker or outcomes topic then a few steps can be skipped

Which steps can be skipped?

@untitaker

untitaker commented Jun 23, 2026

Copy link
Copy Markdown
Member

see getsentry/sentry-kafka-schemas#490 (review) -- this is not how we create taskworker or outcomes topics. we already have a schema definition for those, and sentry-kafka-schemas should not contain infra-specific topics that are not present in self-hosted or elsewhere

EDIT: after discussing in slack we did create taskworker topics like that. I do think we should still add a caveat about outcomes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants