Skip to content

fix(pipeline): align actions, repair filters, and classify no-data replies - #158

Closed
magnus188 wants to merge 3 commits into
drumandbytes:mainfrom
magnus188:pipeilne-fixes
Closed

magnus188 wants to merge 3 commits into
drumandbytes:mainfrom
magnus188:pipeilne-fixes

Conversation

@magnus188

@magnus188 magnus188 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

This PR fixes inconsistencies in the Pipeline page and makes it easier to find and act on broker responses.

I created it after seeing 18 pending tasks in the status cards while the Action Required section said “No pending tasks.” Changing the response-type dropdown also did not update the list. When a broker replied that it had no records of me, there was no search to help find the message or a clear category to distinguish that outcome from a rejected request.

The changes address those problems together:

  • Make action counts agree with the list. Pipeline and Action Needed now use the same source for pending tasks, detected forms without a task, and responses needing review. Forms already represented by a task are counted once. Database read failures show an error instead of an empty “all clear” state.
  • Make the filters work and give status cards a useful action. The response dropdown now submits its selected type correctly. Clicking a status card filters the results below, consistent with the existing behavior on Action Needed. Pipeline uses one results panel with Action Required, Responses, and Requests views; request counts and rows represent the latest request for each broker.
  • Add search for saved responses and outstanding work. Search covers broker details, email subjects and bodies, and task notes in the relevant view. It combines with response-type and review filters, and searches the full active profile before applying the 50-item page limit. Filters and pagination are stored in the URL, so refresh and browser back/forward restore the view. Response results include a text preview and a link to the full saved message.
  • Separate “No data held” from refusal or deletion. Clear replies such as “we have no records of you” now receive the no_data classification. This means the broker reports having no matching data; it does not imply that data was deleted. Conditional statements, conflicting replies, and requests for verification remain flagged for review. The category is reflected in the web UI, CLI summaries, and evidence export.

Existing saved replies benefit from the classification fix too. On the first history open after upgrading, a versioned transaction reclassifies qualifying stored messages without fetching mail. A confident latest reply can resolve the current request and superseded pending work, while newer requests, replies, and tasks are preserved. The upgrade also preserves completed/skipped work, manual pipeline progress, original messages, and existing task notes. It is safe to retry after a rollback and runs only once per database.

Most of the small changes across CLI commands route history opening through that shared upgrade helper, so the fix applies whether someone starts the web UI or uses the CLI. Existing export JSON field names are unchanged. The documentation explains the new category and Pipeline behavior.

Validation completed locally:

  • go test ./..., go test -race ./..., go vet ./..., build, and golangci-lint run ./... (v2.14.0, zero issues) passed. Both embedded broker lists also passed validation.
  • Regression tests cover the 18-review/zero-task mismatch, form deduplication, profile isolation, Unicode and body search beyond the first page, filter restoration, database errors, ambiguous and multilingual replies, and migration rollback, concurrency, and timestamp handling.
  • Chromium/Playwright checks passed at desktop (1440 × 1100) and mobile (390 × 844) sizes: status cards, response filters, search, pagination, back/refresh, opening a full response, and keyboard navigation. No browser errors or horizontal overflow were observed. These checks used synthetic data; live email providers and other browsers were not exercised.

@JustMaris JustMaris left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The mismatch between the status cards and the action list was a real bug, and the regression tests are thorough. Not merging for now. Requests before it can go in:

  1. Name. no_data reads like "data missing". It lands in the export JSON (response_type, pipeline_status), and docs/stability.md makes export values add-only in 1.x, so we can't rename it later without 2.0. Could we use no_data_held, matching the "No data held" label? Please also keep it separate from success/confirmed in the evidence export.
  2. Title/scope. This adds a category, search and a data migration, so the squash title should be feat(pipeline): ..., not fix. If it's easy, splitting the classifier and migration from the counts/filters/search UI would make both easier to review.
  3. Migration. MigrateNoData opens a write transaction and inserts into data_migrations on every openHistoryStore, so even eraser status takes the writer lock. Please do a read-only SELECT for the marker first and only take the write path when it's missing.
  4. Auto-resolve. The migration also sets requests to the new status and completes pending tasks from a regex verdict, which is hard to undo. Can it only reclassify (and leave resolution to the existing reclassify action), or at least be limited to confidence >= 0.7 && !needs_review with a test for the multilingual/conditional cases?
  5. Classifier. subjectRejectionPatterns lost not found, no record and unable to locate|find. That's intended, but please note in docs/broker-replies.md how "no data" replies should now be triaged.
  6. Nits. Redundant if z.Err() == io.EOF { break }; break in responsePreview; lowered[:at] is a byte offset into the lowercased string and can drift from runes for some Unicode.

CI is gated on maintainer approval and will run once these changes are pushed.

@JustMaris

Copy link
Copy Markdown
Member

Superseded by #165, which carries these commits forward with the review asks applied. Thanks for the work on this, Magnus.

JustMaris added a commit that referenced this pull request Oct 17, 2026
Picks up magnus188's #158 and applies the 2026-10-13 review asks: no_data_held rename, read-only migration marker, reclassify-only migration, responsePreview nits, triage docs, and direct tests for the pipeline queries and no-data resolution. Commits keep magnus188's authorship.
@JustMaris JustMaris closed this Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants