Skip to content

RPCN - Pipelines listing page speed + UX improvements - #2593

Open
SpicyPete wants to merge 22 commits into
masterfrom
rpcn/many-many-pipelines-listings
Open

RPCN - Pipelines listing page speed + UX improvements#2593
SpicyPete wants to merge 22 commits into
masterfrom
rpcn/many-many-pipelines-listings

Conversation

@SpicyPete

@SpicyPete SpicyPete commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Rebuild the Redpanda Connect pipelines list for large clusters

The pipeline list was built for a handful of pipelines. On clusters with hundreds it was slow to
render, offered no way to narrow down what you were looking for, and buried broken pipelines
wherever the server happened to return them. This reworks the page around finding one pipeline in
a long list, and fixes the cost of getting that list in the first place.

How it ships

The page's only mount used to be isFeatureFlagEnabled('enableRpcnTiles') && isEmbedded(), and
#2591 removed that flag as dead code — the flag has been off in LaunchDarkly, so this list has
never actually rendered for a user
. It now mounts on a capability instead:

  • Features.pipelinesApi — i.e. the backend serves redpanda.api.console.v1alpha1.PipelineService.
    Cloud gets the new list whether or not it's embedded, with no flag to flip.
  • Self-hosted reports that service unsupported (OSS defaults in endpoint_compatibility.go) and
    keeps the legacy tabs, including the Redpanda Connect install intro.
  • Checked ahead of the Kafka Connect spinner deliberately: the list handles a pending Kafka Connect
    probe itself, so waiting there would delay its own multi-page fetch behind an unrelated request —
    and a failed /console/endpoints leaves feature detection pending forever, so falling through to
    the legacy path beats parking the page on a spinner that never resolves.

Kafka Connect stays reachable: the new page renders the same TabKafkaConnect component behind its
own Kafka Connect tab when a cluster is configured, and the /connect-clusters/$clusterName routes
are untouched.

List page

  • Status tabs — All / Running / Stopped / Error, each with a live count. Transitional states ride
    with their destination (starting counts as running, stopping as stopped). Counts come from the
    status column's faceted row model, so each tab shows what selecting it would yield under the
    current search and filters.
  • Search by name or ID, debounced, matching case-insensitively against both.
  • Faceted filters for input, output, and tag, with per-option counts. Clear filters appears
    only when something is actually filtered — the status tabs are views, not filters, so they aren't
    swept up by it — and it tracks the input directly rather than lagging 200ms behind the debounce.
  • Sortable Pipeline and Status columns. Status sorts attention-first by default (errors and
    transitions above healthy pipelines, idle at the bottom), so a broken pipeline lands on page 1 of a
    large cluster instead of wherever the server put it.
  • Rows are clickable, with the same guard the registry DataTable uses: clicks on portaled
    content (open menus, the delete-confirm backdrop) and on interactive descendants don't navigate,
    and neither does a click that ends a text selection — the pipeline ID is select-all, so one click
    grabs the whole thing for copying. ⌘/Ctrl/Shift-click and middle-click are left to the browser, so
    "open in a new tab" still means that.
  • Connector badges collapse duplicates into redpanda ×2 rather than repeating the same logo
    across the column.
  • Progressive rendering. The list used to wait for every page before showing anything; it now
    renders as soon as the first page has rows and streams the rest in behind the table, with a
    distinct line for "still loading pages" vs "background refresh failed" — partial data and stale
    data read differently now.
  • Search, facets and page survive a trip to the Kafka Connect tab and back (keepMounted — Base UI
    panels unmount by default).
  • Dropped the Processors column and the old DataTableFilter stack from this page.

Accessibility

  • The status tabs own a panel. They sit outside the table (one table filtered per tab, not four
    panels), so without an aria-controls target a screen reader announced "tab, 1 of 4" with nowhere
    to move into. Each tab now points at the table region, which is labelled by the active tab.
  • Async status is announced. "Loading more pipelines" and "Couldn't refresh pipelines" appear
    without user action, so each has an always-mounted sr-only live region (polite for the drain,
    role="alert" for the failure) — the visible lines animate in and out, and a live region only
    announces changes made while it's already in the DOM.
  • Row click stays a pointer shortcut with no row tab stop: the name cell is already a real link, so a
    tab stop per row would just duplicate it.

Fetch and render cost

  • Page size 500 instead of MAX_PAGE_SIZE (which is 25) — 20× fewer sequential round trips to
    drain. The server does the same work per call at any page size: it lists everything and slices.
  • Deduplicate the drain by pipeline ID. The dataplane's keyset page token names the first ID of
    the next page; when that pipeline is deleted mid-drain, a server resolving the token by exact match
    restarts at page one and replays rows we already have.
  • Stop draining on any token the drain already requested. Keyset tokens only move forward, so a
    repeat means the server sent us backwards. The first cut only caught an immediate A → A repeat,
    which a A → B → A cycle walks straight past — the drain then alternates forever, adding a page to
    the query cache every round. Now checked against every prior page param, with a test for each shape.
  • Memoize the YAML parse per config text. The transform re-ran a full parse for every row on every
    drain step and poll tick; it's now O(new rows), with a bounded cache that evicts its oldest half.
  • Row identity keyed on pipeline ID (not row index) and autoResetPageIndex: false, so streaming
    pages don't yank you back to page 1 or repaint a shifted window of rows. Filter and sort changes
    still reset to page 1, and a shrinking row set is clamped before paint.
  • Cache the facet icon component per connector name, so logos in an open filter popover don't remount
    and flash on every poll; count the status tabs in a single pass; and memoize the per-row connector
    aggregation so cells don't re-derive it on every keystroke.

SpicyPete and others added 13 commits July 28, 2026 11:38
* Full-screen page mode for SQL and RPCN editors, console-owned layout

- Footer pins to the viewport bottom on short pages (CSS flex chain in
  standalone, measured min-height in embedded) and keeps centering to the
  content column; bottom padding 8px -> 16px.
- Topics and security-tab pages drop ListLayout's forced min-h-screen
  (min-h-0 override), removing large dead whitespace.
- Embedded Console cancels the Cloud UI host gutters with measured negative
  margins and owns its page gutter (px-12) — deploy-order-safe with cloud-ui
  removing its p-10 later.
- New expanded-page mode: data-page-expanded on <html> (utils/page-expanded)
  + useExpandedPageMode hook release every shell's horizontal constraints via
  global CSS while the page stays in document flow, footer below. The SQL
  studio's fixed-overlay fullscreen is replaced by this in-flow mode, and the
  RPCN pipeline editor gains the same toggle; both place the shared
  ExpandedPageToggle at the top-right of their work surface, clear of Save.
- /sql becomes a normal route; new breadcrumbOnlyHeader staticData flag keeps
  the app header breadcrumb-only for pages with their own title bar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Comment reduction pass

* some Pr feedback

* Code review and cleanup passes

* Small improvements from review

* More changes from code review

* More simplification

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
# Conflicts:
#	frontend/src/components/layout/header.tsx
#	frontend/src/components/pages/rp-connect/pipeline/index.tsx
@SpicyPete
SpicyPete requested review from a team and eblairmckee August 5, 2026 15:00
@SpicyPete SpicyPete self-assigned this Aug 5, 2026
@SpicyPete
SpicyPete requested review from Mateoc, datamali and yougotashovel and removed request for a team August 5, 2026 15:00
@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

🚨 Registry drift detected

App: frontend · Scope: diff vs origin/master · Files: 32

Count
⚠️ Outdated registry components 0
🛠 Locally-modified components 1
❓ Unknown to registry 0
🎨 Off-token palette colours 68
🔢 Ad-hoc utility classes 1
Components needing attention
Status Component Uses Detail
🛠 locally-modified data-table no tagged release matches installed bytes
🎨 Off-token colours (palette literals)

Use semantic tokens (primary, muted-foreground, border, …) instead of raw palette names.

Class Uses Files
indigo-400 14 1
indigo-500 9 1
indigo-300 7 1
indigo-alpha-200 7 1
blue-400 6 1
indigo-600 6 1
indigo-800 6 1
blue-900 5 1
green-500 5 1
indigo-100 5 1
orange-400 5 1
red-600 5 1
blue-500 4 1
blue-alpha-200 4 1
green-400 4 1
indigo-200 4 1
indigo-900 4 1
indigo-alpha-100 4 1
indigo-alpha-300 4 1
orange-500 4 1
red-400 4 1
red-500 4 1
red-alpha-200 4 1
blue-600 3 1
blue-800 3 1
green-300 3 1
green-600 3 1
orange-200 3 1
blue-100 2 1
blue-200 2 1
blue-300 2 1
blue-alpha-100 2 1
blue-alpha-300 2 1
blue-alpha-500 2 1
green-100 2 1
green-50 2 1
green-700 2 1
green-800 2 1
green-900 2 1
indigo-50 2 1
indigo-700 2 1
indigo-alpha-500 2 1
orange-100 2 1
orange-300 2 1
orange-800 2 1
orange-900 2 1
red-200 2 1
red-300 2 1
red-50 2 1
red-alpha-100 2 1
red-alpha-300 2 1
red-alpha-500 2 1
blue-alpha-400 1 1
green-200 1 1
green-alpha-100 1 1
green-alpha-200 1 1
indigo-alpha-400 1 1
orange-50 1 1
orange-600 1 1
orange-700 1 1
orange-950 1 1
purple-500 1 1
purple-600 1 1
red-100 1 1
red-700 1 1
red-800 1 1
red-950 1 1
red-alpha-400 1 1
🔢 Ad-hoc utility classes (arbitrary values)

These bypass the design tokens. Prefer a named scale entry or add a new token.

Class Category Uses Files
rounded-[4px] 🔢 radius 1 1

Generated by lookout audit-changes.

@SpicyPete

Copy link
Copy Markdown
Contributor Author

@claude review

@github-actions

This comment was marked as resolved.

@SpicyPete

Copy link
Copy Markdown
Contributor Author

@claude review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant