Refactor LocalExecutorService and enhance testing coverage - #967
Merged
Merged
Conversation
Pin current behaviour of LocalExecutorService before refactoring: streaming/legacy paths, pre-count, three cancel routes, NULL handling and column projection. Backed by a network-free FakePluginService so the suite runs without a real database. Two assertions intentionally capture known quirks (JDBC-abort cancel reporting count=0, and STOPPED state after all rows committed) to be fixed in a later phase.
Extract the 576-line class into focused units without behaviour change: - SyncStrategy + SyncContext with StreamingSyncStrategy / LegacySyncStrategy - TaskRegistry (object) for the running-task map and cancel executor, kept a singleton so stop() still works across service instances - TaskHandle / TaskCancelledException and ValueCodec for value/SQL codecs start() now only resolves inputs, runs pre-count and picks a strategy; stop() delegates to TaskRegistry. Characterization suite stays green and spotbugs reports no findings.
Unify cancellation behind a single CancellationToken and a runCancelable writer wrapper, collapsing the three former detection points (in-loop throw, post-loop flag, driver-abort exception) into one path that always surfaces TaskCancelledException with the committed count. Previously a stop that aborted the JDBC fetch reported count=0 because it read the stale rowsAtStop counter (updated only every 1000 rows); it now reports the rows actually committed to the target. Row counts are unified on writtenCount() across success and cancellation. Also precompute source-column indexes at onSchema instead of a per-row map lookup in the streaming path.
Add a per-executor timeout (seconds) to the local executor. A watchdog cancels the running sync once the limit is exceeded, surfacing it as RunState.TIMEOUT with the committed row count via the shared CancellationToken (USER stop vs TIMEOUT). The timeout defaults to 0 (disabled) in LocalExecutor's config, so existing syncs keep running to completion. DataSetServiceImpl now reads the value from each executor's effective config; executors without the field (Seatunnel) still fall back to 600s. TIMEOUT is treated as a terminal state like STOPPED so it is not overwritten to FAILURE.
Every plugin is reached through the datacap JDBC-conversion driver (type() defaults to JDBC, supportsStreaming()==true) and the sync target is always the JDBC dataset store, so the both-ends-non-streaming branch that hand-built INSERT strings was unreachable in practice. Remove that branch along with ValueCodec.formatSqlLiteral / escapeSqlString (manual quoting was an injection and type-coercion hazard, and carried a stray NUL byte in a comment). The legacy path now requires a batch-write-capable target and fails fast with a clear message otherwise; the streaming path is unaffected.
Drive LocalExecutorService against a real PostgreSQL container (source and
target) through a real-JDBC PluginService, verifying an actual source->target
sync: streaming copy with column projection/rename, NULL and numeric handling,
pre-count totals, and a real Statement.cancel() stop.
Pin the docker-java API version (1.41) in-code so Testcontainers negotiates
with newer Docker Desktop engines, and skip gracefully when Docker is absent.
Add spotbugs suppressions on the test helpers for false positives (Kotlin
use{} obligations, list casts, test data exposure).
Extract SeaTunnelCommander construction from SeatunnelExecutorService.start() into a package-visible buildCommander(request) (no behaviour change) and add tests asserting the resulting command line for the Spark and SeaTunnel engines. This locks the ExecutorRequest-to-command mapping so the upcoming ExecutorRequest restructure can be verified against a stable expected command.
Collapse the executor-specific typed fields (executorHome, startScript, runWay, runMode, runEngine, fetchSize, batchSize, preCount) into a single options string map, leaving only executor-agnostic fields on ExecutorRequest (taskName, userName, input, output, workHome, timeout, transform, progressListener, pluginManager). Reduce the six telescoping constructors to the primary plus one workHome convenience constructor. Each executor now reads what it needs from options: the local executor takes fetchSize/batchSize/preCount, seatunnel takes home/startScript/way/mode/engine. Callers pass the executor's effective config map straight through as options. Adding a new executor no longer requires touching this shared class. Behaviour is preserved: local characterization + PostgreSQL E2E tests stay green, and the seatunnel command-mapping test asserts the same command line.
Introduce ant-design-vue 4 and wire it globally next to the existing view-shadcn-ui so components can be migrated page by page without breaking untouched pages. Add composables for cross-cutting concerns: - useTheme: single-source dark/light switch (persisted, toggles the html `dark` class and drives antd's dark/default algorithm) - useAntdLocale: maps the vue-i18n locale to antd's built-in locale App.vue now wraps the router view in a-config-provider (theme + locale). No reset.css yet to avoid affecting not-yet-migrated pages. Pin pnpm onlyBuiltDependencies to keep pnpm 10 CI from failing on ignored builds.
Replace ShadcnSelect with a-select (grouped options) and drop the view-shadcn-ui setLocale call; antd's built-in strings now follow the vue-i18n locale through a-config-provider. loadLocale already updates the global locale, so switching language reactively updates both app and antd text.
Replace ShadcnException with a-result (status 404/403/500, title/sub-title, extra + icon slots) across NotFound, NotAuthorized and NotNetwork. Swap ShadcnButton to a-button and its `to` navigation to router pushes on click; NotNetwork's custom icon now uses antd's WarningOutlined.
Rebuild the sign-in form on antd: a-form (:model + @finish/@finishFailed) with a-form-item rules, a-input / a-input-password, submit via html-type=submit, and a-card / a-avatar / a-divider / a-space / a-spin. Global $Message calls become antd message; validation-error reporting adapts to antd's errorFields shape. Field rules and behaviour are unchanged.
Mirror the sign-in migration for sign-up: antd a-form with a-input / a-input-password, submit via html-type=submit, a-card / a-avatar / a-divider / a-space / a-spin, and antd message. The confirm-password custom validator now uses antd's (rule, value) signature; rules and behaviour unchanged.
First list-page migration proving the hard patterns: - ShadcnTable -> a-table with #bodyCell slot (role tags, action buttons); useHeaders now emits antd columns (title/dataIndex/key) - ShadcnPagination -> a-pagination (v-model:current, show-size-changer, change / show-size-change events) - ShadcnSpin -> a-spin wrapper (:spinning); a-card / a-tag / a-tooltip / a-button / a-space Also convert the page from Options API to <script setup> composition style (refs + functions + onMounted). Icons keep ShadcnIcon for now (migrated in a later dedicated pass). UserRole / UserInfo dialogs untouched (coexistence).
Convert NotFound / NotAuthorized / NotNetwork from Options API to <script setup> (defineOptions for name), completing their composition-style refactor alongside the earlier ant-design-vue swap. Templates unchanged.
Convert AuthSignin / AuthSignup from Options API to <script setup> (refs + functions, useI18n for translations, defineOptions for name), finishing the composition-style refactor of the already-migrated pages. Templates and behaviour unchanged.
…n API Proves the modal pattern: ShadcnModal -> a-modal (v-model:open, footer=null with the form's own submit button); ShadcnCheckboxGroup/Checkbox -> a-checkbox-group / a-checkbox; a-spin wrapper. Convert to <script setup> (defineProps/withDefaults, defineEmits, computed visible get/set, useI18n, antd message). Behaviour and validation unchanged.
…position Migrate the user create/edit flow: ShadcnDrawer -> a-drawer (UserInfo) and the inner ShadcnForm -> a-form with a-input / a-input-password + submit (UserForm). Both converted to <script setup> (defineProps/withDefaults, defineEmits, computed visible, useI18n, antd message); the confirm-password validator uses antd's (rule, value) signature. Behaviour unchanged.
Migrate the whole system/role module: RoleHome list (a-table #bodyCell + a-pagination), RoleInfo dialog (a-modal + a-form with a-input / a-textarea), and RoleMenu assign-menu dialog (a-modal + a-tree checkable with field-names mapping value/label to key/title). RoleUtils emits antd columns. All pages converted to <script setup> composition with useI18n and antd message.
Migrate system/menu: MenuHome list (adds a-switch in the active column via #bodyCell) and MenuInfo drawer form using a-row/a-col grid, a-input, a-input-number, a-select (type/parent/redirect), a-switch and a-textarea. MenuUtils emits antd columns; both pages are <script setup> with useI18n and antd message. Grid spans doubled for antd's 24-column system.
…osition Migrate ConfigureHome (custom two-pane layout kept): a-card / a-spin, a-input / a-input-number / a-switch / a-input-password for the dynamic field editor, a-button. Convert to <script setup> composition with useI18n and antd message; behaviour unchanged.
… API Migrate system/function: FunctionHome list (a-table with a-avatar-group in the plugin column, a-tag, a-pagination), FunctionInfo drawer form (a-select multiple via :options, a-textarea), and FunctionImport drawer with a-tabs (content vs import-from-url), a-alert warnings and danger button. FunctionUtils emits antd columns. All <script setup> composition with useI18n / antd message.
antd a-pagination renders show-total content before the pager, so the raw total count showed up as a leading number. Remove :show-total from the user, role and menu list pages; the page-size selector already conveys volume.
antd Select dropdown items (page-size changer, all a-select) sat flush against each other, so the hovered and selected options touched. Add a small margin + rounded corners to .ant-select-item globally for clearer separation.
Migrate admin/snippet: SnippetHome list adds the last common components - a-dropdown (#overlay a-menu/a-menu-item) for the row action menu and router-link (hash-router aware, target=_blank) for the quote link, plus a-avatar and a-pagination. SnippetInfo drawer form (a-input / a-textarea / AceEditor) and SnippetDelete confirm dialog (a-modal + a-alert list + name match validator). All <script setup> with useI18n / antd message.
Introduce a generic DataTable (components/table/DataTable.vue) that lays out the shared list-page structure: an optional filter slot on top, a toolbar with left page-action slot and built-in right tools (refresh, column show/hide popover, CSV download), then the antd table and pagination. Columns, data, paging and loading are props; refresh / page-change / size-change / download are events; #bodyCell passes through. Refactor UserHome to consume it as the reference usage (create button in #actions, cells via #bodyCell), removing its hand-written table/pagination.
Replace the hand-written a-table + a-pagination + toolbar in RoleHome, MenuHome, FunctionHome and SnippetHome with the shared DataTable component. Page create/import actions move into #actions; cell rendering stays in #bodyCell; refresh / paging are wired to the component's events. Consolidates the list-page structure and gives every list the built-in refresh, column toggle and CSV download tools for free.
Move the DataTable toolbar/filter/pagination layout from Tailwind utility classes (provided by view-shadcn-ui, which will be removed) to scoped CSS, so the component renders consistently now and survives the library removal. Add scroll.x=max-content so table columns size to their content instead of being stretched equally across wide viewports.
Migrate admin/history: HistoryHome list on DataTable (state a-tag with success/error color, plugin/format a-avatar, mode a-tag, a-dropdown row menu with router-link) and HistoryData dialog (a-modal + footer cancel wrapping VisualTable). Both <script setup> with useI18n / antd message. HistoryUtils emits antd columns.
Migrate admin/report on DataTable: ReportHome list (a-switch realtime, a-avatar/a-tag source, a-dropdown row menu with a disabled modify item and router-link), ReportView (a-modal + VisualView) and ReportDelete confirm (a-modal + a-alert list + name match validator). All <script setup> with useI18n / antd message; ReportUtils emits antd columns.
… components to ant-design-vue
…e to ant-design-vue
…readcrumb, metadata tree) to ant-design-vue
- replace remaining Shadcn* components (icon, logger, contribution, workflow editor/view, data builder, code editor) with local antd-based implementations under src/views/components - reference @ant-design/icons-vue components directly at every call site and drop the DcIcon wrapper; backend menu icon names keep a small name-to-component map in LayoutHeader - vendor the tailwind utilities used by templates into src/assets/utilities.local.css and restore antd icon baseline alignment broken by the preflight svg rule - convert table toolbar and row action icon buttons to text style - replace the monaco based shadcn code editor in query page with the shared AceEditor plus metadata suggest auto completion - drop the router loading bar and legacy setLocale calls - pin typescript to 5.4.5 for vue-tsc 1.8 compatibility
The ant layout renders a gray background behind the transparent header row, which showed as gray bands on both sides on wide screens
- rebuild profile sidebar as a card with section header, per-item icons and a bottom promo panel - add page header with description and slogan, a profile hero card with avatar upload (inline cropper) and role/status badges - show basic info fields, a language preference selector and four account overview tiles (role, status, register time with day count, last login with ip from login logs) - add matching i18n keys for en and zh-cn
The ml-20 utility never existed in the vendored styles so the content column sat flush against the sidebar; use the available ml-6 spacing and render the breadcrumb like the prototype
Apply the soft green wave artwork as the fixed page backdrop, offset below the header so its curves show through the content margins
The default gray ant-layout background painted over the body backdrop image, hiding it on every page including the user settings
- add DataCap wordmark next to the logo - render horizontal menu items as rounded pills with tinted selected and hover states instead of the antd underline style - highlight the active menu by longest path-prefix match so child pages mark their parent menu - add a Help label beside the feedback icon and show the username with a chevron in the account dropdown trigger
- introduce --dc-* css variables from design/ui-color-system.md covering brand scale, backgrounds, text hierarchy, borders, status colors, radius, shadows and gradients, with dark-mode variable overrides - expose brand palette and antd theme tokens through the useTheme composable (colorPrimary green, transparent layout background) and a brand registry so user preferences can switch themes later - migrate header, profile sidebar and profile page styles to consume tokens; add hero gradient band and info/success status tags - replace the body backdrop image with css radial-gradient ambience per the design document - align log viewer level colors and contribution heat scale with tokens
The image url is exposed as --dc-body-image so user preferences can switch it later; dark theme falls back to the gradient ambience
Menu items were stretched and used oversized 18px icons; pin items to their natural width with tight padding and 16px icons
The ci installed the latest pnpm while the lockfile was still in the v6 format produced by pnpm 8, breaking pnpm install with ERR_PNPM_BROKEN_LOCKFILE. Upgrade the lockfile to v9 and pin the ci pnpm to 9.13.2, matching the frontend-maven-plugin version; declare packageManager for corepack users
antd derives colorPrimaryBg from the seed color and produced a muted sage for the green brand, which leaked into selected menu items; pin colorPrimaryBg/BgHover/Border/BorderHover to the documented values for light and dark
The header bar collapsed to the menu height with no vertical padding; give it a 64px height with centered content. The 1400px centered container cap also fought the prototype proportions; switch to a 1520px cap with 40px side padding (roughly full-width like the mock)
Add the documented per-state colors: neutral item text, hover surface, selected mint background with darker green text and the 3px brand bar on the leading edge; expose menu hover/text tokens for dark mode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one)
Changelog entry (Details of this change)
Affected version
latest version