Skip to content

[fix](fe) Preserve Broker Load diagnostics - #68371

Open
yonchicy wants to merge 1 commit into
apache:masterfrom
yonchicy:fix_broker_load_err_url_miss
Open

yonchicy wants to merge 1 commit into
apache:masterfrom
yonchicy:fix_broker_load_err_url_miss

Conversation

@yonchicy

@yonchicy yonchicy commented Sep 22, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: None

Related PR: #67002

Problem Summary:

A Broker Load fragment may report DATA_QUALITY_ERROR together with its
error URL and first error message. Coordinator handles the failure and
releases the completion latch before storing those diagnostics. The
loading task can then snapshot empty values into the final job state.
SHOW LOAD consequently loses the URL or FirstErrorMsg even though the
backend supplied them.

Publish both fields before cancellation wakes the loading task in the
legacy Coordinator and Nereids LoadProcessor. Use the pre-status hook
for Nereids diagnostics while retaining final report aggregation in its
existing position. The related PR addresses the URL race; this change
also preserves FirstErrorMsg and tests the production cancellation path.

The unit tests inspect diagnostics at the cancellation/latch boundary,
so they exercise the race deterministically without sleeps. They also
cover normal completion and reports that omit diagnostic fields.

Scope and compatibility:

  • Change diagnostic publication order only; retain transaction, counter,
    retry, and persistence behavior.
  • Add no SQL syntax, configuration, RPC field, or storage format.
  • This fixes diagnostics carried by the report that triggers failure;
    it does not add collection of reports arriving after task teardown.

Release note

Fix missing error URLs and first error messages in SHOW LOAD when a
Broker Load data quality error report includes those diagnostics.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test
    • No need to test or manual test
  • Behavior changed:

    • No.
    • Yes. Preserve reported diagnostics when load tasks wake on
      failure.
  • Does this need documentation?

    • No. Restore existing SHOW LOAD diagnostic fields without
      introducing a new user-facing interface.
    • Yes.

Validation

Local validation passed with JDK 17, Maven 3.9.9, and Thrift 0.24.0:

  • CoordinatorLoadDiagnosticsTest: 2 tests
  • LoadProcessorTest: 3 tests
  • AbstractJobProcessorTest: 2 tests
  • QeProcessorImplReportAckTest: 7 tests

Total: 14 tests, 0 failures, 0 errors, and 0 skipped; 5 tests are new.
FE Checkstyle reported 0 violations. git diff --check passed.
No cluster regression test was run.

bash ./run-fe-ut.sh --run \
'org.apache.doris.qe.CoordinatorLoadDiagnosticsTest,'\
'org.apache.doris.qe.runtime.LoadProcessorTest,'\
'org.apache.doris.qe.AbstractJobProcessorTest,'\
'org.apache.doris.qe.QeProcessorImplReportAckTest'

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Issue Number: None

Related PR: apache#67002

Problem Summary:

A Broker Load fragment may report DATA_QUALITY_ERROR together with its
error URL and first error message. Coordinator handles the failure and
releases the completion latch before storing those diagnostics. The
loading task can then snapshot empty values into the final job state.
SHOW LOAD consequently loses the URL or FirstErrorMsg even though the
backend supplied them.

Publish both fields before cancellation wakes the loading task in the
legacy Coordinator and Nereids LoadProcessor. Use the pre-status hook
for Nereids diagnostics while retaining final report aggregation in its
existing position. The related PR addresses the URL race; this change
also preserves FirstErrorMsg and tests the production cancellation path.

The unit tests inspect diagnostics at the cancellation/latch boundary,
so they exercise the race deterministically without sleeps. They also
cover normal completion and reports that omit diagnostic fields.

Scope and compatibility:

- Change diagnostic publication order only; retain transaction, counter,
  retry, and persistence behavior.
- Add no SQL syntax, configuration, RPC field, or storage format.
- This fixes diagnostics carried by the report that triggers failure;
  it does not add collection of reports arriving after task teardown.

### Release note

Fix missing error URLs and first error messages in SHOW LOAD when a
Broker Load data quality error report includes those diagnostics.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test
    - [ ] No need to test or manual test

- Behavior changed:
    - [ ] No.
    - [x] Yes. Preserve reported diagnostics when load tasks wake on
      failure.

- Does this need documentation?
    - [x] No. Restore existing SHOW LOAD diagnostic fields without
      introducing a new user-facing interface.
    - [ ] Yes.

### Validation

Local validation passed with JDK 17, Maven 3.9.9, and Thrift 0.24.0:

- CoordinatorLoadDiagnosticsTest: 2 tests
- LoadProcessorTest: 3 tests
- AbstractJobProcessorTest: 2 tests
- QeProcessorImplReportAckTest: 7 tests

Total: 14 tests, 0 failures, 0 errors, and 0 skipped; 5 tests are new.
FE Checkstyle reported 0 violations. git diff --check passed.
No cluster regression test was run.

```bash
bash ./run-fe-ut.sh --run \
'org.apache.doris.qe.CoordinatorLoadDiagnosticsTest,'\
'org.apache.doris.qe.runtime.LoadProcessorTest,'\
'org.apache.doris.qe.AbstractJobProcessorTest,'\
'org.apache.doris.qe.QeProcessorImplReportAckTest'
```
@yonchicy
yonchicy force-pushed the fix_broker_load_err_url_miss branch from d2b8476 to 35f81a5 Compare September 22, 2026 06:53
@yonchicy yonchicy changed the title [fix](fe) Preserve Broker Load error URL and first error message [fix](fe) Preserve Broker Load diagnostics Sep 22, 2026
@yonchicy

Copy link
Copy Markdown
Author

Local pipeline review — ✅ PASS

schema: doris-repo-review/v1
status: PASS
pr: apache/doris#68371
commit: 35f81a5b1569ae997c73099499dd8cd614e6c9fe
base: 5c4dbdd07fc43ea287168e1a203beaff4998517f
reviewed_at: 2026-09-22T15:24+08:00
reviewer: yonchicy
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: true

Notes for maintainers

  • Source review only: the production report/cancellation paths and all five new unit tests were inspected; no builds, tests, or cluster runs were performed during this review.

Reviewed locally with the doris-repo-review pipeline. Repository policy may accept this receipt for the matching commit; it is not a human Apache approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants