Modify the optimizer to support obtaining tasks from each AMS node for processing.

[wardlican]

Why are the changes needed?

In master-slave mode, the optimizer needs to support pulling optimization tasks from each ams node for processing.

Close #3928.

Brief change log

1. In master-slave mode, the optimizer can pull tasks from each currently active AMS.
2. In master-slave mode, the AMS automatically detects the addition and timeout of the optimizer from the database.

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before making a pull request

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.

[wardlican]

Please help with the code review.

[czy006]

After resolving any conflicts in the code, I will initiate the first review on my end cc @wardlican

[czy006]

After resolving any conflicts in the code, I will initiate the first review on my end cc @wardlican

[wardlican]

After resolving any conflicts in the code, I will initiate the first review on my end cc @wardlican

Okay, I will resolve the conflicts based on the latest master branch and add support for a database-based storage solution for bucket allocation information, based on the latest HA (High Availability) scheme. @czy006

[codecov-commenter]

Codecov Report

❌ Patch coverage is 31.85247% with 813 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.15%. Comparing base (cbdc517) to head (b6be852).
⚠️ Report is 23 commits behind head on master.

Files with missing lines	Patch %	Lines
...apache/amoro/server/table/DefaultTableService.java	9.17%	179 Missing and 9 partials ⚠️
...ache/amoro/optimizer/common/OptimizerExecutor.java	2.04%	94 Missing and 2 partials ⚠️
...apache/amoro/server/DatabaseBucketAssignStore.java	0.00%	80 Missing ⚠️
.../apache/amoro/server/DefaultOptimizingService.java	19.78%	71 Missing and 2 partials ⚠️
...ain/java/org/apache/amoro/client/AmsThriftUrl.java	0.00%	60 Missing ⚠️
...java/org/apache/amoro/server/AmsAssignService.java	75.84%	40 Missing and 17 partials ⚠️
.../apache/amoro/optimizer/common/AmsNodeManager.java	0.00%	47 Missing ⚠️
...o/server/ha/DataBaseHighAvailabilityContainer.java	0.00%	45 Missing ⚠️
...a/org/apache/amoro/server/ZkBucketAssignStore.java	66.66%	34 Missing and 5 partials ⚠️
...org/apache/amoro/server/AmoroServiceContainer.java	7.89%	33 Missing and 2 partials ⚠️
... and 9 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3950      +/-   ##
============================================
+ Coverage     22.12%   30.15%   +8.02%     
- Complexity     2461     4165    +1704     
============================================
  Files           445      658     +213     
  Lines         40897    53207   +12310     
  Branches       5767     6808    +1041     
============================================
+ Hits           9050    16043    +6993     
- Misses        31089    35957    +4868     
- Partials        758     1207     +449     

Flag	Coverage Δ
core	30.15% <31.85%> (?)
trino	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.