fix(automl): safely append training logs

[林SO (Linxiushen)]

Why are these changes needed?

Appending to an AutoML training log restarts record_id at zero, so the appended run's checkpoint can reconstruct an older configuration. Resume IDs after the greatest existing ID, skipping checkpoints and retaining each run's own best-loss state. Existing duplicate IDs are not rewritten.

The writer also handles an existing final record or checkpoint without a trailing newline, and serializes writers sharing a log with a cross-process sidecar lock. Both normal and append writers acquire the lock before opening or scanning the log and keep it through flush and close. filelock is added to the AutoML and test extras; core-only imports remain supported. The append_log docs explain that waiting for another writer counts toward time_budget, and concurrent runs should use distinct log files.

Validation:

• The original ID regression suite produced 5 failures and 2 passes on base 16bc42e071b3320625dc8358f52e241a88de3b19.
• Before this review follow-up, the Linux regression suite produced 4 failures and 10 passes: unterminated record/checkpoint tails corrupted JSON, and real spawned appenders produced duplicate IDs [0, 0] and [0, 1, 1].
• The final focused suite passes all 25 cases on Linux Python 3.10, 3.11, 3.12, and 3.13. It covers public AutoML.get_estimator_from_log reconstruction, sparse/existing duplicate IDs, line endings, real concurrent writers, independent files, and lock release on failure.
• Windows Python 3.10 and 3.12 each pass 21 nonprocess cases. Actual spawn attempts are blocked locally by CreateProcess returning WinError 5, so the four multiprocessing cases were excluded from the final Windows runs; their passing concurrency results are from Linux.
• All 15 pre-commit hooks pass, as does an isolated core-only import without filelock installed.

The existing log is read once when opening for append. Malformed logs fail without appending content. Full AutoML, forecasting, and Spark suites were not run. An additional small random-forest AutoML.fit smoke reaches an existing _t1 attribute error before writing any records, both with the original training-log module and with this change; no successful fit smoke is claimed.

Checks

• I've used pre-commit to lint the changes in this PR.
• I've included doc changes for shared-log serialization and its time-budget behavior.
• I've added tests corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[Li Jiang (thinkall)]

Overall review of the complete PR: changes are required.

1. flaml/automl/training_log.py:74: appending after a valid final JSON record without a trailing newline concatenates records as }{ and corrupts all subsequent log readers. Detect an unterminated nonempty tail and add a newline before writing; test record and checkpoint tails.
2. flaml/automl/training_log.py:68-74: concurrent appenders scan and allocate IDs without a lock, so both can choose the same next ID. Hold a cross-process lock from ID discovery through writer close and add a two-writer uniqueness regression.

Posted by thinkall-agent-auto-reviewer

[林SO (Linxiushen)]

Addressed both findings in the follow-up commit:

1. Appending now adds a separator only when a nonempty log ends with neither LF nor CR. The tests cover record and checkpoint tails with no newline, LF, CRLF, and CR, including Windows text-mode behavior.
2. Both normal and append writers now hold a filelock sidecar lock from before opening/scanning the log until flush and close. Real spawned-process tests cover missing and existing shared logs, a normal writer followed by an appender, and independent files. Error-path tests verify release after malformed logs, failed opens, and context exceptions. The docs explain the serialization and its effect on time_budget.

The original follow-up regressions fail on Linux with corrupt JSON and duplicate IDs ([0, 0] / [0, 1, 1]). The final 25-case suite passes on Linux Python 3.10, 3.11, 3.12, and 3.13; all pre-commit hooks pass. Windows Python 3.10/3.12 each pass the 21 nonprocess cases, while local CreateProcess access denial prevents Windows multiprocessing validation. Core-only imports were also checked without the new optional dependency installed.

The PR description now reflects the locking behavior, expanded validation, and remaining validation limits. Ready for another review.

[Li Jiang (thinkall)]

Overall review of the complete current PR: approved.

The writer lock now covers ID discovery through file close, preventing duplicate IDs across concurrent writers. Appends safely insert separators after valid unterminated record or checkpoint tails. Regression and stress coverage exercises missing/existing logs, legacy sparse IDs, LF/CRLF/CR/no terminator, threads/processes, malformed files, failed opens, crash recovery, and retrain/log readers. No blocking issues found.

Posted by thinkall-agent-auto-reviewer