Skip to content

Fix: commit score/tailor/cover-letter results incrementally instead of in one final batch - #90

Open
mprecilio20 wants to merge 1 commit into
Pickle-Pixel:mainfrom
mprecilio20:fix/incremental-db-commits
Open

Fix: commit score/tailor/cover-letter results incrementally instead of in one final batch#90
mprecilio20 wants to merge 1 commit into
Pickle-Pixel:mainfrom
mprecilio20:fix/incremental-db-commits

Conversation

@mprecilio20

Copy link
Copy Markdown

Summary

The score, tailor, and cover pipeline stages each accumulate all per-job results in memory and only write/commit to the database once, after every job in the batch has finished — not incrementally per job.

This causes two real problems on any run of meaningful size:

  • A crash, kill, or interruption partway through a stage loses all progress for that stage, even though the expensive part (the LLM calls, and for tailor/cover the resume/cover-letter files already written to disk) already completed successfully. On a batch of a few hundred jobs at several seconds-to-a-minute per LLM call, that's hours of redone work after any interruption.
  • applypilot status is misleading while a stage is running. Scored/tailored/cover-letter counts stay at zero for the entire duration of a stage, even while it's actively making progress, because nothing has been committed yet for status to read. It looks like the pipeline is stuck when it isn't.

Fix

Each stage now commits immediately after processing each individual job, instead of accumulating and writing once at the end.

The "don't reprocess already-completed jobs" behavior needed no new logic — it already existed via the job-selection queries filtering on the relevant column being NULL (fit_score, tailored_resume_path, cover_letter_path). It just wasn't reachable in practice, since nothing was ever committed for those filters to see mid-run. Restarting after an interruption now naturally resumes from where it left off.

Scope

Deliberately narrow — just the three run_scoring / run_tailoring / run_cover_letters functions in scoring/scorer.py, scoring/tailor.py, scoring/cover_letter.py. No behavior change on a clean, uninterrupted run, other than status reflecting real progress instead of zeros throughout.

Test plan

  • Verified all three files parse (ast.parse)
  • Verified a job's fit_score commits and is correctly excluded from the next pending_score query (confirms resume-skip behavior works)
  • Ran applypilot status mid-run before and after the fix — before: stuck at 0 scored for the entire stage; after: count increments in real time as each job completes
  • Would appreciate a maintainer sanity-check on tailor/cover_letter's attempt-counter semantics, since those two also increment a retry counter for failed jobs (unchanged behavior, just moved inside the loop)

Per-row commits add negligible overhead relative to the LLM call each row already waits on (milliseconds vs. seconds-to-minutes).

…nal batch

Each of these three stages previously accumulated all results in memory
and only wrote/committed to the database once, after every job in the
batch had been processed. On a run of any real size this means a crash,
kill, or interruption partway through loses all progress for that stage,
even though most of the work (LLM calls, resume/cover-letter files
already written to disk) had already completed successfully.

This also made `applypilot status` misleading mid-run: scored/tailored/
cover-letter counts stayed at zero for the entire duration of a stage,
even while it was actively making progress, since nothing had been
committed yet for status to see.

Each stage now commits right after processing each individual job. Since
the job-selection queries already filter on the relevant column being
NULL (fit_score, tailored_resume_path, cover_letter_path), a rerun after
an interruption naturally skips already-completed jobs with no changes
needed there -- this was already the intended resume behavior, it just
wasn't reachable in practice.

No behavior change on a clean, uninterrupted run other than `status`
reflecting progress in real time. Per-row commits add negligible
overhead relative to the LLM call each row already waits on.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants