Skip to content

log_parsers: read mocha's spec reporter - #162

Open
k21993 wants to merge 1 commit into
huggingface:mainfrom
k21993:fix/mocha-spec-reporter
Open

k21993 wants to merge 1 commit into
huggingface:mainfrom
k21993:fix/mocha-spec-reporter

Conversation

@k21993

@k21993 k21993 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Parse mocha's spec reporter as its own format in both log_parsers/jest_parser.py and the standalone copy in _pr_runtime_verifier.py. A mocha log previously parsed to {}, so mocha repos yielded no tasks at all (Mocha logs parse to nothing, so mocha repos yield no tasks #161).
  • Mocha is detected by its N passing (Xms) epilogue or its log-symbols check mark. Jest's path is untouched, since jest prints neither.
  • Names are built from the indentation (Math > add() > adds two numbers), because mocha has no file header. ✔, ✓ and √ are passes, N) is a failure, - is pending, and a trailing (123ms) is dropped.
  • The epilogue is skipped, so its numbered blocks, which repeat each failing title above a stack trace, are not recorded as tests. A later check mark means a second mocha run began in the same log, which is what mocha a && mocha b produces.

On the real before/after runs from #161 (mocha 12.0.2):

Tests parsed pre / post F2P P2P Outcome
main 0 / 0 none none dropped: "no fail-to-pass tests after validation"
this PR 2 / 2 div() > rejects a zero divisor div() > divides graded 1.0, resolved=True

Test plan

  • uv run --all-extras pytest -q: 2015 passed, 6 skipped (live-network tests)
  • uv run ruff check . and uv run ruff format --check .
  • New tests/test_mocha_output.py runs every case against both parsers. Fixtures are verbatim npx mocha output (mocha 12.0.2): a suite with passes, a numbered failure, two pending tests and a slow test's duration; suites nested three deep; and two runs concatenated in one log.
  • 18 of the 20 new tests fail on main; all pass with this change.
  • The two-run log records all 11 tests and nothing from either epilogue, so repeated failure titles and stack traces stay out of the map.
  • A jest log with ✓ and a PASS header still parses as jest, keeping its file prefix.
  • The standalone verifier, run with python -I, grades the post-fix log at reward 1.0.
  • Pathological 200k-character lines (long titles, long pending lines, 50k repeated durations) parse in milliseconds, with a subprocess timeout guard like Fix pytest parsing of parametrized node IDs containing spaces #91's.

Out of scope

  • normalize_test_cmds_for_runtime still appends --verbose to mocha commands. Mocha accepts it and prints identical output, so this PR leaves it alone; happy to drop it separately.
  • Mocha reporters other than spec (dot, nyan, json), and its --parallel interleaving.
  • Suites whose titles collide across files, which mocha's output cannot distinguish since it never prints the file.

Closes #161

@adithya-s-k

Copy link
Copy Markdown
Collaborator

Thanks, Mocha support is needed for #161. I reproduced two cases to fix before merging:

  • Concatenated runs: if the second run starts with a numbered failure before its first pass, that failure disappears. Please cover a second run with only failures too.
  • Jest output with a PASS a.test.js header and √ passes (4 ms) gets treated as Mocha, losing the file prefix and keeping the duration in the test ID.

Please fix run boundaries and format detection in both parser copies, with regression tests. The rest can stay scoped as proposed.

@k21993
k21993 force-pushed the fix/mocha-spec-reporter branch from cfcb07a to a6693a5 Compare September 24, 2026 17:40
The JS parser only understood jest's format, so a mocha log produced an
empty status map: modern mocha prints log-symbols' check mark, numbered
failures and dashed pending tests, with suites nested by indentation and
no file header to anchor them.

Every mocha repo therefore yielded no tasks, since F2P needs a test that
fails before the fix and passes after, and neither run recorded anything.

Detect mocha by its `N passing (Xms)` epilogue or its ✔, then read the
listing. The epilogue repeats each failing title above a stack trace, so
those blocks are consumed by counting the entries its `N failing` line
announces; anything after that is the first result of the next mocha run
in the same log, which `mocha a && mocha b` produces.

√ cannot identify mocha, because jest prints it on Windows too, so a log
carrying a jest file header is read as jest: treating one as mocha lost
the file prefix, kept jest's ` (4 ms)` duration in the test ID and
dropped its ✕ failures, which mocha's reporter never prints.

Closes huggingface#161
@k21993
k21993 force-pushed the fix/mocha-spec-reporter branch from a6693a5 to 76c6360 Compare September 24, 2026 23:03
@k21993

k21993 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

Both reproduced, and the jest one was worse than it looked: reading that log as mocha also dropped the × line entirely, since mocha's spec reporter never prints it. Before this push:

second run starting with a failure -> {'Math > adds': PASSED, 'Math > breaks': FAILED, 'API > works': PASSED}   # 'API > also breaks' gone
second run with only failures      -> {'Math > adds': PASSED}                                                   # whole run gone
jest with √ and a PASS header      -> {'Suite > passes (4 ms)': PASSED}                                         # no file prefix, duration kept, × lost

Run boundaries now key off mocha's own count: N failing says how many numbered blocks the epilogue repeats, each at indent 2, and once those are consumed the next numbered or dashed line is the following run's listing rather than a repeat. That covers a second run whose first result is a failure, and one with only failures.

Format detection now prefers jest whenever the log carries a PASS/FAIL file header, which mocha never prints. √ alone no longer selects mocha, since jest uses it on Windows; a √ mocha log is still recognized through its epilogue. After the fix:

second run starting with a failure -> 4 tests, including 'API > also breaks'
second run with only failures      -> 3 tests
jest with √ and a PASS header      -> {'a.test.js > Suite > passes': PASSED, 'a.test.js > Suite > fails': FAILED}

Both parser copies changed together. Added tests: the two concatenated-run shapes, a three-failure epilogue consumed by count, jest-wins-on-header for both √ and ✓, a √ mocha log identified by its epilogue, and a jest log with no glyph ambiguity. 34 tests in the file, 6 of which fail on main. Full suite 2029 passed, 6 skipped; ruff clean. Scope is otherwise unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mocha logs parse to nothing, so mocha repos yield no tasks

2 participants