✅ test: gate GraalPy gaps on probed capabilities#682
Merged
Conversation
GraalPy 24.2.2 ran the suite with 55 failures. Each test now gates on a probe of the behavior it needs rather than on an interpreter name. Three tests needed a fix, not a gate: - test_write_non_starvation counted reader releases from the moment it started the writer process, so a slow interpreter start read as starvation. It now counts from the moment the writer reaches the lock. - Two heartbeat tests compared a utime round-trip for bit equality, and GraalPy returns the timestamp one nanosecond off. They now compare with a tolerance, which also covers filesystems with coarse timestamps. - test_singleton_instance_tracking_is_unique_per_subclass carried a PyPy skip it never needed. It touches no finalizer, so it now runs everywhere. The new capabilities live in tasks/coverage_pragmas.py beside the existing probes, keeping a skip and a coverage exclusion from disagreeing: - prompt-finalization replaces three hasattr(sys, "pypy_version_info") gates. It holds only on a refcounting collector, so PyPy keeps skipping those tests and CPython keeps running them. - collected-finalization and class-collection cover a GraalPy that runs no __del__ and reclaims no class, even after a forced gc.collect(). - audit-events holds where sys.audit reaches an installed hook. GraalPy accepts the hook and never calls it, so seven tests that drive filelock through sqlite3.connect events observe nothing. - link-dir-fd is narrower than dir-fd. GraalPy takes os.open relative to a directory descriptor but not os.link, so the backend never opens the directory that two fault-injection tests count their faults against. - o-nofollow now probes the refusal instead of the constant, because GraalPy defines O_NOFOLLOW and follows the link anyway. Twenty cancellation tests are xfail rather than skipped. GraalPy answers athrow() with "RuntimeError: generator didn't stop after athrow()" raised from its own contextlib, so a cancellation crossing an async context manager arrives as the wrong exception. The gate drives a coroutine by hand to observe that, and stays non-strict because the deviation depends on where the cancellation lands. src/filelock is untouched and no pre-existing capability value changes, so CPython measures the lines it measured before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #681, whose ENOTSUP fix filelock needs before it will import on GraalPy at all. Review that one first;
the base here is its branch, so this diff is the test-only commit on top.
GraalPy 24.2.2 (macos-aarch64,
uv python install graalpy-3.11.0) ran the suite with 55 failures. GraalPy is not inCI, so CI cannot validate any of this. I wrote every gate to hold on Linux, macOS, Windows, Termux/Android and
pypy3.11 too, and checked pypy3.11 by running it.
Where the 55 went
contextlibbug, characterized)audit-eventsprompt-finalization/collected-finalizationos.mkfifomissingfifocapabilitylink-dir-fdclass-collectionO_NOFOLLOWignoredo-nofollow, now probedFinal GraalPy run: 1105 passed, 153 skipped, 28 xfailed, 5 xpassed, 0 failed.
Genuinely fixed (3)
These three encoded an assumption of the harness, so they now run on every runtime instead of skipping:
test_write_non_starvationcounted reader releases from the moment it spawned the writer process. GraalPy needsseconds to boot an interpreter, so the reader chain drained before the writer ever reached the lock and a slow
start read as starvation. It now snapshots the count when the writer signals it has reached the lock and asserts
on the delta. On CPython the snapshot is 0, so the assertion is unchanged.
utimeround-trip for bit equality. GraalPy returns the timestamp one nanosecondoff, which made them fail about one run in three. They compare with a tolerance now, which also covers filesystems
with coarse timestamp granularity. The property under test spans 1000 seconds, so a millisecond tolerance weakens nothing.
test_singleton_instance_tracking_is_unique_per_subclasscarried ahasattr(sys, "pypy_version_info")skip itnever needed, since it touches no finalizer. I ran it on PyPy 3.11 without the skip and it passes.
Guarded on a probed capability (16)
These live in
tasks/coverage_pragmas.pyalongside the existing probes, keeping a skip and a coverage exclusionfrom drifting apart. Each one probes the behavior instead of reading
sys.implementation.name:prompt-finalizationholds where__del__runs as the last reference drops, and replaces the threehasattr(sys, "pypy_version_info")gates. I measured CPythonTrue, PyPyFalse, GraalPyFalse, so PyPy keepsskipping exactly what it skipped and CPython keeps running it.
collected-finalizationholds wheregc.collect()runs a pending__del__: CPythonTrue, PyPyTrue,GraalPy
False. GraalPy hands finalizers to the host collector and never gets them back.class-collectionholds wheregc.collect()reclaims a dynamically built class. GraalPy never does, evenafter 30 collections.
audit-eventsholds wheresys.auditreaches a hook fromsys.addaudithook. GraalPy accepts the hook andcalls it for nothing, so seven tests that drive filelock through
sqlite3.connectevents see an empty stream.link-dir-fdnarrows the existingdir-fd, which probesos.open. GraalPy hasos.open in os.supports_dir_fdbut notos.link, so_link_no_replaceskips its dir_fd branch and never opensthe directory that two fault-injection tests count their faults against (7 directory closes vs CPython's 8).
o-nofollowused to readhasattr(os, "O_NOFOLLOW"), and GraalPy defines the flag then follows the linkanyway. It now opens a real symlink and asks for the refusal, matching how
link-follow-symlinksalready handlesPyPy. It falls back to the constant where symlinks cannot be created, leaving Windows unaffected.
xfail (33 instances across 20 tests)
GraalPy raises
RuntimeError: generator didn't stop after athrow()from its owncontextlib._GeneratorContextManagerBase.__aexit__. Reduced to ten lines with no filelock involved:Cancelling
secondwhile it is suspended inside__aenter__raisesRuntimeErrorinstead ofCancelledErroronGraalPy 24.2.2, and
CancelledErroron CPython 3.11 and PyPy 3.11. filelock's@asynccontextmanageruse isidiomatic and the raising frame belongs to the interpreter stdlib, so these are xfail rather than skip and should
start passing once GraalPy fixes it. The gate drives a coroutine by hand, needing no event loop, and is
strict=Falsebecause thedeviation depends on where the cancellation lands (5 of the 33 still pass).
The same defect clears
__context__when an exception is thrown into a suspended frame, which is whatgenerator-exception-contextcovers fortest_finish_connection_chains_rollback_then_close_errorandtest_coroutine_on_acquired_error_preserves_context.CPython is unaffected
tox r -e 3.14on darwin: 100% line and branch coverage, 12710/12710 lines, diff-cover 100%.src/filelockis untouched.o-nofollowstaysTrue), and no new pragma appearsin
src/, so CPython measures an identical line set. The one new# pragma: needs link-dir-fdsits on a testdecorator and excludes nothing where the capability is present.
tests never provide the sole cover for a
src/line.tox r -e typeandtox r -e fixclean.No changelog entry: #679 made the same kind of test-and-coverage-infrastructure change without one.