fix(pc-sampling): count _not_issued stall samples separately - #166
Merged
Merged
Conversation
CUPTI's PC Sampling API counts every sample under its warp state and, when the scheduler issued nothing that cycle, again under the state's _not_issued twin. The analyzer, both text reports and the collect log summed the two, so each stall share came out at about half its value (long_scoreboard at 93% showed as 49% plus 46%) and per-kernel sample totals were about 1.9x CUPTI's count. Shares now use the warp states only, which add up to totalSamples less non-user-kernel and dropped samples. The _not_issued samples are shown as a separate breakdown with their own total, and inspect_stalls(not_issued=True) shows that view per kernel. The analyzer also labelled PC Sampling API rows through the Activity API index table, so long_scoreboard appeared as "Barrier". Rows now use the CUPTI reason name, and the index table, still used for Activity API rows, matches CUpti_ActivityPCSamplingStallReason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #164. This commit was reviewed in #165, but #165 targeted
pc-sampling-sampled-launch-thresholdafter #164 had already merged it intomain, so the change never reachedmainand CI never ran on it.Problem
CUPTI's PC Sampling API reports every sample under its warp state (
smsp__pcsamp_warps_issue_stalled_<state>) and, when the scheduler issued no instruction that cycle, again under<state>_not_issued. The_not_issuedcounts are a subset of the state's counts (Nsight Compute Profiling Guide, sections 2.4.7 and 2.4.8). The analyzer, both text reports and the collect-summary debug log summed the two, so each stall share came out at about half its value and per-kernel sample totals were about 1.9x CUPTI's count.The analyzer also labelled PC Sampling API rows through the Activity API index table, so
long_scoreboard(index 12 on an RTX 5060) appeared as "Barrier" and its_not_issuedtwin as "Sleeping".Change
selectedandnot_selectedincluded). These add up tototalSamples - nonUsrKernelsTotalSamples - droppedSamples._not_issuedsamples are a separate breakdown with their own total: extra columns ininspect_profile_samplesand the Python report, a "Not issued" block in the C++ report, andinspect_stalls(not_issued=True).smsp__pcsamp_sample_countandsmsp__pcsamp_samples_data_droppedare not warp states and are left out.CUpti_ActivityPCSamplingStallReason.N samples (+M not issued), so N lines up with CUPTI's counters.Evidence
Four vector-add PcSampling captures on an RTX 5060 (CUDA 13.3), 200 to 6,000 launches:
totalSamples - nonUsrKernelsTotalSamples - droppedSamplesexactly in all four runs._not_issuedsamples than state samples.On the 1,000-launch capture the analyzer's reason table showed "Barrier 49.2%" and "Sleeping 44.6%" before this change and shows
long_scoreboard93.4% after it.Tests
tests/python/test_pc_stall_shares.py(5 tests) andTextReportTest.PcStallSharesExcludeNotIssuedSamples. All six fail on cf2e392 and pass with this change.test_analyzer.py).TextReportTest7/7 on a CUDA 13.3 build; fullgpufl_tests609/609 on a no-CUDA build.EngineCoverageTest.EmitsExpectedEvents/PcSamplingpasses on an RTX 5060. Its collect summary reads195316 samples (+132816 not issued)withtotalSamples=195663 dropped=0 nonUsrKernels=347(195,663 - 347 = 195,316).🤖 Generated with Claude Code