Skip to content

nvproxy: expose CLOCK_MONOTONIC_RAW as a distinct clock for capProfiling - #14002

Open
luiscape wants to merge 1 commit into
google:masterfrom
luiscape:luis/allign-clocks-in-capprofiling
Open

nvproxy: expose CLOCK_MONOTONIC_RAW as a distinct clock for capProfiling#14002
luiscape wants to merge 1 commit into
google:masterfrom
luiscape:luis/allign-clocks-in-capprofiling

Conversation

@luiscape

@luiscape luiscape commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

GPU profilers (Nsight Systems/CUPTI) build their trace timeline from CLOCK_MONOTONIC_RAW. gVisor serves CLOCK_MONOTONIC_RAW as an alias of CLOCK_MONOTONIC, this will drift from CLOCK_MONOTONIC_RAW breaking profiling.

When nvproxy is enabled with capProfiling, this change exposes CLOCK_MONOTONIC_RAW as a clock distinct from CLOCK_MONOTONIC that tracks the host's CLOCK_MONOTONIC_RAW (absolute, unadjusted). CLOCK_MONOTONIC is unchanged. When profiling is disabled, CLOCK_MONOTONIC_RAW continues to alias CLOCK_MONOTONIC as before.

This is an example program that won't store kernel traces without this change:

  nsys profile --trace=cuda --sample=none --cpuctxsw=none \
    --export=sqlite -o /out/torch \
    python3 -c 'import torch; a=torch.randn(1024,1024,device="cuda"); (a@a).sum().item()'

One can verify that's the case by looking at the resulting SQLite database:

sqlite3 /tmp/torch.sqlite "SELECT count(*) FROM CUPTI_ACTIVITY_KIND_KERNEL"

Assisted-by: Claude

Comment thread pkg/sentry/kernel/timekeeper.go
Comment thread pkg/sentry/kernel/timekeeper.go Outdated
@luiscape
luiscape force-pushed the luis/allign-clocks-in-capprofiling branch from f7395ae to 9be64c1 Compare August 5, 2026 23:47
@luiscape

luiscape commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

@aaltinaydev thank you for the review. I added a warning and panic.

@ayushr2

ayushr2 commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

We may be iterating on this PR. Not ready to pull yet!

@EtiennePerot
EtiennePerot marked this pull request as draft August 6, 2026 00:39
@luiscape
luiscape force-pushed the luis/allign-clocks-in-capprofiling branch from 9be64c1 to 98bcf81 Compare August 6, 2026 00:53
@luiscape luiscape changed the title capProfiling: anchor clocks to CLOCK_MONOTONIC_RAW nvproxy: expose CLOCK_MONOTONIC_RAW as a distinct clock for capProfiling Aug 6, 2026
@luiscape
luiscape force-pushed the luis/allign-clocks-in-capprofiling branch from 98bcf81 to 055c868 Compare August 6, 2026 01:16
@luiscape
luiscape requested a review from aaltinaydev August 6, 2026 02:30
@ayushr2
ayushr2 marked this pull request as ready for review August 6, 2026 03:28
@EtiennePerot
EtiennePerot requested review from ayushr2 and removed request for aaltinaydev and fvoznika August 6, 2026 04:18

@ayushr2 ayushr2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Luis!! Just nits.

Comment thread pkg/sentry/kernel/kernel.go Outdated
Comment thread pkg/sentry/kernel/timekeeper.go Outdated
Comment thread vdso/vdso_time.cc Outdated
Comment thread pkg/sentry/kernel/timekeeper.go Outdated
Comment thread pkg/sentry/time/calibrated_clock.go Outdated
Comment thread runsc/boot/loader.go Outdated
Comment thread runsc/boot/loader.go Outdated
Comment thread pkg/sentry/time/calibrated_clock.go
Comment thread pkg/sentry/time/clock_id.go
@ayushr2
ayushr2 dismissed aaltinaydev’s stale review August 6, 2026 04:52

PR has been reworked.

Comment thread vdso/vdso_time.cc
// clock_gettime(CLOCK_MONOTONIC_RAW).
int ClockMonotonicRaw(struct timespec* ts) {
if (get_params()->monotonic_raw_syscall) {
return sys_clock_gettime(CLOCK_MONOTONIC_RAW, ts);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Profilers timestamp at high rates inside the process being measured, so a syscall per timestamp will probably hurt and give poor timestamp measurements. VDSO exists to avoid the syscall context switch.

Can we serve this from the VDSO instead of falling back to the syscall? The param page has plenty of room (currently seq + 8 u64s in a 4KiB page), and all clocks calibrate against the same TSC, so it's mostly mechanical:

  • add monotonicRaw{Ready,BaseCycles,BaseRef,Frequency} to vdsoParams and mirror them in struct params
  • publish them in Timekeeper.update(), without monotonicOffset
  • make ClockMonotonicRaw() a copy of ClockMonotonic() reading the raw fields

Clocks.Update() would need to return the raw params too. Six return values is a lot; maybe fold them into a struct.

One thing to be careful about: we need to distinguish "raw disabled" from "raw enabled but not yet calibrated". If we only check ready and fall back to the syscall, every non-profiling sandbox regresses from today's free VDSO alias to a syscall. So !enabled should call ClockMonotonic() directly, and only enabled && !ready should hit the syscall:

int ClockMonotonicRaw(struct timespec* ts) {
  // ...seqcount loop reading enabled, ready, base_ref, base_cycles, frequency...
  if (!enabled) return ClockMonotonic(ts);                    // in-VDSO alias, as today
  if (!ready)   return sys_clock_gettime(CLOCK_MONOTONIC_RAW, ts);
  // ...same delta_cycles/cycles_to_ns computation...
}

I think accuracy should not be impacted, since the syscall path uses the same calibrated parameters the VDSO would recompute.

@ayushr2
ayushr2 self-requested a review August 6, 2026 05:17
@luiscape
luiscape force-pushed the luis/allign-clocks-in-capprofiling branch from 055c868 to 9c529bd Compare August 8, 2026 14:56
GPU profilers (Nsight Systems/CUPTI) build their trace timeline from
CLOCK_MONOTONIC_RAW. gVisor serves CLOCK_MONOTONIC_RAW as an alias of
CLOCK_MONOTONIC, this will drift from CLOCK_MONOTONIC_RAW breaking profiling.

When nvproxy is enabled with `capProfiling`, this change exposes
CLOCK_MONOTONIC_RAW as a clock distinct from CLOCK_MONOTONIC that tracks the
host's CLOCK_MONOTONIC_RAW (absolute, unadjusted). CLOCK_MONOTONIC is
unchanged. When profiling is disabled, CLOCK_MONOTONIC_RAW continues to alias
CLOCK_MONOTONIC as before.
@luiscape
luiscape force-pushed the luis/allign-clocks-in-capprofiling branch from 9c529bd to b12dd08 Compare August 8, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants