diff --git a/changelog.d/8221-descriptor-keys-edge.md b/changelog.d/8221-descriptor-keys-edge.md new file mode 100644 index 0000000000..6b57508bed --- /dev/null +++ b/changelog.d/8221-descriptor-keys-edge.md @@ -0,0 +1,94 @@ +### `fix(gc/shape)`: root and rewrite the keys edge from the ShapeId descriptor, not `ObjectHeader.keys_array` (#8112) + +`ObjectHeader.keys_array` was the last blocker for #8047's header shrink. +#8086 made the `ShapeId` descriptor authoritative for the keys *value*, but the +header word was still the only thing that **rooted** the keys array and the +only **rewritable location** the evacuator could hand to a slot visitor — +so deleting it would have unrooted every keys array in the heap. This lands the +missing GC protocol. `ObjectHeader`'s size is unchanged; #8047 shrinks it. + +**The edge now lives in the descriptor.** `ShapeDescriptor` records are BOXED, +so each record's address is fixed for its lifetime, and a lifted descriptor +carries `record` — the address it came from — alongside the `keys` snapshot. +`object::gc_shape_keys_edge_slot` hands the collector `&mut record.keys`, an +ordinary child slot it marks through and rewrites in place. The address rides +on the descriptor `gc_child_slots` already resolved for the receiver, so the +edge costs no extra shape-table probe (#8122's one-probe rule). + +`synchronize_live_object_shape_descriptor_after_header_visit` is **deleted**, +along with the fact-capture block in `visit_gc_layout_slot_descriptors` that +fed it. That callback existed only because the header was the strong edge and +the descriptor a weak copy to be repaired from it, under exact-facts validation, +once per traced receiver whose keys array had moved. With the descriptor holding +the edge there is nothing to reconcile — and nothing in the slot visitor reads +`keys_array` for a fact any more, which is what makes #8047 a deletion rather +than a rewrite. The header word is demoted to a derived mirror the collector +refreshes and keeps forwarding. + +**Liveness is a real ephemeron gate, and it had to be.** The obvious +simplification — emit the descriptor edge once per TRACED receiver and let +per-object liveness be the ephemeron relation — is *wrong for a generational +minor*, and `PERRY_GC_VERIFY_EVACUATION` said so on the first end-to-end run: + +``` +old-young-edge-verifier failed: missing_edges=1 + parent=0x…7b8 type=object old_arena=true marked=false + slot=0x…680 child=0x…308 child_type=array slot_page_ever_dirty=false +``` + +A minor never enumerates old objects, so an old carrier's edge is never emitted; +and the record is SHARED, so one sibling's rewrite creates an old→young edge for +a parent the minor never visits and which no per-parent remembered-set page can +describe. The gate is therefore per-descriptor state: `old_carrier`, set by the +slot visitor whenever it observes the shape on an old-generation receiver, and +consulted by `scan_shape_table_rekey_mut`, which roots exactly those records and +leaves the rest to metadata rewriting. A full trace enumerates every live object, +so `rotate_old_carrier_epoch_after_full_trace` recomputes the gate from what that +trace saw — the gate over-approximates by at most one full collection, which is +the generational contract, and never becomes unconditional rooting. + +`gc/shape_keys_edge.rs` (its own file: `barrier/mod.rs` is at 1995 lines and +`cycle.rs` at 1991) answers the one question the old→young verifier has to ask +about a shared word — coverage for it is the shape table's root, not any +parent's page. + +**Census first, per the issue's requirement (2) framing.** Peak descriptor +population over the 14 `gc_ratchet` probes plus a 1 MB JSON-parse kernel is +**~4 250 descriptors naming ~1 150 distinct keys arrays**, dominated by a fixed +bootstrap cohort; steady-state workload contribution is 1–229. So the protocol +is not a scaling problem, and the issue's requirement (1) — stable-address +descriptor storage — turns out to be **necessary after all**: the lift/write-back +the metadata pass already performs covers the *rewrite* only, never a strong, +rewritable edge. + +**Validation.** `shape_churn.ts` (400 rounds × 400 records over 7 shapes, a +120-record retained window re-read by name after every collection window) under +`PERRY_GC_SCHEDULE_RATE` + `PERRY_GC_FORCE_EVACUATE` + `PERRY_GC_VERIFY_EVACUATION` ++ `PERRY_GC_PROTECT_FROMSPACE`, with the instruments proven armed +(`[gc-fromspace-protect] retired_set=#N`, `copied_objects` and +`promoted_objects` both non-zero over tens of thousands of copying minors). + +`gc/tests/shape_keys_descriptor_edge.rs` adds four fixtures, all gating on +`copied_objects > 0` **and** on the receiver having actually moved: + +* the descriptor record — not the header word — is enumerated as a child slot, + and siblings of one shape share exactly one edge; +* with the header mirror suppressed, a keys array reachable only through the + descriptor survives a copying minor and the record is rewritten to the live + array (the #8047 rehearsal); +* **sabotage arm**: with the descriptor edge suppressed *as well*, the same + workload leaves the record stale or pruned — so a green run of the previous + fixture cannot be satisfied by something other than the edge under test; +* a keys array whose last carrier died is still reclaimed — an immortality bug + is as real as a use-after-free. + +The suppression switches are `#[cfg(test)]` thread-locals, not env knobs: the +GC-knob kill policy requires a required CI arm for every shipped knob's +off-state, and neither state may be reachable in a shipped binary. + +`scripts/shape_descriptor_census.py` gains four authority surfaces and three +sabotage arms: un-boxing the record, un-gating the rooting arm, visiting the +derived mirror before the authoritative edge, and reading the mirror for a fact +are each red. The raw `keys_array` callsite count drops 183 → 180. +`object/shapes.rs` reached 2048 lines, so its three unit suites moved verbatim +to `object/shapes_tests.rs`. diff --git a/crates/perry-runtime/src/gc/dead_owner.rs b/crates/perry-runtime/src/gc/dead_owner.rs index 2c89b7a0e7..62a53466a7 100644 --- a/crates/perry-runtime/src/gc/dead_owner.rs +++ b/crates/perry-runtime/src/gc/dead_owner.rs @@ -204,6 +204,14 @@ fn owner_type_matches(header: &GcHeader, expected_obj_type: Option) -> bool /// entry, before any header is finalized or freed, so deadness probes read /// intact headers. pub(super) fn prune_dead_owner_side_tables_post_trace(full_trace: bool) { + if full_trace { + // #8112: a full trace enumerated every live object, so the old-carrier + // notes it accumulated are exactly the shapes old objects still carry. + // Adopting them here is what lets the gate SHED a shape — minors only + // ever add notes, so without this the table's root set would grow + // monotonically and no keys array would ever be reclaimed again. + crate::object::shapes::rotate_old_carrier_epoch_after_full_trace(); + } let probe = PostTraceProbe::new(full_trace); fan_out( &|addr| probe.owner_is_dead(addr, None), diff --git a/crates/perry-runtime/src/gc/layout_slot_visit.rs b/crates/perry-runtime/src/gc/layout_slot_visit.rs index d2579a8f63..673c04e571 100644 --- a/crates/perry-runtime/src/gc/layout_slot_visit.rs +++ b/crates/perry-runtime/src/gc/layout_slot_visit.rs @@ -15,80 +15,48 @@ pub(super) unsafe fn visit_gc_layout_slot_descriptors( visit: &mut dyn FnMut(GcMutableSlotDescriptor), ) { let mut child_slots = gc_child_slots(header); - // Capture the authoritative descriptor facts. `gc_child_slots` has already - // copied the descriptor's keys edge into the compatibility header scratch - // slot; a copying visit can rewrite that slot, after which the descriptor - // table is updated below. - let object_shape_facts = if (*header).obj_type == GC_TYPE_OBJECT { - let obj = (header as *mut u8).add(GC_HEADER_SIZE) as *mut crate::object::ObjectHeader; - // #8122: the descriptor `gc_child_slots` resolved for this receiver. - // Nothing between that probe and here can allocate or mutate the - // shape table, so it is the same value a fresh probe would return. - let descriptor = child_slots.object_shape; - let old_keys = descriptor - .map(|facts| facts.keys as usize as *mut crate::array::ArrayHeader) - .unwrap_or((*obj).keys_array); - let live_inline_slot_count = descriptor - .map(|facts| facts.live_inline_slot_count) - // #8113: 0, not a second descriptor probe. `unwrap_or` is EAGER, - // so re-deriving the bound here cost a whole extra shape-table - // lookup on every call — and the bound has no other source now, so - // the fallback could only ever have returned 0 anyway. - .unwrap_or(0); - if old_keys.is_null() { - Some((obj, 0, 0, live_inline_slot_count)) - } else if crate::value::addr_class::try_read_tracked_gc_header(old_keys as usize) - .is_some_and(|keys_header| (*keys_header.as_ptr()).obj_type == GC_TYPE_ARRAY) - { - // A forwarded tracked array still carries GC_TYPE_ARRAY in - // its from-space header. The length helper follows that stub, - // so a sibling whose shared keys edge was already rewritten - // can still validate against the descriptor's new pointer. - Some(( - obj, - old_keys as u64, - descriptor - .map(|facts| facts.logical_key_count) - .unwrap_or_else(|| { - crate::array::keys_array_len_capped_to_capacity(old_keys) as u32 - }), - live_inline_slot_count, - )) - } else { - // Do not dereference corrupt/unmapped header words merely because - // their sibling word happens to look like a ShapeId. The - // authoritative header edge below is still enumerated; only - // redundant descriptor synchronization is skipped. - None + // #8112: the authoritative ordered-keys edge, taken from the descriptor + // `gc_child_slots` already resolved for this receiver. It is the boxed + // record's OWN `keys` word, so the collector marks through it and rewrites + // it in place — the descriptor is the root and the rewritable location, + // and the header word below is a derived mirror. + // + // Never enumerate the HashMap BUCKET as a GC slot: dirty-page work may + // retain enumerated slot addresses across budgeted resumptions, during + // which descriptor insertion can reallocate the table. Boxing the record + // is what answers that — the bucket moves, the record does not. + // + // Liveness is an ephemeron relation with two halves. A YOUNG carrier is + // traced, so emitting the edge here marks the keys array exactly while + // that receiver lives. An OLD carrier is not traced by a minor at all — + // and because the record is SHARED, one sibling's rewrite creates an + // old→young edge for a parent the minor never visits, which no per-parent + // remembered-set page can describe. That half is the `old_carrier` gate + // armed below and rooted by `shapes::scan_shape_table_rekey_mut`. + // `PERRY_GC_VERIFY_EVACUATION` is what established the second half is + // needed: without it the verifier aborts on a `slot_page_ever_dirty=false` + // old→young edge through this word. + let shape_keys_edge = if (*header).obj_type == GC_TYPE_OBJECT { + // A receiver the minor will not enumerate for itself arms the table's + // ephemeron gate. The test is "not in the nursery", not "in old-gen": + // a `gc_malloc`'d large object and an immortal bootstrap resident are + // both outside the young generation and both invisible to a minor, and + // over-arming the gate only costs one extra rooted record until the + // next full trace recomputes it. A to-space survivor IS nursery, so a + // young carrier still relies on the edge emitted just below. + let user_ptr = (header as *mut u8).add(GC_HEADER_SIZE); + if !crate::arena::pointer_in_nursery(user_ptr as usize) { + crate::object::shapes::note_old_generation_carrier(child_slots.object_shape); } + crate::object::gc_shape_keys_edge_slot(child_slots.object_shape) } else { None }; if let Some(slot) = child_slots.take_prefix_child_slot() { visit(fixed_slot(slot).with_layout(HeapChildSlotReadKind::Prefix)); } - // #8067: the header keys slot above is the sole strong edge. Once its - // visitor callback has run, mirror an immediate rewrite into the weak - // descriptor. Never enumerate the HashMap bucket as a GC slot: dirty-page - // work may retain enumerated slot addresses across budgeted resumptions, - // during which descriptor insertion can reallocate the table. A deferred - // visitor leaves old==new here; the metadata forwarding pass repairs it - // after copying. RegExp uses its dedicated GC slot kind and never enters - // the ObjectHeader branch above. - if let Some((obj, old_keys, logical_key_count, live_inline_slot_count)) = object_shape_facts { - let new_keys = (*obj).keys_array as u64; - // Mark, verify, and deferred dirty scans leave the header edge - // unchanged. Only a copying rewrite needs to borrow and update the - // weak descriptor table. - if new_keys != old_keys { - crate::object::shapes::synchronize_live_object_shape_descriptor_after_header_visit( - obj, - old_keys, - new_keys, - logical_key_count, - live_inline_slot_count, - ); - } + if let Some(slot) = shape_keys_edge { + visit(fixed_slot(slot).with_layout(HeapChildSlotReadKind::Prefix)); } if let Some(slot) = child_slots.take_meta_child_slot() { visit(fixed_slot(slot).with_layout(HeapChildSlotReadKind::Prefix)); diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 01a5dc0a0f..61ed52223d 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -79,6 +79,11 @@ use root_words::*; mod layout; mod layout_slot_visit; use layout_slot_visit::*; +/// #8112: the one question the remembered set asks about the shape table's +/// shared keys word. Its own file because both `barrier/mod.rs` (1995 lines) +/// and `cycle.rs` (1991) are at the 2000-line cap. +mod shape_keys_edge; +use shape_keys_edge::slot_is_shared_shape_keys_word; /// #7510: the per-object slot-layout side tables and the emptiness flag that /// keeps them off the allocation, store, death and trace paths. Split out of /// `layout.rs` so it stays under the repo's 2000-line-per-file cap. diff --git a/crates/perry-runtime/src/gc/shape_keys_edge.rs b/crates/perry-runtime/src/gc/shape_keys_edge.rs new file mode 100644 index 0000000000..8bb99f8440 --- /dev/null +++ b/crates/perry-runtime/src/gc/shape_keys_edge.rs @@ -0,0 +1,53 @@ +//! #8112 — the shape table's shared keys edge, seen from old→young +//! verification. +//! +//! `object/shapes.rs` owns the edge itself; this module owns the one question +//! `verify_old_young_parent_slots_covered` has to ask about it. +//! +//! A receiver's ordered-keys edge is the `keys` word of its boxed +//! `ShapeDescriptor`, and every sibling of the shape enumerates that same word. +//! So it is not a slot any ONE parent owns, and per-parent coverage — "is this +//! parent's page in the remembered set?" — is the wrong question to ask of it: +//! +//! * one sibling's rewrite changes the edge of every other, including old +//! receivers a minor never visits and never gets a chance to remember; +//! * the word lives outside the GC heap, so no page a barrier can dirty +//! contains it; +//! * and even a per-parent entry that DOES get recorded (the promoted-object +//! rebuild still records one whenever it walks a carrier, and that entry is +//! useful — it re-enters the owner next cycle) cannot speak for the parents +//! it is not attached to. When its own parent dies, the surviving old +//! carriers are left with an edge nothing describes. +//! +//! Recording is therefore left alone and only the VERIFIER skips the word. +//! What actually covers it is the shape table's own root scanner under the +//! `old_carrier` gate (`shapes::scan_shape_table_rekey_mut`). Skipping without +//! that gate would be a missing-edge bug; the gate without the skip is the +//! `slot_page_ever_dirty=false` abort `PERRY_GC_VERIFY_EVACUATION` produced +//! while this issue was being built. + +use super::{GcHeader, GC_HEADER_SIZE, GC_TYPE_OBJECT}; + +/// Is `slot` the shared shape-table `keys` word of `parent_header`'s receiver? +/// +/// The cheap term comes first and is false for every ordinary slot: an object +/// field, an array element and a closure capture all live inside the GC heap, +/// while a descriptor record is a `Box` on the Rust heap. Only then does this +/// pay for a shape probe. +#[inline] +pub(super) unsafe fn slot_is_shared_shape_keys_word( + parent_header: *mut GcHeader, + slot: *mut u64, +) -> bool { + if parent_header.is_null() || slot.is_null() || (*parent_header).obj_type != GC_TYPE_OBJECT { + return false; + } + if crate::arena::classify_heap_generation(slot as usize) + != crate::arena::HeapGeneration::Unknown + { + return false; + } + let obj = (parent_header as *mut u8).add(GC_HEADER_SIZE) as *const crate::object::ObjectHeader; + let shape_id = crate::object::shapes::object_shape_stamp(obj); + crate::object::shapes::shape_id_owns_keys_slot(shape_id, slot) +} diff --git a/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs b/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs index e2ecf93076..821e8e2091 100644 --- a/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs +++ b/crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs @@ -1013,9 +1013,11 @@ fn test_dead_shape_descriptor_churn_returns_to_baseline_after_full_gc() { ); } -/// #8067: the header is the sole strong edge; a live stamped object's scan -/// synchronizes its weak descriptor mirror. Two siblings share one descriptor; -/// after copied-minor evacuation both headers and that descriptor must agree. +/// #8112: the DESCRIPTOR record is the strong edge and the rewritten location; +/// the header word is a derived mirror the collector also keeps valid until +/// #8047 deletes it. Two siblings share one descriptor — and therefore one +/// edge — so after copied-minor evacuation both mirrors and that record must +/// agree. #[test] fn test_shared_live_shape_descriptor_survives_and_rekeys_once() { let _guard = CopyingNurseryTestGuard::new(2); @@ -1035,9 +1037,19 @@ fn test_shared_live_shape_descriptor_survives_and_rekeys_once() { (*b).parent_class_id = id; } assert_eq!( - crate::gc::test_gc_rewrite_slot_count(a as usize), - Some(1), - "a stamped object must enumerate only its authoritative header keys slot" + crate::gc::test_gc_rewrite_slot_addresses(a as usize), + Some(vec![ + unsafe { std::ptr::addr_of_mut!((*a).keys_array) } as usize, + crate::object::shapes::shape_descriptor_keys_slot(id) + .expect("a table-resident descriptor exposes its keys word") as usize, + ]), + "#8112: a stamped object enumerates its derived header mirror AND the \ + authoritative descriptor record" + ); + assert_eq!( + crate::gc::test_gc_rewrite_slot_addresses(b as usize).map(|slots| slots[1]), + crate::gc::test_gc_rewrite_slot_addresses(a as usize).map(|slots| slots[1]), + "#8112: siblings of one shape must share ONE keys edge" ); js_shadow_slot_set(0, ptr_bits(a as usize)); js_shadow_slot_set(1, ptr_bits(b as usize)); @@ -1066,12 +1078,20 @@ fn test_shared_live_shape_descriptor_survives_and_rekeys_once() { js_shadow_slot_set(1, 0); } -/// #8074 review: a forwarded array's from-space payload contains its forwarding -/// address, not a usable `(length, capacity)` pair. Descriptor fact capture -/// must resolve both values from the live array or the stale capacity word can -/// truncate the logical count and make an immediate header rewrite fail closed. +/// #8074 review, retired by #8112 and kept as its regression guard. +/// +/// The hazard was real under the old model: a forwarded array's from-space +/// payload holds its forwarding address, not a usable `(length, capacity)` +/// pair, and the post-visit callback CAPTURED descriptor facts from the header +/// edge — so a stale capacity word could truncate the logical count and make +/// the rewrite fail closed. There is no fact capture any more. The visitor +/// writes the record it was handed, reading no `ArrayHeader` field at all, so +/// this fixture now asserts the structural reason the hazard cannot recur: +/// rewriting through the enumerated descriptor slot lands exactly, with every +/// other fact untouched, while the array's own length/capacity words say +/// something impossible. #[test] -fn test_forwarded_keys_capacity_preserves_immediate_descriptor_sync() { +fn test_forwarded_keys_capacity_cannot_disturb_the_descriptor_rewrite() { let _guard = GcTestIsolationGuard::new(); crate::object::shapes::test_clear_shape_table(); @@ -1079,7 +1099,7 @@ fn test_forwarded_keys_capacity_preserves_immediate_descriptor_sync() { let live_keys = unsafe { alloc_nursery_test_array() }; // `set_forwarding_address` stores this pointer over the old length/capacity // pair. Pick a logical count one above the resulting stale capacity word, - // so the pre-fix mixed old/new read deterministically truncates it. + // so a re-derivation from the array would deterministically truncate it. let stale_capacity = ((live_keys as u64) >> 32) as u32; let logical_key_count = stale_capacity .checked_add(1) @@ -1106,17 +1126,18 @@ fn test_forwarded_keys_capacity_preserves_immediate_descriptor_sync() { assert_eq!((*old_keys).capacity, stale_capacity); assert!( (*old_keys).capacity < logical_key_count, - "test premise: the stale capacity must truncate the live count" + "test premise: the stale capacity would truncate the live count" ); } let owner_header = unsafe { header_from_user_ptr(owner.cast()) } as *mut GcHeader; - let header_keys_slot = unsafe { std::ptr::addr_of_mut!((*owner).keys_array) as *mut u64 }; + let descriptor_keys_slot = crate::object::shapes::shape_descriptor_keys_slot(id) + .expect("a table-resident descriptor exposes its keys word"); let mut rewritten = 0usize; unsafe { visit_gc_layout_slot_descriptors(owner_header, &mut |descriptor| { descriptor.visit_slots(&mut |slot| { - if slot.slot == header_keys_slot { + if slot.slot == descriptor_keys_slot { *slot.slot = live_keys as u64; rewritten += 1; } @@ -1125,12 +1146,16 @@ fn test_forwarded_keys_capacity_preserves_immediate_descriptor_sync() { } assert_eq!( rewritten, 1, - "the authoritative header edge must be rewritten once" + "the authoritative descriptor edge must be enumerated exactly once" ); let descriptor = crate::object::shapes::shape_descriptor_by_id(id) .expect("rewritten live descriptor disappeared"); assert_eq!(descriptor.keys, live_keys as u64); - assert_eq!(descriptor.logical_key_count, logical_key_count); + assert_eq!( + descriptor.logical_key_count, logical_key_count, + "#8112: nothing in the visit re-derives a count from the array, so the \ + stale capacity word cannot truncate it" + ); assert_eq!(descriptor.live_inline_slot_count, 0); unsafe { @@ -1140,11 +1165,13 @@ fn test_forwarded_keys_capacity_preserves_immediate_descriptor_sync() { crate::object::shapes::test_clear_shape_table(); } -/// #8067 release fail-closed guard: descriptor fact capture must classify the -/// keys word before reading ArrayHeader fields. A live GC_TYPE_OBJECT can -/// carry a corrupt header edge and a real ShapeId at the same time; the -/// authoritative header edge remains visible, but descriptor synchronization -/// must skip without dereferencing that word. +/// #8067 release fail-closed guard, restated for #8112. A live GC_TYPE_OBJECT +/// can carry a corrupt header word and a real ShapeId at the same time. Under +/// the old model the collector captured descriptor FACTS from that word, so it +/// had to classify it before dereferencing. The descriptor is now the source, +/// so the corrupt word is never read for facts at all — it is only a mirror +/// slot the collector still rewrites, and the unrelated descriptor must come +/// through untouched. #[test] fn test_shape_descriptor_skips_a_plausible_misaligned_corrupt_keys_word() { let _guard = GcTestIsolationGuard::new(); @@ -1167,8 +1194,9 @@ fn test_shape_descriptor_skips_a_plausible_misaligned_corrupt_keys_word() { assert_eq!( crate::gc::test_gc_rewrite_slot_count(owner as usize), - Some(1), - "only the authoritative corrupt header slot may be enumerated" + Some(2), + "#8112: the corrupt mirror stays enumerated (the collector still has to \ + rewrite it) alongside the authoritative descriptor record" ); let descriptor = crate::object::shapes::shape_descriptor_by_id(id) .expect("invalid header facts must not retire the unrelated descriptor"); @@ -1180,10 +1208,13 @@ fn test_shape_descriptor_skips_a_plausible_misaligned_corrupt_keys_word() { crate::object::shapes::test_drop_shape_descriptors(valid_keys as usize); } -/// #8067: DirtyHeaderSlotScan retains enumerated raw slot pointers between -/// budgeted work units. Descriptor-table growth in that mutator window must -/// not invalidate any saved pointer, so a stamped object's enumeration may -/// contain its stable ObjectHeader slot but never a HashMap bucket address. +/// DirtyHeaderSlotScan retains enumerated raw slot pointers between budgeted +/// work units, and descriptor-table growth in that mutator window must not +/// invalidate any saved pointer. #8067 answered that by refusing to enumerate +/// the descriptor at all; #8112 answers it by BOXING the record, so the map +/// may rehash freely while every enumerated `keys` word stays put. This test +/// is the difference between those two answers: it saves the enumeration, +/// forces a thousand insertions, and demands the identical addresses back. #[test] fn test_deferred_shape_slot_enumeration_survives_descriptor_table_reallocation() { let _guard = GcTestIsolationGuard::new(); @@ -1204,10 +1235,14 @@ fn test_deferred_shape_slot_enumeration_survives_descriptor_table_reallocation() let saved_slots = crate::gc::test_gc_rewrite_slot_addresses(owner as usize) .expect("tracked object must have a rewrite descriptor"); let header_keys_slot = unsafe { std::ptr::addr_of_mut!((*owner).keys_array) as *mut u64 }; + let descriptor_keys_slot = crate::object::shapes::shape_descriptor_keys_slot(id) + .expect("a table-resident descriptor exposes its keys word") + as usize; assert_eq!( saved_slots, - vec![header_keys_slot as usize], - "deferred work retained a descriptor-table bucket address" + vec![header_keys_slot as usize, descriptor_keys_slot], + "#8112: the enumeration is the derived mirror plus the authoritative \ + descriptor record" ); for i in 0..1024usize { @@ -1220,8 +1255,14 @@ fn test_deferred_shape_slot_enumeration_survives_descriptor_table_reallocation() .expect("rooted object must remain enumerable after table growth"); assert_eq!( slots_after, - vec![header_keys_slot as usize], - "descriptor-table growth changed the stable authoritative slot address" + vec![header_keys_slot as usize, descriptor_keys_slot], + "descriptor-table growth moved a keys word that deferred dirty-page \ + work may still be holding — the box did not keep the record put" + ); + assert_eq!( + crate::object::shapes::shape_descriptor_keys_slot(id).map(|slot| slot as usize), + Some(descriptor_keys_slot), + "1024 insertions rehashed the map and moved the record with it" ); js_shadow_slot_set(0, 0); diff --git a/crates/perry-runtime/src/gc/tests/mod.rs b/crates/perry-runtime/src/gc/tests/mod.rs index ac4862fa19..d39d3ce047 100644 --- a/crates/perry-runtime/src/gc/tests/mod.rs +++ b/crates/perry-runtime/src/gc/tests/mod.rs @@ -45,6 +45,7 @@ mod scan_fallback; mod schedule; mod shadow_stack_ops; mod shape_descriptor_authority; +mod shape_keys_descriptor_edge; mod smoke; mod step_bounds; pub(super) mod support; diff --git a/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs new file mode 100644 index 0000000000..c0c50b6828 --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/shape_keys_descriptor_edge.rs @@ -0,0 +1,276 @@ +//! #8112 — the ordered-keys edge is rooted and rewritten from the ShapeId +//! descriptor, not from `ObjectHeader::keys_array`. +//! +//! Invariant under test: +//! +//! > A keys array is marked, and its pointer rewritten after a move, through +//! > the boxed `ShapeDescriptor` record named by a live receiver's ShapeId. +//! > The header word is a derived mirror: deleting it (#8047) must unroot +//! > nothing and leave nothing stale. +//! +//! Why this needs a fixture rather than an argument. An unrooted-but-reachable +//! keys array is the quietest failure the collector has: nothing is missing at +//! the collection, and the damage surfaces cycles later as a wrong property +//! name or a `TypeError` at an unrelated call site. So every test below runs a +//! REAL copying minor, gates on `copied_objects > 0` (a cycle that moved +//! nothing exercises none of this) and on the receiver's address having +//! actually changed, and then asserts a discriminating quantity — the +//! descriptor's `keys` word before versus after. +//! +//! And the detector is sabotage-tested. `keys_edge_sabotage_is_detected` +//! re-runs the identical workload with the descriptor edge ALSO suppressed and +//! asserts the record comes back stale or pruned. Without that arm, a green run +//! here could not distinguish "the edge works" from "the keys array was kept +//! alive by something else entirely". + +use super::super::*; +use super::support::{collect_minor_trace, ptr_bits, CopyingNurseryTestGuard}; +use crate::object::shapes; + +/// Facts the assertions compare, read exclusively through the descriptor — +/// never through `ObjectHeader::keys_array`, which these tests suppress. +#[derive(Clone, Copy, Debug)] +struct KeysEdge { + keys: u64, + forwarded: bool, + is_array: bool, + logical_key_count: u32, + old_carrier: bool, +} + +unsafe fn keys_edge_of(obj: *const crate::ObjectHeader) -> Option { + let descriptor = shapes::object_shape_descriptor(obj)?; + let keys = descriptor.keys; + let (forwarded, is_array) = if keys == 0 { + (false, false) + } else { + match crate::value::addr_class::try_read_tracked_gc_header(keys as usize) { + Some(header) => ( + (*header.as_ptr()).gc_flags & GC_FLAG_FORWARDED != 0, + (*header.as_ptr()).obj_type == GC_TYPE_ARRAY, + ), + None => (false, false), + } + }; + Some(KeysEdge { + keys, + forwarded, + is_array, + logical_key_count: descriptor.logical_key_count, + old_carrier: descriptor.old_carrier, + }) +} + +/// Build a two-key object in shadow slot `slot`, returning nothing: every later +/// read re-derives the receiver from the slot, because each store below can +/// allocate and therefore collect. +fn build_two_key_object(slot: u32, prefix: &[u8]) { + js_shadow_slot_set( + slot, + ptr_bits(crate::object::js_object_alloc(0, 2) as usize), + ); + for suffix in [b'a', b'b'] { + let mut name = prefix.to_vec(); + name.push(suffix); + let key = crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32); + let obj = (js_shadow_slot_get(slot) & POINTER_MASK) as *mut crate::ObjectHeader; + crate::object::js_object_set_field_by_name( + obj, + key, + crate::value::js_nanbox_pointer(key as i64), + ); + } +} + +/// The collector's own rewrite enumeration for `user_ptr`. +fn rewrite_slots(user_ptr: usize) -> Vec { + crate::gc::test_gc_rewrite_slot_addresses(user_ptr) + .expect("a live object has a rewrite-slot enumeration") +} + +#[test] +fn the_descriptor_record_is_enumerated_as_a_child_slot() { + let _guard = CopyingNurseryTestGuard::new(1); + build_two_key_object(0, b"e8112_enum_"); + let obj = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + + let descriptor = unsafe { shapes::object_shape_descriptor(obj) } + .expect("a published object has an authoritative descriptor"); + let keys_slot = descriptor + .keys_slot() + .expect("a table-resident descriptor exposes the address of its own keys word") + as usize; + + let slots = rewrite_slots(obj as usize); + assert!( + slots.contains(&keys_slot), + "#8112: the collector must enumerate the DESCRIPTOR's keys word as a \ + child slot; it reported {slots:?} and the record lives at {keys_slot:#x}" + ); + assert!( + keys_slot < obj as usize + || keys_slot >= obj as usize + std::mem::size_of::(), + "#8112: the enumerated edge must be the descriptor record, not the \ + header mirror inside the receiver" + ); + + // Siblings of one shape hand the collector the SAME edge. That is the + // ephemeron relation stated structurally: one shape, one keys array, one + // slot, marked exactly while some live receiver still carries the id. + build_two_key_object(0, b"e8112_enum_"); + let sibling = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + let sibling_descriptor = + unsafe { shapes::object_shape_descriptor(sibling) }.expect("the sibling is published too"); + assert_eq!( + sibling_descriptor.keys_slot(), + descriptor.keys_slot(), + "two receivers of one shape must share one descriptor record" + ); +} + +/// Run the shared workload under `suppression` and report what the descriptor +/// says afterwards. Returns `None` when the cycle was not discriminating (no +/// copy, or the receiver did not move), so a caller can fail loudly instead of +/// passing vacuously. +fn collect_and_report(suppress_edge: bool) -> Option<(KeysEdge, KeysEdge)> { + build_two_key_object(0, b"e8112_move_"); + let obj_before = (js_shadow_slot_get(0) & POINTER_MASK) as usize; + let before = unsafe { keys_edge_of(obj_before as *const crate::ObjectHeader) } + .expect("the receiver is published before the collection"); + assert!( + crate::arena::pointer_in_nursery(before.keys as usize), + "#8112 test setup: the keys array must be YOUNG so a copying minor \ + actually has to move it (keys at {:#x})", + before.keys + ); + + assert!( + !before.old_carrier, + "#8112 test setup: this shape must have YOUNG carriers only. Once the \ + old-carrier gate is armed, the shape table roots the keys array on its \ + own and the sabotage arm below would stop discriminating — it would \ + pass because the TABLE kept the array alive, not because the edge \ + under test did." + ); + + // Rooted young canary: keeps `copied_objects > 0` true independently of the + // subject, so the liveness gate below cannot be satisfied by the very edge + // under test. + js_shadow_slot_set(1, ptr_bits(crate::object::js_object_alloc(0, 0) as usize)); + + let _suppression = if suppress_edge { + shapes::TestKeysEdgeSuppression::without_any_keys_edge() + } else { + shapes::TestKeysEdgeSuppression::without_header_mirror() + }; + let trace = collect_minor_trace(GcTriggerKind::Direct); + assert!( + trace.copying_nursery.copied_objects > 0, + "#8112 requires a COPYING minor; a cycle that moved nothing exercises \ + no rewrite at all (copied_objects=0)" + ); + + let obj_after = (js_shadow_slot_get(0) & POINTER_MASK) as usize; + if obj_after == obj_before { + return None; + } + let after = unsafe { keys_edge_of(obj_after as *const crate::ObjectHeader) }; + Some(( + before, + after.unwrap_or(KeysEdge { + keys: 0, + forwarded: false, + is_array: false, + logical_key_count: 0, + old_carrier: false, + }), + )) +} + +#[test] +fn a_keys_array_reachable_only_through_the_descriptor_survives_and_is_rewritten() { + let _guard = CopyingNurseryTestGuard::new(2); + let (before, after) = collect_and_report(false) + .expect("#8112: the receiver must move for this cycle to be discriminating"); + + assert_ne!( + after.keys, before.keys, + "#8112: the receiver moved but the descriptor still names the \ + from-space keys array at {:#x} — the record was not rewritten", + before.keys + ); + assert!( + !after.forwarded, + "#8112: the descriptor names a FORWARDED header at {:#x}; the rewrite \ + followed only one hop", + after.keys + ); + assert!( + after.is_array, + "#8112: the descriptor no longer names an array after the move \ + (keys {:#x})", + after.keys + ); + assert_eq!( + after.logical_key_count, before.logical_key_count, + "#8112: the rewrite must not disturb the descriptor's other facts" + ); +} + +#[test] +fn keys_edge_sabotage_is_detected() { + let _guard = CopyingNurseryTestGuard::new(2); + let (before, after) = collect_and_report(true) + .expect("#8112: the receiver must move for this cycle to be discriminating"); + + // With BOTH edges gone, nothing marked the keys array and nothing rewrote + // the record. The detector above must be able to SEE that: either the + // descriptor still names the from-space address, or the dead-owner prune + // removed the descriptor outright. What it must NOT be is a live, moved, + // correctly-named array — that would mean the assertions in the test above + // are satisfied by something other than the descriptor edge. + let correctly_rewritten = after.keys != 0 + && after.keys != before.keys + && !after.forwarded + && after.is_array + && after.logical_key_count == before.logical_key_count; + assert!( + !correctly_rewritten, + "#8112 SABOTAGE ARM: with the descriptor edge and the header mirror \ + both suppressed, the keys array was still rooted and rewritten \ + ({:#x} -> {:#x}). Something other than the edge under test is keeping \ + it alive, so a green run of \ + `a_keys_array_reachable_only_through_the_descriptor_survives_and_is_rewritten` \ + proves nothing.", + before.keys, after.keys + ); +} + +#[test] +fn a_keys_array_whose_last_carrier_died_is_still_reclaimed() { + let _guard = CopyingNurseryTestGuard::new(2); + build_two_key_object(0, b"e8112_dead_"); + let obj = (js_shadow_slot_get(0) & POINTER_MASK) as *mut crate::ObjectHeader; + let shape_id = unsafe { shapes::object_shape_stamp(obj) }; + assert!(shapes::shape_descriptor_by_id(shape_id).is_some()); + + // Drop the only root. An immortality bug is as real as a use-after-free: + // if the descriptor table ever became an UNCONDITIONAL root for `keys`, + // this descriptor would outlive every object that ever carried it and the + // dead-key prune would go circular ("is the keys array dead?" can never be + // yes once the asker roots it). + js_shadow_slot_set(0, crate::value::TAG_UNDEFINED); + js_shadow_slot_set(1, ptr_bits(crate::object::js_object_alloc(0, 0) as usize)); + let trace = collect_minor_trace(GcTriggerKind::Direct); + assert!( + trace.copying_nursery.copied_objects > 0, + "#8112: a cycle that copied nothing cannot demonstrate reclamation" + ); + + assert!( + shapes::shape_descriptor_by_id(shape_id).is_none(), + "#8112: the descriptor for a shape no live object carries any more must \ + be reclaimed by the dead-key prune; it survived, which is what \ + unconditional table rooting would look like" + ); +} diff --git a/crates/perry-runtime/src/gc/verify.rs b/crates/perry-runtime/src/gc/verify.rs index bbfeb6952d..cc378269bf 100644 --- a/crates/perry-runtime/src/gc/verify.rs +++ b/crates/perry-runtime/src/gc/verify.rs @@ -561,6 +561,13 @@ pub(super) unsafe fn verify_old_young_parent_slots_covered( if crate::weakref::is_weak_target_trace_slot(header, slot.slot) { return; } + // #8112: the shape table's shared keys word is not a slot this parent + // owns, so per-parent coverage is the wrong question to ask of it. + // `gc/shape_keys_edge.rs` says why; the table's `old_carrier` root is + // what covers it instead. + if slot_is_shared_shape_keys_word(header, slot.slot) { + return; + } slot.record_layout_read(); verify_old_young_slot_covered(snapshot, stats, header, slot.slot); }); diff --git a/crates/perry-runtime/src/object/gc_slots.rs b/crates/perry-runtime/src/object/gc_slots.rs index e13bf0cde5..144cbe17d9 100644 --- a/crates/perry-runtime/src/object/gc_slots.rs +++ b/crates/perry-runtime/src/object/gc_slots.rs @@ -1,15 +1,52 @@ use super::{shapes, ObjectHeader}; use crate::ArrayHeader; -/// The object's keys-array child slot, given the receiver's `ShapeDescriptor`. +/// The AUTHORITATIVE ordered-keys edge of a traced receiver (#8112). /// -/// #8122: the collector resolves the receiver's `ShapeDescriptor` ONCE per -/// traced object (`gc::layout::gc_child_slots`) and threads it through the -/// keys-slot, field-range, payload-mask and slot-visit steps, which used to -/// probe the shape table independently — five `shape_descriptor_by_id` -/// lookups per traced object, the top leaf of a traced in-place-promotion -/// cycle's profile. Callers therefore pass the descriptor rather than the -/// function re-deriving it. +/// This is the descriptor's own `keys` word, not a copy of it: the record is +/// boxed (`object::shapes::ShapeDescriptor`), so its address is fixed for the +/// record's lifetime and the collector can mark through it and rewrite it in +/// place like any other child slot. The address rides along on the descriptor +/// `gc::layout::gc_child_slots` already resolved for this receiver, so the +/// edge costs no extra shape-table probe (#8122's one-probe rule) and needs no +/// post-visit write-back callback. +/// +/// Siblings sharing a ShapeId hand the collector the SAME slot: one shape, one +/// edge. For a YOUNG carrier that is the whole liveness rule — the receiver is +/// traced, so the edge is emitted exactly while it lives. An OLD carrier needs +/// more, because a minor never enumerates it: the visitor also arms +/// `ShapeDescriptor::old_carrier`, and `shapes::scan_shape_table_rekey_mut` +/// roots the record for it. Emitting here regardless of generation is +/// deliberate — the alternative loses a shape whose only carrier is promoted +/// during the very drain that would have emitted it. +#[inline] +pub(crate) fn gc_shape_keys_edge_slot( + descriptor: Option, +) -> Option<*mut u64> { + #[cfg(test)] + if shapes::test_keys_edge_suppressed() { + // Sabotage arm: with BOTH this edge and the header mirror gone, a keys + // array has no root and no rewritable location at all. The fixtures use + // it to prove their detector fires — a green protected run then means + // the detector works, not that nothing was tried. + return None; + } + let descriptor = descriptor?; + if descriptor.keys == 0 { + return None; + } + descriptor.keys_slot() +} + +/// The object's keys-array MIRROR slot. +/// +/// #8112 demoted this word. It is no longer the strong edge and no longer the +/// scratch buffer the descriptor is repaired from — `gc_shape_keys_edge_slot` +/// above is both. What remains is an ABI mirror the mutator still loads +/// directly (codegen emits the load), so the collector refreshes it from the +/// authoritative record and keeps rewriting it across a move. #8047 deletes +/// the word, this function, and its call site together; nothing else has to +/// change, which is exactly what this issue had to establish. pub(crate) unsafe fn gc_keys_array_slot( obj: *mut ObjectHeader, descriptor: Option, @@ -17,18 +54,21 @@ pub(crate) unsafe fn gc_keys_array_slot( if obj.is_null() { return None; } + #[cfg(test)] + if shapes::test_keys_mirror_suppressed() { + // #8047 rehearsal: with the mirror gone, the descriptor edge is the + // only thing that can keep this receiver's keys array alive and + // correctly forwarded. `gc/tests/shape_keys_descriptor_edge.rs` runs + // real collections in this mode. + return None; + } if let Some(descriptor) = descriptor { - // Compatibility scratch slot: GC obtains the authoritative edge from - // the ShapeId descriptor, then lets the existing slot visitor rewrite - // it in place. #8047 can replace this scratch with a descriptor-table - // rewrite without changing the source of the edge. - // - // The descriptor lookup immediately precedes this collector-side - // materialization: no allocation or callback can change its `keys` - // edge before the visitor receives the slot. That ordering is what - // makes the descriptor authoritative while this legacy field remains - // only the mutable scratch location used for pointer rewriting. - // GC_STORE_AUDIT(ROOT): collector materializes the authoritative descriptor root into its compatibility rewrite slot. + // Mirror refresh, not an edge: a sibling traced earlier in this cycle + // may already have rewritten the shared record, in which case this + // receiver's word is stale and the slot visitor below would otherwise + // hand the collector a from-space address it has no forwarding record + // for. + // GC_STORE_AUDIT(ROOT): collector refreshes the derived keys mirror from the authoritative descriptor record. (*obj).keys_array = descriptor.keys as usize as *mut ArrayHeader; } if (*obj).keys_array.is_null() { diff --git a/crates/perry-runtime/src/object/mod.rs b/crates/perry-runtime/src/object/mod.rs index 5cd839d2de..ca34fd09ca 100644 --- a/crates/perry-runtime/src/object/mod.rs +++ b/crates/perry-runtime/src/object/mod.rs @@ -77,8 +77,8 @@ pub(crate) use field_get_set::scan_accessor_receiver_override_root_mut; mod field_set_by_name; mod gc_slots; pub(crate) use gc_slots::{ - gc_field_slot_range, gc_keys_array_slot, rebuild_array_layout_from_slots, - rebuild_object_field_layout, + gc_field_slot_range, gc_keys_array_slot, gc_shape_keys_edge_slot, + rebuild_array_layout_from_slots, rebuild_object_field_layout, }; mod global_fetch; pub(crate) use global_fetch::scan_pending_fetch_signal_root_mut; diff --git a/crates/perry-runtime/src/object/shapes.rs b/crates/perry-runtime/src/object/shapes.rs index 296d11d7a0..430e8aad7e 100644 --- a/crates/perry-runtime/src/object/shapes.rs +++ b/crates/perry-runtime/src/object/shapes.rs @@ -42,15 +42,45 @@ pub(crate) struct ShapeIndex { slots: HashMap>, } -/// Immutable facts named by one ShapeId. The raw keys pointer is a weak mirror -/// of the authoritative `ObjectHeader::keys_array` edge: live-object scans -/// rekey it immediately after visiting that header slot, and the metadata pass -/// repairs deferred forwarding. The table itself never exposes a GC slot. -#[derive(Clone, Copy, Debug, Eq, PartialEq)] +/// Immutable facts named by one ShapeId. +/// +/// #8112: `keys` is the AUTHORITATIVE ordered-keys edge — the collector marks +/// it and rewrites it in place, and `ObjectHeader::keys_array` is the derived +/// mirror. It used to be the other way round: the header word was the sole +/// strong edge and this field a weak copy that a post-visit callback repaired. +/// The inversion is what #8047 needs, because deleting the header word must +/// not unroot anything. +/// +/// The table is a rehashing `PtrHashMap`, so the bucket address is NOT stable +/// across descriptor insertion — and the incremental collector retains +/// enumerated slot addresses across budgeted resumptions. Descriptors are +/// therefore BOXED (`ShapeTableInner::descriptors`), which makes each record's +/// address fixed for its lifetime, and `record` carries the address of THIS +/// boxed descriptor so a traced receiver can hand the collector a rewritable +/// `keys` location without a second table probe (#8122's one-probe rule). +#[derive(Clone, Copy, Debug)] pub(crate) struct ShapeDescriptor { /// Raw ArrayHeader address in Perry's fixed-width heap-word ABI. Keeping - /// this weak mirror u64 preserves identical representation on ILP32/LP64. + /// this u64 preserves identical representation on ILP32/LP64. pub(crate) keys: u64, + /// Address of the BOXED record this value was lifted from, or 0 for a + /// descriptor built outside the table (equality comparisons, tests). + /// Never part of shape IDENTITY — see the hand-written `PartialEq` below. + pub(crate) record: usize, + /// Is this shape carried by at least one OLD-generation object? + /// + /// #8112's liveness gate. A minor never enumerates old objects, so the + /// per-receiver edge cannot express "an old object still carries this + /// shape" — and the record is SHARED, so no per-parent remembered-set + /// entry can either (one sibling's rewrite creates an old→young edge for a + /// parent the minor never visits). This flag is what the shape table roots + /// on. It is sticky within an epoch and recomputed by every full trace, so + /// it over-approximates by at most one full collection: exactly the + /// generational contract, and never unconditional rooting. + pub(crate) old_carrier: bool, + /// Notes accumulated since the last full trace; adopted into `old_carrier` + /// by [`rotate_old_carrier_epoch_after_full_trace`]. + pub(crate) old_carrier_seen: bool, pub(crate) logical_key_count: u32, pub(crate) live_inline_slot_count: u32, /// Zero for ordinary structural shapes. Descriptor/prototype mutations @@ -63,6 +93,28 @@ pub(crate) struct ShapeDescriptor { pub(crate) object_kind: ShapeObjectKind, } +/// Shape identity is the FACTS, never the storage address. A descriptor value +/// lifted out of the table compares equal to the boxed record it came from. +impl ShapeDescriptor { + /// The one `keys` word the collector rewrites for this shape, or `None` + /// for a descriptor value that was never lifted out of the table. + #[inline] + pub(crate) fn keys_slot(&self) -> Option<*mut u64> { + if self.record == 0 { + return None; + } + Some(unsafe { std::ptr::addr_of_mut!((*(self.record as *mut ShapeDescriptor)).keys) }) + } +} + +impl PartialEq for ShapeDescriptor { + fn eq(&self, other: &Self) -> bool { + descriptor_facts(*self) == descriptor_facts(*other) + } +} + +impl Eq for ShapeDescriptor {} + #[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)] pub(crate) enum ShapeObjectKind { Ordinary, @@ -96,7 +148,12 @@ struct ShapeTableInner { /// No external input reaches it, so hash-flooding resistance buys nothing /// here for the same reason it buys nothing on the pointer-keyed /// registries `fast_hash` already serves. - descriptors: crate::fast_hash::PtrHashMap, + /// BOXED (#8112): the collector enumerates `&mut record.keys` as an + /// ordinary GC slot, so the record's address must survive every descriptor + /// insertion that can happen while a budgeted scan holds it. A `Box` keeps + /// the payload put when the map rehashes; the map only ever moves the + /// eight-byte owning pointer. + descriptors: crate::fast_hash::PtrHashMap>, /// Exact-facts reverse index. More than one id is legal when a worker /// minted a local descriptor before a process-global module id arrived. /// @@ -140,18 +197,7 @@ fn descriptor_facts(descriptor: ShapeDescriptor) -> ShapeFacts { } } -fn remove_id_from_keys_index(inner: &mut ShapeTableInner, keys: u64, id: u32) { - let remove_entry = if let Some(ids) = inner.ids_by_keys.get_mut(&keys) { - ids.retain(|&candidate| candidate != id); - ids.is_empty() - } else { - false - }; - if remove_entry { - inner.ids_by_keys.remove(&keys); - } -} - +#[cfg(test)] fn remove_id_from_facts_index(inner: &mut ShapeTableInner, facts: ShapeFacts, id: u32) { let remove_entry = if let Some(ids) = inner.ids_by_facts.get_mut(&facts) { ids.retain(|&candidate| candidate != id); @@ -169,9 +215,9 @@ fn rebuild_descriptor_reverse_indices(inner: &mut ShapeTableInner) { HashMap::with_capacity(inner.descriptors.len()); let mut ids_by_keys: crate::fast_hash::PtrHashMap> = crate::fast_hash::new_ptr_hash_map(); - for (&id, &descriptor) in &inner.descriptors { + for (&id, descriptor) in &inner.descriptors { ids_by_facts - .entry(descriptor_facts(descriptor)) + .entry(descriptor_facts(**descriptor)) .or_default() .push(id); ids_by_keys.entry(descriptor.keys).or_default().push(id); @@ -302,6 +348,9 @@ fn shape_descriptor_ensure_with_generation( let id = alloc_shape_id().map_err(|_| ShapeDescriptorError::IdExhausted)?; let descriptor = ShapeDescriptor { keys: keys_id as u64, + record: 0, + old_carrier: false, + old_carrier_seen: false, logical_key_count, live_inline_slot_count, semantic_generation, @@ -310,7 +359,7 @@ fn shape_descriptor_ensure_with_generation( // Publish by-id first, then the reverse accelerator. An ObjectHeader is // stamped only after this function returns, so a visible id always has a // complete descriptor. - inner.descriptors.insert(id, descriptor); + inner.descriptors.insert(id, box_descriptor(descriptor)); inner.ids_by_facts.entry(facts).or_default().push(id); inner.ids_by_keys.entry(facts.keys).or_default().push(id); Ok(id) @@ -376,7 +425,76 @@ pub(crate) fn shape_descriptor_by_id(shape_id: u32) -> Option { .borrow() .descriptors .get(&shape_id) - .copied() + .map(|record| lift_descriptor(record)) +} + +/// Box a descriptor and stamp the record with its OWN address (#8112). +/// +/// Self-referential on purpose. The alternative — deriving the address in +/// `lift_descriptor` from the `&ShapeDescriptor` a shared table borrow yields — +/// would hand the collector a pointer with SHARED provenance and then write +/// through it. Taking it from the box while it is still uniquely owned keeps +/// the write well-formed. +fn box_descriptor(descriptor: ShapeDescriptor) -> Box { + let mut boxed = Box::new(descriptor); + boxed.record = std::ptr::addr_of_mut!(*boxed) as usize; + boxed +} + +/// Copy a boxed record out of the table (#8112). +/// +/// The copy's `keys` is a snapshot; `record` — stamped by [`box_descriptor`] — +/// names the one storage the collector rewrites. A caller that only reads facts +/// uses the snapshot; the GC hands `keys_slot()` to the slot visitor, so a +/// moved keys array lands back in the table with no second probe and no +/// write-back callback. +#[inline] +fn lift_descriptor(record: &ShapeDescriptor) -> ShapeDescriptor { + *record +} + +/// Record that a shape is carried by an OLD-generation receiver. +/// +/// Called from the collector's slot visitor, which resolved the descriptor for +/// this receiver already, so the note costs a generation range check and a +/// byte store — no second shape-table probe (#8122's one-probe rule). The +/// store goes straight through the boxed record's address rather than +/// re-borrowing `ShapeTableInner`: the visitor runs inside walks that already +/// hold that borrow. +/// +/// # Safety +/// +/// `descriptor.record`, when non-zero, is the address of a live boxed record +/// owned by this agent's shape table. Records are freed only by +/// `prune_dead_shape_keys`, which runs at sweep — after every enumeration of +/// the cycle that produced this descriptor. +#[inline] +pub(crate) unsafe fn note_old_generation_carrier(descriptor: Option) { + let Some(descriptor) = descriptor else { + return; + }; + if descriptor.record == 0 { + return; + } + let record = descriptor.record as *mut ShapeDescriptor; + // GC_STORE_AUDIT(POINTER_FREE): liveness bookkeeping byte, never a heap reference. + (*record).old_carrier = true; + (*record).old_carrier_seen = true; +} + +/// Recompute the old-carrier gate from the trace that just finished. +/// +/// A FULL trace enumerates every live object, so the notes it accumulated are +/// exactly the shapes old objects still carry; adopt them and clear the +/// accumulator. Minors only ever ADD notes, which is why the gate needs a full +/// trace to shed a shape whose last old carrier died — the same rule that +/// governs every other old-generation reclamation. +pub(crate) fn rotate_old_carrier_epoch_after_full_trace() { + let mut inner = crate::state::state().shapes.inner.borrow_mut(); + for record in inner.descriptors.values_mut() { + record.old_carrier = record.old_carrier_seen; + record.old_carrier_seen = false; + } } /// Mint (or retrieve) the ShapeId paired with canonical keys and equal @@ -407,6 +525,9 @@ fn install_external_shape_id( } let descriptor = ShapeDescriptor { keys: keys as usize as u64, + record: 0, + old_carrier: false, + old_carrier_seen: false, logical_key_count, live_inline_slot_count, semantic_generation: 0, @@ -415,13 +536,13 @@ fn install_external_shape_id( let facts = descriptor_facts(descriptor); let mut inner = crate::state::state().shapes.inner.borrow_mut(); if let Some(existing) = inner.descriptors.get(&id) { - return *existing == descriptor; + return **existing == descriptor; } // A worker can have minted an equivalent local descriptor before module // initialization installs the process-global codegen id. Keep both id // descriptors valid for already-published objects and make the external // id canonical for subsequent births in this agent. - inner.descriptors.insert(id, descriptor); + inner.descriptors.insert(id, box_descriptor(descriptor)); // An equivalent local descriptor can predate module initialization. Keep // both reverse-index entries and prefer the external id for subsequent // births in this agent; already-published local ids remain resolvable. @@ -789,7 +910,7 @@ fn retain_key_count_versions(keys: u64) { }; let mut current_ids = Vec::with_capacity(ids.len()); for id in ids { - let Some(descriptor) = inner.descriptors.get(&id).copied() else { + let Some(descriptor) = inner.descriptors.get(&id).map(|record| **record) else { continue; }; debug_assert_eq!( @@ -877,67 +998,56 @@ pub(crate) unsafe fn debug_assert_object_shape_parity_for_keys( } } -/// Validate and immediately mirror a live object's authoritative keys edge -/// after that header slot has been visited. No address inside the descriptor -/// HashMap is ever handed to a generic visitor: remembered-set enumeration can -/// save slot pointers across budgeted mutator resumptions, while descriptor -/// insertion/pruning may reallocate the table in between. +/// The address of the ONE `keys` word the collector rewrites for `shape_id`, +/// or `None` when the id names no descriptor in this agent (#8112). /// -/// An immediate copying visitor has already rewritten `new_header_keys`; a -/// deferred dirty-work visitor leaves it equal to `old_header_keys`, and the -/// registered metadata forwarding pass repairs the weak mirror after copying. -/// Exact release-mode facts prevent a stale or foreign id from rekeying an -/// unrelated descriptor. Returns whether the descriptor facts validated. -pub(crate) unsafe fn synchronize_live_object_shape_descriptor_after_header_visit( - obj: *const crate::object::ObjectHeader, - old_header_keys: u64, - new_header_keys: u64, - logical_key_count: u32, - live_inline_slot_count: u32, -) -> bool { - let shape_id = object_shape_stamp(obj); - if shape_id == 0 { - return false; +/// This is the seam that replaced the post-visit write-back callback. The +/// callback existed because the header word was the strong edge and the +/// descriptor a weak copy that had to be repaired from it, under exact-facts +/// validation, once per traced receiver whose keys array had moved. With the +/// descriptor holding the edge, the slot visitor writes the record directly +/// and there is nothing left to reconcile. +/// +/// The returned address belongs to a BOXED record, so it is stable across +/// descriptor insertion; only `prune_dead_shape_keys` frees one, and that runs +/// at sweep, after every enumeration of the cycle that produced it. +#[cfg(test)] +#[inline] +pub(crate) fn shape_descriptor_keys_slot(shape_id: u32) -> Option<*mut u64> { + if !is_shape_id(shape_id) { + return None; } + crate::state::state() + .shapes + .inner + .borrow_mut() + .descriptors + .get_mut(&shape_id) + .map(|record| std::ptr::addr_of_mut!(record.keys)) +} - let mut inner = crate::state::state().shapes.inner.borrow_mut(); - let (old_facts, new_facts) = { - let Some(descriptor) = inner.descriptors.get_mut(&shape_id) else { - // A foreign-agent/stale id fails closed; the authoritative header - // edge is still traced by the caller. - return false; - }; - // Release-mode fail-closed gate. An id hit is insufficient: a foreign - // or stale id must never cause an unrelated descriptor pointer to be - // rekeyed. `new_header_keys` may differ after evacuation; a sibling - // may also have rewritten the shared descriptor before this object. - if descriptor.logical_key_count != logical_key_count - || descriptor.live_inline_slot_count != live_inline_slot_count - || (descriptor.keys != old_header_keys && descriptor.keys != new_header_keys) - { - return false; - } - let old_facts = descriptor_facts(*descriptor); - if descriptor.keys == old_header_keys && new_header_keys != old_header_keys { - descriptor.keys = new_header_keys; - } - (old_facts, descriptor_facts(*descriptor)) - }; - if new_facts != old_facts { - remove_id_from_facts_index(&mut inner, old_facts, shape_id); - inner - .ids_by_facts - .entry(new_facts) - .or_default() - .push(shape_id); - remove_id_from_keys_index(&mut inner, old_facts.keys, shape_id); - inner - .ids_by_keys - .entry(new_facts.keys) - .or_default() - .push(shape_id); +/// Is `slot` the shared `keys` word of `shape_id`'s descriptor record? +/// +/// #8112: that word is a TABLE root, not a slot any receiver owns. Every +/// sibling of the shape enumerates it, so a rewrite performed while tracing +/// one receiver silently changes the edge of every other — including old +/// receivers a minor never visits, for which no per-parent remembered-set page +/// could ever be armed. The remembered-set and old→young verification paths +/// therefore skip it and let the shape table's own root scanner cover it. +#[inline] +pub(crate) fn shape_id_owns_keys_slot(shape_id: u32, slot: *mut u64) -> bool { + if !is_shape_id(shape_id) { + return false; } - true + // Immutable borrow on purpose: this runs inside collector walks, and a + // `borrow_mut` here would make the predicate itself a re-entrancy hazard. + crate::state::state() + .shapes + .inner + .borrow() + .descriptors + .get(&shape_id) + .is_some_and(|record| std::ptr::addr_of!(record.keys) as *mut u64 == slot) } /// Drop the stamp iff the word currently holds one, leaving a real @@ -1130,7 +1240,19 @@ pub(crate) fn scan_shape_table_rekey_mut(visitor: &mut crate::gc::RuntimeRootVis let mut descriptor_moved = false; for descriptor in inner.descriptors.values_mut() { let mut addr = descriptor.keys as usize; - if visitor.visit_metadata_usize_slot(&mut addr) { + // #8112 ephemeron gate. A shape with an OLD carrier is rooted here: + // the minor that has to keep its keys array alive never enumerates the + // object that carries it. A shape with only young carriers is NOT — + // those receivers are traced, and each one emits the edge itself, so + // rooting them from the table would make every keys array ever minted + // immortal and turn `prune_dead_shape_keys`'s "is the keys array + // dead?" into a question it asks of itself. + let moved = if descriptor.old_carrier { + visitor.visit_usize_slot(&mut addr) + } else { + visitor.visit_metadata_usize_slot(&mut addr) + }; + if moved { descriptor.keys = addr as u64; descriptor_moved = true; } @@ -1161,6 +1283,70 @@ pub(crate) fn scan_shape_table_rekey_mut(visitor: &mut crate::gc::RuntimeRootVis } } +// #8112 / #8047 rehearsal switches. Suppressing the derived `ObjectHeader` +// keys mirror leaves the descriptor edge as the ONLY thing the collector has; +// suppressing the descriptor edge too is the SABOTAGE arm that proves the +// fixture's detector distinguishes a rewritten record from a stale one. +// +// Deliberately `#[cfg(test)]` thread-locals and not env knobs: the GC-knob +// kill policy requires every shipped knob's off-state to be exercised by a +// required CI arm, and neither state may be reachable in a shipped binary — +// with the mirror off, the mutator's direct header loads are stale by +// construction. Only collector-level fixtures may turn them on. +#[cfg(test)] +thread_local! { + static KEYS_MIRROR_SUPPRESSED: std::cell::Cell = const { std::cell::Cell::new(false) }; + static KEYS_EDGE_SUPPRESSED: std::cell::Cell = const { std::cell::Cell::new(false) }; +} + +#[cfg(test)] +#[inline] +pub(crate) fn test_keys_mirror_suppressed() -> bool { + KEYS_MIRROR_SUPPRESSED.with(std::cell::Cell::get) +} + +#[cfg(test)] +#[inline] +pub(crate) fn test_keys_edge_suppressed() -> bool { + KEYS_EDGE_SUPPRESSED.with(std::cell::Cell::get) +} + +/// RAII guard so a panicking fixture cannot leave a suppression on for the +/// next test on this thread. +#[cfg(test)] +pub(crate) struct TestKeysEdgeSuppression { + mirror: bool, + edge: bool, +} + +#[cfg(test)] +impl TestKeysEdgeSuppression { + /// Drop the header mirror; keep the descriptor edge. This is the state + /// #8047 ships. + pub(crate) fn without_header_mirror() -> Self { + Self { + mirror: KEYS_MIRROR_SUPPRESSED.with(|c| c.replace(true)), + edge: KEYS_EDGE_SUPPRESSED.with(std::cell::Cell::get), + } + } + + /// Drop BOTH. Nothing roots or rewrites the keys array — the sabotage arm. + pub(crate) fn without_any_keys_edge() -> Self { + Self { + mirror: KEYS_MIRROR_SUPPRESSED.with(|c| c.replace(true)), + edge: KEYS_EDGE_SUPPRESSED.with(|c| c.replace(true)), + } + } +} + +#[cfg(test)] +impl Drop for TestKeysEdgeSuppression { + fn drop(&mut self) { + KEYS_MIRROR_SUPPRESSED.with(|c| c.set(self.mirror)); + KEYS_EDGE_SUPPRESSED.with(|c| c.set(self.edge)); + } +} + #[cfg(test)] pub(crate) fn test_shape_entry_exists(keys_id: usize) -> bool { crate::state::state() @@ -1200,7 +1386,7 @@ pub(crate) fn test_drop_shape_descriptors(keys_id: usize) { .unwrap_or_default(); for id in stale { if let Some(descriptor) = inner.descriptors.remove(&id) { - remove_id_from_facts_index(&mut inner, descriptor_facts(descriptor), id); + remove_id_from_facts_index(&mut inner, descriptor_facts(*descriptor), id); } } } @@ -1232,624 +1418,9 @@ pub(crate) fn test_shape_id_for_keys(keys_id: usize) -> Option { .and_then(|ids| ids.first().copied()) } +/// The shape-table unit suites, in a sibling file: `shapes.rs` sits close to +/// the repo's 2000-line-per-file cap and #8112 added the descriptor record's +/// keys slot and old-carrier gate to it. Moved verbatim. #[cfg(test)] -mod c3c_tests { - use super::*; - - fn key(name: &str) -> *mut crate::StringHeader { - crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32) - } - - /// #6759 C3c: ids come from the dedicated range (disjoint from real and - /// builtin class ids), are stable per exact descriptor facts, and distinct - /// across identities. - #[test] - fn shape_ids_are_range_disjoint_and_stable() { - let _lock = crate::gc::global_side_table_test_lock(); - let a: usize = 0xC3C0_0000_0000_1000; - let b: usize = 0xC3C0_0000_0000_2000; - let ida = shape_id_for_keys_ensure(a as *const ArrayHeader, 4); - let idb = shape_id_for_keys_ensure(b as *const ArrayHeader, 4); - assert!(is_shape_id(ida) && is_shape_id(idb)); - assert_ne!(ida, idb); - assert_eq!(shape_id_for_keys_ensure(a as *const ArrayHeader, 4), ida); - // Real class-id space must never classify as a shape id. - assert!(!is_shape_id(0)); - assert!(!is_shape_id(1)); - assert!(!is_shape_id(0x7FFF_FF30)); - assert!(!is_shape_id(0xFFFF_0005)); - shape_drop(a as *const ArrayHeader); - shape_drop(b as *const ArrayHeader); - test_drop_shape_descriptors(a); - test_drop_shape_descriptors(b); - } - - /// #6759 C3 rung 2: the codegen-facing allocator receives the id minted - /// beside its canonical keys global and installs it before the newborn - /// instance is published to user code. No by-name lookup is allowed in - /// this fixture: observing a stamp therefore proves it was present at - /// birth rather than lazily self-healed by rung 1. - #[test] - fn compiled_class_allocator_stamps_the_canonical_shape_at_birth() { - let _lock = crate::gc::global_side_table_test_lock(); - const CID: u32 = 0x0C3C_7902; - let packed = b"birth_a\0birth_b"; - let keys = - crate::object::js_build_class_keys_array(CID, 2, packed.as_ptr(), packed.len() as u32); - let shape_id = js_object_shape_id_for_keys(keys as usize as u64, 2); - assert!( - is_shape_id(shape_id), - "module init must mint a real ShapeId" - ); - - let obj = - crate::object::js_object_alloc_class_inline_keys_stamped(CID, 0, 2, keys, shape_id); - let birth_word = unsafe { (*obj).parent_class_id }; - assert_eq!( - birth_word, shape_id, - "a fresh compiled class instance waited for a by-name lookup to stamp" - ); - assert_eq!( - unsafe { (*obj).keys_array }, - keys, - "the stamp and canonical keys global must describe the same shape" - ); - } - - /// #6759 C3c stamp invariant on a REAL object through the real - /// write/read paths: a read resolution stamps a shape id into the - /// plain object's `parent_class_id`; after further appends any surviving - /// stamp resolves to exact current pointer/logical/live facts. This fixture - /// deliberately reserves eight live inline slots while owning fewer keys, - /// so the old key-count-only compatibility mint is not the expected id. - #[test] - fn plain_object_stamp_lifecycle() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let obj = crate::object::js_object_alloc(0, 8); - for name in ["c3c_a", "c3c_b", "c3c_c"] { - crate::object::js_object_set_field_by_name(obj, key(name), 1.0); - } - assert_eq!((*obj).class_id, 0, "test premise: plain object"); - let _ = crate::object::js_object_get_field_by_name(obj, key("c3c_b")); - let stamp = (*obj).parent_class_id; - assert!( - is_shape_id(stamp), - "read resolution must stamp a shape id, got {stamp:#x}" - ); - - crate::object::js_object_set_field_by_name(obj, key("c3c_d"), 2.0); - crate::object::js_object_set_field_by_name(obj, key("c3c_e"), 3.0); - let stamp2 = (*obj).parent_class_id; - if stamp2 != 0 { - assert!(is_shape_id(stamp2)); - let descriptor = shape_descriptor_by_id(stamp2) - .expect("a surviving stamp must resolve in this agent"); - assert_eq!(descriptor.keys, (*obj).keys_array as u64); - assert_eq!( - descriptor.logical_key_count, - crate::array::js_array_length((*obj).keys_array) - ); - assert_eq!( - descriptor.live_inline_slot_count, - crate::object::object_live_slot_count(obj) - ); - debug_assert_object_shape_parity(obj); - } - - // Reads still resolve correctly through the id-keyed cache. - let v = crate::object::js_object_get_field_by_name(obj, key("c3c_d")); - assert_eq!(f64::from_bits(v.bits()), 2.0); - } - } -} - -#[cfg(test)] -mod c6804_tests { - use super::*; - - /// #6804: shape-cached literal allocation birth-stamps the runtime - /// ShapeId, and siblings of one shape share one id. - #[test] - fn alloc_with_shape_birth_stamps_shared_id() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let packed = b"m6804_a\0m6804_b\0m6804_c"; - let a = crate::object::js_object_alloc_with_shape( - 0x0C3C_6804, - 3, - packed.as_ptr(), - packed.len() as u32, - ); - let b = crate::object::js_object_alloc_with_shape( - 0x0C3C_6804, - 3, - packed.as_ptr(), - packed.len() as u32, - ); - let stamp_a = (*a).parent_class_id; - let stamp_b = (*b).parent_class_id; - assert!( - is_shape_id(stamp_a), - "newborn literal must carry a runtime ShapeId, got {stamp_a:#x}" - ); - assert_eq!( - stamp_a, stamp_b, - "siblings of one literal shape must share one id" - ); - assert_eq!( - (*a).keys_array, - (*b).keys_array, - "test premise: shared keys" - ); - } - } - - /// #6804 wanted "no pre/post-stamp token split", and got it with a - /// self-heal inside `object_shape()`. #8113 removes the self-heal and keeps - /// the property, by a stronger route: **the split population is empty**, - /// because every allocator birth-stamps. - /// - /// The self-heal had to go because it derived the live inline-slot bound - /// from `ObjectHeader::field_count`. With that word deleted, healing an - /// unstamped receiver would publish a descriptor claiming a bound of ZERO — - /// a read-only observation silently truncating the object's traced and - /// writable payload. Missing closed costs a PIC miss; healing wrongly loses - /// fields. - #[test] - fn object_shape_token_is_birth_stamped_and_an_unstamped_one_misses_closed() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let packed = b"m6804_x\0m6804_y"; - let obj = crate::object::js_object_alloc_with_shape( - 0x0C3C_6805, - 2, - packed.as_ptr(), - packed.len() as u32, - ); - let birth_stamp = (*obj).parent_class_id; - assert!(is_shape_id(birth_stamp), "every literal is birth-stamped"); - assert_eq!( - crate::typed_feedback::test_object_shape_token(obj as usize), - birth_stamp as usize, - "the observed token is the birth stamp — no split to heal" - ); - assert_eq!( - shape_descriptor_by_id(birth_stamp) - .expect("birth descriptor") - .live_inline_slot_count, - 2 - ); - - // Manufacture the pre-#6804 unstamped state and prove observing it - // is INERT: no token, no descriptor, and — the part that matters — - // no rewritten live-slot bound. - (*obj).parent_class_id = 0; - assert_eq!( - crate::typed_feedback::test_object_shape_token(obj as usize), - 0, - "an unstamped receiver must miss closed, not be re-stamped" - ); - assert_eq!( - (*obj).parent_class_id, - 0, - "observation must not publish a descriptor for an unstamped receiver" - ); - - // Restoring the birth stamp restores the exact bound, which is the - // proof that nothing was lost by refusing to heal. - (*obj).parent_class_id = birth_stamp; - assert_eq!(crate::object::object_live_slot_count(obj), 2); - } - } - - /// #6804: the first dynamic key on a fresh `{}` births a stamped shape. - #[test] - fn fresh_dynamic_shape_birth_stamps() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let obj = crate::object::js_object_alloc(0, 8); - let key = crate::string::js_string_from_bytes(b"m6804_first".as_ptr(), 11); - crate::object::js_object_set_field_by_name(obj, key, 42.0); - let stamp = (*obj).parent_class_id; - // Either stamped at the null-branch birth, or (for a sibling - // adopting a cached transition edge) still 0 until first read - // — but THIS test allocates a unique key, so the null branch - // ran and must have stamped. - assert!( - is_shape_id(stamp), - "first-key birth must stamp the new shape, got {stamp:#x}" - ); - } - } -} - -#[cfg(test)] -mod descriptor_tests_8067 { - use super::*; - - fn key(name: &str) -> *mut crate::StringHeader { - crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32) - } - - #[test] - fn every_keyless_runtime_allocator_publishes_a_shape_id() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - for obj in [ - crate::object::js_object_alloc(0, 0), - crate::object::js_object_alloc_fast(0, 0), - crate::object::js_object_alloc_with_parent(0x8067_0101, 0, 0), - crate::object::js_object_alloc_fast_with_parent(0x8067_0102, 0, 0), - ] { - let id = object_shape_id(obj); - assert!(is_shape_id(id), "newborn keyless object has no ShapeId"); - let facts = object_shape_descriptor(obj).expect("keyless descriptor"); - assert_eq!(facts.keys, 0); - assert_eq!(facts.logical_key_count, 0); - assert_eq!(facts.live_inline_slot_count, 0); - } - } - } - - #[test] - fn descriptor_and_prototype_changes_mint_semantic_successors() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let obj = crate::object::js_object_alloc(0, 1); - crate::object::js_object_set_field_by_name(obj, key("semantic8067"), 1.0); - let structural = object_shape_id(obj); - - crate::object::descriptor_state::set_property_attrs( - obj as usize, - "semantic8067".to_string(), - crate::object::descriptor_state::PropertyAttrs::new(false, true, true), - ); - let described = object_shape_id(obj); - assert_ne!(described, structural); - let described_facts = object_shape_descriptor(obj).unwrap(); - assert_ne!(described_facts.semantic_generation, 0); - - crate::object::prototype_chain::object_set_static_prototype( - obj as usize, - crate::value::TAG_NULL, - ); - let reparented = object_shape_id(obj); - assert_ne!(reparented, described); - assert_eq!( - object_shape_descriptor(obj).unwrap().keys, - described_facts.keys, - "semantic transitions must preserve the rooted ordered keys edge" - ); - } - } - - #[test] - fn absent_descriptor_clears_do_not_mint_semantic_successors() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let obj = crate::object::js_object_alloc(0, 1); - let addr = obj as usize; - let initial = object_shape_id(obj); - - crate::object::descriptor_state::clear_property_attrs(addr, "missing8067"); - crate::object::descriptor_state::clear_accessor_descriptor(addr, "missing8067"); - assert_eq!(object_shape_id(obj), initial); - - crate::object::descriptor_state::set_property_attrs( - addr, - "attrs8067".to_string(), - crate::object::descriptor_state::PropertyAttrs::new(false, true, true), - ); - crate::object::descriptor_state::clear_property_attrs(addr, "attrs8067"); - let after_real_attr_clear = object_shape_id(obj); - crate::object::descriptor_state::clear_property_attrs(addr, "attrs8067"); - assert_eq!(object_shape_id(obj), after_real_attr_clear); - - crate::object::descriptor_state::set_accessor_descriptor( - addr, - "accessor8067".to_string(), - crate::object::descriptor_state::AccessorDescriptor::default(), - ); - crate::object::descriptor_state::clear_accessor_descriptor(addr, "accessor8067"); - let after_real_accessor_clear = object_shape_id(obj); - crate::object::descriptor_state::clear_accessor_descriptor(addr, "accessor8067"); - assert_eq!(object_shape_id(obj), after_real_accessor_clear); - } - } - - #[test] - fn delete_compaction_never_compares_equal_to_the_predelete_layout() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let obj = crate::object::js_object_alloc(0, 3); - let a = key("delete8067_a"); - let b = key("delete8067_b"); - let c = key("delete8067_c"); - crate::object::js_object_set_field_by_name(obj, a, 1.0); - crate::object::js_object_set_field_by_name(obj, b, 2.0); - crate::object::js_object_set_field_by_name(obj, c, 3.0); - let before = object_shape_id(obj); - assert_eq!(crate::object::js_object_delete_field(obj, a), 1); - let after = object_shape_id(obj); - assert_ne!(after, before); - let facts = object_shape_descriptor(obj).unwrap(); - assert_eq!(facts.logical_key_count, 2); - assert_eq!(facts.live_inline_slot_count, 2); - assert_eq!( - crate::object::js_object_get_field_by_name_f64(obj, b), - 2.0, - "middle-field lookup used a stale pre-delete slot mapping" - ); - } - } - - #[test] - fn exhaustion_parks_without_reuse_or_alias() { - let next = std::sync::atomic::AtomicU32::new(SHAPE_ID_END - 1); - assert_eq!(alloc_shape_id_from(&next), Ok(SHAPE_ID_END - 1)); - assert_eq!(alloc_shape_id_from(&next), Err(ShapeIdExhausted)); - assert_eq!(alloc_shape_id_from(&next), Err(ShapeIdExhausted)); - assert_eq!( - next.load(std::sync::atomic::Ordering::Relaxed), - SHAPE_ID_END, - "exhaustion must park instead of wrapping into an alias" - ); - } - - #[test] - fn inconsistent_facts_are_not_reported_as_id_exhaustion() { - assert_eq!( - shape_descriptor_ensure(std::ptr::null(), 1, 1), - Err(ShapeDescriptorError::InvalidFacts) - ); - } - - #[test] - fn equivalent_local_and_external_ids_remain_resolvable() { - let _lock = crate::gc::global_side_table_test_lock(); - let keys = 0x8067_0000_0000_1700usize; - let local = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) - .expect("shape range unexpectedly exhausted"); - let external = alloc_shape_id().expect("shape range unexpectedly exhausted"); - assert!(install_external_shape_id( - external, - keys as *const ArrayHeader, - 1, - 1, - )); - - assert_eq!( - shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1).unwrap(), - external, - "the process-global id should be preferred for later births" - ); - retain_key_count_versions(keys as u64); - assert!(shape_descriptor_by_id(local).is_some()); - assert!(shape_descriptor_by_id(external).is_some()); - - test_drop_shape_descriptors(keys); - } - - #[test] - fn a_foreign_agent_id_misses_instead_of_aliasing_same_address() { - let _lock = crate::gc::global_side_table_test_lock(); - let fake_keys = 0x8067_0000_0000_1000usize; - let local = shape_descriptor_ensure(fake_keys as *const ArrayHeader, 2, 2) - .expect("shape range unexpectedly exhausted"); - let foreign = std::thread::spawn(move || { - assert_eq!( - shape_descriptor_by_id(local), - None, - "another RuntimeState resolved a foreign agent's ShapeId" - ); - shape_descriptor_ensure(fake_keys as *const ArrayHeader, 2, 2) - .expect("shape range unexpectedly exhausted") - }) - .join() - .expect("agent-isolation thread panicked"); - assert_ne!( - local, foreign, - "process-global ids must not alias by address" - ); - shape_drop(fake_keys as *const ArrayHeader); - test_drop_shape_descriptors(fake_keys); - } - - #[test] - fn process_global_module_shape_id_installs_with_agent_local_keys() { - let _lock = crate::gc::global_side_table_test_lock(); - let module_keys = 0x8067_0000_0000_1800usize; - let module_id = shape_descriptor_ensure(module_keys as *const ArrayHeader, 2, 2) - .expect("shape range unexpectedly exhausted"); - let worker_keys = 0x8067_0000_0000_1900usize; - std::thread::spawn(move || { - assert!(install_external_shape_id( - module_id, - worker_keys as *const ArrayHeader, - 2, - 2, - )); - assert_eq!( - shape_descriptor_by_id(module_id).unwrap().keys, - worker_keys as u64, - "worker resolved a module ShapeId to another agent's keys pointer" - ); - }) - .join() - .expect("worker shape installation panicked"); - test_drop_shape_descriptors(module_keys); - } - - #[test] - fn gc_descriptor_mirror_requires_exact_release_facts() { - let _lock = crate::gc::global_side_table_test_lock(); - let keys = 0x8067_0000_0000_2000usize; - let id = shape_descriptor_ensure(keys as *const ArrayHeader, 3, 2) - .expect("shape range unexpectedly exhausted"); - let obj = crate::object::ObjectHeader { - class_id: 0, - parent_class_id: id, - keys_array: keys as *mut ArrayHeader, - meta: std::ptr::null_mut(), - }; - - unsafe { - assert!( - !synchronize_live_object_shape_descriptor_after_header_visit( - &obj, - keys as u64 + 0x1000, - keys as u64 + 0x2000, - 3, - 2, - ) - ); - assert!( - !synchronize_live_object_shape_descriptor_after_header_visit( - &obj, - keys as u64, - keys as u64, - 4, - 2, - ) - ); - } - assert_eq!(shape_descriptor_by_id(id).unwrap().keys, keys as u64); - - let moved_keys = keys as u64 + 0x3000; - assert!(unsafe { - synchronize_live_object_shape_descriptor_after_header_visit( - &obj, - keys as u64, - moved_keys, - 3, - 2, - ) - }); - assert_eq!(shape_descriptor_by_id(id).unwrap().keys, moved_keys); - test_drop_shape_descriptors(moved_keys as usize); - assert_eq!( - shape_descriptor_by_id(id), - None, - "descriptor rekey did not update the keys-address index" - ); - } - - #[test] - fn key_count_versions_remain_resolvable_until_the_keys_die() { - let _lock = crate::gc::global_side_table_test_lock(); - let keys = 0x8067_0000_0000_2100usize; - let unrelated_keys = 0x8067_0000_0000_2200usize; - let stale_a = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) - .expect("shape range unexpectedly exhausted"); - let stale_b = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 2) - .expect("shape range unexpectedly exhausted"); - let current = shape_descriptor_ensure(keys as *const ArrayHeader, 2, 2) - .expect("shape range unexpectedly exhausted"); - let unrelated = shape_descriptor_ensure(unrelated_keys as *const ArrayHeader, 1, 1) - .expect("shape range unexpectedly exhausted"); - - retain_key_count_versions(keys as u64); - - assert!(shape_descriptor_by_id(stale_a).is_some()); - assert!(shape_descriptor_by_id(stale_b).is_some()); - assert!(shape_descriptor_by_id(current).is_some()); - assert!(shape_descriptor_by_id(unrelated).is_some()); - let inner = crate::state::state().shapes.inner.borrow(); - let current_ids = inner - .ids_by_keys - .get(&(keys as u64)) - .expect("keys identity disappeared from descriptor index"); - assert_eq!(current_ids.as_slice(), &[stale_a, stale_b, current]); - drop(inner); - - test_drop_shape_descriptors(keys); - test_drop_shape_descriptors(unrelated_keys); - } - - #[test] - fn shape_drop_does_not_delete_a_potential_siblings_descriptor() { - let _lock = crate::gc::global_side_table_test_lock(); - let keys = 0x8067_0000_0000_3000usize; - let id = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) - .expect("shape range unexpectedly exhausted"); - - shape_drop(keys as *const ArrayHeader); - - assert_eq!( - shape_descriptor_by_id(id).map(|descriptor| descriptor.keys), - Some(keys as u64), - "shape_drop eagerly invalidated a descriptor a sibling may still name" - ); - test_drop_shape_descriptors(keys); - } - - #[test] - fn live_slot_growth_versions_descriptor_before_value_publication() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let packed = b"slot8067_a"; - let obj = crate::object::js_object_alloc_with_shape( - 0x8067_1001, - 1, - packed.as_ptr(), - packed.len() as u32, - ); - let keys = (*obj).keys_array as usize; - let before = (*obj).parent_class_id; - let before_descriptor = shape_descriptor_by_id(before).expect("birth descriptor"); - assert_eq!(before_descriptor.live_inline_slot_count, 1); - - crate::object::js_object_set_field(obj, 1, crate::JSValue::string_ptr(key("value"))); - let after = (*obj).parent_class_id; - assert_ne!(before, after); - let after_descriptor = shape_descriptor_by_id(after).expect("grown descriptor"); - assert_eq!(after_descriptor.keys, keys as u64); - assert_eq!(after_descriptor.logical_key_count, 1); - assert_eq!(after_descriptor.live_inline_slot_count, 2); - debug_assert_object_shape_parity(obj); - } - } - - #[test] - fn shared_sibling_append_clones_before_descriptor_version_changes() { - let _lock = crate::gc::global_side_table_test_lock(); - unsafe { - let packed = b"sib8067_a"; - let a = crate::object::js_object_alloc_with_shape( - 0x8067_1002, - 1, - packed.as_ptr(), - packed.len() as u32, - ); - let b = crate::object::js_object_alloc_with_shape( - 0x8067_1002, - 1, - packed.as_ptr(), - packed.len() as u32, - ); - let shared_keys = (*a).keys_array; - let shared_id = (*a).parent_class_id; - assert_eq!(shared_keys, (*b).keys_array); - assert_eq!(shared_id, (*b).parent_class_id); - - crate::object::js_object_set_field_by_name(a, key("sib8067_b"), 2.0); - - assert_ne!((*a).keys_array, shared_keys); - assert_eq!((*b).keys_array, shared_keys); - assert_eq!((*b).parent_class_id, shared_id); - assert_ne!((*a).parent_class_id, shared_id); - assert_eq!( - shape_descriptor_by_id(shared_id) - .expect("untouched sibling descriptor") - .logical_key_count, - 1 - ); - let transitioned = - shape_descriptor_by_id((*a).parent_class_id).expect("transitioned descriptor"); - assert_eq!(transitioned.keys, (*a).keys_array as u64); - assert_eq!(transitioned.logical_key_count, 2); - assert_eq!(transitioned.live_inline_slot_count, 2); - } - } -} +#[path = "shapes_tests.rs"] +mod shapes_tests; diff --git a/crates/perry-runtime/src/object/shapes_tests.rs b/crates/perry-runtime/src/object/shapes_tests.rs new file mode 100644 index 0000000000..f6b92716e8 --- /dev/null +++ b/crates/perry-runtime/src/object/shapes_tests.rs @@ -0,0 +1,645 @@ +//! Shape-table unit suites, split out of `object/shapes.rs` to keep it under +//! the repo's 2000-line-per-file cap. Moved verbatim. + +use super::*; + +#[cfg(test)] +mod c3c_tests { + use super::*; + + fn key(name: &str) -> *mut crate::StringHeader { + crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32) + } + + /// #6759 C3c: ids come from the dedicated range (disjoint from real and + /// builtin class ids), are stable per exact descriptor facts, and distinct + /// across identities. + #[test] + fn shape_ids_are_range_disjoint_and_stable() { + let _lock = crate::gc::global_side_table_test_lock(); + let a: usize = 0xC3C0_0000_0000_1000; + let b: usize = 0xC3C0_0000_0000_2000; + let ida = shape_id_for_keys_ensure(a as *const ArrayHeader, 4); + let idb = shape_id_for_keys_ensure(b as *const ArrayHeader, 4); + assert!(is_shape_id(ida) && is_shape_id(idb)); + assert_ne!(ida, idb); + assert_eq!(shape_id_for_keys_ensure(a as *const ArrayHeader, 4), ida); + // Real class-id space must never classify as a shape id. + assert!(!is_shape_id(0)); + assert!(!is_shape_id(1)); + assert!(!is_shape_id(0x7FFF_FF30)); + assert!(!is_shape_id(0xFFFF_0005)); + shape_drop(a as *const ArrayHeader); + shape_drop(b as *const ArrayHeader); + test_drop_shape_descriptors(a); + test_drop_shape_descriptors(b); + } + + /// #6759 C3 rung 2: the codegen-facing allocator receives the id minted + /// beside its canonical keys global and installs it before the newborn + /// instance is published to user code. No by-name lookup is allowed in + /// this fixture: observing a stamp therefore proves it was present at + /// birth rather than lazily self-healed by rung 1. + #[test] + fn compiled_class_allocator_stamps_the_canonical_shape_at_birth() { + let _lock = crate::gc::global_side_table_test_lock(); + const CID: u32 = 0x0C3C_7902; + let packed = b"birth_a\0birth_b"; + let keys = + crate::object::js_build_class_keys_array(CID, 2, packed.as_ptr(), packed.len() as u32); + let shape_id = js_object_shape_id_for_keys(keys as usize as u64, 2); + assert!( + is_shape_id(shape_id), + "module init must mint a real ShapeId" + ); + + let obj = + crate::object::js_object_alloc_class_inline_keys_stamped(CID, 0, 2, keys, shape_id); + let birth_word = unsafe { (*obj).parent_class_id }; + assert_eq!( + birth_word, shape_id, + "a fresh compiled class instance waited for a by-name lookup to stamp" + ); + assert_eq!( + unsafe { (*obj).keys_array }, + keys, + "the stamp and canonical keys global must describe the same shape" + ); + } + + /// #6759 C3c stamp invariant on a REAL object through the real + /// write/read paths: a read resolution stamps a shape id into the + /// plain object's `parent_class_id`; after further appends any surviving + /// stamp resolves to exact current pointer/logical/live facts. This fixture + /// deliberately reserves eight live inline slots while owning fewer keys, + /// so the old key-count-only compatibility mint is not the expected id. + #[test] + fn plain_object_stamp_lifecycle() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let obj = crate::object::js_object_alloc(0, 8); + for name in ["c3c_a", "c3c_b", "c3c_c"] { + crate::object::js_object_set_field_by_name(obj, key(name), 1.0); + } + assert_eq!((*obj).class_id, 0, "test premise: plain object"); + let _ = crate::object::js_object_get_field_by_name(obj, key("c3c_b")); + let stamp = (*obj).parent_class_id; + assert!( + is_shape_id(stamp), + "read resolution must stamp a shape id, got {stamp:#x}" + ); + + crate::object::js_object_set_field_by_name(obj, key("c3c_d"), 2.0); + crate::object::js_object_set_field_by_name(obj, key("c3c_e"), 3.0); + let stamp2 = (*obj).parent_class_id; + if stamp2 != 0 { + assert!(is_shape_id(stamp2)); + let descriptor = shape_descriptor_by_id(stamp2) + .expect("a surviving stamp must resolve in this agent"); + assert_eq!(descriptor.keys, (*obj).keys_array as u64); + assert_eq!( + descriptor.logical_key_count, + crate::array::js_array_length((*obj).keys_array) + ); + assert_eq!( + descriptor.live_inline_slot_count, + crate::object::object_live_slot_count(obj) + ); + debug_assert_object_shape_parity(obj); + } + + // Reads still resolve correctly through the id-keyed cache. + let v = crate::object::js_object_get_field_by_name(obj, key("c3c_d")); + assert_eq!(f64::from_bits(v.bits()), 2.0); + } + } +} + +#[cfg(test)] +mod c6804_tests { + use super::*; + + /// #6804: shape-cached literal allocation birth-stamps the runtime + /// ShapeId, and siblings of one shape share one id. + #[test] + fn alloc_with_shape_birth_stamps_shared_id() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let packed = b"m6804_a\0m6804_b\0m6804_c"; + let a = crate::object::js_object_alloc_with_shape( + 0x0C3C_6804, + 3, + packed.as_ptr(), + packed.len() as u32, + ); + let b = crate::object::js_object_alloc_with_shape( + 0x0C3C_6804, + 3, + packed.as_ptr(), + packed.len() as u32, + ); + let stamp_a = (*a).parent_class_id; + let stamp_b = (*b).parent_class_id; + assert!( + is_shape_id(stamp_a), + "newborn literal must carry a runtime ShapeId, got {stamp_a:#x}" + ); + assert_eq!( + stamp_a, stamp_b, + "siblings of one literal shape must share one id" + ); + assert_eq!( + (*a).keys_array, + (*b).keys_array, + "test premise: shared keys" + ); + } + } + + /// #6804 wanted "no pre/post-stamp token split", and got it with a + /// self-heal inside `object_shape()`. #8113 removes the self-heal and keeps + /// the property, by a stronger route: **the split population is empty**, + /// because every allocator birth-stamps. + /// + /// The self-heal had to go because it derived the live inline-slot bound + /// from `ObjectHeader::field_count`. With that word deleted, healing an + /// unstamped receiver would publish a descriptor claiming a bound of ZERO — + /// a read-only observation silently truncating the object's traced and + /// writable payload. Missing closed costs a PIC miss; healing wrongly loses + /// fields. + #[test] + fn object_shape_token_is_birth_stamped_and_an_unstamped_one_misses_closed() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let packed = b"m6804_x\0m6804_y"; + let obj = crate::object::js_object_alloc_with_shape( + 0x0C3C_6805, + 2, + packed.as_ptr(), + packed.len() as u32, + ); + let birth_stamp = (*obj).parent_class_id; + assert!(is_shape_id(birth_stamp), "every literal is birth-stamped"); + assert_eq!( + crate::typed_feedback::test_object_shape_token(obj as usize), + birth_stamp as usize, + "the observed token is the birth stamp — no split to heal" + ); + assert_eq!( + shape_descriptor_by_id(birth_stamp) + .expect("birth descriptor") + .live_inline_slot_count, + 2 + ); + + // Manufacture the pre-#6804 unstamped state and prove observing it + // is INERT: no token, no descriptor, and — the part that matters — + // no rewritten live-slot bound. + (*obj).parent_class_id = 0; + assert_eq!( + crate::typed_feedback::test_object_shape_token(obj as usize), + 0, + "an unstamped receiver must miss closed, not be re-stamped" + ); + assert_eq!( + (*obj).parent_class_id, + 0, + "observation must not publish a descriptor for an unstamped receiver" + ); + + // Restoring the birth stamp restores the exact bound, which is the + // proof that nothing was lost by refusing to heal. + (*obj).parent_class_id = birth_stamp; + assert_eq!(crate::object::object_live_slot_count(obj), 2); + } + } + + /// #6804: the first dynamic key on a fresh `{}` births a stamped shape. + #[test] + fn fresh_dynamic_shape_birth_stamps() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let obj = crate::object::js_object_alloc(0, 8); + let key = crate::string::js_string_from_bytes(b"m6804_first".as_ptr(), 11); + crate::object::js_object_set_field_by_name(obj, key, 42.0); + let stamp = (*obj).parent_class_id; + // Either stamped at the null-branch birth, or (for a sibling + // adopting a cached transition edge) still 0 until first read + // — but THIS test allocates a unique key, so the null branch + // ran and must have stamped. + assert!( + is_shape_id(stamp), + "first-key birth must stamp the new shape, got {stamp:#x}" + ); + } + } +} + +#[cfg(test)] +mod descriptor_tests_8067 { + use super::*; + + fn key(name: &str) -> *mut crate::StringHeader { + crate::string::js_string_from_bytes(name.as_ptr(), name.len() as u32) + } + + #[test] + fn every_keyless_runtime_allocator_publishes_a_shape_id() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + for obj in [ + crate::object::js_object_alloc(0, 0), + crate::object::js_object_alloc_fast(0, 0), + crate::object::js_object_alloc_with_parent(0x8067_0101, 0, 0), + crate::object::js_object_alloc_fast_with_parent(0x8067_0102, 0, 0), + ] { + let id = object_shape_id(obj); + assert!(is_shape_id(id), "newborn keyless object has no ShapeId"); + let facts = object_shape_descriptor(obj).expect("keyless descriptor"); + assert_eq!(facts.keys, 0); + assert_eq!(facts.logical_key_count, 0); + assert_eq!(facts.live_inline_slot_count, 0); + } + } + } + + #[test] + fn descriptor_and_prototype_changes_mint_semantic_successors() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let obj = crate::object::js_object_alloc(0, 1); + crate::object::js_object_set_field_by_name(obj, key("semantic8067"), 1.0); + let structural = object_shape_id(obj); + + crate::object::descriptor_state::set_property_attrs( + obj as usize, + "semantic8067".to_string(), + crate::object::descriptor_state::PropertyAttrs::new(false, true, true), + ); + let described = object_shape_id(obj); + assert_ne!(described, structural); + let described_facts = object_shape_descriptor(obj).unwrap(); + assert_ne!(described_facts.semantic_generation, 0); + + crate::object::prototype_chain::object_set_static_prototype( + obj as usize, + crate::value::TAG_NULL, + ); + let reparented = object_shape_id(obj); + assert_ne!(reparented, described); + assert_eq!( + object_shape_descriptor(obj).unwrap().keys, + described_facts.keys, + "semantic transitions must preserve the rooted ordered keys edge" + ); + } + } + + #[test] + fn absent_descriptor_clears_do_not_mint_semantic_successors() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let obj = crate::object::js_object_alloc(0, 1); + let addr = obj as usize; + let initial = object_shape_id(obj); + + crate::object::descriptor_state::clear_property_attrs(addr, "missing8067"); + crate::object::descriptor_state::clear_accessor_descriptor(addr, "missing8067"); + assert_eq!(object_shape_id(obj), initial); + + crate::object::descriptor_state::set_property_attrs( + addr, + "attrs8067".to_string(), + crate::object::descriptor_state::PropertyAttrs::new(false, true, true), + ); + crate::object::descriptor_state::clear_property_attrs(addr, "attrs8067"); + let after_real_attr_clear = object_shape_id(obj); + crate::object::descriptor_state::clear_property_attrs(addr, "attrs8067"); + assert_eq!(object_shape_id(obj), after_real_attr_clear); + + crate::object::descriptor_state::set_accessor_descriptor( + addr, + "accessor8067".to_string(), + crate::object::descriptor_state::AccessorDescriptor::default(), + ); + crate::object::descriptor_state::clear_accessor_descriptor(addr, "accessor8067"); + let after_real_accessor_clear = object_shape_id(obj); + crate::object::descriptor_state::clear_accessor_descriptor(addr, "accessor8067"); + assert_eq!(object_shape_id(obj), after_real_accessor_clear); + } + } + + #[test] + fn delete_compaction_never_compares_equal_to_the_predelete_layout() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let obj = crate::object::js_object_alloc(0, 3); + let a = key("delete8067_a"); + let b = key("delete8067_b"); + let c = key("delete8067_c"); + crate::object::js_object_set_field_by_name(obj, a, 1.0); + crate::object::js_object_set_field_by_name(obj, b, 2.0); + crate::object::js_object_set_field_by_name(obj, c, 3.0); + let before = object_shape_id(obj); + assert_eq!(crate::object::js_object_delete_field(obj, a), 1); + let after = object_shape_id(obj); + assert_ne!(after, before); + let facts = object_shape_descriptor(obj).unwrap(); + assert_eq!(facts.logical_key_count, 2); + assert_eq!(facts.live_inline_slot_count, 2); + assert_eq!( + crate::object::js_object_get_field_by_name_f64(obj, b), + 2.0, + "middle-field lookup used a stale pre-delete slot mapping" + ); + } + } + + #[test] + fn exhaustion_parks_without_reuse_or_alias() { + let next = std::sync::atomic::AtomicU32::new(SHAPE_ID_END - 1); + assert_eq!(alloc_shape_id_from(&next), Ok(SHAPE_ID_END - 1)); + assert_eq!(alloc_shape_id_from(&next), Err(ShapeIdExhausted)); + assert_eq!(alloc_shape_id_from(&next), Err(ShapeIdExhausted)); + assert_eq!( + next.load(std::sync::atomic::Ordering::Relaxed), + SHAPE_ID_END, + "exhaustion must park instead of wrapping into an alias" + ); + } + + #[test] + fn inconsistent_facts_are_not_reported_as_id_exhaustion() { + assert_eq!( + shape_descriptor_ensure(std::ptr::null(), 1, 1), + Err(ShapeDescriptorError::InvalidFacts) + ); + } + + #[test] + fn equivalent_local_and_external_ids_remain_resolvable() { + let _lock = crate::gc::global_side_table_test_lock(); + let keys = 0x8067_0000_0000_1700usize; + let local = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) + .expect("shape range unexpectedly exhausted"); + let external = alloc_shape_id().expect("shape range unexpectedly exhausted"); + assert!(install_external_shape_id( + external, + keys as *const ArrayHeader, + 1, + 1, + )); + + assert_eq!( + shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1).unwrap(), + external, + "the process-global id should be preferred for later births" + ); + retain_key_count_versions(keys as u64); + assert!(shape_descriptor_by_id(local).is_some()); + assert!(shape_descriptor_by_id(external).is_some()); + + test_drop_shape_descriptors(keys); + } + + #[test] + fn a_foreign_agent_id_misses_instead_of_aliasing_same_address() { + let _lock = crate::gc::global_side_table_test_lock(); + let fake_keys = 0x8067_0000_0000_1000usize; + let local = shape_descriptor_ensure(fake_keys as *const ArrayHeader, 2, 2) + .expect("shape range unexpectedly exhausted"); + let foreign = std::thread::spawn(move || { + assert_eq!( + shape_descriptor_by_id(local), + None, + "another RuntimeState resolved a foreign agent's ShapeId" + ); + shape_descriptor_ensure(fake_keys as *const ArrayHeader, 2, 2) + .expect("shape range unexpectedly exhausted") + }) + .join() + .expect("agent-isolation thread panicked"); + assert_ne!( + local, foreign, + "process-global ids must not alias by address" + ); + shape_drop(fake_keys as *const ArrayHeader); + test_drop_shape_descriptors(fake_keys); + } + + #[test] + fn process_global_module_shape_id_installs_with_agent_local_keys() { + let _lock = crate::gc::global_side_table_test_lock(); + let module_keys = 0x8067_0000_0000_1800usize; + let module_id = shape_descriptor_ensure(module_keys as *const ArrayHeader, 2, 2) + .expect("shape range unexpectedly exhausted"); + let worker_keys = 0x8067_0000_0000_1900usize; + std::thread::spawn(move || { + assert!(install_external_shape_id( + module_id, + worker_keys as *const ArrayHeader, + 2, + 2, + )); + assert_eq!( + shape_descriptor_by_id(module_id).unwrap().keys, + worker_keys as u64, + "worker resolved a module ShapeId to another agent's keys pointer" + ); + }) + .join() + .expect("worker shape installation panicked"); + test_drop_shape_descriptors(module_keys); + } + + #[test] + fn the_descriptor_keys_slot_is_the_record_the_collector_rewrites() { + let _lock = crate::gc::global_side_table_test_lock(); + let keys = 0x8067_0000_0000_2000usize; + let id = shape_descriptor_ensure(keys as *const ArrayHeader, 3, 2) + .expect("shape range unexpectedly exhausted"); + + // A foreign / never-minted id has no slot: the collector emits no edge + // rather than rewriting an unrelated record. + assert_eq!(shape_descriptor_keys_slot(0), None); + assert_eq!(shape_descriptor_keys_slot(SHAPE_ID_END - 1), None); + + let slot = shape_descriptor_keys_slot(id).expect("minted id has a keys slot"); + assert_eq!( + Some(slot), + shape_descriptor_by_id(id).unwrap().keys_slot(), + "the lifted descriptor must name the boxed record's own keys word" + ); + assert_eq!(unsafe { *slot }, keys as u64); + + // Writing THROUGH the slot is what an evacuating visitor does. The + // table must observe it with no write-back callback of any kind. + let moved_keys = keys as u64 + 0x3000; + unsafe { *slot = moved_keys }; + assert_eq!(shape_descriptor_by_id(id).unwrap().keys, moved_keys); + + // The keys-address reverse index is repaired by the metadata pass, not + // by the store; force it the way `scan_shape_table_rekey_mut` does. + { + let mut inner = crate::state::state().shapes.inner.borrow_mut(); + rebuild_descriptor_reverse_indices(&mut inner); + } + test_drop_shape_descriptors(moved_keys as usize); + assert_eq!( + shape_descriptor_by_id(id), + None, + "descriptor rekey did not update the keys-address index" + ); + } + + #[test] + fn a_boxed_record_keeps_its_keys_slot_across_table_growth() { + let _lock = crate::gc::global_side_table_test_lock(); + // The prohibition #8067 recorded — "descriptor insertion can reallocate + // the table" — is what BOXING answers. Mint one descriptor, take its + // slot, then mint enough siblings to force several rehashes and assert + // the address never moved. Without the box this fails. + let keys = 0x8112_0000_0000_1000usize; + let id = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) + .expect("shape range unexpectedly exhausted"); + let slot = shape_descriptor_keys_slot(id).expect("minted id has a keys slot"); + + let mut minted = Vec::new(); + for i in 1..512usize { + let sibling = keys + i * 0x40; + minted.push( + shape_descriptor_ensure(sibling as *const ArrayHeader, 1, 1) + .expect("shape range unexpectedly exhausted"), + ); + } + assert_eq!( + shape_descriptor_keys_slot(id), + Some(slot), + "descriptor insertion moved a keys slot the collector may still hold" + ); + assert_eq!(unsafe { *slot }, keys as u64); + + test_drop_shape_descriptors(keys); + for i in 1..512usize { + test_drop_shape_descriptors(keys + i * 0x40); + } + } + + #[test] + fn key_count_versions_remain_resolvable_until_the_keys_die() { + let _lock = crate::gc::global_side_table_test_lock(); + let keys = 0x8067_0000_0000_2100usize; + let unrelated_keys = 0x8067_0000_0000_2200usize; + let stale_a = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) + .expect("shape range unexpectedly exhausted"); + let stale_b = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 2) + .expect("shape range unexpectedly exhausted"); + let current = shape_descriptor_ensure(keys as *const ArrayHeader, 2, 2) + .expect("shape range unexpectedly exhausted"); + let unrelated = shape_descriptor_ensure(unrelated_keys as *const ArrayHeader, 1, 1) + .expect("shape range unexpectedly exhausted"); + + retain_key_count_versions(keys as u64); + + assert!(shape_descriptor_by_id(stale_a).is_some()); + assert!(shape_descriptor_by_id(stale_b).is_some()); + assert!(shape_descriptor_by_id(current).is_some()); + assert!(shape_descriptor_by_id(unrelated).is_some()); + let inner = crate::state::state().shapes.inner.borrow(); + let current_ids = inner + .ids_by_keys + .get(&(keys as u64)) + .expect("keys identity disappeared from descriptor index"); + assert_eq!(current_ids.as_slice(), &[stale_a, stale_b, current]); + drop(inner); + + test_drop_shape_descriptors(keys); + test_drop_shape_descriptors(unrelated_keys); + } + + #[test] + fn shape_drop_does_not_delete_a_potential_siblings_descriptor() { + let _lock = crate::gc::global_side_table_test_lock(); + let keys = 0x8067_0000_0000_3000usize; + let id = shape_descriptor_ensure(keys as *const ArrayHeader, 1, 1) + .expect("shape range unexpectedly exhausted"); + + shape_drop(keys as *const ArrayHeader); + + assert_eq!( + shape_descriptor_by_id(id).map(|descriptor| descriptor.keys), + Some(keys as u64), + "shape_drop eagerly invalidated a descriptor a sibling may still name" + ); + test_drop_shape_descriptors(keys); + } + + #[test] + fn live_slot_growth_versions_descriptor_before_value_publication() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let packed = b"slot8067_a"; + let obj = crate::object::js_object_alloc_with_shape( + 0x8067_1001, + 1, + packed.as_ptr(), + packed.len() as u32, + ); + let keys = (*obj).keys_array as usize; + let before = (*obj).parent_class_id; + let before_descriptor = shape_descriptor_by_id(before).expect("birth descriptor"); + assert_eq!(before_descriptor.live_inline_slot_count, 1); + + crate::object::js_object_set_field(obj, 1, crate::JSValue::string_ptr(key("value"))); + let after = (*obj).parent_class_id; + assert_ne!(before, after); + let after_descriptor = shape_descriptor_by_id(after).expect("grown descriptor"); + assert_eq!(after_descriptor.keys, keys as u64); + assert_eq!(after_descriptor.logical_key_count, 1); + assert_eq!(after_descriptor.live_inline_slot_count, 2); + debug_assert_object_shape_parity(obj); + } + } + + #[test] + fn shared_sibling_append_clones_before_descriptor_version_changes() { + let _lock = crate::gc::global_side_table_test_lock(); + unsafe { + let packed = b"sib8067_a"; + let a = crate::object::js_object_alloc_with_shape( + 0x8067_1002, + 1, + packed.as_ptr(), + packed.len() as u32, + ); + let b = crate::object::js_object_alloc_with_shape( + 0x8067_1002, + 1, + packed.as_ptr(), + packed.len() as u32, + ); + let shared_keys = (*a).keys_array; + let shared_id = (*a).parent_class_id; + assert_eq!(shared_keys, (*b).keys_array); + assert_eq!(shared_id, (*b).parent_class_id); + + crate::object::js_object_set_field_by_name(a, key("sib8067_b"), 2.0); + + assert_ne!((*a).keys_array, shared_keys); + assert_eq!((*b).keys_array, shared_keys); + assert_eq!((*b).parent_class_id, shared_id); + assert_ne!((*a).parent_class_id, shared_id); + assert_eq!( + shape_descriptor_by_id(shared_id) + .expect("untouched sibling descriptor") + .logical_key_count, + 1 + ); + let transitioned = + shape_descriptor_by_id((*a).parent_class_id).expect("transitioned descriptor"); + assert_eq!(transitioned.keys, (*a).keys_array as u64); + assert_eq!(transitioned.logical_key_count, 2); + assert_eq!(transitioned.live_inline_slot_count, 2); + } + } +} diff --git a/scripts/shape_descriptor_census.py b/scripts/shape_descriptor_census.py index 487b1204c7..9238c92ed0 100644 --- a/scripts/shape_descriptor_census.py +++ b/scripts/shape_descriptor_census.py @@ -293,16 +293,31 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: for pattern, label in ( # `PtrHashMap` since #8157 (SipHash on a bare u32 was 25% of self time in - # `shapes`). The fact this asserts is that a by-id table EXISTS, not which - # hasher backs it, so accept either spelling. - (r"descriptors\s*:\s*(?:[\w:]+::)?(?:Ptr)?HashMap\s*<\s*u32\s*,\s*ShapeDescriptor", "by-id descriptor table"), + # `shapes`). The hasher is free; the BOX is not. Since #8112 the + # collector enumerates `&mut record.keys` as an ordinary GC slot, and a + # budgeted dirty scan can hold that address across mutator resumptions + # that insert descriptors. Un-boxing the value puts the record back in + # the bucket, where a rehash moves it under the collector's feet. + (r"descriptors\s*:\s*(?:[\w:]+::)?(?:Ptr)?HashMap\s*<\s*u32\s*,\s*Box\s*<\s*ShapeDescriptor\s*>", "by-id descriptor table, boxed for a stable keys slot"), (r"logical_key_count\s*:\s*u32", "exact logical-key fact"), (r"live_inline_slot_count\s*:\s*u32", "exact live-slot fact"), (r"semantic_generation\s*:\s*u64", "semantic transition fact"), (r"object_kind\s*:\s*ShapeObjectKind", "authoritative receiver-kind fact"), (r"\bfn\s+shape_descriptor_by_id\b", "by-id lookup"), (r"\bfn\s+debug_assert_object_shape_parity\b", "parity assertion"), - (r"\bfn\s+synchronize_live_object_shape_descriptor_after_header_visit\b", "live-object descriptor mirror"), + # #8112 replaced the post-visit write-back callback with a rewritable + # location: the lifted descriptor carries the address of its own BOXED + # record, so the slot visitor writes that record and there is nothing + # left to reconcile. + (r"pub\(crate\)\s+record\s*:\s*usize", "authoritative descriptor record address"), + (r"\bfn\s+keys_slot\b", "authoritative keys-edge slot"), + # The liveness gate. Rooting the table unconditionally would make every + # keys array ever minted immortal and turn `prune_dead_shape_keys`'s + # "is the keys array dead?" into a question it asks of itself; rooting + # nothing loses the keys array of a shape only OLD objects carry, which + # a minor never enumerates. + (r"pub\(crate\)\s+old_carrier\s*:\s*bool", "old-carrier ephemeron gate"), + (r"\bfn\s+rotate_old_carrier_epoch_after_full_trace\b", "old-carrier gate recomputed by a full trace"), (r"is_dead_owner\s*\(\s*descriptor\.keys\s+as\s+usize\s*\)", "dead descriptor pruning"), ): require_code(shapes, pattern, label) @@ -321,31 +336,57 @@ def assert_authority_surfaces(sources: dict[str, str]) -> None: scanner = function_body(shapes, "scan_shape_table_rekey_mut") require_code(scanner, r"\bvisit_metadata_usize_slot\s*\(", "weak metadata rewrite") + # #8112: the scanner has exactly TWO arms, and which one a descriptor takes + # IS the liveness protocol. `visit_usize_slot` ROOTS — reserved for a shape + # an OLD object still carries, which a minor cannot enumerate for itself. + # `visit_metadata_usize_slot` does not root — a young carrier is traced and + # emits its own edge, and rooting those from the table would make every + # keys array ever minted immortal. Pin the whole two-armed expression, not + # just the set of APIs called: a sabotage that widens the gate, or that + # swaps the arms, has to be red. + if not re.search( + r"if\s+descriptor\.old_carrier\s*\{\s*" + r"visitor\.visit_usize_slot\(&mut addr\)\s*\}\s*else\s*\{\s*" + r"visitor\.visit_metadata_usize_slot\(&mut addr\)\s*\}", + scanner, + ): + raise CensusError( + "descriptor rooting is not gated on `old_carrier`: the shape table " + "either roots unconditionally (every keys array immortal) or not at " + "all (a shape only old objects carry loses its keys array)" + ) scanner_slot_apis = set(re.findall(r"\b(visit_[A-Za-z0-9_]*slot)\s*\(", scanner)) - if scanner_slot_apis != {"visit_metadata_usize_slot"}: + if scanner_slot_apis != {"visit_metadata_usize_slot", "visit_usize_slot"}: raise CensusError( "descriptor scanner slot API allowlist failed: " + ", ".join(sorted(scanner_slot_apis)) ) layout_body = function_body(layout_visit, "visit_gc_layout_slot_descriptors") - assert_before( + require_code( layout_body, - "child_slots.take_prefix_child_slot()", - "synchronize_live_object_shape_descriptor_after_header_visit(", - "authoritative header visit before descriptor mirror", + r"gc_shape_keys_edge_slot\s*\(", + "descriptor keys edge enumerated as a child slot", ) + # #8112: the authoritative edge is resolved from the descriptor BEFORE the + # derived header mirror is handed to the visitor. Reversing that would let + # a mirror rewrite feed back into the edge, which is the inversion this + # issue exists to remove. assert_before( layout_body, - "try_read_tracked_gc_header(old_keys as usize)", - "keys_array_len_capped_to_capacity(old_keys)", - "array-header validation before descriptor fact read", - ) - require_code( - layout_body, - r"\(\s*\*\s*keys_header\.as_ptr\s*\(\s*\)\s*\)\.obj_type\s*==\s*GC_TYPE_ARRAY", - "descriptor fact capture exact array type", + "gc_shape_keys_edge_slot(", + "child_slots.take_prefix_child_slot()", + "authoritative descriptor edge before derived mirror visit", ) + # And nothing in the visit reads the mirror for a FACT. The old model + # captured `logical_key_count` from the header's array and validated the + # header word before dereferencing it; both are gone, which is why #8047 + # can delete the word without touching this function's logic. + if re.search(r"keys_array", layout_body): + raise CensusError( + "the GC slot visitor reads ObjectHeader::keys_array again; the " + "descriptor is the authoritative edge since #8112" + ) ensure = function_body(shapes, "shape_descriptor_ensure_with_generation") assert_before( @@ -705,10 +746,47 @@ def run_sabotage_selftests(sources: dict[str, str], baseline: dict[str, object]) path = "crates/perry-runtime/src/gc/layout_slot_visit.rs" inverted_gc[path] = swap_once( inverted_gc[path], + "gc_shape_keys_edge_slot(", "child_slots.take_prefix_child_slot()", - "synchronize_live_object_shape_descriptor_after_header_visit(", ) - expect_rejected("descriptor before header visit", lambda: assert_authority_surfaces(inverted_gc)) + expect_rejected( + "derived mirror visited before the authoritative descriptor edge", + lambda: assert_authority_surfaces(inverted_gc), + ) + + shapes_path = "crates/perry-runtime/src/object/shapes.rs" + unboxed_table = dict(sources) + unboxed_table[shapes_path] = unboxed_table[shapes_path].replace( + "PtrHashMap>", + "PtrHashMap", + 1, + ) + expect_rejected( + "descriptor record un-boxed back into a rehashing bucket", + lambda: assert_authority_surfaces(unboxed_table), + ) + + ungated_root = dict(sources) + ungated_root[shapes_path] = ungated_root[shapes_path].replace( + "let moved = if descriptor.old_carrier {", + "let moved = if true {", + 1, + ) + expect_rejected( + "descriptor rooting un-gated into an unconditional table root", + lambda: assert_authority_surfaces(ungated_root), + ) + + header_fact_read = dict(sources) + header_fact_read[path] = header_fact_read[path].replace( + "let shape_keys_edge = if", + "let _mirror = (*obj).keys_array;\n let shape_keys_edge = if", + 1, + ) + expect_rejected( + "GC slot visitor reads the header mirror for a fact", + lambda: assert_authority_surfaces(header_fact_read), + ) inverted_publication = dict(sources) path = "crates/perry-runtime/src/object/shapes.rs" diff --git a/scripts/shape_descriptor_census_baseline.json b/scripts/shape_descriptor_census_baseline.json index e892a78c94..b71eeef6c6 100644 --- a/scripts/shape_descriptor_census_baseline.json +++ b/scripts/shape_descriptor_census_baseline.json @@ -39,8 +39,6 @@ "crates/perry-runtime/src/gc/heap_snapshot.rs|keys_array|access|.unwrap_or((*obj).keys_array as u64);": 1, "crates/perry-runtime/src/gc/layout.rs|keys_array|access|.unwrap_or((*object).keys_array as usize);": 1, "crates/perry-runtime/src/gc/layout/typed_shape.rs|keys_array|access|.unwrap_or((*obj_header).keys_array as usize);": 1, - "crates/perry-runtime/src/gc/layout_slot_visit.rs|keys_array|access|.unwrap_or((*obj).keys_array);": 1, - "crates/perry-runtime/src/gc/layout_slot_visit.rs|keys_array|access|let new_keys = (*obj).keys_array as u64;": 1, "crates/perry-runtime/src/gc/tests/copying.rs|keys_array|access|let keys = (*obj_after).keys_array;": 1, "crates/perry-runtime/src/gc/tests/cycle_state.rs|keys_array|access|(*child).keys_array = std::ptr::null_mut();": 1, "crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs|keys_array|access|(*a).keys_array = keys;": 1, @@ -52,7 +50,8 @@ "crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs|keys_array|access|assert_eq!((*a_after).keys_array, (*b_after).keys_array);": 2, "crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs|keys_array|access|assert_eq!(descriptor.keys, (*a_after).keys_array as u64);": 1, "crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs|keys_array|access|assert_ne!((*a_after).keys_array as usize, old_keys);": 1, - "crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs|keys_array|access|let header_keys_slot = unsafe { std::ptr::addr_of_mut!((*owner).keys_array) as *mut u64 };": 2, + "crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs|keys_array|access|let header_keys_slot = unsafe { std::ptr::addr_of_mut!((*owner).keys_array) as *mut u64 };": 1, + "crates/perry-runtime/src/gc/tests/dead_owner_side_tables.rs|keys_array|access|unsafe { std::ptr::addr_of_mut!((*a).keys_array) } as usize,": 1, "crates/perry-runtime/src/gc/tests/layout_trace/typed_shape.rs|keys_array|access|(*first).keys_array,": 1, "crates/perry-runtime/src/gc/tests/layout_trace/typed_shape.rs|keys_array|access|(*second).keys_array,": 1, "crates/perry-runtime/src/gc/tests/runtime_roots/json_shape_template.rs|keys_array|access|let keys = (*obj).keys_array;": 1, @@ -144,22 +143,21 @@ "crates/perry-runtime/src/object/object_ops/keys_array.rs|keys_array|access|let keys = (*obj).keys_array;": 4, "crates/perry-runtime/src/object/object_ops_frozen.rs|keys_array|access|let keys = (*obj).keys_array;": 1, "crates/perry-runtime/src/object/reflect_support.rs|keys_array|access|let keys_handle = scope.root_raw_mut_ptr((*obj).keys_array);": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|(*a).keys_array,": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|(*b).keys_array,": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|assert_eq!((*b).keys_array, shared_keys);": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|assert_eq!(descriptor.keys, (*obj).keys_array as u64);": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|assert_eq!(shared_keys, (*b).keys_array);": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|assert_eq!(transitioned.keys, (*a).keys_array as u64);": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|assert_ne!((*a).keys_array, shared_keys);": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|crate::array::js_array_length((*obj).keys_array)": 1, "crates/perry-runtime/src/object/shapes.rs|keys_array|access|d.keys == (*obj).keys_array as u64": 1, "crates/perry-runtime/src/object/shapes.rs|keys_array|access|debug_assert_object_shape_parity_for_keys(obj, (*obj).keys_array);": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|let keys = (*obj).keys_array as usize;": 1, "crates/perry-runtime/src/object/shapes.rs|keys_array|access|let keys = (*obj).keys_array;": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|let shared_keys = (*a).keys_array;": 1, "crates/perry-runtime/src/object/shapes.rs|keys_array|access|publish_object_shape_from(obj, predecessor, (*obj).keys_array, live_inline_slot_count)": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|access|unsafe { (*obj).keys_array },": 1, - "crates/perry-runtime/src/object/shapes.rs|keys_array|declaration|keys_array: keys as *mut ArrayHeader,": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|(*a).keys_array,": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|(*b).keys_array,": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|assert_eq!((*b).keys_array, shared_keys);": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|assert_eq!(descriptor.keys, (*obj).keys_array as u64);": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|assert_eq!(shared_keys, (*b).keys_array);": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|assert_eq!(transitioned.keys, (*a).keys_array as u64);": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|assert_ne!((*a).keys_array, shared_keys);": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|crate::array::js_array_length((*obj).keys_array)": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|let keys = (*obj).keys_array as usize;": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|let shared_keys = (*a).keys_array;": 1, + "crates/perry-runtime/src/object/shapes_tests.rs|keys_array|access|unsafe { (*obj).keys_array },": 1, "crates/perry-runtime/src/param_type_guard.rs|keys_array|access|let keys = (*object).keys_array;": 1, "crates/perry-runtime/src/perf_hooks.rs|keys_array|access|let keys_ptr = (*obj).keys_array as usize;": 1, "crates/perry-runtime/src/perf_hooks.rs|keys_array|access|recorded != 0 && (*obj).keys_array as usize == recorded": 1, @@ -181,7 +179,7 @@ "codegen_object_header_size_sites": 34, "raw_member_files": 66, "raw_member_sites": { - "keys_array": 183 + "keys_array": 180 } } }