Show GC results for every ruby when comparing builds

With three or more rubies, the GC time ratios and GC counts tables only
had one row per comparison (base → other), so no ruby had its own row
and the baseline appeared only on the left of each arrow.

harness-gc also printed 0 live and free slots in the heap utilisation
table on Ruby 3.4, because GC.stat_heap before 4.0 has no
heap_live_slots or heap_free_slots.

This commit adds a "GC per ruby" table that prints before the GC time
ratios table. It has one row per ruby per benchmark, including the
base, with the same absolute columns as the single-ruby GC summary. The
table is left out when only one ruby runs, because the GC summary
already shows those values. A benchmark is left out only when no ruby
has GC activity.

The legend and GC metric notes now say which table each sentence
describes, and they describe each table's own skip rule. The Ractor GC
ms/worker note now appears whenever a table shows that column.

After the full GC, harness-gc now derives live and free slots from the
per-heap total_allocated_objects and total_freed_objects counters when
GC.stat_heap does not report them.

Author: eightbitraptor

README.md

@@ -521,18 +521,21 @@ Ractor's own object space. The JSON output records the scope as
 
 The text summary shows GC data in separate tables after the timing table.
 A single-executable report has one `GC summary` table. A comparison report
-has a `GC time ratios` table (base/comparison) and a `GC counts` table
-(base → comparison). A table hides a column that has no data in any row and
+has a `GC per ruby` table, a `GC time ratios` table (base/comparison), and a
+`GC counts` table (base → comparison). The `GC per ruby` table has one row
+for each ruby, including the base, with the absolute `GC summary` columns.
+A table hides a column that has no data in any row and
 lists the hidden columns below the table. A ratio column has no data when it
 is `N/A` in every row; a `0.000` ratio stays visible. Any other column has no
 data when it is zero or `N/A` in every row.
 
 * Tables marked `worker sum` add the Ractor-local counters and GC times of
   the sampled workers of each iteration. `GCs/iter` is the sum of `minor/iter`,
   `major/iter`, and `global/iter`; a global cycle counts under `global` on
-  the Ractor that initiated it, not under `major`. Single-executable reports
-  also show `GC ms/worker`, which divides each iteration's worker-sum GC
-  time by its sampled worker count, then averages.
+  the Ractor that initiated it, not under `major`. The absolute GC tables
+  (the single-executable `GC summary` and the `GC per ruby` table) also show
+  `GC ms/worker`, which divides each iteration's worker-sum GC time by its
+  sampled worker count, then averages.
 * `compacts*` shows the main Ractor's
   `GC.stat(:compact_count)` delta. Every global compacting cycle increments
   it in every object space, so it is not summed across workers.

harness/gc.rb

@@ -148,8 +148,9 @@ def run_benchmark(_num_itrs_hint, **, &block)
     heap_snapshot.each do |idx, stats|
       slot_size = stats[:slot_size] || 0
       eden_slots = stats[:heap_eden_slots] || 0
-      live_slots = stats[:heap_live_slots] || 0
-      free_slots = stats[:heap_free_slots] || 0
+      # Ruby < 4.0 omits heap_live_slots/heap_free_slots; derive them from the per-heap counters.
+      live_slots = stats[:heap_live_slots] || stats[:total_allocated_objects].to_i - stats[:total_freed_objects].to_i
+      free_slots = stats[:heap_free_slots] || eden_slots - live_slots
       eden_pages = stats[:heap_eden_pages] || 0
       live_pct = eden_slots > 0 ? (live_slots * 100.0 / eden_slots) : 0.0
       mem_kib = page_size ? (eden_pages * page_size / 1024.0) : 0.0

lib/benchmark_runner.rb

@@ -74,6 +74,8 @@ def build_output_text(ruby_descriptions, table, format, bench_failures, include_
       sections ||= [{ table: table, format: format, failures: bench_failures, include_gc: include_gc, gc_tables: gc_tables, memory_tables: memory_tables }]
       has_gc_summary = sections.any? { |section| section[:include_gc] && section[:gc_tables] }
       has_memory_summary = sections.any? { |section| section[:memory_tables] }
+      gc_table_list = sections.flat_map { |section| Array(section[:gc_tables]) }
+      per_ruby_headers = gc_table_list.select { |gc_table| gc_table[:name] == 'GC per ruby' }.flat_map { |gc_table| gc_table[:rows].first }
       sections.each do |section|
         title = section[:title]
         output_str << "#{title}:\n" if title
@@ -93,10 +95,11 @@ def build_output_text(ruby_descriptions, table, format, bench_failures, include_
           end
         end
         if has_gc_summary
+          output_str << "- GC per ruby shows the mean GC values per benchmark iteration for each ruby. These values are not ratios. Benchmarks with no GC activity on any ruby are omitted.\n" unless per_ruby_headers.empty?
           output_str << "- GC time ratios are #{base_name}/comparison; above 1 means the comparison spent less GC time.\n"
           output_str << "- gc/iter, mark/iter, and sweep/iter compare total GC (or phase) time per benchmark iteration, so they include both per-GC cost and GC frequency changes.\n"
           output_str << "- gc/GC, mark/GC, and sweep/GC divide average GC (or phase) time by the same run's GCs/iter count; they are not complete per-cycle attribution of process-wide GC.\n"
-          output_str << "- GC counts show #{base_name} → comparison values, not ratios. Rows with no GC activity are omitted.\n"
+          output_str << "- GC counts show #{base_name} → comparison values, not ratios. In GC time ratios and GC counts, a row is omitted when neither #{base_name} nor the comparison has GC activity.\n"
         end
         if has_memory_summary
           output_str << "- Scenario memory shows #{base_name} → comparison values, not ratios. Lower is better.\n"
@@ -105,10 +108,13 @@ def build_output_text(ruby_descriptions, table, format, bench_failures, include_
           output_str << "- ***: p < 0.001, **: p < 0.01, *: p < 0.05 (Welch's t-test)\n"
         end
       end
-      gc_headers = sections.flat_map { |section| Array(section[:gc_tables]).flat_map { |gc_table| gc_table[:rows].first } }
+      gc_headers = gc_table_list.flat_map { |gc_table| gc_table[:rows].first }
       if gc_headers.include?('compacts*')
         output_str << "GC metric notes:\n"
-        output_str << "- compacts*: the main Ractor's GC.stat(:compact_count) delta per iteration. Every global compacting cycle increments compact_count in every object space. Do not sum it across workers.#{other_names.empty? ? '' : " Comparison tables show #{base_name} → comparison values."}\n"
+        compaction_note = +""
+        compaction_note << " GC counts show #{base_name} → comparison values." unless other_names.empty?
+        compaction_note << " GC per ruby shows the value for each ruby." if per_ruby_headers.include?('compacts*')
+        output_str << "- compacts*: the main Ractor's GC.stat(:compact_count) delta per iteration. Every global compacting cycle increments compact_count in every object space. Do not sum it across workers.#{compaction_note}\n"
       end
 
       ractor_gc_sections = sections.select { |section| section[:gc_scope] == 'ractor-local-workload' && section[:gc_tables] }
@@ -118,7 +124,7 @@ def build_output_text(ruby_descriptions, table, format, bench_failures, include_
         output_str << "Ractor GC scope note:\n"
         scope_columns = +"- Tables marked worker sum add Ractor-local counters and GC times across the sampled workers of each iteration"
         scope_columns << (worker_mode ? "; the main Ractor performs the count-0 workload of per-worker benchmarks." : ".")
-        scope_columns << " GC ms/worker divides each iteration's worker-sum GC time by its sampled worker count, then averages." if other_names.empty?
+        scope_columns << " GC ms/worker divides each iteration's worker-sum GC time by its sampled worker count, then averages." if gc_headers.include?('GC ms/worker')
         output_str << "#{scope_columns} Controller snapshots and per-worker heap detail are in the JSON output, not these tables.\n"
         output_str << "- Ruby's Ractor-retirement GC (after a worker's stack is torn down) is not sampled."
         output_str << " Ractors created by the workload of a per-worker benchmark are not sampled." if worker_mode

lib/results_table_builder.rb

@@ -124,7 +124,7 @@ def build_gc_tables
     return nil unless @include_gc
 
     label_header = ["bench", *@row_layout.extra_header_columns]
-    tables = @other_names.empty? ? [build_gc_absolute_table(label_header)] : build_gc_comparison_tables(label_header)
+    tables = @other_names.empty? ? [build_gc_absolute_table(label_header)] : [build_gc_per_ruby_table(label_header), *build_gc_comparison_tables(label_header)]
     tables.compact!
     tables.empty? ? nil : tables
   end
@@ -192,6 +192,19 @@ def gc_count_columns
     columns
   end
 
+  def build_gc_per_ruby_table(label_header)
+    rows = []
+    gc_entries.each do |entry|
+      data_by_name = @executable_names.to_h { |name| [name, bench_data_for(name, entry.data_key)] }
+      next unless gc_activity?(*data_by_name.values.flat_map { |data| data.values_at(*GC_SERIES_KEYS) })
+
+      data_by_name.each do |name, data|
+        rows << [gc_label_cells(entry) + [name], [data]] if GC_SERIES_KEYS.any? { |key| data.key?(key) }
+      end
+    end
+    assemble_gc_table("GC per ruby", label_header + ["ruby"], rows, gc_absolute_columns)
+  end
+
   def build_gc_absolute_table(label_header)
     rows = gc_entries.filter_map do |entry|
       data = bench_data_for(@base_name, entry.data_key)

test/benchmark_runner_test.rb

@@ -524,7 +524,8 @@ def git(dir, *args)
       assert_includes result, "- compacts*: the main Ractor's GC.stat(:compact_count) delta per iteration."
       assert_includes result, 'Every global compacting cycle increments compact_count in every object space.'
       assert_includes result, 'Do not sum it across workers.'
-      assert_includes result, 'Comparison tables show ruby-base → comparison values.'
+      assert_includes result, 'GC counts show ruby-base → comparison values.'
+      refute_includes result, 'GC per ruby shows the value for each ruby.'
       refute_includes result, 'global GCs/iter'
     end
 
@@ -546,7 +547,7 @@ def git(dir, *args)
       refute_includes result, 'Legend:'
       assert_includes result, "GC metric notes:\n"
       assert_includes result, "- compacts*: the main Ractor's GC.stat(:compact_count) delta per iteration."
-      refute_includes result, 'Comparison tables show'
+      refute_includes result, 'GC counts show'
       refute_includes result, 'global GCs/iter'
     end
 
@@ -573,7 +574,7 @@ def git(dir, *args)
 
       refute_includes result, 'GC metric notes:'
       assert_includes result, "the same run's GCs/iter count"
-      assert_includes result, 'GC counts show ruby-base → comparison values, not ratios'
+      assert_includes result, 'GC counts show ruby-base → comparison values, not ratios. In GC time ratios and GC counts, a row is omitted when neither ruby-base nor the comparison has GC activity.'
     end
 
     it 'omits the Ractor scope note when the section rendered no GC table' do
@@ -597,6 +598,80 @@ def git(dir, *args)
       refute_includes result, 'Ractor GC scope note:'
     end
 
+    it 'prints the GC per ruby table before the GC ratio table, with its legend line' do
+      ruby_descriptions = { 'master' => 'ruby 4.1.0dev', 'four' => 'ruby 4.0.6', 'three' => 'ruby 3.4.10' }
+      sections = [
+        {
+          title: 'harness-gc',
+          table: [['bench', 'master (ms)', 'four (ms)', 'three (ms)'], ['gcbench', '1.0', '2.0', '3.0']],
+          format: ['%s', '%s', '%s', '%s'],
+          failures: {},
+          include_gc: true,
+          gc_tables: [
+            { name: 'GC per ruby', scope: nil, hidden: [], rows: [
+              ['bench', 'ruby', 'GC ms/iter', 'compacts*'],
+              ['gcbench', 'master', '1.000', '0.0'],
+              ['gcbench', 'four', '2.000', '0.0'],
+              ['gcbench', 'three', '3.000', '0.0']
+            ] },
+            { name: 'GC time ratios', scope: nil, hidden: [], rows: [
+              ['bench', 'comparison', 'gc/iter'], ['gcbench', 'four', '0.500'], ['gcbench', 'three', '0.333']
+            ] },
+          ],
+        },
+        {
+          title: 'harness',
+          table: [['bench', 'master (ms)', 'four (ms)', 'three (ms)'], ['fib', '1.0', '2.0', '3.0']],
+          format: ['%s', '%s', '%s', '%s'],
+          failures: {},
+          include_gc: false,
+        }
+      ]
+
+      result = BenchmarkRunner.build_output_text(
+        ruby_descriptions, sections.first[:table], sections.first[:format], {}, sections: sections
+      )
+
+      per_ruby_at = result.index("GC per ruby (harness-gc):\n")
+      summary_at = result.index("GC time ratios (harness-gc):\n")
+      refute_nil per_ruby_at
+      refute_nil summary_at
+      assert_operator per_ruby_at, :<, summary_at
+      assert_match(/^gcbench\s+master\s+1\.000\s+0\.0$/, result)
+      assert_includes result, '- GC per ruby shows the mean GC values per benchmark iteration for each ruby. These values are not ratios. Benchmarks with no GC activity on any ruby are omitted.'
+      assert_includes result, 'Do not sum it across workers. GC counts show master → comparison values. GC per ruby shows the value for each ruby.'
+      refute_includes result, 'GC per ruby (harness):'
+    end
+
+    it 'explains GC ms/worker in the Ractor scope note when only the GC per ruby table shows it' do
+      ruby_descriptions = { 'base' => 'ruby 4.1.0dev', 'exp' => 'ruby 4.1.0dev experiment' }
+      sections = [
+        {
+          title: 'harness-ractor',
+          table: [['bench', 'ractors', 'base (ms)', 'exp (ms)'], ['object-new', '2', '1.0', '2.0']],
+          format: ['%s', '%s', '%s', '%s'],
+          failures: {},
+          include_gc: true,
+          gc_tables: [
+            { name: 'GC per ruby', scope: 'worker sum', hidden: [], rows: [
+              ['bench', 'ractors', 'ruby', 'GC ms/iter', 'GC ms/worker'], ['object-new', '2', 'base', '4.000', '2.000']
+            ] },
+            { name: 'GC time ratios', scope: 'worker sum', hidden: [], rows: [
+              ['bench', 'ractors', 'gc/iter'], ['object-new', '2', '2.000']
+            ] },
+          ],
+          gc_scope: 'ractor-local-workload',
+        }
+      ]
+
+      result = BenchmarkRunner.build_output_text(
+        ruby_descriptions, sections.first[:table], sections.first[:format], {}, sections: sections
+      )
+
+      assert_includes result, "Ractor GC scope note:\n"
+      assert_includes result, "GC ms/worker divides each iteration's worker-sum GC time by its sampled worker count, then averages."
+    end
+
     it 'includes RSS ratio legend when include_rss is true' do
       ruby_descriptions = {
         'ruby-base' => 'ruby 3.3.0',