The columnar scan is priced 1.9x to 3.4x below the heap per unit of work, and seq_page_cost decides whether that is a defect

[jdatcmd]

Measured while re-deriving #753 (see that issue's measurement comment for the full run). The
finding there splits in two, and this is the half that belongs to pgColumnar rather than to
core.

The columnar scan cost model disagrees with itself as the projection widens

Cost per millisecond of real serial execution, same query, same 2,000,197 rows, same data, on
a 4,000,000-row 14-column table and its heap twin. Core's seqscan calibration is the reference
scale: not a claim that it is true, but it is the scale every other path in the planner is
judged against.

projection	columnar cost	columnar min ms	cost/ms	heap cost/ms	heap ÷ columnar
sel	61,065	176.0	347	858	2.47x
sel, a, b	83,196	216.4	385	846	2.20x
8 columns	138,522	268.3	516	765	1.48x
*	239,001	1279.1	187	944	5.05x

Two things, and the second is the one worth an issue:

1. The columnar scan is under-priced relative to that scale by 1.5x to 5x. A path priced
   at half what it costs wins comparisons it should lose.

2. It is not a constant offset. The heap's cost/ms is stable across projection widths, 765
   to 944, a 1.24x spread. Columnar's runs 187 to 516, a 2.8x spread, and not
   monotonically. So the model does not merely need a multiplier; its shape in projection
   width is wrong.

The * row is the same defect at its extreme: a full-row columnar scan of this table takes
1279 ms against the heap's 167 ms, 7.6x slower, while being priced at 1.5x the heap. That
columnar loses at SELECT * is expected and fine. That the cost model says 1.5x when reality
is 7.6x is not.

Why this cannot just be corrected

The scan cost is the scale every other path comparison is judged on. Raising it toward reality
would change, at least:

• the index-fetch penalty (Planner chooses an Index Scan on columnar for ordering; per-row fetch cost is not modelled and the plan is orders of magnitude slower #355, The #355 index-fetch penalty changes the estimate but not the plan: the cheaper path is already freed when it is applied #362), which is calibrated as a fraction of
  pgcolumnar_full_scan_cost;
• Point lookup regressed 23.75 ms to 1251.88 ms once ANALYZE has run, and ANALYZE itself is pathologically slow on some shapes #171, where a point lookup went from 23.75 ms to 1251.88 ms because ANALYZE made the
  full scan look cheap enough to beat an index scan. That regression is pinned in
  test/analyze_stats.sh and it lives on exactly this number;
• Index-fetch penalty collapses to zero on a correlated index, and the custom scan is priced as if zone maps did not exist #434's zone-map survival scaling and A columnar scan is priced with no decode cost, so the planner declines a parallel projection that is 2.9x faster #503's per-column decode CPU, which are the two
  refinements already layered on pgcolumnar_refined_scan_cost;
• the grouped vector aggregate, which shares the same input estimate.

So this is not a one-line recalibration, and it must not be attempted as a side effect of a
parallelism fix.

What the work is

1. Establish whether the divergence is in the per-column decode term (A columnar scan is priced with no decode cost, so the planner declines a parallel projection that is 2.9x faster #503), the
   projected-width I/O term (Point lookup regressed 23.75 ms to 1251.88 ms once ANALYZE has run, and ANALYZE itself is pathologically slow on some shapes #171), or the base. The 2.8x spread across widths says at
   least one width-dependent term is mis-shaped, and the * outlier says the widest case is
   worst, which points at decode rather than I/O.
2. Re-derive each term against measured time on a fixture with real entropy, at several
   projection widths and several column types (the * case here includes four md5 text
   columns and a numeric; text decode may be the whole story).
3. Change one term at a time, with the suites above as the controls, and require
   analyze_stats and the index-path suites green at each step rather than only at the end.

What would make this closable

A cost/ms spread for columnar comparable to the heap's 1.24x across the same projection
widths, with the point-lookup and index-fetch behaviour unchanged. The absolute scale matters
less than the consistency: a model that is uniformly 2x low is a multiplier away from correct,
and one that varies 2.8x with projection width is choosing wrong plans in both directions.

[jdatcmd]

Correction: the headline in the issue body is wrong, and this replaces it

The first sweep varied which columns as well as how many: * was the only projection
that touched the md5 text columns, so its jump could be text decode wearing a width costume.
It was. Re-run with width and type separated, 2,000,197 rows, PG 17, min of 7 reps.

Int-only ladder: only the NUMBER of int4 columns varies

projection	columnar cost	ms	cost x	time x	cost/ms	heap cost/ms
sel	61,065	167.5	1.00	1.00	364.7	891.7
sel, a	72,130	198.1	1.18	1.18	364.1	844.9
sel, a, b	83,196	214.5	1.36	1.28	387.8	823.4
sel, a, b, c, d	105,326	233.1	1.72	1.39	451.9	809.5
8 int columns	138,522	271.2	2.27	1.62	510.8	716.6

The width defect is real but modest, and it is nothing like what this issue was filed
with. Columnar's cost/ms spread across the ladder is 1.40x, against the heap's 1.24x.
The "2.8x against 1.24x" in the issue body above is withdrawn: the 2.8x was 516/187 and the
187 came entirely from the * cell.

There is also no sign change on int columns. Cost grows 2.27x while time grows 1.62x,
monotonically, so on this ladder the model gets steadily dearer than reality as width grows,
while the heap's cost is flat at 158,109 and its time grows 1.24x.

What does survive is the level. Columnar is priced at 365 to 511 cost-units per
millisecond against the heap's 717 to 892 — under-costed by roughly 1.7x to 2.4x on the
scale every other path comparison is judged against. That is still worth fixing and it is
still risky to fix, for the reasons in the issue body.

Type cells: same rows, one projection each, type is the only variable

projection	columnar cost	columnar ms	heap ms	columnar ÷ heap
sel (int4)	61,065	173.7	170.3	1.02x
n1 (numeric)	74,261	269.2	226.8	1.19x
n2 (float8)	73,196	351.5	232.0	1.52x
t1 (one md5 text)	79,853	353.1	212.2	1.66x
t1, t2, t3, t4	136,218	1082.0	235.3	4.60x
*	239,001	1297.6	186.2	6.97x

Four text columns account for 1082 ms of *'s 1298 ms, while all eight int columns account
for 271 ms. So the * result reported earlier is a text result, not a width result, and
the reviewer's confound suspicion was correct.

That is a separate defect from this one and is filed separately: it is an execution gap, not a
costing gap, and the costing makes it worse rather than causing it (four text columns are
priced at 136,218 against the heap's 158,109 — less than the heap, for 4.6x the time).

What this issue is now about

Only the int-only ladder: a scan cost that is 1.7x to 2.4x low against the reference scale,
with a 1.40x drift across projection width where the heap drifts 1.24x the other way. That is
worth correcting and it carries the same risk the body describes, plus one the body did not
name: the suite that encodes the regressions a recalibration could re-break,
planner_choice_quality, is never invoked in CI under PGC_SKIP_TIMING. #764 owns that
decision, and it is a prerequisite here.

[jdatcmd]

The error is one constant, not a width-dependent shape. And correcting it flips four of thirteen plan choices.

Two results, both from the reviewer's decomposition of the int-only ladder. Neither was in the
issue when it was filed.

1. The model is exactly linear in column count, and only the base is wrong

Per-column differences across the ladder are 11065.0, 11066.0, 11065.0, 11065.3 — machine
exact. So:

cost = 50,000 + 11,065 * ncols
time = 164.9 ms + 13.69 ms * ncols

Calibrating each term rather than the total:

term	cost/ms	heap's measured band
per-column	11,065 / 13.69 = 808	717 to 892 — inside it
base	50,000 / 164.9 = 303	2.6x below it

The per-column decode coefficient is already right. The whole defect is the fixed term.
That is why k ran 2.45 at one column down to 1.40 at eight: not a width-dependent error, a
constant error seen through a varying denominator. Correcting only the base collapses the
spread from 1.40x to 1.11x, tighter than the heap's own 1.24x.

This matches the code. pgcolumnar_refined_scan_cost is a base (the inherited seqscan cost,
width-scaled per #171) plus cpu_operator_cost * ntuples * pgcolumnar_projected_ncols(rel)
(#503). The term the fit says is correct is #503's, and it is what makes "on a wide projection
it now chooses the parallel path" true. A global scaling would have moved it for no reason.

2. Measured consequence: 4 of 13 serial-chosen cells flip, 0 regress

Not arithmetic. A throwaway GUC added the correction to the scan's run cost, and the plan
sweep re-ran across all 30 cells. The knob was reverted; it is not in any branch.

cells	result
serial-chosen	13
flipped to parallel	4 — all three 999,987-row narrow projections, plus * at 2,000,197
parallel-chosen that regressed to serial	0

On the issue's own shape, sel, a, b at 999,987 rows: serial 83,196 → 165,096, and the plan
becomes Gather at 140,925, a 17.2% margin (predicted 16.1%).

But it is necessary and not sufficient. The timing run says the parallel plan is faster in
all 13 cells, and a correct cost model would flip all 13. It flips 4. The nine that remain
are the 2,000,197 and 3,000,052-row cells, where the Gather tuple charge is 200,020 and
300,005 — far past what an 81,900 base correction can reach. Those are held by the
width-blind per-tuple charge, which is core's cost_gather and not fixable here.

So #766 subsumes #753's stated shape but not all of #753's evidence.

The instrument was wrong first, and the way it was wrong is worth recording

The first run reported 0 of 13 flipped and looked like a clean refutation. The knob added
the correction to *out_startup as well as *out_total. A startup cost is paid by every
worker and is never divided among them, so both sides rose by exactly the same amount and no
cell could flip whatever the correction. Adding to the run cost only, the prediction holds.

A null result from an instrument that cannot produce a positive one is not a null result.

What this does to the risk, and it cuts both ways

The correction raises the scan cost where the base dominates, which is selective lookups.
The two regressions named in the issue body pull in opposite directions on this one number:

• Point lookup regressed 23.75 ms to 1251.88 ms once ANALYZE has run, and ANALYZE itself is pathologically slow on some shapes #171 is the case where columnar was priced too cheap and wrongly beat an index (a full
  scan at 1.00 against an index scan at 174.29; a point lookup went 23.75 ms to 1251.88 ms).
  Raising the base pushes toward the index. This correction should help there.
• Planner chooses an Index Scan on columnar for ordering; per-row fetch cost is not modelled and the plan is orders of magnitude slower #355 is the opposite: the planner reached for an index scan and that plan was
  catastrophic (a 12-hour window cancelled at 79 minutes), and the fix raised the index side so
  columnar would win. Raising columnar's base erodes exactly that margin. This correction
  pushes against it.

So the gate is not "run the suites and hope". It is a specific opposition on one number, with
both arms named: analyze_stats.sh carries #171 (the code comment in columnar_customscan.c
says so outright), and the index-fetch-penalty path carries #355/#362. There may be no value
that satisfies both, and that would itself be the finding — a more interesting one than a
recalibration.

That also sharpens the #764 prerequisite rather than softening it: with the Gather half
unfixable here, this base term is the only lever, and the suite that would catch its
regressions is never invoked in CI.

Credit for the base/slope decomposition and for both regression directions to the reviewer.

[jdatcmd]

The proposed floor cannot be derived, and the arithmetic says it could not close the gap anyway

Two measurements, both against the "floor the page-cost removal with a per-group term"
hypothesis. Both come back against it, and the second changes what this issue is about.

1. Per-group overhead accounts for essentially none of it

Rows and projection held fixed, group count varied 148x by building the same 4,000,000 rows at
four stripe_row_limit values. If the width-independent time were per-group it would scale
with the group count.

stripe_row_limit	groups	cost	min ms	ms/group
150,000	27	83,419.73	169.6	6.2825
25,000	160	83,416.27	163.2	1.0198
5,000	800	83,413.87	186.2	0.2328
1,000	4,000	84,267.20	202.7	0.0507

148 times the groups, 1.2 times the time, and not monotonically — 160 groups is the
fastest cell of the four. Taking the whole 27→4,000 difference as per-group overhead gives
0.0083 ms per group, so over the 27 groups of the original fixture it is 0.22 ms of a ~165 ms
intercept: about 0.1%.

So a per-group floor cannot be derived from the locate cost. It would be a fitted constant
wearing a derivation, which is the thing this issue is trying not to do.

2. The entire removal is only 56% of the gap

Measured directly rather than inferred:

relpages   = 49,001        pageCost = seq_page_cost * relpages = 49,001.00
widthFrac  = 12 / 184 = 0.0652
removed    = 49,001 * (1 - 0.0652) = 45,805.28      <- 93.5% of the page cost is stripped

The gap to the heap's calibration is 81,900. The entire page-cost removal is 45,805. So a
floor that removed nothing at all — charging full page cost — closes 56% of the gap and no
more. Flooring it cannot be the whole fix, whatever the floor.

3. Where the rest is, and why it argues against the fix shape

The width-independent time is per row, not per group: ~165 ms for 4,000,000 rows scanned,
half of them filtered before materialisation, 2,000,197 materialised. The model does charge
per row — cpu_tuple_cost * ntuples = 40,000, plus qual evaluation — and that charge is about
2.6x low against measured time.

But the heap pays exactly the same per-row charge. Decomposing both totals:

	pages	per-row	decode	total	measured ms	cost/ms
heap	~108,160	40,000	—	158,109	192.0	823
columnar	49,001 − 45,805 = 3,196	40,000	33,195	83,196	216.4	385

The heap's total lands near the right scale because 68% of it is a page term. On a warm
cache that term buys nothing: both tables are resident, and the heap reads 845 MB against
columnar's 383 MB in almost the same wall time.

So columnar is not obviously mis-modelled here. It removed a page cost it genuinely does not
pay, and was left with per-row terms that are ~2.6x low — an under-pricing it shares with
the heap, and which is core's cpu_tuple_cost, not this extension's.

Flooring the page removal would restore a fictional I/O charge to compensate for an
under-priced per-row constant. Two errors cancelling is not a fix, and the compensation
would be wrong by a different amount on every table, since it scales with relpages rather than
with rows.

What this means for the issue

The code defect is real and worth recording: saved = pageCost * (1.0 - widthFrac) is clamped
only against going negative, never against going too far, and at widthFrac = 0.065 it strips
93.5% of the page cost. But "columnar is 2.6x under-costed against the heap" is no longer a
safe framing, because the reference is a model whose dominant term is fiction on a warm
cache.

Stated limitation: every number here is warm-cache. shared_buffers is 2 GB and both
tables total 1.2 GB, so no arm pays real I/O. Cold, the page term becomes real, the heap's
cost becomes defensible, and columnar's smaller footprint should show as a genuine advantage
the current model already captures. A cold-cache repeat is the next thing this issue needs,
and it may reverse the sign of the whole finding.

Credit to the reviewer for locating the unfloored removal and for insisting the floor be
derived rather than fitted. It was the insistence that produced this: the derivation was
attempted and it failed, which is more useful than a constant that worked.

[jdatcmd]

The discrepancy is inside the range PostgreSQL's own documentation prescribes for this fixture

seq_page_cost is the anchor the CPU cost parameters are defined against, and config.sgml
says of the cache case outright: "in a heavily-cached database you should lower both values
relative to the CPU parameters". Every number on this issue was taken with
shared_buffers = 2 GB holding 1.2 GB of tables — fully cached, exactly the situation the
defaults are documented as wrong for.

So the framing "columnar is under-costed and the heap is the reference" was never well posed.
You cannot hold seq_page_cost fixed as truth and declare cpu_tuple_cost too low: their
ratio is the calibration, and keeping the page term while rescaling the per-row term is
choosing a unit, not finding a defect.

Swept directly. Execution time cannot depend on cost parameters, so times are measured once
and reused — and the plan is asserted unchanged at every setting, since a flipped plan would
mean a different time and a void row. It held at all nine (plans unchanged? yes throughout).

seq_page_cost	columnar cost	heap cost	col c/ms	heap c/ms	heap ÷ col
1.0 (default)	83,195.72	158,109.00	391.5	665.4	1.70
0.7	82,237.00	125,676.30	386.9	528.9	1.37
0.5	81,597.86	104,054.50	383.9	437.9	1.14
0.4	81,278.29	93,243.60	382.4	392.4	1.03
0.3	80,958.72	82,432.70	380.9	346.9	0.91
0.2	80,639.14	71,621.80	379.4	301.4	0.79
0.1	80,319.57	60,810.90	377.9	255.9	0.68

They converge at about seq_page_cost = 0.38, and past it the sign reverses — columnar
then costs more per millisecond than the heap. 0.2 to 0.4 is an ordinary setting for an SSD
or a cached system, and it is what the documentation prescribes for the environment this was
measured in.

And the headline number is not stable enough to claim

The convergence point predicted from the earlier timing run was 0.20; this run gives 0.38. The
difference is entirely in the measured times, which moved between runs:

	earlier run	this run
heap min ms	192.0	237.6
columnar min ms	216.4	212.5
heap ÷ col at seq_page_cost = 1.0	2.14	1.70

The same quantity has now been measured at 2.6x, 2.14x and 1.70x. That is a 1.5x spread in
the headline itself, driven by the heap arm's time varying 24% between runs on a contended
host. So there is no defensible "columnar is Nx under-costed" figure here, quite apart
from whether the comparison is well posed.

What follows

• The code observation stands on its own terms: saved = pageCost * (1.0 - widthFrac) is
  clamped only against going negative, and at widthFrac = 0.065 it strips 93.5% of the page
  cost. Whether that is a defect or the correct modelling of a scan that genuinely does not
  read those pages is a separate question from any of the above.
• A calibration fix would not help The Gather cost is charged per row and is blind to tuple width, which holds nine of eleven measured cells and an extension cannot change it #753 — it slightly hurts it. The Gather tuple charge
  does not scale with seq_page_cost, so lowering the anchor makes the columnar serial scan
  cheaper (83,196 → 80,639 at 0.2) and the fixed charge goes from 120% of serial to about
  124%. The serial plan becomes marginally more favoured. So The Gather cost is charged per row and is blind to tuple width, which holds nine of eleven measured cells and an extension cannot change it #753's remaining half really is
  the width-blind per-tuple charge and nothing else, and the two issues are fully separable.
• A genuine cold-cache run is not available here. Clearing the page cache means
  drop_caches on the host, which runs nine containers and a live desktop session; a
  container-local restart clears only shared_buffers while the OS cache stays warm, which
  measures a third thing that is neither arm. Answering the cold question needs a machine that
  may be made cold, which is a hardware request rather than a measurement to schedule.

The open question is therefore not "run it cold" but whether this issue is about the scan
cost model or about running default cost parameters on a fully-cached fixture. The sweep
above is the evidence for that decision.

Credit to the reviewer for the seq_page_cost-as-anchor argument, for the convergence
arithmetic, and for pointing out that a cold run would perturb every other tenant on the box.

[jdatcmd]

The convergence point is not a quantity either, and that bounds what this issue can conclude here

The convergence point above was reported as 0.38, against 0.20 predicted from an earlier
timing run. That gap is not a disagreement. It is the same instability, propagated.

Solving for the crossing as a function of the heap arm's measured time, with the columnar arm
held at its measured 212.5 ms (both cost lines checked against the measured totals: heap(1.0)
= 158,109 against 158,109.00, columnar(1.0) = 83,196 against 83,195.72, columnar(0.2) = 80,640
against 80,639.14):

heap min ms	convergence seq_page_cost
192.0	0.212
200.0	0.241
210.0	0.277
216.4	0.300
225.0	0.332
237.6	0.378

The heap arm measured 192.0 ms and 237.6 ms on two runs of the identical query, 24% apart. So
the convergence point moves 0.21 to 0.38 on run-to-run noise in one arm, which is the same
noise that moved the headline ratio 2.6x, 2.14x, 1.70x. Neither figure is a property of the
code. Quoting either to three decimals off a single pair of times overstates both.

So the real limit is the hardware, not the cache state

The earlier note said a cold-cache run is unavailable because drop_caches is host-wide
across nine containers and a live desktop session. That is true and it is the secondary
reason.

The primary reason is that this box cannot resolve the effect at all. A quantity that
reads 1.70x to 2.6x across three runs of the same query is not going to be settled by a fourth
run on the same hardware, cold or warm. A cold run here would produce a fourth draw from the
same distribution and we would learn which draw we got, not whether there is a defect.

That holds independently of the other tenants, which makes it the cleaner statement: the
question is not whether we may perturb someone else's container, it is that the instrument in
this room does not have the resolution.

What this issue can and cannot say

Can, on the evidence gathered:

• saved = pageCost * (1.0 - widthFrac) is clamped only against going negative, and strips
  93.5% of the page cost at widthFrac = 0.065. That is a fact about the code.
• Per-group overhead accounts for ~0.1% of the width-independent time (148x the groups, 1.2x
  the time, non-monotonic), so the per-group floor cannot be derived.
• The whole removal is 56% of the gap to the heap's calibration at the default anchor.
• The apparent discrepancy vanishes at seq_page_cost somewhere in 0.21 to 0.38, and reverses
  below it, on a fixture the documentation says should not be run at 1.0 anyway.

Cannot, on this hardware:

• Whether the columnar scan cost model is wrong at all, and by how much. Every candidate figure
  moves by more than the effect being claimed.

The decision this needs

Not "run it cold". The two options are:

1. Hardware that may be made cold and is not shared, to answer whether there is a defect;
2. Close The columnar scan is priced 1.9x to 3.4x below the heap per unit of work, and seq_page_cost decides whether that is a defect #766 as unresolvable here, with the sweep, the failed floor derivation and the
   instability recorded, and the code observation kept as a documented fact rather than a
   defect.

Option 2 is defensible on what is already here, and it is a better outcome than the fitted
constant that would otherwise have shipped: +81,900 flipped 4 of 13 plan choices and would
have looked exactly like a fix.

Credit to the reviewer for propagating the noise through the convergence point, and for
pointing out that the resolution argument stands on its own without needing the shared-host
one.