Bazel- and CMake-built openroad from the same source produce different QoR

[oharboe]

What

floorplan_gcd_sky130hd_base.tar.gz

Same OpenROAD source (45b7772) built two ways gives different QoR on the same input.

Reproduction

sky130hd/gcd through make finish, three consecutive runs of each binary:

Build	tns max	wns max	runs identical
./build_openroad.sh (CMake)	-76.42	-1.72	3/3
bazelisk build //:openroad (root: tools/OpenROAD)	-76.45	-1.71	3/3

Each build is deterministic in isolation.

Floorplan stage outputs match byte-for-byte. The divergence first appears later in the flow.

Untar floorplan_gcd_sky130hd_base.tar.gz

# Build A: CMake
./build_openroad.sh
OR_CMAKE=$(realpath tools/install/OpenROAD/bin/openroad)

# Build B: Bazel (run from tools/OpenROAD/)
( cd tools/OpenROAD && bazelisk build //:openroad )
OR_BAZEL=$(realpath tools/OpenROAD/bazel-bin/openroad)

cd flow
for or in $OR_CMAKE $OR_BAZEL; do
    rm -rf logs/sky130hd/gcd objects/sky130hd/gcd reports/sky130hd/gcd results/sky130hd/gcd
    OPENROAD_EXE=$or make DESIGN_CONFIG=designs/sky130hd/gcd/config.mk finish > /dev/null 2>&1
    echo "$(basename $or) ($(stat -c%s $or)B): $(grep 'tns max\|wns max' reports/sky130hd/gcd/base/6_finish.rpt | tr '\n' ' ')"
done

Amplification on harder designs

Same setup, larger designs:

Design	finish__timing__setup__tns (CMake)	(Bazel)
nangate45/bp_fe_top	-1.21838	-16.0461
nangate45/mempool_group	-11515.9	-23362.2

The 0.04% drift on gcd compounds through placement/CTS/routing into multi-x deltas.

Build environment delta

Same source, same tools/OpenROAD/.bazelrc and tools/OpenROAD/CMakeLists.txt. The two builds differ in:

• toolchain — CMake uses the system clang++ 18.1; Bazel uses toolchains_llvm 1.5.0 → LLVM 20.1.8.
• linked deps — CMake uses system Boost / abseil / spdlog / tcmalloc / libomp; Bazel pins all of them via BCR.
• cxx flags via tools/OpenROAD/.bazelrc vs the CMake build's defaults.

[oharboe]

@hzeller FYI

[hzeller]

also, abc comes from third_party/ in cmake, from BCR in bazel.

[oharboe]

This is about openroad, there is an .odb that is input to this stage.

yosys is not idempontent, otherwise we could have extended rules-base.json with a hash for synthesis. There's a hash for .odb output file from synth_odb.tcl though.

[hzeller]

CC @QuantamHD as he might have either insight or at least interest :)

[maliberty]

If you can isolate the first step to differ that helps scope the problem. Dump a def from each stage and compare them.

It likely originate in the different compiler used and some optimization or undefined behavior

[oharboe]

Re-ran this issue's repro on current master to check whether #10812 changes the picture, then root-caused what remains. Setup: same source tree built two ways — CMake with Ubuntu 22.04's g++ 11.4 (orfs docker image, system deps) vs Bazel with the #10812 toolchain (hermetic-llvm clang 22.1.8, libc++, BCR deps). Same container, same ORFS flow and yosys for all runs; EQUIVALENCE_CHECK=0 LEC_CHECK=0.

QoR delta has collapsed

design	metric	CMake/gcc11	Bazel/clang22
sky130hd/gcd	tns max / wns max	−77.04 / −1.75	−77.09 / −1.75
nangate45/bp_fe_top	finish tns / wns	−1.69 / −0.08	−2.07 / −0.07

Each build is run-to-run deterministic (3/3 identical on gcd). The multi-x amplification from the issue description (bp_fe_top −1.22 vs −16.05) is gone — whatever contributed most has been fixed somewhere between May's master and today's.

First stage to differ (per-stage .odb sha1)

• gcd: byte-identical through synth, floorplan, place, CTS and global route; first divergence is 5_2_route (detailed route).
• bp_fe_top: diverges already at global placement (3_3_place_gp) — expected compiler sensitivity in FP-heavy gpl; not investigated further.

Root cause of the drt divergence (input guides identical)

1. boost::icl ruled out: applying the icl strict-weak-ordering fix (Refactored interval lookup operations boostorg/icl#54, carried for libc++ in bazel: switch C++ toolchain to hermetic-llvm; add etc/bazel-hermetic #10812) to the CMake build's system boost and rebuilding all icl consumers changes nothing under libstdc++ — output byte-identical to unpatched.
2. Priority-queue comparators that are not total orders. GuidePathFinder::Wavefront::operator< compares (cost, node_idx) but not prev_idx; two wavefronts reaching the same node at the same cost via different predecessors compare equivalent, and std::priority_queue pop order for equivalent elements is implementation-defined — libstdc++ and libc++ heaps disagree, so the chosen predecessor (and the traceback path, and which guides survive) differs by stdlib. Proof: adding a prev_idx tie-break makes guide processing output byte-identical (content and order, 2470 guides on gcd) across gcc11/libstdc++ ↔ clang22/libc++. FlexDRConnectivityChecker::astar's wf and FlexDR.cpp's expandBoxes Wavefront have the same defect.
3. Region-query result order. After (2), track assignment still diverges with bit-identical guides and bit-identical pin access points. The remaining mechanism: boost::geometry rtree node splitting (rstar/redistribute_elements.hpp) sorts elements with std::sort keyed on bbox corners, which tie heavily for track-aligned shapes — the permutation of equal elements is stdlib-specific, so queryGuide/queryGRPin return results in different orders. Consumers that take the first match (e.g. FlexTAWorker::initIroute_helper_pin's queryGRPin loop) or assign ids in iteration order (taPin, used as an assignment tie-breaker) then diverge. Fixing this class needs a policy decision — canonically ordering query results (hot path) vs canonicalizing at each order-sensitive consumer — so I'm leaving that to discussion here rather than folding it into a fix PR.

PR with the (2) comparator fixes incoming; it makes guide processing stdlib-independent and removes the same latent tie-order dependence in two more drt spots, but full bit-identical drt across stdlibs additionally needs (3).