You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Data accuracy: corrections, duplicates and dates (workstream) #296
Tracker for data accuracy: correcting existing values, merging duplicates, fixing dates and links. It is one of the workstreams split out of the #1 roadmap on 2026-09-27.
PRs that fix data (fix(data…), or label data + bug) link here with Closes #<this>. The tracker stays open.
Scope
Cross-checks against Wikipedia and manufacturer spec pages (CPU, GPU, SoC, watch)
Plausibility outliers (storage recorded as RAM, unit slips, impossible values)
Duplicate or near-duplicate records (merging while keeping the better record)
Placeholder dates (YYYY-01-01) and placeholder SoC links
Removing values that turn out to be estimates (e.g. DX12 benchmarks on pre-DX12 GPUs)
Rules
Only change a value when a source unambiguously gives the right one for that exact SKU or variant. If sources conflict, document the conflict instead of picking a side.
Add the evidence URL to source_urls; keep verified as it is.
Each fix PR lists its changed fields and the unresolved items in the body.
Summary: Scanned 3,459 tablet records across 63 brands and 447 watch records across 13 brands using normalized slugs/names and spec cross-checks (release date, SoC, display size/resolution, RAM/storage). Found 192 confirmed duplicate pairs/groups and 38 possible pairs needing human/source verification, plus identified several distinct multi-year hardware refreshes that should not be merged.
1. Watch Findings
Confirmed Duplicates (30 pairs / groups)
Identical GSMArena scrape duplicates (3 pairs): Scraped twice from cigarplug/scrape-gsma under different row IDs with 100% identical specs:
huawei/2019/watch-gt-2/: huawei-watch-gt-2-scrapegsma-103 vs huawei-watch-gt-2-scrapegsma-986
huawei/2020/watch-gt-2e/: huawei-watch-gt-2e-scrapegsma-77 vs huawei-watch-gt-2e-scrapegsma-960
lg/2015/watch-urbane-lte/: lg-watch-urbane-lte-scrapegsma-1082 vs lg-watch-urbane-lte-scrapegsma-199
Cross-directory base model duplicates (26 pairs): Same underlying device imported into sibling directories due to brand-prefixed directory names (one from sarangzambare/smartdevices_pricing, one from msainani/gsmarena-mobile-devices):
Apple (17 pairs):
2014: apple-watch-38mm-1st-gen vs watch-38mm-1st-gen, apple-watch-42mm-1st-gen vs watch-42mm-1st-gen, apple-watch-edition-38mm-1st-gen vs watch-edition-38mm-1st-gen, apple-watch-edition-42mm-1st-gen vs watch-edition-42mm-1st-gen, apple-watch-sport-38mm-1st-gen vs watch-sport-38mm-1st-gen, apple-watch-sport-42mm-1st-gen vs watch-sport-42mm-1st-gen
2016: apple-watch-series-1-aluminum-38mm vs watch-series-1-aluminum-38mm, 42mm vs 42mm, apple-watch-series-2-38mm vs watch-series-2-38mm, 42mm vs 42mm, apple-watch-series-2-aluminum-38mm vs watch-series-2-aluminum-38mm, 42mm vs 42mm, apple-watch-edition-series-2-38mm vs watch-edition-series-2-38mm, 42mm vs 42mm
2017: apple-watch-series-3 vs watch-series-3, apple-watch-series-3-aluminum vs watch-series-3-aluminum, apple-watch-edition-series-3 vs watch-edition-series-3
Asus (1 pair): 2016 asus-zenwatch-3-wi503q vs zenwatch-3-wi503q
Huawei (3 pairs): 2018 huawei-watch-2-2018 vs watch-2-2018, huawei-watch-gt vs watch-gt, huawei-watch-magic vs watch-magic
LG (2 pairs): 2017 lg-watch-style vs watch-style, 2018 lg-watch-w7 vs watch-w7
Samsung (3 pairs): 2018 samsung-galaxy-watch vs galaxy-watch, 2016 samsung-gear-s3-classic vs gear-s3-classic, 2016 samsung-gear-s3-frontier vs gear-s3-frontier
Intra-directory brand prefix duplicate (1 pair):
intex/2015/irist-smartwatch/: intex-irist-smartwatch vs irist-smartwatch
Possible Duplicates / Needs Human Check (5 pairs)
lg/watch-urbane-2nd-edition-lte: 2015 vs 2016 year split across sources (lg-watch-urbane-2nd-edition-lte-0-75gb-4gb-4g-lte vs lg-watch-urbane-2nd-edition-lte-aitoolbuzz-5427-0-75gb-4gb-4g-lte).
samsung/galaxy-watch: 2018 42mm bluetooth record vs base record with slightly differing listed screen diameter (1.2" vs 1.3").
samsung/gear-s2-classic: 2015 Wi-Fi vs 2016 3G release year split.
samsung/gear-s3-classic: 2016 Wi-Fi vs 2017 LTE release year split.
samsung/gear-s3-classic-lte: duplicate across gear-s3-classic-lte and samsung-gear-s3-classic-lte directories.
2. Tablet Findings
Confirmed Duplicates (162 pairs / groups)
Identical GSMArena scrape duplicates (15 pairs):
hp/2013/slate8-pro/: hp-slate8-pro-scrapegsma-52 vs hp-slate8-pro-scrapegsma-935
huawei/2013/mediapad-7-youth/: huawei-mediapad-7-youth-scrapegsma-1002 vs ...-119
huawei/2019/matepad-pro/: huawei-matepad-pro-scrapegsma-93 vs ...-976
huawei/2019/mediapad-m6-10-8/: huawei-mediapad-m6-10-8-scrapegsma-107 vs ...-990
huawei/2019/mediapad-m6-8-4/: huawei-mediapad-m6-8-4-scrapegsma-109 vs ...-992
huawei/2019/mediapad-m6-turbo-8-4/: huawei-mediapad-m6-turbo-8-4-scrapegsma-108 vs ...-991
huawei/2020/matepad/: huawei-matepad-scrapegsma-76 vs ...-959
huawei/2020/matepad-pro-5g/: huawei-matepad-pro-5g-scrapegsma-83 vs ...-966
huawei/2020/matepad-t8/: huawei-matepad-t8-scrapegsma-68 vs ...-951
lenovo/2016/yoga-tab-3-10/: lenovo-yoga-tab-3-10-scrapegsma-1054 vs ...-171
Additional pairs in Lenovo, LG, and Samsung tablets.
Cross-directory base model duplicates (50 pairs):
Acer (6 pairs): 2018 acer-chromebook-tab-10 vs chromebook-tab-10, 2015 acer-iconia-tab-10-a3-a30 vs iconia-tab-10-a3-a30, 2014 acer-iconia-tab-7-a1-713 vs iconia-tab-7-a1-713, a1-713hd, a3-a20, a3-a20fhd
Apple (7 pairs): 2017 apple-ipad-9-7-2017 vs ipad-9-7-2017, apple-ipad-pro-10-5-2017 vs ipad-pro-10-5-2017, 2018 apple-ipad-pro-11 vs ipad-pro-11, 2015 apple-ipad-pro-12-9-2015 vs ipad-pro-12-9-2015, 2017 apple-ipad-pro-12-9-2017 vs ipad-pro-12-9-2017, 2018 apple-ipad-pro-12-9-2018 vs ipad-pro-12-9-2018, 2016 apple-ipad-pro-9-7-2016 vs ipad-pro-9-7-2016
HP (3 pairs): 2015 hp-pro-slate-10-ee-g1 vs pro-slate-10-ee-g1, hp-pro-slate-8 vs pro-slate-8, 2013 hp-slate10-hd vs slate10-hd
Huawei (2 pairs): 2018 huawei-mediapad-m5-lite vs mediapad-m5-lite, huawei-mediapad-t5 vs mediapad-t5
Lenovo (3 pairs): 2017 lenovo-tab-7 vs tab-7, lenovo-tab-7-essential vs tab-7-essential, 2016 lenovo-yoga-tab-3-plus vs yoga-tab-3-plus
Samsung (4 pairs): 2017 samsung-galaxy-tab-a-8-0-2017 vs galaxy-tab-a-8-0-2017, 2018 samsung-galaxy-tab-a-10-5 vs galaxy-tab-a-10-5, 2018 samsung-galaxy-tab-s4-10-5 vs galaxy-tab-s4-10-5, 2017 galaxy-tab-active-2 vs galaxy-tab-active2 vs samsung-galaxy-tab-active-2 (triplet)
Vodafone (1 pair): 2017 vodafone-smart-tab-n8 vs smart-tab-n8
Other brands: Amazon Fire (2015/2017 Fire 7, HD 8, HD 10), Allview Viva, etc.
Same directory containing <brand>-<model>.json (from arwinneil) and <model>.json (from msainani) with identical release year, display size, SoC, and memory.
Examples: acer-iconia-tab-a100 vs iconia-tab-a100, apple-ipad-air vs ipad-air, apple-ipad-mini-2 vs ipad-mini-2, apple-ipad-air-2 vs ipad-air-2, amazon-kindle-fire vs kindle-fire, asus-google-nexus-7 vs google-nexus-7, hp-slate7-extreme vs slate7-extreme.
Possible Duplicates / Needs Human Check (33 pairs)
RAM tier mismatch on same device (20 pairs): Same device model where one source recorded the base RAM tier and another recorded the higher tier:
Huawei: huawei-mediapad-m2-8-0 (2GB vs 3GB), x2 (2GB vs 3GB), m2-10-0 (2GB vs 3GB), t2-10-0-pro (2GB vs 3GB), t2-7-0 (1GB vs 2GB), m3-lite-10 (3GB vs 4GB), t3-10 (2GB vs 3GB), etc.
Lenovo: lenovo-tab-4-10-plus (3GB vs 4GB), tab-4-8-plus (3GB vs 4GB)
Samsung: samsung-galaxy-tab-3-8-0 (1.5GB vs 1GB), tab-4-7-0 (1.5GB vs 1GB), tab-active (1.5GB vs 1GB), tab-a-8-0 2015 (1.5GB vs 2GB), tab-e-9-6 (1.5GB vs 1GB), tab-e-8-0 (1.5GB vs 1GB), tab-j (1.5GB vs 1GB)
1-year source date discrepancies (13 pairs): Same model specs but release dates span year boundary:
alcatel/one-touch-evo-8hd (2013 vs 2014, Rockchip RK3066)
alcatel/pixi-3-3g (2014 vs 2015, MediaTek MT8312)
blu/touch-book-m7 vs touchbook-m7 (2016 vs 2017, Intel Atom x3)
htc/shift (2007 vs 2008, Intel Stealey 800MHz)
huawei/mediapad-7-vogue (HiSilicon K3V2 Hi3620 vs Huawei K3V2, 7.01" vs 7.0")
microsoft/surface-duo: dual directories microsoft-surface-duo/ and surface-duo/ for global/US 128GB/256GB SKUs
Confirmed False Positives (Do Not Merge)
Legitimate annual revisions / refreshes with different silicon:
Resolved 30 of the 45 slugs #266 skipped. The change is the field the matching Wikipedia rows agree on, or the row selected by socket, segment, clock, or TDP. Fifteen stay unresolved: package rows that still disagree and the record cannot tell them apart, month phrases that are not a day ("Mid 2008", "Q2 2009"), and slugs with no record (celeron-d-320, celeron-d-325, celeron-d-335, core-m3, core-ultra-x9, mobile-pentium-mmx-166).
Examples: Core 2 Duo T-series release dates moved back to the 2006–2008 Merom/Penryn months both package rows list; E8335 base clock 2.66 → 2.93 GHz; Celeron 2.00 GHz desktop date 2009-02-01 → 2002-09-18; Xeon Gold 5215 date 2019-12-01 → 2019-04-02.
2026-09-29 batch: tablet/watch duplicate merge + placeholder-data root cause
PR #324 merged: 183 duplicate records removed across 182 verified tablet/watch groups (18 identical GSMArena re-imports, 70 cross-directory model duplicates, 95 intra-directory brand-prefix duplicates). Ambiguous cases (RAM-tier variants, year splits, LTE/Wi-Fi distinctions, Surface Duo, MediaPad Vogue, 3 genuine Prestigio revisions, Watch Magic's unsourced OS) were deliberately left unmerged.
Root cause found along the way: a whole import batch had stamped placeholder values (battery_mah=1500, weight_g=150.0) on 43 watch and 217 tablet records regardless of actual device — not real conflicting data. 170 fields were repaired using same-device sibling evidence; 362 with no confident replacement are now null rather than fake. Also caught a comma-parsing bug: iPad Pro 12.9 (2015) battery was stored as 307 mAh instead of 10,307.
This required a schema change: TechEngine#109 makes mobile battery_mah/weight_g nullable (previously non-nullable, which is very likely why the placeholder bug existed in the first place - some import had to put a number to satisfy the schema).
Also this batch: #323 resolved 30 of the 45 ambiguous Intel CPU slugs from #266 by matching each record's socket/clock/segment/TDP against the correct Wikipedia row; 15 remain unresolved.
Scanned the smartphone category (93,407 records) for near-duplicates, using the
same method as the tablet/watch pass (#324). Opened #329.
Merged: 50 confirmed brand-prefix duplicate pairs. These are the cases where a
device exists under both a bare slug (nord-n100) and a brand-prefixed twin
(oneplus-nord-n100) and the core specs (release_date / soc / ram / battery /
weight / os) are identical. Kept the shorter canonical slug, unioned source_urls,
carried the richer side's fields onto the keeper, deleted the redundant file — same
convention as the Intel Atom CPU dedup. One real conflict inside an apparent
duplicate (OnePlus 5T Star Wars main camera 20 MP vs 16 MP) was resolved to 16 MP
per GSMArena/OnePlus (the 20 MP is the secondary sensor), not blind-merged.
Bigger finding — a systemic placeholder-value bug, reported not mass-edited.
487 distinct device names across 624 files share the exact triple battery_mah=1500, weight_g=150.0, ram_gb=1, spanning 1994–2020 and even including
watches/tablets, mostly from the gsmarena-mobile-devices Kaggle import. Same
fabricated-defaults signature (1500/150) #324 flagged. More triples with the 150.0 g / 1 GB + generic YYYY-01-01 tell-tale: (2000,150,1)×182, (1800,150,1)×92, (1000,150,1)×83, (2500,150,1)×80.
This is also why 717 of the 767 brand-prefix pairs read as "conflicts" rather than
duplicates: the bare (Kaggle) side carries these placeholder defaults while the twin
has real specs. They're very likely the same devices, but merging them correctly
means repairing the fabricated side against a real source per record — inventing
values for hundreds of devices, which violates the "never invent values" rule. That
warrants a dedicated repair pass; I left those 717 pairs untouched and documented
rather than guessing.
Both gates pass on the merged tree: app.validate and integrity_check.py --strict
(0 hard anomalies). Dump refreshed in the same PR.
Worked through the 717 brand-prefix conflict pairs that #329 left documented but
unresolved (the pairs that disagreed on soc / release_date / ram / battery / weight /
os). Opened #330.
Approach: re-detected the pairs on current develop, then classified every conflict
field against evidence rather than guessing. The soc disagreement (572 pairs) is
almost always the same physical chip under a different slug spelling
(qualcomm-msm7227-snapdragon-s1 vs qualcomm-msm7227), and the other conflicts on
those pairs are the fabricated bulk-import placeholders (weight_g=150.0, battery_mah=1500, generic YYYY-01-01).
Exact counts (all 717 accounted for):
resolved-and-merged: 364 — same-chip pairs sourced Kaggle-GSMArena (bare) vs
phonedb.net (twin). Kept the GSMArena record (real specs; spot-verified Acer DX650
= 133 g and CloudMobile S500 = 122 g against GSMArena/imei.org, vs the phonedb
twins' 150.0 g placeholder), unioned source_urls, deleted the twin. No value
invented — in every one of these the keeper already held the real value and only
the deleted twin carried the placeholder. verified was not flipped (the
phonedb scalars proved unreliable), and release_date was not changed
(phonedb dates are unreliable too — it lists DX650 as 2009-03 while GSMArena/imei
say 2009-06 — so importing them would be guessing).
confirmed-different-devices-not-duplicates: 31 — pairs whose two sides name
genuinely different silicon, i.e. different SKUs/devices that merely share a base
name. Left separate and dropped from the duplicate classification. Examples:
Huawei P50 Pro (Kirin 9000 and Snapdragon 888 — GSMArena confirms both
chipset variants shipped), Honor Magic V2/V3, HTC One M9 (SD810) vs One M9+
(Helio X10), OnePlus Nord CE 2 (Dimensity 900) vs Nord CE 2 Lite (SD695), Sony
Xperia Z (S4 Pro) vs Z1 (SD800), Motorola razr generations, Oppo Find X8 Pro
(Dimensity 9400), ZTE Grand Memo (SD800 vs SD600).
still-unresolved-no-source: 322 — left untouched rather than guessed:
37 same-chip pairs sourced against vendor/Wikipedia pages instead of phonedb
(different provenance, want individual checks);
157 with a genuine ram/weight/battery question needing a per-record source;
104 where one side's soc is a vague descriptor (dual-core 1.0 GHz, N/A, <brand> mobile platform) — same device, but the date/scalars still need a source;
23 same-silicon-spelling pairs that are mergeable in a follow-up pass;
1 odd pair (blu selfie vs vivo-selfie).
Plus the release_date field on the 364 merged records, deliberately left as-is.
Merging the remaining 322 correctly needs per-record web sourcing of hundreds of
individual devices; per the "never invent a value" rule I've left them documented
rather than guessed. Both gates pass on #330 (app.validate, integrity_check.py --strict → 0 hard anomalies); dump refreshed in the same PR.
Picking up the tracker's open "near-duplicates in other categories not yet scanned" item, I hand-reviewed the whole brand category — the same duplicate-hunt CPU/tablet/watch/smartphone/laptop/monitor/pda got today. Scope: 207 brand records under data/brand/<country>/<slug>.json, on develop 9ce1979.
Result: the brand category is already clean — no confirmed duplicates, no merge, no PR. Reporting here rather than forcing a change.
What I checked
Brand records are manufacturer entities (name, slug, country, categories, founded_year, website, logo, descriptions), not product specs, so I looked for the three failure modes that apply to entities:
Same company under two slugs (e.g. hp vs hewlett-packard, casing/punctuation variants). Automated check: 0 normalized-name collisions and 0 normalized-slug collisions across all 207 records (normalizing to lowercase alphanumerics). None of the classic synonym pairs are double-listed — hp, google, lg (LG Electronics), samsung, at-t, t-mobile each appear exactly once.
Sub-brand/product-line mis-split from a parent. 11 records carry the sub-brand tag or are sub-brand-like (alcatel, infinix, iqoo, itel, nubia, poco, redmi, tecno, xolo, redmagic, alienware). Each is a legitimately separate entity with correct parent attribution in its description (TCL, Transsion, vivo, ZTE, Xiaomi, Lava, Dell). These are the "Beats-under-Apple" case — kept distinct, not merged.
Placeholder/near-empty stubs duplicating a fuller record. 0 found — no sparse stub shadows a fuller record for the same company.
Related-but-distinct entities (deliberately NOT merged)
The only pairs that look superficially like duplicates are joint-venture / brand-licensing lineages, and each side is a genuinely separate entity with its own device fan-out:
BenQ (TW, 2001, monitors/projectors) vs BenQ-Siemens (DE, the 2005-2007 JV mobile brand) — distinct.
Fujitsu (JP, 1935, cpu-designer) vs Fujitsu Siemens (DE, the 1999-2009 JV that made Pocket LOOX PDAs) vs Siemens (DE, 1847) — three distinct entities.
Sony vs Sony Ericsson (the 2001-2012 JV) — distinct.
HMD Global vs Nokia (HMD licenses the Nokia brand; separate company) — distinct.
HONOR vs Huawei (HONOR independent since its 2020 spin-off) — distinct.
Near-name prefixes that are unrelated companies: blackberry/blackview, infineon/infinix, microchip/micromax/microsoft, transmeta/transsion. All distinct.
This is consistent with the recent develop history — the "364 resolved brand-prefix conflicts" work already normalized brand slugs, so the category was in good shape going in.
Verification
No data files were changed. Confirmed the current tree is green:
python -m app.validate → Data validation passed (exit 0), including brand path/slug convention and the brand foreign-key set used by device records.
python integrity_check.py <data> --strict → no hard anomalies (exit 0); the structural section reports no duplicate brand slugs/names, no slug/filename mismatches, and no verified-without-source brands.
No FK repoints were needed (nothing merged), and no dump refresh is warranted since no data changed.
Continuing the "near-duplicates in other categories not yet scanned" item after the brand pass, I scanned the software category. Scope: 42,493 software records under data/software/<2char>/<slug>.json, on develop ea3f31e.
Result: no true duplicates — the software category is clean. No merge, no PR. Reporting here, same standard as brand.
Method (adapted to how this dataset is keyed)
Software records here are Wikidata-derived, and every one of the 42,493 records carries a Wikidata QID in source_urls (0 without). The QID is the entity identity, and slugs suffixed -qNNNN are deliberate disambiguators for records that share a display name. So "same name, different slug" is expected and is not itself a duplicate — a real duplicate is the same Wikidata entity imported twice, an unkeyed name-and-developer collision, or a content clone. I checked all three:
Same Wikidata QID cited by more than one record — the decisive entity-level check. 0 shared-QID clusters. No entity is imported twice.
Same normalized name + same developer set with no distinguishing QID — 0 clusters.
Byte-identical record bodies (ignoring slug) — 0 clusters. No content clones.
Name collisions are genuinely distinct entities
There are 993 normalized-name collisions covering 2,165 records, and all 993 are fully QID-disambiguated (0 ambiguous). Spot-checking the largest:
"Mail" (6): Apple Mail (distinct watchOS / iOS / macOS Wikidata items), Microsoft's Windows Mail, a Linux mail client, etc. — different QIDs and developers.
"Calculator" (4): Apple's, Google's, and Sabec's calculators — distinct products.
"Arc" (4): The Browser Company's Arc browser vs Paul Graham's Arc Lisp dialect vs others.
"GEOS" (4): the GeoWorks GEOS vs the Unix-like and other GEOS entities.
"Apollo", "Opus", "R", "Spark", "Snap!", "Flow", "PRISM", "Visi On" — same pattern: each is a set of unrelated products that happen to share a name, separated by distinct QIDs.
These are exactly the "don't merge real distinct entities that happen to be related/similarly named" case.
The "stub-looking" QID-suffixed records confirm the pattern rather than break it: e.g. zune (Q229851) vs zune-q1970794 (Q1970794), zoa-space (Q118225284) vs zoa-space-q118225291 (Q118225291), and zsnes vs zsnes-2 (ZSNES 2, Q116868010, a distinct 2021 project) all cite different Wikidata items — they are separate entities, not stubs shadowing a fuller record.
Verification
No data files changed. Confirmed the current tree is green:
python integrity_check.py <data> --strict → no hard anomalies (exit 0); the structural section reports no duplicate software slugs, no slug/filename mismatches, and no verified-without-source software records.
No FK repoints were needed (nothing merged) and no dump refresh is warranted since no data changed.
Laptop near-duplicate scan is done — #332 (against develop).
Ran the same method as the tablet/watch (#324) and smartphone (#329) passes on all 1,995 data/laptop/ records: slug/name normalization (punctuation- and brand-prefix-agnostic) plus a spec cross-check (release_date, cpu ref/name, ram_gb, storage_gb, display size/resolution, gpu, weight, os).
Per-pattern results:
Pattern 1 (same scrape imported twice under different row-id slugs): 29 groups merged. Each pair was byte-identical except slug and variant.source_row, all within Ammok/laptop_price_prediction. Kept the lower source_row, deleted the redundant file; source_urls identical so the union was a no-op and no values were changed. Breakdown: acer 1, dell 13, hp 3, lenovo 8, msi 2, toshiba 2. 1995 → 1966 records.
Pattern 2 (brand-prefix path variance, e.g. dell-xps-13 vs xps-13): none. All records live under a consistent brand-prefixed base dir.
Pattern 3 (cross-dataset dup under different slug conventions): none. No device name appears in both bulk datasets; the datasets cover disjoint models. Identical-coarse-spec collisions between different SKUs (e.g. Acer Aspire ES1-572 vs Extensa EX2540) were correctly left unmerged since their names differ.
Configuration variants (same model, different CPU/RAM/storage) were left untouched — distinct real SKUs, not duplicates.
Placeholder / fabricated-defaults check: clean. No equivalent of the phone battery_mah=1500 / weight_g=150.0 stamping. ram_gb, storage_gb, weight_g, and display.size_inch all show organic distributions with distinct-name diversity proportional to frequency (e.g. ram_gb=16 across 735 distinct models, storage_gb=512 across 739). No single value is stamped across unrelated devices.
app.validate and integrity_check.py --strict both pass on the final state (only the pre-existing advisory CPU/GPU ratio outliers print, which the gate treats as review-only). Dump refreshed as the PR's last commit, scoped to the laptop collection.
That closes out the "near-duplicates in other categories not yet scanned" item for laptop — no ambiguous groups were left pending.
Monitor near-duplicate scan is done — #333 (against develop).
Continuing the "near-duplicates in other categories not yet scanned" item after the tablet/watch (#324), smartphone (#329) and laptop (#332) passes. Scope: all 882 data/monitor/ records, every one sourced from a single Kaggle Amazon monitor dataset. Same method: slug/name normalization (punctuation- and brand-prefix-agnostic) plus a spec cross-check (release_date, size_inch, resolution, aspect_ratio, refresh_hz, panel_type, ppi, and the model number in name).
Pattern 1 (same scrape imported twice under different row-id slugs): 650 files merged across 14 base models. Each cluster was byte-identical except slug and variant.source_row. Kept the lowest source_row, deleted the redundant files; source_urls were identical so the union was a no-op and no values changed. Largest clusters: asus-va329he −83, aopen-aopen-by-acer −68, acer-cb272 −62, acer-sh242y −59, acer-xb273k −59, acer-xz342cu −46, acer-kb252q −44, acer-sb240y −43, acer-sb272 −38, acer-xz320qu −30, acer-hdr10 −72, acer-sb270 −26, acer-kc242y −13, acer-xb273u −6.
Pattern 2 (brand-prefix path variance, e.g. dell-u2723qe vs u2723qe): none. Every record already lives under a consistent brand-prefixed base dir.
Pattern 3 (cross-source dup under a different title convention): 1 merged.asus-va24ehe rows 60 & 404 — the same VA24EHE (23.8", 1080p, 75 Hz, IPS, $99) imported with two Amazon title strings. Kept the cleaner title and adopted the more complete adaptive_sync = "FreeSync" from the other side (a real, documented VA24EHE spec, cited in the PR).
Configuration variants (same panel, different size/resolution/refresh/SKU) were left untouched — distinct real products, not duplicates. Documented and intentionally left unmerged: acer-eda343cur (100 vs 180 Hz), acer-sb240y (G0bi vs G0wi), acer-sb242y (EBI vs Hbi), acer-sb272 (new vs Renewed), aopen-aopen-by-acer (32SA2QK 4K vs 27E1 FHD), gigabyte-m27q (M27Q vs M27Q PRO), hp-hp-series-monitor (Series 5 24" vs 524sa), lenovo-lenovo-thinkvision-full (M15 vs T22i-30), lg-lg-digital-signage (49" vs 86"), lg-lg-ultragear-gaming (QHD 240 Hz vs 5120×1440 165 Hz), lilliput-fa1014-np (1280×800 non-touch vs 1080p touch), samsung-hdr2000 (Neo G7 vs Neo G8), and the acer-hdr10 residual (its hdr10 slug is an importer fallback shared by a few genuinely different Nitros — left as-is rather than guess).
Placeholder / fabricated-defaults check: clean. No equivalent of the phone-batch stamping — refresh_hz, ppi, size_inch, msrp_usd all show organic distributions with distinct-name diversity proportional to frequency; no single spec value is stamped across unrelated devices. The one systematic artifact is the placeholder date release_date = "2023-01-01" on all 882 records — a uniform dataset-wide import default, not a spec forced onto unrelated devices, so out of scope for this merge but worth flagging for the "placeholder dates (YYYY-01-01)" item: this whole Amazon-monitor import has no real release dates.
app.validate and integrity_check.py --strict both pass on the final state (only the pre-existing advisory CPU/GPU ratio outliers print, which the gate treats as review-only). Dump refreshed as the PR's last commit, scoped to the monitor collection — the 651 orphaned detail pages were pruned, the monitors list rebuilt (231), and the manifest count updated; unrelated pre-existing serialization drift in other collections was kept out to keep the diff monitor-scoped.
That closes out the "near-duplicates in other categories not yet scanned" item for monitor — no ambiguous groups were left pending beyond the documented distinct-SKU cases above.
Continued the smartphone brand-prefix dedup. #329 (50) and #330 (364) left a
residual of 353; #334 resolves 170 more with source-verified fixes and un-pairs 3
genuinely different devices.
resolved-and-merged: 170
37 vendor/Wikipedia-sourced same-chip pairs — same device, same chip (spelling
variance); adopted the twin's reliable Wikipedia/vendor release_date over the
bare YYYY-01-01 placeholder (verified e.g. Huawei P30 Pro 2019-03-26 = GSMArena).
7 same-model pairs whose sides named different chips, resolved against a cited
source: Oppo Find X8 Pro → Dimensity 9400 (oppo.com), Motorola Edge 50 Ultra →
Snapdragon 8s Gen 3 (motorola.com), Honor Magic V3 → Snapdragon 8 Gen 3
(phonearena), plus OnePlus Nord CE 2 Lite / CE 2, Sony Xperia Z, Honor Magic V2
(bare already correct; GSMArena/vendor cited).
126 same-chip-spelling / vague-soc-vs-specific pairs (Kaggle-GSMArena vs phonedb):
kept the GSMArena record, adopted the specific soc where the keeper's was vague,
rejected the phonedb stamped placeholders, deleted the twin.
confirmed-different-devices-not-duplicates (un-paired): 3 — HTC one-m9 holds
the One M9+ (Helio X10) vs One M9 (SD810); Motorola razr the 2023 (SD 8+ Gen 1)
vs 2019 (SD710); Sony xperia-z3 the Z3+ (SD810) vs Z3 (SD801). Left as separate
records.
still-unresolved-no-source: 180 — 21 same-model pairs whose correct chip I
couldn't source this round (includes the Huawei P50 Pro, a genuine dual-chipset
model — Kirin 9000 and Snapdragon 888 both shipped), 157 pairs needing a per-record
ram/weight/battery source, and 2 odd/other pairs. Left documented rather than
guessed.
Running total across #329/#330/#334: 584 of the original 767 brand-prefix pairs
resolved (50 + 364 + 170), plus the genuinely-different devices un-paired. Both
gates pass on #334 (app.validate, integrity_check.py --strict); dump refreshed
in the same PR. Refs #296 (leaving the tracker open for the residual and the
other categories' work).
Brand-prefix twins in sibling dirs: 4 pairs — treo-680 / treo-750v / treo-750 / treo-500v each had a phonedb-sourced palm-treo-… twin. Kept the
bare GSMArena record (same convention as the fix(data): resolve 364 smartphone brand-prefix conflicts #330 merge), upgraded the name to the
brand-inclusive form ("Palm Treo 680" …), unioned source_urls, deleted the twin.
Treo 750 vs 750v and 500 vs 500v were kept separate — different SKUs.
Cross-source clean + scrape-gsma in same base dir: 22 records across 19 groups —
the 16 HP iPAQ pairs, i-mate pda2/pda2k/pdal, and the Palm Treo Pro 3-way (bare +
scrape-gsma + phonedb). Kept the clean record and repaired its ram_gb from the
deleted twin's cited memory_internal (the clean side carried the fabricated ram_gb=1; the twin had the real 64/128 MB). scrape-gsma URL unioned so the RAM is
sourced. No value invented.
Left unmerged and documented (not guessed): the GSM webOS palm-pre-scrapegsma-7065
and palm-pre-plus-scrapegsma-7063 overlap the region-split castle family
(palm-pre[-plus]-palm-castle-c CDMA and palm-pre[-plus]-gsm-{eu,na}-palm-castle-g)
but do not map 1:1. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm,
pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone.
Systemic finding (reported, not mass-edited): the same fabricated-defaults
signature the earlier passes flagged is present in pda too — nearly every clean
record carries ram_gb=1 (real values 16–256 MB) and many Palm records carry os="Feature phone" for Palm OS / webOS devices. I only repaired ram_gb inside the
confirmed merges (where the deleted twin supplied a cited value); the remaining
placeholder records need a per-record source and were left untouched rather than
mass-edited.
Both gates pass on the merged tree (app.validate; integrity_check.py --strict —
0 hard anomalies). Dump refreshed in the same PR (pdas 140 → 112, no other
collection touched).
Using Refs #296 rather than Closes: this clears the pda slice of the
"near-duplicates in other categories not yet scanned" item, but that item stays open
— smartphone still has 322 documented-unresolved pairs (per #330), and the remaining
categories (laptop, monitor, CPU beyond the Atom pass, etc.) haven't been scanned for
near-duplicates yet.
Follow-up to #332: ran a dedicated placeholder-value / fabricated-defaults scan across all 1,966 post-dedup data/laptop/ records (not just duplicate pairs), looking for the phone-import pattern where an import batch stamps an identical fake default (e.g. battery_mah=1500, weight_g=150.0) on unrelated devices regardless of their real specs.
Result: no fabricated-default bug found in laptop data. Nothing to fix beyond what the dedup pass already caught. No files changed, so no PR.
What I checked and why each common-looking value is real, not fabricated:
Per-field value frequency counted by distinct device (brand + base_model_slug, so config variants of one model don't inflate a value). For weight_g, ram_gb, storage_gb, os_version, display.size_inch/resolution/ppi, and combos (size+resolution, ram+storage, weight+size, full 5-tuple). Every high-frequency value is a genuinely market-dominant config (16GB RAM, 512GB SSD, 15.6"/1920x1080, Windows "10") and the distinct-device counts decay smoothly — there is no isolated spike of an implausible value stamped across unrelated models.
ppi is 100% internally consistent with each record's own size_inch + resolution — 0 mismatches >8px across 1,966 records. A fabricated display-spec batch would leave ppi inconsistent with the stated panel; it doesn't.
Field presence per source dataset is complementary, not faked:abhinavflac supplies msrp+ppi but no weight/os_version (0%); Ammok supplies weight+os_version but no msrp. Missing fields are simply absent in each source, not stamped with a default.
Non-round shared weights (the most placeholder-looking case) trace to real causes on spot-check: 1910 g is entirely the HP 15 (2017) chassis reused across configs; 1370 g is the real MacBook Pro 13" (2016-2019) / ThinkPad Yoga weight; 1360/1250/1200 g are genuine ultrabook weights shared by real ~13" machines. None is an implausible value on unrelated devices.
No physically-impossible values: nothing outside sane ranges; the 14 heaviest-for-size items are all real gaming/workstation laptops (Alienware 15/17, ROG G703, ZBook 17, Legion Y720, IdeaPad Y910) at their true weights.
Laptops have no battery_mah/battery_wh field at all, so the phone bug's exact vector doesn't exist here.
validate.py / integrity_check.py --strict were not re-run since no records were modified (both already pass on this merged state per #332).
Follow-up to the monitor dedup (#333): I ran a standalone placeholder-value scan across all 231 post-dedup data/monitor/ records (216 distinct devices), specifically hunting for the fabricated-default pattern the tablet/watch and smartphone passes found (an import stamping an identical fake value — e.g. battery_mah=1500, weight_g=150.0 — onto unrelated devices), including records that have no duplicate twin to reveal it.
Result: no fabricated-default bug in monitors. No data changed, no fix PR. Reporting plainly, as agreed.
Method
For every device-varying field I counted how many distinct devices share each exact value, then applied a decisive correlation test: a real common value is consistent with each device's own name/derivable specs, whereas a stamped default appears regardless of the device. I checked each field's most-repeated value against the model name (and, for ppi, against size+resolution).
Why the high-frequency values are real, not stamped
features.response_time_ms = 1 (65 devices) — the biggest "suspicious" cluster. 64/65 are ≥100 Hz and/or say "gaming"/"1ms" in the title; 0 low-refresh office monitors falsely carry 1 ms, and 0 names state a different response time than the field. 1 ms is the genuine marketed spec for these gaming panels.
refresh_hz — of 125 records that state a refresh rate in the name, 0 disagree with the field. resolution — 0 mismatches across 109 named records. size_inch — only 4 apparent mismatches, all regex artifacts (e.g. "34.18 inch" parsed as 18, a "22-to-24-inch" privacy-screen range, rounding of 24.5→25).
msrp_usd = 100 (19 devices) — all genuinely budget/portable/office 1080p panels (22–27" IPS, ZenScreen portables, touch monitors); the field has 110 distinct prices overall and neighbours at 80/99/110, i.e. an organic distribution, not one price pasted everywhere.
rating — a textbook Amazon distribution: a smooth bell 4.2→4.6 (peak 4.6 = 51) with a tail down to 1.0 and a small 5.0 spike. Not a default.
panel_type = IPS, aspect_ratio = 16:9, resolution = 1920x1080, size_inch = 27" are simply the genuinely dominant real monitor specs, and they co-vary correctly with the names.
ppi — 3 records are internally inconsistent with their size+resolution (Dell Inspiron 24 5400 AiO, MSI MAG274UPF, Lilliput FA1014), but each has a unique wrong value, i.e. individual scrape noise, not the repeated-default fingerprint this pass targets (general per-field accuracy is out of scope here).
The one genuinely-repeated placeholder — and why it's left as-is
release_date = "2023-01-01" sits on all 216 devices. It is a real placeholder, but it's category-wide (not a fake value stamped on a subset of unrelated devices while others vary), it's a required field that validate.py will reject if nulled or removed, and there's no per-device source to derive true dates — so nulling it is impossible and inventing dates is off-limits. This is already the tracked "placeholder dates (YYYY-01-01)" item on this issue and needs an external per-device source, so I'm leaving it rather than touching it under this task.
Verification
No files changed. Confirmed the current tree is green:
python -m app.validate → Data validation passed.
python integrity_check.py <data> --strict → no hard anomalies.
So the monitor category has no hidden fabricated-default values beyond the category-wide placeholder date already on the tracker.
Opened #337 against develop — a first source-citation pass over data/laptop/ and data/monitor/.
Scope (checked live): laptop 1995 total / 1890 unverified; monitor 882
total / 882 unverified. Every unverified record already carried a
bulk-dataset link (huggingface Ammok, abhinavflac, Kaggle Amazon-monitor),
so the gap was uncited, not unsourced.
This PR (45 records touched):
Newly cited-and-verified (no value change): 35
Value-corrected-with-citation: 10 — MacBook Air weight x2, Surface
Laptop weight x3, Pixelbook weight x3, LG 24GN60R size x1, Samsung
Odyssey G9 49" DQHD resolution x1 (all corrected to match the cited
vendor/Wikipedia spec, never invented).
Left uncited, no confirmable source this pass: 2727 (1866 laptop +
861 monitor) — untouched, keep existing provenance.
Each verified record was confirmed field-by-field against a manufacturer
spec page or Wikipedia infobox (Apple, Microsoft, Google, ASUS, Dell,
Gigabyte, MSI, Samsung, LG, BenQ, ViewSonic).
python -m app.validate passes and integrity_check.py --strict reports
no hard anomalies. The static dump was regenerated (python -m app.dump
from TechEngine against the worktree) as the PR's final commit.
The large remaining uncited set is a good candidate for follow-up batches
(the abhinavflac records carry exact model SKUs that are individually
verifiable against vendor pages).
Total ~5,645 refs across ~2,793 base models. All three are verified: false
Kaggle/phonedb bulk-import defaults with "Unknown mobile GPU"/"N/A" fields.
Fixed: 45 records (Google Pixel). Pixel phones use one global SoC per model
(no regional Exynos/SD split), so the chip is unambiguous and Wikipedia-documented.
Re-pointed to existing verified records: Pixel(2016)→snapdragon-821,
Pixel 3→snapdragon-845, Pixel 3a/3a XL→snapdragon-670,
Pixel 4/4 XL→snapdragon-855, Pixel 4a→snapdragon-730g,
Pixel 4a 5G/5/5a 5G→snapdragon-765g (30 off "0", 15 off qualcomm-snapdragon).
Each gained the canonical Wikipedia device-page citation in source_urls; no
values invented; assignments corroborated by sibling Pixel records already on the
same chips.
Left unresolved (documented, not guessed): ~5,600. The device records carry no
embedded chipset signal — only name/RAM/battery/display, overwhelmingly from the aitoolbuzz/phonedb Kaggle imports. Repairing them means per-record web sourcing
of thousands of mostly obscure/regional devices, i.e. inventing at scale — the same
line #324/#329/#330 drew. Notable trap avoided: Samsung Galaxy S/Note region
variants (SM-N960F Exynos vs SM-N960U Snapdragon, etc.) can't be blanket-mapped
and need per-model-number sourcing.
Both gates pass (app.validate ✅; integrity_check.py --strict → 0 hard
anomalies); dump refreshed as the PR's last commit. Follow-up passes can pick off
further single-global-SoC brands the same way (fully sourced, no invention).
Follow-up to #332 (laptop): ran the same distinct-device placeholder-value scan on data/tablet/ (3,305 records) and data/watch/ (418 records), checking whether the fabricated-default pattern (battery_mah=1500, weight_g=150.0) or the RAM-in-storage bug survives outside what #324 and #301 already caught.
Result: both categories are clean. No fabricated-default pattern remains, nothing to fix — no PR.
Distinct-device value frequency (keyed by brand + base_model_slug) for battery_mah, weight_g, ram_gb, storage_options_gb, display.size_inch/resolution/ppi, and combos (battery+weight, size+res+ppi). Every high-frequency value is a genuinely common real spec (4000/5000/6000 mAh tablet batteries, 7"/8"/10.1" sizes, 1GB/2GB/4GB RAM) and — critically — each is spread across 8–11 independent source datasets (aitoolbuzz, madajaswanth, jakubkhalponiak, cigarplug/scrape-gsma, arwinneil, beridzeg45, sady36, lileejass…). The fabricated-default fingerprint is a single value concentrated in one import batch stamped on unrelated devices; that does not appear here — the distributions decay smoothly and are cross-source.
ppi internal consistency with size + resolution: tablets 0 / 2,092 mismatches. Watches showed 11 apparent mismatches, but all are round-display models (255 of 413 watch resolutions are square W×W) where GSMArena publishes a diameter-based ppi rather than the naive pixel-diagonal. Spot-checked the Samsung Gear S3 (1.3", 360×360, stored ppi=278) against GSMArena, which lists exactly "360 x 360 pixels (~278 ppi density)" — so the stored values are correctly sourced, not fabricated. My rectangular formula was simply the wrong model for round panels.
Physical plausibility: no impossible values. The handful of small-for-tablet flags are real phone/tablet hybrids (Asus PadFone family) and Archos PMP-style "Internet Tablets"; the one battery flag (iPad Pro 12.9 2017 = 891 mAh, likely a source typo for ~10,891 mAh) is a unique value, not a repeated default, so it's outside the placeholder-pattern scope of this task.
Field presence per dataset is complementary, not fabricated (e.g. madajaswanth/jakubkhalponiak/lileejass supply battery+weight+ram but no display size/ppi; the gsmarena-derived sets supply full display). Missing ≠ stamped.
validate.py / integrity_check.py --strict were not re-run since no records were modified.
That covers tablet + watch for the "fabricated defaults in other categories" item alongside the laptop pass in #332.
Applied the same placeholder-value methodology from the monitor pass to data/gpu/ — all 2,083 records, hunting for the fabricated-default fingerprint (one value stamped across many unrelated cards whose real specs differ). Checked base_clock_mhz, boost_clock_mhz, memory_gb, tdp_w, release_date, plus memory_bus_bit, memory_type, memory_bandwidth_gbps, core counts, pcie_version, architecture, msrp_usd.
Result: no fabricated-default bug in GPUs. No data changed, no fix PR. Reporting plainly.
Why the repeated values are real, not stamped
This is a curated TechPowerUp/Wikipedia-sourced set (verified: true), and the distinct-card frequency counts plus correlation-against-card-class checks all come back consistent:
Clocks — base_clock_mhz has 435 distinct values, boost_clock_mhz 478. The top repeats (500/600/650 MHz, ~80 cards each) are genuine round clocks that early-2000s GPUs actually ran at. Only 5 post-2016 cards carry a 500–650 base, and all are legitimate: Quadro/RTX Max-Q mobile parts (real 600 MHz base, 1215–1350 MHz boost) and Jetson Orin Nano SoC modules (fixed 625 MHz). boost == base on 53/735 post-2016 cards — every one is an OEM/workstation/datacenter part (FirePro, Radeon Instinct/Pro, Tesla, Radeon R5/R7 OEM) that genuinely ships without a separate boost clock, matching TechPowerUp.
memory_gb — the canonical power-of-two sizes (1/2/0.5/4/8/16 GB). The biggest cluster, 1.0 GB (319 cards), spans 2005–2017 and peaks 2010–2013 — the actual 1 GB-VRAM era — with zero cards released ≥2020. Period-consistent, not pasted onto modern cards.
tdp_w — standard tiers (50/75/150/250 W = slot power, 6-pin, 6+8-pin, flagship). Modern ≥16 GB cards show varied high TDPs (300/250/350/700/600/400 W), i.e. no flat default.
memory_bus_bit / memory_type / pcie_version / architecture — real taxonomies with natural distributions (128/256/64-bit; GDDR5/GDDR3/GDDR6; PCIe 2.0/3.0 x16; TeraScale/Tesla/Kepler…).
release_date — 955 distinct dates. The largest cluster, 2021-04-12 (30 cards), is a real NVIDIA professional/datacenter launch day (RTX A2000/A4000/A5000, A10/A16/A30, T600, Quadro T1200). Only 57/2,083 records use YYYY-01-01, and they are spread across 1996–2026 on old/obscure/OEM/engineering and future-unannounced cards (e.g. 2026-01-01 RTX 5070/5080 SUPER) — the honest "only the year is known" convention, not a single stamped placeholder like the monitor set's uniform 2023-01-01. (release_date is a required field, so it couldn't be nulled regardless, and there's no per-card source to invent exact days.)
Identical full-spec fingerprints — the largest are 4–9 cards, and each is a genuine rebadge/form-factor family that really shares specs (Radeon HD 5610/6510/6530 rebrands; Tesla K40c/d/m; Radeon X800 XL AGP/PCIe; Voodoo family). Real shared values, not fabrication.
Verification
No files changed. Current tree is green:
python -m app.validate → Data validation passed.
python integrity_check.py <data> --strict → no hard anomalies.
Note on the parallel GPU work
Another pass is doing near-duplicate/cross-check on data/gpu/. My scan was strictly the fabricated-placeholder angle and found nothing to fix, so there's no overlap with their dedup edits and no data conflict from my side (I changed nothing). The GPU category has no hidden fabricated-default values.
Follow-up backfill run (targeting a higher --limit 2000 after #339 merged) is blocked by an active GSMArena rate limit — no records were upgraded this run.
Details:
Rebased a fresh branch onto latest develop and ran the tool from the develop-pinned TechEngine commit (147d27d).
Started with a small dry-run to check the limit status before committing to a large batch. On the first fresh HTTP request GSMArena returned 429 with Retry-After: 36000s (10 hours). The tool stopped cleanly before sending any further requests (1 request total, exit 1) and wrote nothing.
Per the standing rules I did not retry/spin, and since this was a no-op (working tree clean, zero writes) I'm reporting here rather than opening a PR. The tool and gate are unchanged and working — the earlier dry-run before #339 was clean — so this is purely a cooldown window, not a tooling regression.
Next step: wait out the Retry-After window (roughly ~7 more hours from now) and re-run the --limit 2000 backfill then. The resume cache already carries the 37 confirmations from #339, so a later run resumes from where this left off without re-fetching them.
Ran a within-record self-consistency pass over data/pda/ (140 records), same pattern as #301 - values derived only from each record's own fields, no external sources.
data/pda/palm/2018/palm/palm.json: storage_options_gb was [3, 32], but the 3 is the RAM value (
am_gb: 3 / �ariant.memory.ram_gb: 3) leaked into the storage list. The record's own �ariant.memory.storage_gb: 32 confirms storage is 32 GB only -> corrected to [32].
Scanned the website category (one of the last untouched in the accuracy tracker) for near-duplicates. Opened #341 against \develop.
Result: merged 153 confirmed near-duplicate groups and removed 154 redundant files (one 3-way group); the \websites\ collection drops 40,084 -> 39,930.
Candidates were grouped by normalized \homepage_url\ + display name, with records sharing the exact same normalized URL as the strong signal (195 such groups reviewed). Confirmed duplicates fell into these patterns:
QID-suffixed clones of an otherwise identical record (<slug>-q);
bare-domain stubs vs the proper record (e.g. \3dmgame-com\ -> \3dm, \�mazon-com\ -> \�mazon);
same-entity name variants incl. acronym/expansion and translated open-data-portal names (e.g. ANNO / AustriaN Newspapers Online, Gosuslugi / State Services).
Each merge kept the fuller/verified/better-sourced side, unioned \source_urls, OR'd \�erified, and filled only missing optional fields — no invented values.
Left as distinct (the known trap): records that only share an index/landing URL but are different real entities — museum exhibition pages under one archive host, imageboard boards, registration services differing only by URL fragment (#ul/#ip), and borderline company-vs-website / subsite / regional-version pairs (itv.com vs ITVX, Daily Mail vs MailOnline, two municipalities sharing one open-data portal URL, etc.).
Verification: \�pp.validate\ passes; \integrity_check.py --strict\ reports no hard anomalies (no dup slugs, no slug/filename mismatch, no verified-without-source). Static dump refreshed as the PR's last commit.
Remaining open sub-item: the parallel software scan.
Ran a fresh within-dataset self-consistency pass over \data/tablet/\ and \data/watch/\ (follow-up to #301, since more data has moved through since — e.g. the tablet/watch dedup in #324 and the mobile-source reclassification).
Found and fixed 261 records (259 tablet + 2 watch) where the device's RAM size had leaked into \storage_options_gb. In every case the polluting value equals the record's own \�ariant.memory.ram_gb\ (and top-level
am_gb) and is strictly smaller than every real storage tier in the same list — those tiers independently confirmed by the record's own \�ariant.memory.storage_gb. A 1-2 GB entry can't be a storage capacity for these devices, so it's the RAM size, and the fix removes it. Example: \iconia-tab-a100\ had \storage_options_gb: [1, 16]\ with
am_gb: 1\ → corrected to [16].
Every correction is derived only from fields already in the same record — no external sources, no invention. Genuinely ambiguous cases (multiple RAM tiers in the list, or a low value that could plausibly be a real storage size) were deliberately left untouched.
Verification:
\python -m app.validate\ → passed
\integrity_check.py --strict\ → exit 0, no hard anomalies, structural section clean, no tablet/watch flags
Mandatory dump refresh committed last, scoped 1:1 to the 261 tablet/watch detail files (pre-existing unrelated dump drift excluded).
Scanned all 2,076 data/gpu/ records with the same method as the CPU / tablet
/ watch / smartphone passes. Found 8 confirmed cross-source duplicate pairs
(same physical product imported twice under different slug conventions, one
side Kaggle-sourced, one TPU/vendor-sourced). Merged each into the more
complete / correctly-sourced record, source-verified every disagreeing field,
unioned source_urls, deleted the redundant file: 2,076 → 2,068.
Notable spec conflicts resolved against authoritative sources:
RTX 3060 8 GB: kept the correct 2022-10-12 launch record.
No configuration variants were merged. Close families that only share a name
prefix — Tesla K40 d/m/s/t, GRID vGPU profiles, PCI/AGP and MXM/PCIe bus
variants, OEM SKUs — are distinct real products and were left untouched, per
the #327 guidance.
Both gates pass: app.validate ✅ and integrity_check.py --strict ✅
(structural section clean). Dump refreshed via python -m app.dump, scoped to v1/gpus/** + manifest.
Followed up the pda dedup pass (#335) with a placeholder-value scan across all
112 post-dedup pda records — looking for the fabricated-default fingerprint (one
fake-precise value repeated across many unrelated devices) independent of whether a
record had a duplicate twin. Opened #344.
Clusters found:
ram_gb = 1 on 83 of 112 records — fabricated (no PDA 2000-2011 had 1 GB;
Dell Axim X5 = 32/64 MB, Palm IIIc = 8 MB, verified against Wikipedia/PCMag/
TidBITS). The Kaggle/phonedb bulk-import placeholder.
battery_mah = 1500 on 33 (mostly the classic Palm line) — fabricated but
unfixable, see below.
ppi = 114 (26), the 3.5"/240x320 combo (26), and weight_g = 190 (9) are real shared values, not fabricated: 240x320 QVGA on a 3.5" panel genuinely is
~114 ppi, and 190 g is real chassis-sharing (iPAQ h63xx family + i-mate HTC
rebrands). Left untouched.
Fixed: 18 ram_gb=1 → real sourced values (ram_gb is a required field, so
repaired to a cited value, not nulled; no value invented):
13 Dell Axim — Wikipedia Dell Axim spec table (low-end "Basic" = 32 MB; every
other model incl. the whole X50/X51 line = 64 MB).
Left documented rather than guessed or nulled (this is the important part for
the tracker):
The classic Palm line (m100 = 2 MB, m105/m125/m130/m500/m505 = 8 MB, m515 =
16 MB, IIIc/IIIxe/VIIx = 8 MB, Tungsten/Zire/Treo/Centro …) has real RAM below
the schema's ram_gb >= 0.016 (16 MB) floor, so it is not representable —
almost certainly why these got the ram=1 placeholder on import. Can't set the
real value (fails validate) and can't null a required field. This is a schema
limitation worth noting: sub-16 MB RAM devices can't be recorded accurately at
all. ~50 records are in this bucket.
Remaining ram_gb=1 (i-mate PDA phones, HP iPAQ Glisten, Mitac Mio 728, later
Palm Treo/webOS) need per-device sourcing; left documented.
battery_mah=1500: classic Palms ran on AAA/AA cells (no mAh rating) or small
internal Li-ions — no clean per-device mAh to cite without inventing, and the
field is required (can't null).
Both gates pass (app.validate; integrity_check.py --strict — 0 hard anomalies).
Dump refreshed in the same PR (only the 18 repaired detail pages change).
Refs #296 — removes a sourced slice of the pda placeholders; the
unrepresentable/unsourceable remainder stays open and documented above.
Batch progress on the uncited-website backlog — opened #345 against develop.
What landed: 7,086 data/website/ records upgraded from Wikidata-only
(verified: false) to source-verified, by adding an independent, checkable
citation — the English Wikipedia article linked from each record's Wikidata item
(enwiki sitelink) — and setting verified: true. No values were fabricated or
altered.
Method: for each verified: false, Wikidata-only record, resolve its QID and
query the Wikidata API. If the item has an English Wikipedia article, that URL is
a real human-checkable source for the same entity, so it's appended to source_urls and the record is verified. Records with no enwiki article are left
untouched — no citation invented.
Left uncited — launch_date vs Wikidata P571 year mismatch, flagged for manual review
17
Already verified before this pass
7,635
Total website records
40,084
0 corrections on purpose: 17 records had a launch_date disagreeing with the
Wikidata inception year (P571). Since a site's launch date and its subject's
inception are different facts (e.g. Netflix 1997 the company vs. 2007 the service),
overwriting one with the other would inject wrong data, so they were left as-is and
flagged for manual review.
Verification:app.validate passes; TechEngine integrity_check.py --strict
exits 0 with no hard anomalies; static dump refreshed for the affected website
records (scoped to data/website/ + its site/public/v1/websites dump).
The remaining uncited records (those without an English Wikipedia article) are
left for future batches.
Opened #346 against develop — a first source-citation pass over data/tablet/ and data/watch/ (companion to the laptop/monitor pass in #337).
Scope (checked live): tablet 3305 total / 3020 unverified; watch 418
total / 325 unverified. As with laptops/monitors, every unverified record
already carried a bulk-dataset link (Kaggle, aitoolbuzz, GSMArena-scrape,
GitHub), so the gap was uncited, not unsourced.
Left uncited, no confirmable source this pass: 3312 (3012 tablet +
300 watch) — untouched, keep existing provenance.
Each verified record was confirmed field-by-field (RAM, battery, weight,
OS, SoC, storage) against a Wikipedia infobox or manufacturer/GSMArena spec
page.
python -m app.validate passes and integrity_check.py --strict reports
no hard anomalies. The static dump was regenerated (python -m app.dump
from TechEngine against the worktree, fresh scratch DB) as the PR's final
commit.
Large remaining uncited set is a strong follow-up target: ~60 tablet/watch
records already cite GSMArena/phonedb per-model pages and just need
field-by-field confirmation once those sources are fetchable (GSMArena was
rate-limiting during this pass).
Followed up on the PDA dedup's systemic finding (uniform ram_gb=1 and os="Feature phone" across the category). PR #347 corrects 83 of the 112data/pda/ records, each value sourced from that device's cited datasheet.
os (50 records):"Feature phone" → real platform: "Palm OS" (Palm III/V/VII/m-series/Tungsten/Zire/TX/Z22/Treo≤755p/Centro), "webOS" (Pre/Pixi families), "Windows Mobile" (Dell Axim X5, MiTAC Mio 728). Already-correct Windows Mobile records were left as-is.
ram_gb: placeholder 1 → true capacity from phonedb.net's "RAM Capacity" field, converted MiB→GB in the existing convention (64→0.0625, 128→0.125, 256→0.25, 512→0.5, 32→0.032, 16→0.016). variant.memory.ram_gb mirrored.
source_urls: added for the 6 kaggle-only records (Wikipedia for Treo 180/270/600/650; imei24 for iPAQ Glisten and i-mate Pocket PC).
Spot-verified several directly (Palm IIIc 8 MiB/Palm OS 3.5, Palm TX 32 MiB/Palm OS 5.4, Palm Pre 2 512 MiB/webOS). No values invented.
Left unresolved (documented, not guessed): 12 pre-2003 Palm classics have 2–8 MiB RAM (m100 & Zire = 2 MiB; IIIc/IIIxe/VIIx/m105/m125/m130/m500/m505/i705/Zire 21 = 8 MiB), which is below the ram_gb schema floor (0.016 GB = 16 MiB in app/validate.py). Their os was corrected to Palm OS, but ram_gb was left rather than writing an overstated in-range value. Raising the validate floor is an engine-side change; noted for a follow-up.
Verified: python -m app.validate passed; integrity_check.py --strict reported no hard anomalies. Dump refreshed as the PR's last commit (83 changed PDA pages only).
Follow-up self-consistency pass over data/monitor/ (fresh scan of all 882 records).
Found and fixed: 18 records with a null aspect_ratio that is unambiguously derivable from the record's own resolution field (and consistent with its own name). Filled using the ratio labels this catalog already uses — 16:9 (1920x1080 / 2560x1440 / 3840x2160), 21:9 (3440x1440 / 3840x1600), 32:9 (3840x1080 / 5120x1440), 16:10 (1280x800). PR: #348 (against develop).
Deliberately left alone:
Resolution vs explicit digits in name: 0 mismatches.
Refresh rate: 0 mismatches (dual/OC rates like "165Hz/180Hz(OC)" already store the base value).
Size: ~19 differences, all marketing "Class"/nominal vs actual-panel size (e.g. 24" Class = 23.8"), which are legitimate and not errors.
9 aspect_ratio records where the record's own resolution and name contradict each other (e.g. 49" "Dual QHD"/DQHD listed as 3440x1440, "27" 1440p" listed as 5120x1440, names asserting 21:9 on a 1920x1080 field). No single value is safely derivable from the record alone, so they were left and noted rather than guessed.
Verification: python -m app.validate passes; integrity_check.py --strict reports no hard anomalies. Mandatory dump refresh included as the PR's last commit — only the 18 affected monitor detail files changed.
Focused follow-up pass for the "these are actually different devices, not
duplicates" pattern (distinct from the "needs a source" residual, which I've left
alone). Opened #349.
Found 7 brand-prefix pairs that are genuinely different products sharing a base
name — the 3 reported earlier plus 4 new ones — and un-paired each by giving the
mis-slugged variant a distinct slug/filename/name (no merges, nothing invented):
HTC U11+ (u11 → u11-plus) vs U11 — distinct fall-2017 model (androidcentral).
HTC One M9+ (one-m9 → one-m9-plus) vs One M9 — Helio X10 vs SD810.
Huawei P40 Pro+ (p40-pro → p40-pro-plus) vs P40 Pro (pocket-lint/CNET).
Sony Xperia Z3+ (xperia-z3 → xperia-z3-plus) vs Xperia Z3 — the Z3+/Z4 is a
distinct 2015 SD810 model with its own Wikipedia page.
Motorola Razr 2023 (razr → razr-2023) vs Razr 2019.
Fairphone 3+ (→ fairphone-3-plus-dual-sim-lte-fp3plus) vs Fairphone 3.
Gigabyte GSmart GS202+ (→ gsmart-gs202-plus) vs GS202.
In each pair the base-model record is left untouched; only the variant's slug/name/
filename change so the two stop colliding as a false pair. Residual brand-prefix
pairs drop 183 → 176.
Counts: confirmed-different-devices-not-duplicates un-paired 7; merged 0
(not this pass); still-needs-source residual (21 same-model wrong-soc + 157
ram/weight/battery) intentionally untouched — those need external sources.
Both gates pass on #349 (app.validate, integrity_check.py --strict, slug==
filename holds); dump refreshed in the same PR. I believe this exhausts the readily
identifiable "genuinely different device" pattern in the smartphone brand-prefix
set — the remainder is the source-needed residual. Refs #296.
Google Play identifiers merged (#359): 31,903 smartphone/tablet/watch records now carry model_numbers / codenames from Google Play supported_devices.csv. Ids only, no spec value changes. ~21.8k lower-confidence rows are held for review.
Google Play ids round 2 merged (#361): 174 more records, written per path with SoC and sibling-id checks (blind audit 38/40; the 2 misses are fixed by the ≥4-char codename rule). The new-device hold pools failed two blind fresh-sample audits (81–84%) and stay in review.
Activity
data(verify): promote 761 CPUs, 6 GPUs and 5 SoCs with a live cited s…
Seungpyo1007 commentedon Sep 28, 2026
tablet/watch near-duplicate scan
Summary: Scanned 3,459 tablet records across 63 brands and 447 watch records across 13 brands using normalized slugs/names and spec cross-checks (release date, SoC, display size/resolution, RAM/storage). Found 192 confirmed duplicate pairs/groups and 38 possible pairs needing human/source verification, plus identified several distinct multi-year hardware refreshes that should not be merged.
1. Watch Findings
Confirmed Duplicates (30 pairs / groups)
Identical GSMArena scrape duplicates (3 pairs): Scraped twice from
cigarplug/scrape-gsmaunder different row IDs with 100% identical specs:huawei/2019/watch-gt-2/:huawei-watch-gt-2-scrapegsma-103vshuawei-watch-gt-2-scrapegsma-986huawei/2020/watch-gt-2e/:huawei-watch-gt-2e-scrapegsma-77vshuawei-watch-gt-2e-scrapegsma-960lg/2015/watch-urbane-lte/:lg-watch-urbane-lte-scrapegsma-1082vslg-watch-urbane-lte-scrapegsma-199Cross-directory base model duplicates (26 pairs): Same underlying device imported into sibling directories due to brand-prefixed directory names (one from
sarangzambare/smartdevices_pricing, one frommsainani/gsmarena-mobile-devices):apple-watch-38mm-1st-genvswatch-38mm-1st-gen,apple-watch-42mm-1st-genvswatch-42mm-1st-gen,apple-watch-edition-38mm-1st-genvswatch-edition-38mm-1st-gen,apple-watch-edition-42mm-1st-genvswatch-edition-42mm-1st-gen,apple-watch-sport-38mm-1st-genvswatch-sport-38mm-1st-gen,apple-watch-sport-42mm-1st-genvswatch-sport-42mm-1st-genapple-watch-series-1-aluminum-38mmvswatch-series-1-aluminum-38mm,42mmvs42mm,apple-watch-series-2-38mmvswatch-series-2-38mm,42mmvs42mm,apple-watch-series-2-aluminum-38mmvswatch-series-2-aluminum-38mm,42mmvs42mm,apple-watch-edition-series-2-38mmvswatch-edition-series-2-38mm,42mmvs42mmapple-watch-series-3vswatch-series-3,apple-watch-series-3-aluminumvswatch-series-3-aluminum,apple-watch-edition-series-3vswatch-edition-series-3asus-zenwatch-3-wi503qvszenwatch-3-wi503qhuawei-watch-2-2018vswatch-2-2018,huawei-watch-gtvswatch-gt,huawei-watch-magicvswatch-magiclg-watch-stylevswatch-style, 2018lg-watch-w7vswatch-w7samsung-galaxy-watchvsgalaxy-watch, 2016samsung-gear-s3-classicvsgear-s3-classic, 2016samsung-gear-s3-frontiervsgear-s3-frontierIntra-directory brand prefix duplicate (1 pair):
intex/2015/irist-smartwatch/:intex-irist-smartwatchvsirist-smartwatchPossible Duplicates / Needs Human Check (5 pairs)
lg/watch-urbane-2nd-edition-lte: 2015 vs 2016 year split across sources (lg-watch-urbane-2nd-edition-lte-0-75gb-4gb-4g-ltevslg-watch-urbane-2nd-edition-lte-aitoolbuzz-5427-0-75gb-4gb-4g-lte).samsung/galaxy-watch: 2018 42mm bluetooth record vs base record with slightly differing listed screen diameter (1.2" vs 1.3").samsung/gear-s2-classic: 2015 Wi-Fi vs 2016 3G release year split.samsung/gear-s3-classic: 2016 Wi-Fi vs 2017 LTE release year split.samsung/gear-s3-classic-lte: duplicate acrossgear-s3-classic-lteandsamsung-gear-s3-classic-ltedirectories.2. Tablet Findings
Confirmed Duplicates (162 pairs / groups)
Identical GSMArena scrape duplicates (15 pairs):
hp/2013/slate8-pro/:hp-slate8-pro-scrapegsma-52vshp-slate8-pro-scrapegsma-935huawei/2013/mediapad-7-youth/:huawei-mediapad-7-youth-scrapegsma-1002vs...-119huawei/2019/matepad-pro/:huawei-matepad-pro-scrapegsma-93vs...-976huawei/2019/mediapad-m6-10-8/:huawei-mediapad-m6-10-8-scrapegsma-107vs...-990huawei/2019/mediapad-m6-8-4/:huawei-mediapad-m6-8-4-scrapegsma-109vs...-992huawei/2019/mediapad-m6-turbo-8-4/:huawei-mediapad-m6-turbo-8-4-scrapegsma-108vs...-991huawei/2020/matepad/:huawei-matepad-scrapegsma-76vs...-959huawei/2020/matepad-pro-5g/:huawei-matepad-pro-5g-scrapegsma-83vs...-966huawei/2020/matepad-t8/:huawei-matepad-t8-scrapegsma-68vs...-951lenovo/2016/yoga-tab-3-10/:lenovo-yoga-tab-3-10-scrapegsma-1054vs...-171Cross-directory base model duplicates (50 pairs):
acer-chromebook-tab-10vschromebook-tab-10, 2015acer-iconia-tab-10-a3-a30vsiconia-tab-10-a3-a30, 2014acer-iconia-tab-7-a1-713vsiconia-tab-7-a1-713,a1-713hd,a3-a20,a3-a20fhdapple-ipad-9-7-2017vsipad-9-7-2017,apple-ipad-pro-10-5-2017vsipad-pro-10-5-2017, 2018apple-ipad-pro-11vsipad-pro-11, 2015apple-ipad-pro-12-9-2015vsipad-pro-12-9-2015, 2017apple-ipad-pro-12-9-2017vsipad-pro-12-9-2017, 2018apple-ipad-pro-12-9-2018vsipad-pro-12-9-2018, 2016apple-ipad-pro-9-7-2016vsipad-pro-9-7-2016hp-pro-slate-10-ee-g1vspro-slate-10-ee-g1,hp-pro-slate-8vspro-slate-8, 2013hp-slate10-hdvsslate10-hdhuawei-mediapad-m5-litevsmediapad-m5-lite,huawei-mediapad-t5vsmediapad-t5lenovo-tab-7vstab-7,lenovo-tab-7-essentialvstab-7-essential, 2016lenovo-yoga-tab-3-plusvsyoga-tab-3-plusprestigio-multipad-10-1-ultimatevsmultipad-10-1-ultimate,multipad-4-quantum-10-1,9-7, 2014multipad-4-quantum-9-7-colombiasamsung-galaxy-tab-a-8-0-2017vsgalaxy-tab-a-8-0-2017, 2018samsung-galaxy-tab-a-10-5vsgalaxy-tab-a-10-5, 2018samsung-galaxy-tab-s4-10-5vsgalaxy-tab-s4-10-5, 2017galaxy-tab-active-2vsgalaxy-tab-active2vssamsung-galaxy-tab-active-2(triplet)vodafone-smart-tab-n8vssmart-tab-n8Intra-directory brand prefix duplicates (97 pairs):
<brand>-<model>.json(fromarwinneil) and<model>.json(frommsainani) with identical release year, display size, SoC, and memory.acer-iconia-tab-a100vsiconia-tab-a100,apple-ipad-airvsipad-air,apple-ipad-mini-2vsipad-mini-2,apple-ipad-air-2vsipad-air-2,amazon-kindle-firevskindle-fire,asus-google-nexus-7vsgoogle-nexus-7,hp-slate7-extremevsslate7-extreme.Possible Duplicates / Needs Human Check (33 pairs)
huawei-mediapad-m2-8-0(2GB vs 3GB),x2(2GB vs 3GB),m2-10-0(2GB vs 3GB),t2-10-0-pro(2GB vs 3GB),t2-7-0(1GB vs 2GB),m3-lite-10(3GB vs 4GB),t3-10(2GB vs 3GB), etc.lenovo-tab-4-10-plus(3GB vs 4GB),tab-4-8-plus(3GB vs 4GB)samsung-galaxy-tab-3-8-0(1.5GB vs 1GB),tab-4-7-0(1.5GB vs 1GB),tab-active(1.5GB vs 1GB),tab-a-8-02015 (1.5GB vs 2GB),tab-e-9-6(1.5GB vs 1GB),tab-e-8-0(1.5GB vs 1GB),tab-j(1.5GB vs 1GB)alcatel/one-touch-evo-8hd(2013 vs 2014, Rockchip RK3066)alcatel/pixi-3-3g(2014 vs 2015, MediaTek MT8312)blu/touch-book-m7vstouchbook-m7(2016 vs 2017, Intel Atom x3)htc/shift(2007 vs 2008, Intel Stealey 800MHz)huawei/mediapad-7-vogue(HiSilicon K3V2 Hi3620 vs Huawei K3V2, 7.01" vs 7.0")microsoft/surface-duo: dual directoriesmicrosoft-surface-duo/andsurface-duo/for global/US 128GB/256GB SKUsConfirmed False Positives (Do Not Merge)
Total Candidate Near-Duplicates: 192 confirmed duplicate pairs/groups, 38 possible pairs needing human verification.
fix(data): backfill Commons photos for 7 Wikipedia-cited smartphones
fix(data): merge 18 duplicate Intel Atom CPU records
Seungpyo1007 commentedon Sep 28, 2026
Intel CPU ambiguous slugs (#323)
Resolved 30 of the 45 slugs #266 skipped. The change is the field the matching Wikipedia rows agree on, or the row selected by socket, segment, clock, or TDP. Fifteen stay unresolved: package rows that still disagree and the record cannot tell them apart, month phrases that are not a day ("Mid 2008", "Q2 2009"), and slugs with no record (
celeron-d-320,celeron-d-325,celeron-d-335,core-m3,core-ultra-x9,mobile-pentium-mmx-166).Examples: Core 2 Duo T-series release dates moved back to the 2006–2008 Merom/Penryn months both package rows list; E8335 base clock 2.66 → 2.93 GHz; Celeron 2.00 GHz desktop date 2009-02-01 → 2002-09-18; Xeon Gold 5215 date 2019-12-01 → 2019-04-02.
Seungpyo1007 commentedon Sep 28, 2026
2026-09-29 batch: tablet/watch duplicate merge + placeholder-data root cause
PR #324 merged: 183 duplicate records removed across 182 verified tablet/watch groups (18 identical GSMArena re-imports, 70 cross-directory model duplicates, 95 intra-directory brand-prefix duplicates). Ambiguous cases (RAM-tier variants, year splits, LTE/Wi-Fi distinctions, Surface Duo, MediaPad Vogue, 3 genuine Prestigio revisions, Watch Magic's unsourced OS) were deliberately left unmerged.
Root cause found along the way: a whole import batch had stamped placeholder values (battery_mah=1500, weight_g=150.0) on 43 watch and 217 tablet records regardless of actual device — not real conflicting data. 170 fields were repaired using same-device sibling evidence; 362 with no confident replacement are now null rather than fake. Also caught a comma-parsing bug: iPad Pro 12.9 (2015) battery was stored as 307 mAh instead of 10,307.
This required a schema change: TechEngine#109 makes mobile battery_mah/weight_g nullable (previously non-nullable, which is very likely why the placeholder bug existed in the first place - some import had to put a number to satisfy the schema).
Also this batch: #323 resolved 30 of the 45 ambiguous Intel CPU slugs from #266 by matching each record's socket/clock/segment/TDP against the correct Wikipedia row; 15 remain unresolved.
Seungpyo1007 commentedon Sep 29, 2026
Scanned the smartphone category (93,407 records) for near-duplicates, using the
same method as the tablet/watch pass (#324). Opened #329.
Merged: 50 confirmed brand-prefix duplicate pairs. These are the cases where a
device exists under both a bare slug (
nord-n100) and a brand-prefixed twin(
oneplus-nord-n100) and the core specs (release_date / soc / ram / battery /weight / os) are identical. Kept the shorter canonical slug, unioned
source_urls,carried the richer side's fields onto the keeper, deleted the redundant file — same
convention as the Intel Atom CPU dedup. One real conflict inside an apparent
duplicate (OnePlus 5T Star Wars main camera 20 MP vs 16 MP) was resolved to 16 MP
per GSMArena/OnePlus (the 20 MP is the secondary sensor), not blind-merged.
Bigger finding — a systemic placeholder-value bug, reported not mass-edited.
487 distinct device names across 624 files share the exact triple
battery_mah=1500, weight_g=150.0, ram_gb=1, spanning 1994–2020 and even includingwatches/tablets, mostly from the
gsmarena-mobile-devicesKaggle import. Samefabricated-defaults signature (1500/150) #324 flagged. More triples with the
150.0 g / 1 GB+ genericYYYY-01-01tell-tale:(2000,150,1)×182,(1800,150,1)×92,(1000,150,1)×83,(2500,150,1)×80.This is also why 717 of the 767 brand-prefix pairs read as "conflicts" rather than
duplicates: the bare (Kaggle) side carries these placeholder defaults while the twin
has real specs. They're very likely the same devices, but merging them correctly
means repairing the fabricated side against a real source per record — inventing
values for hundreds of devices, which violates the "never invent values" rule. That
warrants a dedicated repair pass; I left those 717 pairs untouched and documented
rather than guessing.
Both gates pass on the merged tree:
app.validateandintegrity_check.py --strict(0 hard anomalies). Dump refreshed in the same PR.
fix(data): merge 50 confirmed smartphone near-duplicate records
fix(data): merge 50 confirmed smartphone near-duplicate records
Seungpyo1007 commentedon Sep 29, 2026
Worked through the 717 brand-prefix conflict pairs that #329 left documented but
unresolved (the pairs that disagreed on soc / release_date / ram / battery / weight /
os). Opened #330.
Approach: re-detected the pairs on current develop, then classified every conflict
field against evidence rather than guessing. The
socdisagreement (572 pairs) isalmost always the same physical chip under a different slug spelling
(
qualcomm-msm7227-snapdragon-s1vsqualcomm-msm7227), and the other conflicts onthose pairs are the fabricated bulk-import placeholders (
weight_g=150.0,battery_mah=1500, genericYYYY-01-01).Exact counts (all 717 accounted for):
resolved-and-merged: 364 — same-chip pairs sourced Kaggle-GSMArena (bare) vs
phonedb.net (twin). Kept the GSMArena record (real specs; spot-verified Acer DX650
= 133 g and CloudMobile S500 = 122 g against GSMArena/imei.org, vs the phonedb
twins' 150.0 g placeholder), unioned source_urls, deleted the twin. No value
invented — in every one of these the keeper already held the real value and only
the deleted twin carried the placeholder.
verifiedwas not flipped (thephonedb scalars proved unreliable), and
release_datewas not changed(phonedb dates are unreliable too — it lists DX650 as 2009-03 while GSMArena/imei
say 2009-06 — so importing them would be guessing).
confirmed-different-devices-not-duplicates: 31 — pairs whose two sides name
genuinely different silicon, i.e. different SKUs/devices that merely share a base
name. Left separate and dropped from the duplicate classification. Examples:
Huawei P50 Pro (Kirin 9000 and Snapdragon 888 — GSMArena confirms both
chipset variants shipped), Honor Magic V2/V3, HTC One M9 (SD810) vs One M9+
(Helio X10), OnePlus Nord CE 2 (Dimensity 900) vs Nord CE 2 Lite (SD695), Sony
Xperia Z (S4 Pro) vs Z1 (SD800), Motorola razr generations, Oppo Find X8 Pro
(Dimensity 9400), ZTE Grand Memo (SD800 vs SD600).
still-unresolved-no-source: 322 — left untouched rather than guessed:
(different provenance, want individual checks);
dual-core 1.0 GHz,N/A,<brand> mobile platform) — same device, but the date/scalars still need a source;blu selfievsvivo-selfie).Plus the
release_datefield on the 364 merged records, deliberately left as-is.Merging the remaining 322 correctly needs per-record web sourcing of hundreds of
individual devices; per the "never invent a value" rule I've left them documented
rather than guessed. Both gates pass on #330 (
app.validate,integrity_check.py --strict→ 0 hard anomalies); dump refreshed in the same PR.fix(data): resolve 364 smartphone brand-prefix conflicts (same-chip d…
Seungpyo1007 commentedon Sep 29, 2026
Brand near-duplicate scan, 2026-09-29
Picking up the tracker's open "near-duplicates in other categories not yet scanned" item, I hand-reviewed the whole brand category — the same duplicate-hunt CPU/tablet/watch/smartphone/laptop/monitor/pda got today. Scope: 207 brand records under
data/brand/<country>/<slug>.json, on develop9ce1979.Result: the brand category is already clean — no confirmed duplicates, no merge, no PR. Reporting here rather than forcing a change.
What I checked
Brand records are manufacturer entities (name, slug, country, categories, founded_year, website, logo, descriptions), not product specs, so I looked for the three failure modes that apply to entities:
hpvshewlett-packard, casing/punctuation variants). Automated check: 0 normalized-name collisions and 0 normalized-slug collisions across all 207 records (normalizing to lowercase alphanumerics). None of the classic synonym pairs are double-listed —hp,google,lg(LG Electronics),samsung,at-t,t-mobileeach appear exactly once.sub-brandtag or are sub-brand-like (alcatel,infinix,iqoo,itel,nubia,poco,redmi,tecno,xolo,redmagic,alienware). Each is a legitimately separate entity with correct parent attribution in its description (TCL, Transsion, vivo, ZTE, Xiaomi, Lava, Dell). These are the "Beats-under-Apple" case — kept distinct, not merged.Related-but-distinct entities (deliberately NOT merged)
The only pairs that look superficially like duplicates are joint-venture / brand-licensing lineages, and each side is a genuinely separate entity with its own device fan-out:
Near-name prefixes that are unrelated companies:
blackberry/blackview,infineon/infinix,microchip/micromax/microsoft,transmeta/transsion. All distinct.This is consistent with the recent develop history — the "364 resolved brand-prefix conflicts" work already normalized brand slugs, so the category was in good shape going in.
Verification
No data files were changed. Confirmed the current tree is green:
python -m app.validate→ Data validation passed (exit 0), including brand path/slug convention and the brand foreign-key set used by device records.python integrity_check.py <data> --strict→ no hard anomalies (exit 0); the structural section reports no duplicate brand slugs/names, no slug/filename mismatches, and no verified-without-source brands.No FK repoints were needed (nothing merged), and no dump refresh is warranted since no data changed.
Seungpyo1007 commentedon Sep 29, 2026
Software near-duplicate scan, 2026-09-29
Continuing the "near-duplicates in other categories not yet scanned" item after the brand pass, I scanned the software category. Scope: 42,493 software records under
data/software/<2char>/<slug>.json, on developea3f31e.Result: no true duplicates — the software category is clean. No merge, no PR. Reporting here, same standard as brand.
Method (adapted to how this dataset is keyed)
Software records here are Wikidata-derived, and every one of the 42,493 records carries a Wikidata QID in
source_urls(0 without). The QID is the entity identity, and slugs suffixed-qNNNNare deliberate disambiguators for records that share a display name. So "same name, different slug" is expected and is not itself a duplicate — a real duplicate is the same Wikidata entity imported twice, an unkeyed name-and-developer collision, or a content clone. I checked all three:Name collisions are genuinely distinct entities
There are 993 normalized-name collisions covering 2,165 records, and all 993 are fully QID-disambiguated (0 ambiguous). Spot-checking the largest:
These are exactly the "don't merge real distinct entities that happen to be related/similarly named" case.
The "stub-looking" QID-suffixed records confirm the pattern rather than break it: e.g.
zune(Q229851) vszune-q1970794(Q1970794),zoa-space(Q118225284) vszoa-space-q118225291(Q118225291), andzsnesvszsnes-2(ZSNES 2, Q116868010, a distinct 2021 project) all cite different Wikidata items — they are separate entities, not stubs shadowing a fuller record.Verification
No data files changed. Confirmed the current tree is green:
python -m app.validate→ Data validation passed (exit 0), incl. software required-field / slug-convention / filename-match checks.python integrity_check.py <data> --strict→ no hard anomalies (exit 0); the structural section reports no duplicate software slugs, no slug/filename mismatches, and no verified-without-source software records.No FK repoints were needed (nothing merged) and no dump refresh is warranted since no data changed.
Seungpyo1007 commentedon Sep 29, 2026
Laptop near-duplicate scan is done — #332 (against
develop).Ran the same method as the tablet/watch (#324) and smartphone (#329) passes on all 1,995
data/laptop/records: slug/name normalization (punctuation- and brand-prefix-agnostic) plus a spec cross-check (release_date, cpu ref/name, ram_gb, storage_gb, display size/resolution, gpu, weight, os).Per-pattern results:
slugandvariant.source_row, all withinAmmok/laptop_price_prediction. Kept the lowersource_row, deleted the redundant file;source_urlsidentical so the union was a no-op and no values were changed. Breakdown: acer 1, dell 13, hp 3, lenovo 8, msi 2, toshiba 2.1995 → 1966records.dell-xps-13vsxps-13): none. All records live under a consistent brand-prefixed base dir.Configuration variants (same model, different CPU/RAM/storage) were left untouched — distinct real SKUs, not duplicates.
Placeholder / fabricated-defaults check: clean. No equivalent of the phone
battery_mah=1500/weight_g=150.0stamping.ram_gb,storage_gb,weight_g, anddisplay.size_inchall show organic distributions with distinct-name diversity proportional to frequency (e.g.ram_gb=16across 735 distinct models,storage_gb=512across 739). No single value is stamped across unrelated devices.app.validateandintegrity_check.py --strictboth pass on the final state (only the pre-existing advisory CPU/GPU ratio outliers print, which the gate treats as review-only). Dump refreshed as the PR's last commit, scoped to the laptop collection.That closes out the "near-duplicates in other categories not yet scanned" item for laptop — no ambiguous groups were left pending.
Seungpyo1007 commentedon Sep 29, 2026
Monitor near-duplicate scan is done — #333 (against
develop).Continuing the "near-duplicates in other categories not yet scanned" item after the tablet/watch (#324), smartphone (#329) and laptop (#332) passes. Scope: all 882
data/monitor/records, every one sourced from a single Kaggle Amazon monitor dataset. Same method: slug/name normalization (punctuation- and brand-prefix-agnostic) plus a spec cross-check (release_date,size_inch,resolution,aspect_ratio,refresh_hz,panel_type,ppi, and the model number inname).Result: 651 confirmed near-duplicates merged.
882 → 231.Per-pattern results:
slugandvariant.source_row. Kept the lowestsource_row, deleted the redundant files;source_urlswere identical so the union was a no-op and no values changed. Largest clusters:asus-va329he−83,aopen-aopen-by-acer−68,acer-cb272−62,acer-sh242y−59,acer-xb273k−59,acer-xz342cu−46,acer-kb252q−44,acer-sb240y−43,acer-sb272−38,acer-xz320qu−30,acer-hdr10−72,acer-sb270−26,acer-kc242y−13,acer-xb273u−6.dell-u2723qevsu2723qe): none. Every record already lives under a consistent brand-prefixed base dir.asus-va24eherows 60 & 404 — the same VA24EHE (23.8", 1080p, 75 Hz, IPS, $99) imported with two Amazon title strings. Kept the cleaner title and adopted the more completeadaptive_sync = "FreeSync"from the other side (a real, documented VA24EHE spec, cited in the PR).Configuration variants (same panel, different size/resolution/refresh/SKU) were left untouched — distinct real products, not duplicates. Documented and intentionally left unmerged:
acer-eda343cur(100 vs 180 Hz),acer-sb240y(G0bi vs G0wi),acer-sb242y(EBI vs Hbi),acer-sb272(new vs Renewed),aopen-aopen-by-acer(32SA2QK 4K vs 27E1 FHD),gigabyte-m27q(M27Q vs M27Q PRO),hp-hp-series-monitor(Series 5 24" vs 524sa),lenovo-lenovo-thinkvision-full(M15 vs T22i-30),lg-lg-digital-signage(49" vs 86"),lg-lg-ultragear-gaming(QHD 240 Hz vs 5120×1440 165 Hz),lilliput-fa1014-np(1280×800 non-touch vs 1080p touch),samsung-hdr2000(Neo G7 vs Neo G8), and theacer-hdr10residual (itshdr10slug is an importer fallback shared by a few genuinely different Nitros — left as-is rather than guess).Placeholder / fabricated-defaults check: clean. No equivalent of the phone-batch stamping —
refresh_hz,ppi,size_inch,msrp_usdall show organic distributions with distinct-name diversity proportional to frequency; no single spec value is stamped across unrelated devices. The one systematic artifact is the placeholder daterelease_date = "2023-01-01"on all 882 records — a uniform dataset-wide import default, not a spec forced onto unrelated devices, so out of scope for this merge but worth flagging for the "placeholder dates (YYYY-01-01)" item: this whole Amazon-monitor import has no real release dates.app.validateandintegrity_check.py --strictboth pass on the final state (only the pre-existing advisory CPU/GPU ratio outliers print, which the gate treats as review-only). Dump refreshed as the PR's last commit, scoped to the monitor collection — the 651 orphaned detail pages were pruned, the monitors list rebuilt (231), and the manifest count updated; unrelated pre-existing serialization drift in other collections was kept out to keep the diff monitor-scoped.That closes out the "near-duplicates in other categories not yet scanned" item for monitor — no ambiguous groups were left pending beyond the documented distinct-SKU cases above.
fix(data): resolve 170 more smartphone brand-prefix conflicts
Seungpyo1007 commentedon Sep 29, 2026
Continued the smartphone brand-prefix dedup. #329 (50) and #330 (364) left a
residual of 353; #334 resolves 170 more with source-verified fixes and un-pairs 3
genuinely different devices.
resolved-and-merged: 170
variance); adopted the twin's reliable Wikipedia/vendor
release_dateover thebare
YYYY-01-01placeholder (verified e.g. Huawei P30 Pro 2019-03-26 = GSMArena).source: Oppo Find X8 Pro → Dimensity 9400 (oppo.com), Motorola Edge 50 Ultra →
Snapdragon 8s Gen 3 (motorola.com), Honor Magic V3 → Snapdragon 8 Gen 3
(phonearena), plus OnePlus Nord CE 2 Lite / CE 2, Sony Xperia Z, Honor Magic V2
(bare already correct; GSMArena/vendor cited).
kept the GSMArena record, adopted the specific soc where the keeper's was vague,
rejected the phonedb stamped placeholders, deleted the twin.
confirmed-different-devices-not-duplicates (un-paired): 3 — HTC
one-m9holdsthe One M9+ (Helio X10) vs One M9 (SD810); Motorola
razrthe 2023 (SD 8+ Gen 1)vs 2019 (SD710); Sony
xperia-z3the Z3+ (SD810) vs Z3 (SD801). Left as separaterecords.
still-unresolved-no-source: 180 — 21 same-model pairs whose correct chip I
couldn't source this round (includes the Huawei P50 Pro, a genuine dual-chipset
model — Kirin 9000 and Snapdragon 888 both shipped), 157 pairs needing a per-record
ram/weight/battery source, and 2 odd/other pairs. Left documented rather than
guessed.
Running total across #329/#330/#334: 584 of the original 767 brand-prefix pairs
resolved (50 + 364 + 170), plus the genuinely-different devices un-paired. Both
gates pass on #334 (
app.validate,integrity_check.py --strict); dump refreshedin the same PR.
Refs #296(leaving the tracker open for the residual and theother categories' work).
Seungpyo1007 commentedon Sep 29, 2026
Scanned the pda category (140 records) for near-duplicates by hand — small
enough to review every match individually. Opened #335.
Merged: 28 confirmed duplicate records → 21 keepers (140 → 112). Same three
patterns as the tablet/watch (#324) and smartphone (#329/#330) passes:
Accompli 388 (
scrapegsma-344/-1227) and Mitac MIO A701 (-278/-1161),byte-identical except
source_row. Kept the lower row-id.treo-680/treo-750v/treo-750/treo-500veach had a phonedb-sourcedpalm-treo-…twin. Kept thebare GSMArena record (same convention as the fix(data): resolve 364 smartphone brand-prefix conflicts #330 merge), upgraded the name to the
brand-inclusive form ("Palm Treo 680" …), unioned
source_urls, deleted the twin.Treo 750 vs 750v and 500 vs 500v were kept separate — different SKUs.
the 16 HP iPAQ pairs, i-mate pda2/pda2k/pdal, and the Palm Treo Pro 3-way (bare +
scrape-gsma + phonedb). Kept the clean record and repaired its
ram_gbfrom thedeleted twin's cited
memory_internal(the clean side carried the fabricatedram_gb=1; the twin had the real 64/128 MB). scrape-gsma URL unioned so the RAM issourced. No value invented.
Left unmerged and documented (not guessed): the GSM webOS
palm-pre-scrapegsma-7065and
palm-pre-plus-scrapegsma-7063overlap the region-split castle family(
palm-pre[-plus]-palm-castle-cCDMA andpalm-pre[-plus]-gsm-{eu,na}-palm-castle-g)but do not map 1:1. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm,
pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone.
Systemic finding (reported, not mass-edited): the same fabricated-defaults
signature the earlier passes flagged is present in pda too — nearly every clean
record carries
ram_gb=1(real values 16–256 MB) and many Palm records carryos="Feature phone"for Palm OS / webOS devices. I only repairedram_gbinside theconfirmed merges (where the deleted twin supplied a cited value); the remaining
placeholder records need a per-record source and were left untouched rather than
mass-edited.
Both gates pass on the merged tree (
app.validate;integrity_check.py --strict—0 hard anomalies). Dump refreshed in the same PR (pdas 140 → 112, no other
collection touched).
Using
Refs #296rather thanCloses: this clears the pda slice of the"near-duplicates in other categories not yet scanned" item, but that item stays open
— smartphone still has 322 documented-unresolved pairs (per #330), and the remaining
categories (laptop, monitor, CPU beyond the Atom pass, etc.) haven't been scanned for
near-duplicates yet.
fix(data): merge 651 confirmed monitor near-duplicate records
chore(dump): refresh static API for the merged monitor records
fix(data): resolve 170 more smartphone brand-prefix conflicts
fix(data): merge 28 confirmed PDA near-duplicate records
Seungpyo1007 commentedon Sep 29, 2026
Follow-up to #332: ran a dedicated placeholder-value / fabricated-defaults scan across all 1,966 post-dedup
data/laptop/records (not just duplicate pairs), looking for the phone-import pattern where an import batch stamps an identical fake default (e.g.battery_mah=1500,weight_g=150.0) on unrelated devices regardless of their real specs.Result: no fabricated-default bug found in laptop data. Nothing to fix beyond what the dedup pass already caught. No files changed, so no PR.
What I checked and why each common-looking value is real, not fabricated:
weight_g,ram_gb,storage_gb,os_version,display.size_inch/resolution/ppi, and combos (size+resolution, ram+storage, weight+size, full 5-tuple). Every high-frequency value is a genuinely market-dominant config (16GB RAM, 512GB SSD, 15.6"/1920x1080, Windows "10") and the distinct-device counts decay smoothly — there is no isolated spike of an implausible value stamped across unrelated models.ppiis 100% internally consistent with each record's ownsize_inch+resolution— 0 mismatches >8px across 1,966 records. A fabricated display-spec batch would leave ppi inconsistent with the stated panel; it doesn't.abhinavflacsupplies msrp+ppi but no weight/os_version (0%);Ammoksupplies weight+os_version but no msrp. Missing fields are simply absent in each source, not stamped with a default.1910 gis entirely the HP 15 (2017) chassis reused across configs;1370 gis the real MacBook Pro 13" (2016-2019) / ThinkPad Yoga weight;1360/1250/1200 gare genuine ultrabook weights shared by real ~13" machines. None is an implausible value on unrelated devices.battery_mah/battery_whfield at all, so the phone bug's exact vector doesn't exist here.validate.py/integrity_check.py --strictwere not re-run since no records were modified (both already pass on this merged state per #332).Seungpyo1007 commentedon Sep 29, 2026
Follow-up to the monitor dedup (#333): I ran a standalone placeholder-value scan across all 231 post-dedup
data/monitor/records (216 distinct devices), specifically hunting for the fabricated-default pattern the tablet/watch and smartphone passes found (an import stamping an identical fake value — e.g.battery_mah=1500,weight_g=150.0— onto unrelated devices), including records that have no duplicate twin to reveal it.Result: no fabricated-default bug in monitors. No data changed, no fix PR. Reporting plainly, as agreed.
Method
For every device-varying field I counted how many distinct devices share each exact value, then applied a decisive correlation test: a real common value is consistent with each device's own name/derivable specs, whereas a stamped default appears regardless of the device. I checked each field's most-repeated value against the model name (and, for
ppi, against size+resolution).Why the high-frequency values are real, not stamped
features.response_time_ms = 1(65 devices) — the biggest "suspicious" cluster. 64/65 are ≥100 Hz and/or say "gaming"/"1ms" in the title; 0 low-refresh office monitors falsely carry 1 ms, and 0 names state a different response time than the field. 1 ms is the genuine marketed spec for these gaming panels.refresh_hz— of 125 records that state a refresh rate in the name, 0 disagree with the field.resolution— 0 mismatches across 109 named records.size_inch— only 4 apparent mismatches, all regex artifacts (e.g. "34.18 inch" parsed as 18, a "22-to-24-inch" privacy-screen range, rounding of 24.5→25).msrp_usd = 100(19 devices) — all genuinely budget/portable/office 1080p panels (22–27" IPS, ZenScreen portables, touch monitors); the field has 110 distinct prices overall and neighbours at 80/99/110, i.e. an organic distribution, not one price pasted everywhere.rating— a textbook Amazon distribution: a smooth bell 4.2→4.6 (peak 4.6 = 51) with a tail down to 1.0 and a small 5.0 spike. Not a default.panel_type = IPS,aspect_ratio = 16:9,resolution = 1920x1080,size_inch = 27"are simply the genuinely dominant real monitor specs, and they co-vary correctly with the names.ppi— 3 records are internally inconsistent with their size+resolution (Dell Inspiron 24 5400 AiO, MSI MAG274UPF, Lilliput FA1014), but each has a unique wrong value, i.e. individual scrape noise, not the repeated-default fingerprint this pass targets (general per-field accuracy is out of scope here).The one genuinely-repeated placeholder — and why it's left as-is
release_date = "2023-01-01"sits on all 216 devices. It is a real placeholder, but it's category-wide (not a fake value stamped on a subset of unrelated devices while others vary), it's a required field thatvalidate.pywill reject if nulled or removed, and there's no per-device source to derive true dates — so nulling it is impossible and inventing dates is off-limits. This is already the tracked "placeholder dates (YYYY-01-01)" item on this issue and needs an external per-device source, so I'm leaving it rather than touching it under this task.Verification
No files changed. Confirmed the current tree is green:
python -m app.validate→ Data validation passed.python integrity_check.py <data> --strict→ no hard anomalies.So the monitor category has no hidden fabricated-default values beyond the category-wide placeholder date already on the tracker.
Seungpyo1007 commentedon Sep 29, 2026
Opened #337 against
develop— a first source-citation pass overdata/laptop/anddata/monitor/.Scope (checked live): laptop 1995 total / 1890 unverified; monitor 882
total / 882 unverified. Every unverified record already carried a
bulk-dataset link (huggingface Ammok, abhinavflac, Kaggle Amazon-monitor),
so the gap was uncited, not unsourced.
This PR (45 records touched):
Laptop weight x3, Pixelbook weight x3, LG 24GN60R size x1, Samsung
Odyssey G9 49" DQHD resolution x1 (all corrected to match the cited
vendor/Wikipedia spec, never invented).
861 monitor) — untouched, keep existing provenance.
Each verified record was confirmed field-by-field against a manufacturer
spec page or Wikipedia infobox (Apple, Microsoft, Google, ASUS, Dell,
Gigabyte, MSI, Samsung, LG, BenQ, ViewSonic).
python -m app.validatepasses andintegrity_check.py --strictreportsno hard anomalies. The static dump was regenerated (
python -m app.dumpfrom TechEngine against the worktree) as the PR's final commit.
The large remaining uncited set is a good candidate for follow-up batches
(the abhinavflac records carry exact model SKUs that are individually
verifiable against vendor pages).
fix(data): add source citations for 45 laptop/monitor records
chore(site): refresh public dump for laptop/monitor citations
Seungpyo1007 commentedon Sep 29, 2026
Generic SoC placeholder sourcing — first pass (PR #338)
Rescanned current
developfor placeholdersocreferences. Three genuineplaceholder SoC records hold essentially all of them:
data/soc/arm/2014/0.json("0") — 3,610 device refsdata/soc/qualcomm/2011/qualcomm-snapdragon.json(bare family) — 1,762data/soc/arm/2008/n-a.json("N/A") — 273Total ~5,645 refs across ~2,793 base models. All three are
verified: falseKaggle/phonedb bulk-import defaults with
"Unknown mobile GPU"/"N/A"fields.Fixed: 45 records (Google Pixel). Pixel phones use one global SoC per model
(no regional Exynos/SD split), so the chip is unambiguous and Wikipedia-documented.
Re-pointed to existing verified records: Pixel(2016)→
snapdragon-821,Pixel 3→
snapdragon-845, Pixel 3a/3a XL→snapdragon-670,Pixel 4/4 XL→
snapdragon-855, Pixel 4a→snapdragon-730g,Pixel 4a 5G/5/5a 5G→
snapdragon-765g(30 off"0", 15 offqualcomm-snapdragon).Each gained the canonical Wikipedia device-page citation in
source_urls; novalues invented; assignments corroborated by sibling Pixel records already on the
same chips.
Left unresolved (documented, not guessed): ~5,600. The device records carry no
embedded chipset signal — only name/RAM/battery/display, overwhelmingly from the
aitoolbuzz/phonedbKaggle imports. Repairing them means per-record web sourcingof thousands of mostly obscure/regional devices, i.e. inventing at scale — the same
line #324/#329/#330 drew. Notable trap avoided: Samsung Galaxy S/Note region
variants (
SM-N960FExynos vsSM-N960USnapdragon, etc.) can't be blanket-mappedand need per-model-number sourcing.
Both gates pass (
app.validate✅;integrity_check.py --strict→ 0 hardanomalies); dump refreshed as the PR's last commit. Follow-up passes can pick off
further single-global-SoC brands the same way (fully sourced, no invention).
Refs #296
Seungpyo1007 commentedon Sep 29, 2026
Follow-up to #332 (laptop): ran the same distinct-device placeholder-value scan on
data/tablet/(3,305 records) anddata/watch/(418 records), checking whether the fabricated-default pattern (battery_mah=1500,weight_g=150.0) or the RAM-in-storage bug survives outside what #324 and #301 already caught.Result: both categories are clean. No fabricated-default pattern remains, nothing to fix — no PR.
What I checked:
battery_mah=1500andweight_g=150have zero occurrences in either tablet or watch. fix(data): merge 183 confirmed tablet/watch near-duplicate records #324 fully eliminated them.battery_mah,weight_g,ram_gb,storage_options_gb,display.size_inch/resolution/ppi, and combos (battery+weight, size+res+ppi). Every high-frequency value is a genuinely common real spec (4000/5000/6000 mAh tablet batteries, 7"/8"/10.1" sizes, 1GB/2GB/4GB RAM) and — critically — each is spread across 8–11 independent source datasets (aitoolbuzz, madajaswanth, jakubkhalponiak, cigarplug/scrape-gsma, arwinneil, beridzeg45, sady36, lileejass…). The fabricated-default fingerprint is a single value concentrated in one import batch stamped on unrelated devices; that does not appear here — the distributions decay smoothly and are cross-source.ppiinternal consistency with size + resolution: tablets 0 / 2,092 mismatches. Watches showed 11 apparent mismatches, but all are round-display models (255 of 413 watch resolutions are square W×W) where GSMArena publishes a diameter-based ppi rather than the naive pixel-diagonal. Spot-checked the Samsung Gear S3 (1.3", 360×360, storedppi=278) against GSMArena, which lists exactly "360 x 360 pixels (~278 ppi density)" — so the stored values are correctly sourced, not fabricated. My rectangular formula was simply the wrong model for round panels.ram_gbexceeding the maxstorage_options_gb— no swap residue.validate.py/integrity_check.py --strictwere not re-run since no records were modified.That covers tablet + watch for the "fabricated defaults in other categories" item alongside the laptop pass in #332.
Seungpyo1007 commentedon Sep 29, 2026
Applied the same placeholder-value methodology from the monitor pass to
data/gpu/— all 2,083 records, hunting for the fabricated-default fingerprint (one value stamped across many unrelated cards whose real specs differ). Checkedbase_clock_mhz,boost_clock_mhz,memory_gb,tdp_w,release_date, plusmemory_bus_bit,memory_type,memory_bandwidth_gbps, core counts,pcie_version,architecture,msrp_usd.Result: no fabricated-default bug in GPUs. No data changed, no fix PR. Reporting plainly.
Why the repeated values are real, not stamped
This is a curated TechPowerUp/Wikipedia-sourced set (
verified: true), and the distinct-card frequency counts plus correlation-against-card-class checks all come back consistent:base_clock_mhzhas 435 distinct values,boost_clock_mhz478. The top repeats (500/600/650 MHz, ~80 cards each) are genuine round clocks that early-2000s GPUs actually ran at. Only 5 post-2016 cards carry a 500–650 base, and all are legitimate: Quadro/RTX Max-Q mobile parts (real 600 MHz base, 1215–1350 MHz boost) and Jetson Orin Nano SoC modules (fixed 625 MHz).boost == baseon 53/735 post-2016 cards — every one is an OEM/workstation/datacenter part (FirePro, Radeon Instinct/Pro, Tesla, Radeon R5/R7 OEM) that genuinely ships without a separate boost clock, matching TechPowerUp.memory_gb— the canonical power-of-two sizes (1/2/0.5/4/8/16 GB). The biggest cluster, 1.0 GB (319 cards), spans 2005–2017 and peaks 2010–2013 — the actual 1 GB-VRAM era — with zero cards released ≥2020. Period-consistent, not pasted onto modern cards.tdp_w— standard tiers (50/75/150/250 W = slot power, 6-pin, 6+8-pin, flagship). Modern ≥16 GB cards show varied high TDPs (300/250/350/700/600/400 W), i.e. no flat default.memory_bus_bit/memory_type/pcie_version/architecture— real taxonomies with natural distributions (128/256/64-bit; GDDR5/GDDR3/GDDR6; PCIe 2.0/3.0 x16; TeraScale/Tesla/Kepler…).release_date— 955 distinct dates. The largest cluster, 2021-04-12 (30 cards), is a real NVIDIA professional/datacenter launch day (RTX A2000/A4000/A5000, A10/A16/A30, T600, Quadro T1200). Only 57/2,083 records useYYYY-01-01, and they are spread across 1996–2026 on old/obscure/OEM/engineering and future-unannounced cards (e.g. 2026-01-01 RTX 5070/5080 SUPER) — the honest "only the year is known" convention, not a single stamped placeholder like the monitor set's uniform2023-01-01. (release_dateis a required field, so it couldn't be nulled regardless, and there's no per-card source to invent exact days.)Verification
No files changed. Current tree is green:
python -m app.validate→ Data validation passed.python integrity_check.py <data> --strict→ no hard anomalies.Note on the parallel GPU work
Another pass is doing near-duplicate/cross-check on
data/gpu/. My scan was strictly the fabricated-placeholder angle and found nothing to fix, so there's no overlap with their dedup edits and no data conflict from my side (I changed nothing). The GPU category has no hidden fabricated-default values.fix(data): backfill 37 confirmed GSMArena URLs via verify tool
chore(dump): refresh smartphone dump pages for GSMArena URL backfill
Seungpyo1007 commentedon Sep 29, 2026
Follow-up backfill run (targeting a higher
--limit 2000after #339 merged) is blocked by an active GSMArena rate limit — no records were upgraded this run.Details:
developand ran the tool from the develop-pinned TechEngine commit (147d27d).429withRetry-After: 36000s(10 hours). The tool stopped cleanly before sending any further requests (1 request total, exit 1) and wrote nothing.Per the standing rules I did not retry/spin, and since this was a no-op (working tree clean, zero writes) I'm reporting here rather than opening a PR. The tool and gate are unchanged and working — the earlier dry-run before #339 was clean — so this is purely a cooldown window, not a tooling regression.
Next step: wait out the
Retry-Afterwindow (roughly ~7 more hours from now) and re-run the--limit 2000backfill then. The resume cache already carries the 37 confirmations from #339, so a later run resumes from where this left off without re-fetching them.Seungpyo1007 commentedon Sep 29, 2026
Ran a within-record self-consistency pass over data/pda/ (140 records), same pattern as #301 - values derived only from each record's own fields, no external sources.
Found and fixed 1 real issue (PR #340):
am_gb: 3 / �ariant.memory.ram_gb: 3) leaked into the storage list. The record's own �ariant.memory.storage_gb: 32 confirms storage is 32 GB only -> corrected to [32].
Reviewed and intentionally left (ambiguous):
fix(data): self-consistency correction for 1 PDA record
Seungpyo1007 commentedon Sep 29, 2026
Scanned the website category (one of the last untouched in the accuracy tracker) for near-duplicates. Opened #341 against \develop.
Result: merged 153 confirmed near-duplicate groups and removed 154 redundant files (one 3-way group); the \websites\ collection drops 40,084 -> 39,930.
Candidates were grouped by normalized \homepage_url\ + display name, with records sharing the exact same normalized URL as the strong signal (195 such groups reviewed). Confirmed duplicates fell into these patterns:
Each merge kept the fuller/verified/better-sourced side, unioned \source_urls, OR'd \�erified, and filled only missing optional fields — no invented values.
Left as distinct (the known trap): records that only share an index/landing URL but are different real entities — museum exhibition pages under one archive host, imageboard boards, registration services differing only by URL fragment (#ul/#ip), and borderline company-vs-website / subsite / regional-version pairs (itv.com vs ITVX, Daily Mail vs MailOnline, two municipalities sharing one open-data portal URL, etc.).
Verification: \�pp.validate\ passes; \integrity_check.py --strict\ reports no hard anomalies (no dup slugs, no slug/filename mismatch, no verified-without-source). Static dump refreshed as the PR's last commit.
Remaining open sub-item: the parallel software scan.
Seungpyo1007 commentedon Sep 29, 2026
Ran a fresh within-dataset self-consistency pass over \data/tablet/\ and \data/watch/\ (follow-up to #301, since more data has moved through since — e.g. the tablet/watch dedup in #324 and the mobile-source reclassification).
Found and fixed 261 records (259 tablet + 2 watch) where the device's RAM size had leaked into \storage_options_gb. In every case the polluting value equals the record's own \�ariant.memory.ram_gb\ (and top-level
am_gb) and is strictly smaller than every real storage tier in the same list — those tiers independently confirmed by the record's own \�ariant.memory.storage_gb. A 1-2 GB entry can't be a storage capacity for these devices, so it's the RAM size, and the fix removes it. Example: \iconia-tab-a100\ had \storage_options_gb: [1, 16]\ with
am_gb: 1\ → corrected to [16].
Every correction is derived only from fields already in the same record — no external sources, no invention. Genuinely ambiguous cases (multiple RAM tiers in the list, or a low value that could plausibly be a real storage size) were deliberately left untouched.
Verification:
PR: #342
Seungpyo1007 commentedon Sep 29, 2026
GPU near-duplicate + spec-accuracy pass done — #343 (against
develop).Scanned all 2,076
data/gpu/records with the same method as the CPU / tablet/ watch / smartphone passes. Found 8 confirmed cross-source duplicate pairs
(same physical product imported twice under different slug conventions, one
side Kaggle-sourced, one TPU/vendor-sourced). Merged each into the more
complete / correctly-sourced record, source-verified every disagreeing field,
unioned
source_urls, deleted the redundant file: 2,076 → 2,068.Notable spec conflicts resolved against authoritative sources:
FP32 16.35 → 15.7 TFLOPS (NVIDIA datasheet).
No configuration variants were merged. Close families that only share a name
prefix — Tesla K40 d/m/s/t, GRID vGPU profiles, PCI/AGP and MXM/PCIe bus
variants, OEM SKUs — are distinct real products and were left untouched, per
the #327 guidance.
Both gates pass:
app.validate✅ andintegrity_check.py --strict✅(structural section clean). Dump refreshed via
python -m app.dump, scoped tov1/gpus/**+ manifest.Seungpyo1007 commentedon Sep 29, 2026
Followed up the pda dedup pass (#335) with a placeholder-value scan across all
112 post-dedup pda records — looking for the fabricated-default fingerprint (one
fake-precise value repeated across many unrelated devices) independent of whether a
record had a duplicate twin. Opened #344.
Clusters found:
ram_gb = 1on 83 of 112 records — fabricated (no PDA 2000-2011 had 1 GB;Dell Axim X5 = 32/64 MB, Palm IIIc = 8 MB, verified against Wikipedia/PCMag/
TidBITS). The Kaggle/phonedb bulk-import placeholder.
battery_mah = 1500on 33 (mostly the classic Palm line) — fabricated butunfixable, see below.
ppi = 114(26), the3.5"/240x320combo (26), andweight_g = 190(9) arereal shared values, not fabricated: 240x320 QVGA on a 3.5" panel genuinely is
~114 ppi, and 190 g is real chassis-sharing (iPAQ h63xx family + i-mate HTC
rebrands). Left untouched.
Fixed: 18
ram_gb=1→ real sourced values (ram_gb is a required field, sorepaired to a cited value, not nulled; no value invented):
other model incl. the whole X50/X51 line = 64 MB).
N560 spec).
Left documented rather than guessed or nulled (this is the important part for
the tracker):
16 MB, IIIc/IIIxe/VIIx = 8 MB, Tungsten/Zire/Treo/Centro …) has real RAM below
the schema's
ram_gb >= 0.016(16 MB) floor, so it is not representable —almost certainly why these got the
ram=1placeholder on import. Can't set thereal value (fails validate) and can't null a required field. This is a schema
limitation worth noting: sub-16 MB RAM devices can't be recorded accurately at
all. ~50 records are in this bucket.
ram_gb=1(i-mate PDA phones, HP iPAQ Glisten, Mitac Mio 728, laterPalm Treo/webOS) need per-device sourcing; left documented.
battery_mah=1500: classic Palms ran on AAA/AA cells (no mAh rating) or smallinternal Li-ions — no clean per-device mAh to cite without inventing, and the
field is required (can't null).
Both gates pass (
app.validate;integrity_check.py --strict— 0 hard anomalies).Dump refreshed in the same PR (only the 18 repaired detail pages change).
Refs #296— removes a sourced slice of the pda placeholders; theunrepresentable/unsourceable remainder stays open and documented above.
fix(data): repair 18 fabricated placeholder RAM values in PDA records
Seungpyo1007 commentedon Sep 29, 2026
Batch progress on the uncited-website backlog — opened #345 against
develop.What landed: 7,086
data/website/records upgraded from Wikidata-only(
verified: false) to source-verified, by adding an independent, checkablecitation — the English Wikipedia article linked from each record's Wikidata item
(
enwikisitelink) — and settingverified: true. No values were fabricated oraltered.
Method: for each
verified: false, Wikidata-only record, resolve its QID andquery the Wikidata API. If the item has an English Wikipedia article, that URL is
a real human-checkable source for the same entity, so it's appended to
source_urlsand the record is verified. Records with no enwiki article are leftuntouched — no citation invented.
Counts (this batch):
verified: true)launch_datevs WikidataP571year mismatch, flagged for manual review0 corrections on purpose: 17 records had a
launch_datedisagreeing with theWikidata inception year (
P571). Since a site's launch date and its subject'sinception are different facts (e.g. Netflix 1997 the company vs. 2007 the service),
overwriting one with the other would inject wrong data, so they were left as-is and
flagged for manual review.
Verification:
app.validatepasses; TechEngineintegrity_check.py --strictexits 0 with no hard anomalies; static dump refreshed for the affected website
records (scoped to
data/website/+ itssite/public/v1/websitesdump).The remaining uncited records (those without an English Wikipedia article) are
left for future batches.
fix(data): add source citations for 7086 website records
chore(dump): refresh static API for the 7086 newly source-verified we…
Seungpyo1007 commentedon Sep 29, 2026
Opened #346 against
develop— a first source-citation pass overdata/tablet/anddata/watch/(companion to the laptop/monitor pass in#337).
Scope (checked live): tablet 3305 total / 3020 unverified; watch 418
total / 325 unverified. As with laptops/monitors, every unverified record
already carried a bulk-dataset link (Kaggle, aitoolbuzz, GSMArena-scrape,
GitHub), so the gap was uncited, not unsourced.
This PR (13 records touched):
S11 Ultra / S10 Ultra / S10+, Galaxy Fold, Galaxy Z Fold 6, Kindle Fire;
Pixel Watch 4 41mm / 45mm, Galaxy Watch Ultra.
os"Windows Mobile" ->"Windows RT 8.1"; Galaxy Watch 7 storage_options [2,32] -> [32]; Galaxy
Watch 6 storage_options [2,16] -> [16] (spurious 2 GB entries removed).
300 watch) — untouched, keep existing provenance.
Each verified record was confirmed field-by-field (RAM, battery, weight,
OS, SoC, storage) against a Wikipedia infobox or manufacturer/GSMArena spec
page.
python -m app.validatepasses andintegrity_check.py --strictreportsno hard anomalies. The static dump was regenerated (
python -m app.dumpfrom TechEngine against the worktree, fresh scratch DB) as the PR's final
commit.
Large remaining uncited set is a strong follow-up target: ~60 tablet/watch
records already cite GSMArena/phonedb per-model pages and just need
field-by-field confirmation once those sources are fetchable (GSMArena was
rate-limiting during this pass).
fix(data): add source citations for 13 tablet/watch records
chore(site): refresh public dump for tablet/watch citations
fix(data): merge 153 confirmed website near-duplicate records
chore(dump): refresh static dump after website dedup merge
Seungpyo1007 commentedon Sep 29, 2026
PDA placeholder ram_gb/os fix, 2026-09-29
Followed up on the PDA dedup's systemic finding (uniform
ram_gb=1andos="Feature phone"across the category). PR #347 corrects 83 of the 112data/pda/records, each value sourced from that device's cited datasheet."Feature phone"→ real platform:"Palm OS"(Palm III/V/VII/m-series/Tungsten/Zire/TX/Z22/Treo≤755p/Centro),"webOS"(Pre/Pixi families),"Windows Mobile"(Dell Axim X5, MiTAC Mio 728). Already-correct Windows Mobile records were left as-is.1→ true capacity from phonedb.net's "RAM Capacity" field, converted MiB→GB in the existing convention (64→0.0625, 128→0.125, 256→0.25, 512→0.5, 32→0.032, 16→0.016).variant.memory.ram_gbmirrored.Spot-verified several directly (Palm IIIc 8 MiB/Palm OS 3.5, Palm TX 32 MiB/Palm OS 5.4, Palm Pre 2 512 MiB/webOS). No values invented.
Left unresolved (documented, not guessed): 12 pre-2003 Palm classics have 2–8 MiB RAM (m100 & Zire = 2 MiB; IIIc/IIIxe/VIIx/m105/m125/m130/m500/m505/i705/Zire 21 = 8 MiB), which is below the
ram_gbschema floor (0.016 GB = 16 MiB inapp/validate.py). Theiroswas corrected to Palm OS, butram_gbwas left rather than writing an overstated in-range value. Raising the validate floor is an engine-side change; noted for a follow-up.Verified:
python -m app.validatepassed;integrity_check.py --strictreported no hard anomalies. Dump refreshed as the PR's last commit (83 changed PDA pages only).Seungpyo1007 commentedon Sep 29, 2026
Follow-up self-consistency pass over
data/monitor/(fresh scan of all 882 records).Found and fixed: 18 records with a null
aspect_ratiothat is unambiguously derivable from the record's ownresolutionfield (and consistent with its ownname). Filled using the ratio labels this catalog already uses — 16:9 (1920x1080 / 2560x1440 / 3840x2160), 21:9 (3440x1440 / 3840x1600), 32:9 (3840x1080 / 5120x1440), 16:10 (1280x800). PR: #348 (againstdevelop).Deliberately left alone:
24" Class= 23.8"), which are legitimate and not errors.aspect_ratiorecords where the record's ownresolutionandnamecontradict each other (e.g. 49" "Dual QHD"/DQHD listed as 3440x1440, "27" 1440p" listed as 5120x1440, names asserting 21:9 on a 1920x1080 field). No single value is safely derivable from the record alone, so they were left and noted rather than guessed.Verification:
python -m app.validatepasses;integrity_check.py --strictreports no hard anomalies. Mandatory dump refresh included as the PR's last commit — only the 18 affected monitor detail files changed.Seungpyo1007 commentedon Sep 29, 2026
Focused follow-up pass for the "these are actually different devices, not
duplicates" pattern (distinct from the "needs a source" residual, which I've left
alone). Opened #349.
Found 7 brand-prefix pairs that are genuinely different products sharing a base
name — the 3 reported earlier plus 4 new ones — and un-paired each by giving the
mis-slugged variant a distinct slug/filename/name (no merges, nothing invented):
u11→u11-plus) vs U11 — distinct fall-2017 model (androidcentral).one-m9→one-m9-plus) vs One M9 — Helio X10 vs SD810.p40-pro→p40-pro-plus) vs P40 Pro (pocket-lint/CNET).xperia-z3→xperia-z3-plus) vs Xperia Z3 — the Z3+/Z4 is adistinct 2015 SD810 model with its own Wikipedia page.
razr→razr-2023) vs Razr 2019.fairphone-3-plus-dual-sim-lte-fp3plus) vs Fairphone 3.gsmart-gs202-plus) vs GS202.In each pair the base-model record is left untouched; only the variant's slug/name/
filename change so the two stop colliding as a false pair. Residual brand-prefix
pairs drop 183 → 176.
Counts: confirmed-different-devices-not-duplicates un-paired 7; merged 0
(not this pass); still-needs-source residual (21 same-model wrong-soc + 157
ram/weight/battery) intentionally untouched — those need external sources.
Both gates pass on #349 (
app.validate,integrity_check.py --strict, slug==filename holds); dump refreshed in the same PR. I believe this exhausts the readily
identifiable "genuinely different device" pattern in the smartphone brand-prefix
set — the remainder is the source-needed residual.
Refs #296.fix(data): resolve 8 more ambiguous Intel CPU slugs
chore(dump): refresh static JSON dump after Intel CPU slug fixes
fix(data): resolve 45 generic SoC placeholder references (Google Pixel)
fix(data): resolve 8 more ambiguous Intel CPU slugs
chore(dump): refresh static JSON dump after Intel CPU slug fixes
fix(data): resolve 45 generic SoC placeholder references (Google Pixel)
fix(data): un-pair 7 genuinely different smartphone devices sharing a…
fix(data): self-consistency corrections for 18 monitor records
chore(dump): refresh static API for the 18 monitor aspect_ratio fills
fix(data): self-consistency corrections for 18 monitor records
chore(dump): refresh static API for the 18 monitor aspect_ratio fills
fix(validate): mirror engine rules for model ids, optional battery/we…
data(smartphone,tablet,watch): add Google Play model numbers and code…
data(smartphone,tablet,watch): add Google Play model numbers and code…
data(smartphone,tablet): add Google Play model numbers and codenames …
data(smartphone,tablet,watch): add Google Play model numbers and code…
data(smartphone,tablet,watch): add Google Play model numbers and code…
data(smartphone,tablet,watch): add Google Play model numbers and code…
data(smartphone,tablet,watch): add Google Play model numbers and code…
data(smartphone,tablet): add Google Play model numbers and codenames …
data(smartphone,tablet,watch): add Google Play model numbers and code…
data(smartphone,tablet): add Google Play model numbers and codenames …
Seungpyo1007 commentedon Oct 6, 2026
Google Play identifiers merged (#359): 31,903 smartphone/tablet/watch records now carry
model_numbers/codenamesfrom Google Playsupported_devices.csv. Ids only, no spec value changes. ~21.8k lower-confidence rows are held for review.data(smartphone,tablet): add Google Play model numbers and codenames …
Seungpyo1007 commentedon Oct 6, 2026
Google Play ids round 2 merged (#361): 174 more records, written per path with SoC and sibling-id checks (blind audit 38/40; the 2 misses are fixed by the ≥4-char codename rule). The new-device hold pools failed two blind fresh-sample audits (81–84%) and stay in review.
fix(data): drop the Galaxy Tab S11 Ultra photo with a false CC0 tag
fix(validate): allow only Wikimedia Commons image_url and logo_url
fix(data): null gsmarena image_url on 2,528 phones, tablets and watch…
fix(data): null mobiledokan image_url on 2,985 smartphones, xiaomi/sa…
fix(data): null aitoolbuzz image_url on 1,166 tablets and watches, sa…
fix(data): null aitoolbuzz image_url on 2,693 smartphones, brands ace…
fix(data): null aitoolbuzz image_url on 2,688 smartphones, brands i-m…
fix(data): null aitoolbuzz image_url on 2,702 smartphones, brands one…
fix(data): null aitoolbuzz image_url on 2,622 smartphones, brands son…
fix(data): null three non-Commons brand logo_url hotlinks
docs(data): note that image and logo URLs keep their Commons licenses
fix(data): stop tracking the Wikipedia HTML verify cache
fix(data): drop the Galaxy Tab S11 Ultra photo with a false CC0 tag
fix(validate): allow only Wikimedia Commons image_url and logo_url
fix(data): resolve 8 GPU cross-source near-duplicates and spec conflicts
fix(data): merge 153 confirmed website near-duplicate records
fix(data): self-consistency corrections for 261 tablet/watch records
fix(data): resolve 83 PDA placeholder ram_gb/os values
fix(data): resolve 8 GPU cross-source near-duplicates and spec conflicts
fix(data): resolve 83 PDA placeholder ram_gb/os values
fix(data): merge 153 confirmed website near-duplicate records
fix(data): self-consistency corrections for 261 tablet/watch records
fix(validate): allow only Wikimedia Commons image_url and logo_url
fix(validate): allow only Wikimedia Commons image_url and logo_url
fix(data/cpu): correct the Xeon X5698's boost clock and release date