Skip to content

chore(data): weekly refresh 2026-10-19 - #370

Open
Seungpyo1007 wants to merge 1 commit into
developfrom
refresh/2026-10-19
Open

Seungpyo1007 wants to merge 1 commit into
developfrom
refresh/2026-10-19

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Weekly data refresh — 2026-10-19

Automated live re-scrape + full-dataset integrity gate + static dump.

Validation

  • app.validate (schema/range/slug/FK): passed
  • integrity_check.py --strict (cross-source anomaly gate): passed

Verification (Tier 0 existence/trust)

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/runner/work/TechEngine/TechEngine/app/verify/main.py", line 8, in
sys.exit(main())
^^^^^^
File "/home/runner/work/TechEngine/TechEngine/app/verify/cli.py", line 850, in main
result: int = args.func(args)
^^^^^^^^^^^^^^^
File "/home/runner/work/TechEngine/TechEngine/app/verify/cli.py", line 82, in cmd_score
records = load_all(only=changed)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/TechEngine/TechEngine/app/verify/common.py", line 92, in load_all
cat: load_category(cat, None if cat in FK_CATEGORIES else only)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/TechEngine/TechEngine/app/verify/common.py", line 80, in load_category
return [Record(category, path, data) for path, data in _load(category, only)]
^^^^^^^^^^^^^^^^^^^^^
TypeError: _load() takes 1 positional argument but 2 were given

Enrichment summaries

Closes #295

@github-actions github-actions Bot added data Dataset changes enhancement New feature or request labels Oct 19, 2026
@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Oct 19, 2026
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate and authoritative for merge.

13/13 categories ? 213,649 records ? diff base develop@db1dea0

Changed records in this PR

4 record(s) assessed.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 4
    "Yellow" : 0
    "Red" : 0
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
cpu 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 4 4 0 0 100.0%
All 🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩 4 4 0 0 100.0%

Full-dataset baseline

As of 2026-10-06T12:12:50Z (daily verify-status aggregate).

category total green yellow red
brand 3022 122 2900 0
soc 2117 284 837 996
smartphone 92824 18216 74555 53
tablet 3305 293 3011 1
watch 418 96 320 2
pda 112 80 32 0
gpu 2083 1536 547 0
cpu 4628 4340 287 1
laptop 1966 126 1840 0
monitor 231 3 228 0
software 42493 366 42127 0
website 40084 13420 26664 0
device_catalog 20366 0 20366 0
all 213649 38882 173714 1053

green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

TechEngine change review: PASS

13/13 categories ? 213,649 records ? diff base develop@db1dea0

Check Result
python -m app.validate PASS
Changed dump JSON and manifest/index counts PASS
New hard integrity anomalies vs PR base PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 0 0 0 0 0
smartphone 0 0 0 0 0 0
tablet 0 0 0 0 0 0
watch 0 0 0 0 0 0
pda 0 0 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 4 0 0 0 0
laptop 0 0 0 0 0 0
monitor 0 0 0 0 0 0
software 0 0 0 0 0 0
website 0 0 0 0 0 0
device_catalog 0 0 0 0 0 0

Changed record examples

cpu modified

  • cpu/intel/2010/enterprise/xeon-e6510.json - Intel Xeon E6510
  • cpu/intel/2010/enterprise/xeon-e6540.json - Intel Xeon E6540
  • cpu/intel/2010/enterprise/xeon-e7520.json - Intel Xeon E7520
  • cpu/intel/2010/enterprise/xeon-l7555.json - Intel Xeon L7555

Heuristic review

  • Heuristic warnings: none found.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

TechEngine validation stats: PASS

13/13 categories ? 213,649 records ? diff base develop@db1dea0

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 3022 123 2899 0 3022 4.1%
soc 2117 284 1833 0 2117 13.4%
smartphone 92824 17235 75589 0 92824 18.6%
tablet 3305 293 3012 0 3305 8.9%
watch 418 98 320 0 418 23.4%
pda 112 68 44 0 112 60.7%
gpu 2083 1496 587 0 2083 71.8%
cpu 4628 4222 406 0 4628 91.2%
laptop 1966 129 1837 0 1966 6.6%
monitor 231 21 210 0 231 9.1%
software 42493 358 42135 0 42493 0.8%
website 40084 14721 25363 0 40084 36.7%
device_catalog 20366 0 20366 0 20366 0.0%
all 213649 39048 174601 0 213649 18.3%

Warning

Tracked verified coverage is below 50% for device_catalog 0.0% (0/20366), software 0.8% (358/42493), brand 4.1% (123/3022), laptop 6.6% (129/1966), tablet 8.9% (293/3305), monitor 9.1% (21/231), soc 13.4% (284/2117), all 18.3% (39048/213649), and 3 more.
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • The integrity gate blocks only hard anomalies introduced by this PR relative to its merge base.
  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py (PR head compared with PR base, changed records only)
loaded CPU=4 GPU=0
Integrity section Flagged lines
structural 0
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 0
CPU cross-source ratio outliers (possible wrong-variant) 0
GPU cross-source ratio outliers + sanity 0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Dataset changes enhancement New feature or request

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

Data collection: new records (workstream)

2 participants