Skip to content

[LLM] Implement build-time seed.db preprocessor pipeline - #7

Merged
menny merged 1 commit into
sentence-scorerfrom
pipeline-1-preprocessor
Oct 16, 2026
Merged

menny merged 1 commit into
sentence-scorerfrom
pipeline-1-preprocessor

Conversation

@menny

@menny menny commented Oct 16, 2026

Copy link
Copy Markdown
Member

Summary

Implements Pipeline 1 (Build-Time seed.db Generation) as specified in docs/INGESTION.md:

  • Step 1: Kaikki Dictionary Streaming: Streams Kaikki Spanish JSONL line-by-line, resolves SUBTLEX-ESP frequency rankings, detects cognates using CognateDetector, and expands base and inflected forms into dictionary.
  • Step 3: False Cognates Expansion: Expands trap seeds across all conjugated and plural forms in the dictionary into false_cognates.
  • Step 4: Tatoeba Processing: Reads bilingual sentence pairs, scores them with SentenceScorer, and packages valid sentences into curated_lessons.
  • Cognate Rules Seeding: Populates cognate_rules from JSON definitions.
  • SQLite Database Management: Transaction-safe batched persistence with schema initialization and indexes.
  • Preprocessor CLI & Tests: //tools/preprocessor binary entry point and end-to-end integration test suite.

Stacked on top of #6 (sentence-scorer).

- Add SeedDatabase managing SQLite schema creation, batched transactions, and queries for dictionary, false_cognates, curated_lessons, and cognate_rules.
- Add SublexReader for parsing SUBTLEX-ESP word frequency rankings.
- Add KaikkiProcessor for streaming Kaikki JSONL dumps with cognate tagging and inflection expansion.
- Add FalseCognatesExpander for mapping false cognate seeds across inflected forms.
- Add TatoebaProcessor for sentence readability scoring and lesson packaging.
- Add CognateRulesProcessor for cognate transformer rules seeding.
- Add PreprocessorPipeline and Main CLI binary.
- Add comprehensive unit and integration tests in PreprocessorTest.
@menny
menny added this pull request to stack #8 October 16, 2026 01:55
@menny
menny merged commit abb6a51 into main Oct 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant