Skip to content

Add support for extracting i18n strings from .astro files#9

Open
taoeffect wants to merge 17 commits into
masterfrom
issue-6
Open

Add support for extracting i18n strings from .astro files#9
taoeffect wants to merge 17 commits into
masterfrom
issue-6

Conversation

@taoeffect

@taoeffect taoeffect commented Jun 12, 2026

Copy link
Copy Markdown
Member

This adds .astro file support to the extractor, so the strings used on groupincome.org (built with Astro) can be managed alongside the app's translations. Closes #6.

What it does:

  • Extracts translation keys from <I18n>...</I18n> components (capitalized, as Astro requires — lowercase <i18n> works too).
  • Extracts L('...') calls from frontmatter, {...} expressions, attribute expressions, and <script> blocks — always parsed as TypeScript, since that's what Astro uses.
  • Keeps the existing {placeholder} syntax instead of switching to pipes (|name|), by relying on Astro's is:raw directive. This means .vue and .astro files produce identical keys, so the app and website share one set of translations. If an <I18n> contains placeholders but is missing is:raw, the extractor prints a friendly warning instead of letting Astro fail confusingly at build time.
  • Adds a --debug-astro flag (alias --da) for inspecting how a .astro file is parsed, matching the existing --debug-pug/--debug-html flags.
  • Includes unit and integration test coverage, plus README/ARCHITECTURE docs.

No changes to existing behavior — .astro files are picked up automatically, no new flags needed.

AI Disclosure

Assisted with Fable 5.

taoeffect added 15 commits June 11, 2026 10:50
Astro {...} expressions were parsed as plain TypeScript, so idiomatic
conditional/mapped JSX rendering failed to parse and any L() calls or
<I18n> blocks inside were silently dropped (no error, no warning).

- Parse all .astro possible-scripts as TSX via a new extractFromTSX
  entry point (ScriptKind.TSX); the plain-TS path is kept for Vue,
  where <T>expr casts must keep parsing as casts.
- Thread a Typescript_tsx kind through Quickjs.extract and a TSX
  template_script through Vue.collect_from_possible_scripts.
- Re-scan expression segments containing <I18n>/<i18n> in Astro.collect
  for nested I18n blocks (recursive, including args={...} and the
  is:raw lint), since the TS extractor only finds L() literals.
- Cover with inline tests, an extended demo.astro fixture, and new
  integration greps; document the TSX behavior in README,
  ARCHITECTURE, and AGENTS.

Addresses issue 1 of REVIEW-1 for issue #6.
A '/' starting a regex literal previously fell through the comment
probe, so an unbalanced brace inside the regex (e.g.
{x.replace(/}/g, '')}) terminated the expression early and silently
dropped the rest. The scanner now consumes regex literals (with escape
and character-class handling) when the preceding significant character
makes division impossible; ambiguous cases keep the old behavior, and
'<'/'>' are excluded so JSX closers are not misread as regex starts.
script_block and style_block discarded the expressions captured by
tag_rest, so L() calls in directives like
<script define:vars={{ message: L('Hello') }}> were silently dropped.
Propagate them as Expression segments, matching generic_tag.
The minimal component snippet called L() without importing it, making
the sample not copy-pasteable.
L is reserved for string literals (that is what the extractor scans
for); the runtime lookup inside the component must use a different
function name, since it receives the rendered slot text as a variable.
The README only addressed translators and never explained how L() and
the i18n/I18n tags work: extraction-time markers for string literals
and runtime lookup functions backed by the generated .strings/.json
mappings. Add that explanation plus a Usage Examples section covering
JS/TS, Vue (HTML and Pug templates), and Astro, with extractor output
verified against the built binary. Examples import from the upcoming
@okturtles/strings npm package; the broken I18n.astro component sample
(which called L with a variable) is removed.
Run tests on pushes and PRs to master (the default branch) instead of
the old lwt/test-suite branches, restrict GITHUB_TOKEN to read-only
contents, and pin actions to commit SHAs to guard against tag hijacking.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class .astro support to the string extraction pipeline so Astro-based sites can share the same translation keys and .strings/.json outputs as the existing app/Vue tooling.

Changes:

  • Introduces a native Astro scanner (Parsing.Astro) that extracts <I18n>/<i18n> slot text and collects frontmatter / {...} / attribute / <script> code as “possible scripts”.
  • Extends the QuickJS/TypeScript extractor with a TSX mode to support JSX inside Astro expressions, and wires .astro handling into the CLI (including a --debug-astro flag) plus warnings output.
  • Adds unit + integration fixtures/tests and updates README/ARCHITECTURE/AGENTS docs and CI workflow configuration.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/test_runner.ml Adds Astro unit tests and refactors some existing assertions for formatting/consistency.
tests/fixtures/demo.astro New integration fixture covering frontmatter, expressions/JSX, nested <I18n>, args, and script blocks.
tests/dune Extends integration test to assert .astro strings are extracted and missing-translation markers are produced.
src/utils/collector.ml Adds a warnings queue and rendering/blitting support for non-fatal diagnostics.
src/quickjs/quickjs.mli Adds Typescript_tsx kind to support TSX parsing.
src/quickjs/quickjs.ml Wires Typescript_tsx through to a new TSX extraction entrypoint.
src/quickjs/parsers.js Implements extractFromTSX (TypeScript parser in TSX mode) and refactors common L() traversal.
src/parsing/dune Enables inline tests for the parsing library (needed for astro.ml inline tests).
src/parsing/astro.mli New Astro parser interface.
src/parsing/astro.ml New Astro scanner/parser + collector logic + extensive inline test coverage.
src/cli/vue.ml Adds Astro to template language handling and adds TSX as a template-script option.
src/cli/strings.ml Auto-detects .astro, runs Astro parsing + TSX script extraction, prints warnings, and adds --debug-astro.
README.md Updates docs with expanded usage examples and Astro-specific guidance.
ARCHITECTURE.md Documents Astro parsing/collection flow and collector warnings.
AGENTS.md Updates repo overview, commands, and flags to include Astro/TSX and warning behavior.
.github/workflows/test.yml Updates workflow triggers/permissions and pins action versions; CI still builds dependencies and runs dune runtest.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread README.md Outdated
Comment thread README.md Outdated
Skip the QuickJS download/compile and the Flow repository clone on
cache hits to speed up test runs. Keys are pinned to the dependency
versions so caches never go stale.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support for .astro file

2 participants