From 6b947868cd5eb57ce6c1ee600a2a2b614ba05b58 Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 10:50:59 -0700 Subject: [PATCH 01/17] Update AGENTS.md via Fable 5 --- AGENTS.md | 71 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 46 insertions(+), 25 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5228f22..7a22eb6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,69 +10,90 @@ This repository contains an OCaml-based internationalization (i18n) string extra - Looking for specific functionality or function definitions before searching. - **[DEVELOPMENT.md](DEVELOPMENT.md)**: Contains instructions for environment setup, build processes for various platforms, and release workflows. **Read this file first** when: - - Setting up the development environment or installing dependencies (OCaml, JS, QuickJS). + - Setting up the development environment or installing dependencies (OCaml, JS, QuickJS, Flow). - Building the project for development or release. - Executing the tool for manual verification or testing. - Managing version numbers or release artifacts. ## Project Overview -- **Language**: OCaml (5.1.1) with some C++ (QuickJS bridge) and JavaScript (parsers via Browserify). +- **Language**: OCaml (5.1.1 in CI) with some C++ (QuickJS bridge) and JavaScript (parsers via Browserify). - **Architecture**: - - `src/cli/`: Main entry point, command-line interface, and output generation logic. + - `src/cli/`: Main entry point (`strings.ml`), command-line interface, output generation (`.strings`/`.json`), and Vue file splitting (`vue.ml`). - `src/parsing/`: OCaml parsers using `Angstrom` for custom formats and `Flow_parser` for JS. - `src/quickjs/`: Bridge to QuickJS to run JavaScript-based parsers (TypeScript/Pug) from OCaml. - `src/utils/`: Common utilities for collection, timing, and I/O. -- **Key Libraries**: `Core`, `Lwt` (concurrency), `Angstrom` (parsing), `Yojson`, `Ppx_jane`. +- **Key Libraries**: `Core`, `Lwt` (concurrency), `Angstrom`, `Yojson`, `Ppx_jane`. +- **Active branch context**: This codebase is the **Lwt** variant (an Eio port exists on other branches). CI runs on branches `lwt` and `test-suite`. Concurrency code uses `Lwt.Syntax`/`Lwt_io`, and `Strings.parse` returns `string Core.String.Table.t Lwt.t`. ## Essential Commands ### Build - **Development build**: `dune build src/cli/strings.exe` - **Watch mode**: `dune build src/cli/strings.exe -w` -- **Release build (MacOS)**: `DUNE_PROFILE=release dune build src/cli/strings.exe` -- **Full release cycle**: See `DEVELOPMENT.md` for `cp`, `strip`, and Docker commands. +- **Release build**: `DUNE_PROFILE=release dune build src/cli/strings.exe` +- **Full release cycle** (strip, Docker/Linux): see `DEVELOPMENT.md`. +- If `dune` is not on PATH, run `eval $(opam env)` first (or prefix with `opam exec --`). + +### Test +```sh +eval $(opam env) +dune runtest tests/ +``` +This runs both the inline unit tests (`tests/test_runner.ml`) and an integration test defined as a `runtest` rule in `tests/dune`, which builds the CLI, runs it against `tests/fixtures/` in a temp directory, and verifies that existing French translations are preserved and `MISSING TRANSLATION` markers are emitted. ### Run - After building: `./_build/default/src/cli/strings.exe [directory-to-extract-from]` -- The CLI expects to be run from the root of a project containing a `strings/` directory (or it will create one if a `.git` folder is present). - -### Installation (Dev Setup) -Refer to `DEVELOPMENT.md` for specific `opam` and `npm` setup steps, as the project has several external dependencies (Flow, QuickJS, pug-lexer, etc.). +- The CLI **fails with "This program must be run from the root of your project"** unless the working directory contains either a `strings/` directory or a `.git` directory. +- Output directory defaults to `strings/`; override with `--output DIR` (`-o`). +- All long flags require the full `--` form (`~full_flag_required` is set everywhere). + +### CLI Flags (actual, from `src/cli/strings.ml`) +- `--output DIR` / `-o`: change output directory (default `strings`). +- `--ts`: treat scripts in HTML/Pug element attributes as TypeScript. +- `--slow-pug` / `--sp`: use the official Pug parser via QuickJS instead of the fast native OCaml one. +- `--debug-pug` / `--dp` and `--debug-html` / `--dh`: debug template parsing in `.vue` files (mutually exclusive). +- There is **no** `--show-debugging` flag. + +## Setup Gotchas (things that break builds) + +- **Flow symlinks**: `src/flow_parser`, `src/sedlex`, and `src/collections` are symlinks into a cloned `flow` repo (v0.183.1) at the project root. If they're missing or dangling, builds fail with module errors. Recreate per `DEVELOPMENT.md`. +- **QuickJS dependency**: Requires a compiled `quickjs` directory (quickjs-2021-03-27, `make` run) at the project root. `dune` rules in `src/quickjs/dune` copy `quickjs.h`, `libquickjs.a`, and invoke `quickjs/qjsc` from there. +- **Generated runtime**: `src/quickjs/runtime.h` is generated at build time from `src/quickjs/parsers.js` via `npx browserify` then `qjsc`. Requires `npm install --no-save typescript browserify pug-lexer pug-parser pug-walk` at the repo root. +- **libomp**: `src/quickjs/dune` searches a hardcoded list of paths for `libomp.a`/`libgomp.a` (Homebrew Cellar paths on macOS, `/usr/lib/...` on Linux). If your system has it elsewhere, the build fails with "Could not find libomp.a" — add your path to the list in `src/quickjs/dune`. +- **Link flags**: Platform/profile-specific link flags live in `src/cli/link_flags.{system}.{dev,release}.dune` (the Linux dev one is just `()`). A missing file for your platform/profile combination breaks the build. +- **Version number**: `let version = "x.y.z"` in `src/cli/strings.ml` must be bumped manually for releases. ## Code Conventions & Patterns ### Parsing Strategy 1. **Direct Parsers**: Simple formats like `.strings`, `HTML`, and basic `Vue` tags are parsed using `Angstrom` in `src/parsing/`. -2. **JS/TS Parsing**: - - Javascript uses `Flow_parser` and a custom AST walker in `src/parsing/js_ast.ml`. +2. **JS/TS Parsing**: + - JavaScript uses `Flow_parser` and a custom AST walker in `src/parsing/js_ast.ml`. - TypeScript uses the official TS parser running inside QuickJS (`src/quickjs/`). -3. **Pug Parsing**: Has a "fast" OCaml implementation (`src/parsing/pug.ml`) and a "slow" official Pug implementation via QuickJS (`src/quickjs/`). +3. **Pug Parsing**: Has a "fast" OCaml implementation (`src/parsing/pug.ml`) and a "slow" official Pug implementation via QuickJS (enabled with `--slow-pug`). ### Extraction Pattern -- Content is extracted into a `Utils.Collector.t`. -- The collector tracks found strings, potential scripts (to be further parsed), and file errors. +- Content is extracted into a `Utils.Collector.t` (`{ path; strings: string Queue.t; ... }`). +- The collector tracks found strings, potential scripts (to be further parsed), and file errors. Use `Collector.blit_transfer` to merge collectors. - **Convention**: Strings found inside `L("...")` calls are treated as translations in JS/TS. ### Concurrency - Uses `Lwt` for cooperative concurrency. - Parallel traversal of directories is handled in `src/cli/strings.ml` via `Lwt_list` and `Lwt_pool`. - JS workers (QuickJS) are managed via `Lwt_pool` and `Lwt_preemptive` in `src/quickjs/quickjs.ml`. +- Angstrom parsing of channels uses `Angstrom_lwt_unix.parse` taking an `Lwt_io.input_channel` (not raw strings or Eio flows). -## Important Gotchas - -- **QuickJS Dependency**: Requires a compiled `quickjs` directory at the project root for building. `dune` rules in `src/quickjs/dune` copy headers and libraries from there. -- **Generated Headers**: `src/quickjs/runtime.h` is generated from `src/quickjs/parsers.js` using `browserify` and `qjsc`. -- **Linking**: MacOS builds use specific link flags (e.g., `ld64.lld`) defined in `src/cli/link_flags.*`. -- **OCamlFormat**: `.ocamlformat` is present; ensure you format OCaml code before submitting. +### Style +- **OCamlFormat**: `.ocamlformat` defines a custom "Asemio Style" (margin 106, `break-cases=all`, `if-then-else=keyword-first`, etc.). Format OCaml code before submitting. - **Memory Safety**: Be cautious with C++ FFI code in `src/quickjs/quickjs.cpp`, particularly regarding OCaml's GC interaction (`CAMLparam`, `CAMLreturn`, `caml_release_runtime_system`). ## Testing Approach -- **Inline Tests**: The project uses `ppx_inline_test`. Parsers in `src/parsing/` can be tested directly within the OCaml files or in the `tests/` directory. -- **Test Suite**: A standard test suite is located in `tests/test_runner.ml`. It covers JS, HTML, Pug, and `.strings` file parsing. -- **Integration Tests**: Verification can be performed by running the built binary against fixtures in `tests/fixtures/` and checking the generated output in the `strings/` directory. -- **Debug Flags**: Use `--show-debugging` or `--debug-pug` / `--debug-html` flags in the CLI to inspect internal parsing results. +- **Inline Tests**: `ppx_inline_test` (`let%test_unit`) with `ppx_assert` (`[%test_eq:]`). Tests live in `tests/test_runner.ml` (a library with `(inline_tests)`); parsers in `src/parsing/` can also be tested inline. +- **Lwt in tests**: Wrap async test bodies in `Lwt_main.run`. For testing `Strings.parse` without the filesystem, build an in-memory channel: `Lwt_io.of_bytes ~mode:Lwt_io.input (Lwt_bytes.of_string content)`. +- **Synchronous parsers**: `Js.extract_to_collector` and the HTML/Pug parsers work on raw strings synchronously — no Lwt needed in those tests. +- **Integration Tests**: `tests/dune` contains a bash-based `runtest` rule using fixtures in `tests/fixtures/` (demo.html, demo.js, demo.pug, demo.vue). The CI workflow (`.github/workflows/test.yml`) requires `mkdir -p strings` before `dune runtest tests/`. ## Troubleshooting From 1efe25fd85bbb8011ad4c94a172b53589c9b56b4 Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 11:42:59 -0700 Subject: [PATCH 02/17] Add non-fatal warnings to Collector for upcoming Astro lint output --- src/cli/strings.ml | 4 ++++ src/cli/vue.ml | 6 +++++- src/utils/collector.ml | 25 +++++++++++++++++++++++-- 3 files changed, 32 insertions(+), 3 deletions(-) diff --git a/src/cli/strings.ml b/src/cli/strings.ml index 62aeee4..2cddb9f 100644 --- a/src/cli/strings.ml +++ b/src/cli/strings.ml @@ -63,6 +63,10 @@ let process_file traversal count filename ~f:get_collector = Utils.Collector.render_errors collector |> Option.value_map ~default:Lwt.return_unit ~f:Lwt_io.eprintl in + let* () = + Utils.Collector.render_warnings collector + |> Option.value_map ~default:Lwt.return_unit ~f:Lwt_io.eprintl + in Queue.iter collector.strings ~f:handler; Vue.collect_from_possible_scripts collector traversal.template_script ~on_string:handler ) diff --git a/src/cli/vue.ml b/src/cli/vue.ml index fe8ac1a..f3575e4 100644 --- a/src/cli/vue.ml +++ b/src/cli/vue.ml @@ -98,7 +98,7 @@ let collect_from_languages collector languages = languages let debug_template ~path languages template_script target = - let print_collector ~error_kind (Utils.Collector.{ strings; file_errors; _ } as collector) = + let print_collector ~error_kind (Utils.Collector.{ strings; file_errors; warnings; _ } as collector) = let* () = collect_from_possible_scripts collector template_script ~on_string:(Queue.enqueue strings) in @@ -106,6 +106,10 @@ let debug_template ~path languages template_script target = let deduped = Queue.fold strings ~init:String.Set.empty ~f:Set.add in let* () = Lwt_io.printlf "Found %d strings:" (Set.length deduped) in Set.iter deduped ~f:(fun s -> bprintf buf !"%{Yojson.Basic}\n" (`String s)); + if not (Queue.is_empty warnings) + then ( + bprintf buf "\n⚠️ %s warnings in %s:\n" error_kind path; + Queue.iter warnings ~f:(bprintf buf "- %s\n") ); if not (Queue.is_empty file_errors) then ( bprintf buf "\n❌ %s errors in %s:\n" error_kind path; diff --git a/src/utils/collector.ml b/src/utils/collector.ml index 313d7d7..fe0ee35 100644 --- a/src/utils/collector.ml +++ b/src/utils/collector.ml @@ -5,11 +5,18 @@ type t = { strings: string Queue.t; possible_scripts: string Queue.t; file_errors: string Queue.t; + warnings: string Queue.t; } [@@deriving sexp] let create ~path = - { path; strings = Queue.create (); possible_scripts = Queue.create (); file_errors = Queue.create () } + { + path; + strings = Queue.create (); + possible_scripts = Queue.create (); + file_errors = Queue.create (); + warnings = Queue.create (); + } let render_errors { file_errors; path; _ } = match Queue.length file_errors with @@ -24,7 +31,21 @@ let render_errors { file_errors; path; _ } = Queue.iter file_errors ~f:(bprintf buf "- %s\n"); Some (Buffer.contents buf) +let render_warnings { warnings; path; _ } = + match Queue.length warnings with + | 0 -> None + | 1 -> + let buf = Buffer.create 256 in + bprintf buf "\n⚠️ 1 warning in %s: %s" path (Queue.get warnings 0); + Some (Buffer.contents buf) + | len -> + let buf = Buffer.create 256 in + bprintf buf "\n⚠️ %d warnings in %s:\n" len path; + Queue.iter warnings ~f:(bprintf buf "- %s\n"); + Some (Buffer.contents buf) + let blit_transfer ~src ~dst = Queue.blit_transfer ~src:src.strings ~dst:dst.strings (); Queue.blit_transfer ~src:src.possible_scripts ~dst:dst.possible_scripts (); - Queue.blit_transfer ~src:src.file_errors ~dst:dst.file_errors () + Queue.blit_transfer ~src:src.file_errors ~dst:dst.file_errors (); + Queue.blit_transfer ~src:src.warnings ~dst:dst.warnings () From 6325744d1457d1dedd352aa664cb618176604481 Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 12:30:26 -0700 Subject: [PATCH 03/17] Add native Angstrom parser for .astro files (issue #6) --- src/parsing/astro.ml | 377 ++++++++++++++++++++++++++++++++++++++++++ src/parsing/astro.mli | 7 + src/parsing/dune | 1 + 3 files changed, 385 insertions(+) create mode 100644 src/parsing/astro.ml create mode 100644 src/parsing/astro.mli diff --git a/src/parsing/astro.ml b/src/parsing/astro.ml new file mode 100644 index 0000000..bb07b40 --- /dev/null +++ b/src/parsing/astro.ml @@ -0,0 +1,377 @@ +open! Core + +type i18n_block = { + tag: string; + is_raw: bool; + args: string option; + text: string option; +} +[@@deriving sexp_of] + +type segment = + | Frontmatter of string + | Script of string + | Expression of string + | I18n of i18n_block +[@@deriving sexp_of] + +type t = segment list [@@deriving sexp_of] + +(* Astro expressions are wrapped in parentheses before being handed to the TypeScript + extractor so that object literals such as [args={{ name: x }}] parse as expressions + instead of block statements *) +let wrap_expression s = sprintf "(%s)" s + +let collect Utils.Collector.{ strings; possible_scripts; warnings; _ } (segments : t) = + let enqueue_expression source = + if not (String.is_empty (String.strip source)) + then Queue.enqueue possible_scripts (wrap_expression source) + in + List.iter segments ~f:(function + | Frontmatter source + |Script source -> + Queue.enqueue possible_scripts source + | Expression source -> enqueue_expression source + | I18n { tag; is_raw; args; text } -> + Option.iter args ~f:enqueue_expression; + Option.iter text ~f:(fun text -> + let key = String.strip text in + if not (String.is_empty key) + then ( + Queue.enqueue strings key; + if (not is_raw) && String.contains key '{' + then + Queue.enqueue warnings + (sprintf + "<%s> contains placeholders but is missing the is:raw directive: %s. Add it like this: \ + <%s is:raw ...> (otherwise Astro evaluates the placeholders as expressions)" + tag + (Yojson.Basic.to_string (`String key)) + tag ) ) ) ) + +let parser () = + let open Angstrom in + let open Basic in + let buf = Buffer.create 256 in + + (* Character-level scanners. They all append the consumed characters to [buf] so that + brace matching ignores braces inside strings, template literals, and comments. *) + let rec braces depth = + any_char >>= fun c -> + match c with + | '}' when depth = 0 -> return () + | '}' -> + Buffer.add_char buf c; + braces (depth - 1) + | '{' -> + Buffer.add_char buf c; + braces (depth + 1) + | ('\'' | '"') as q -> + Buffer.add_char buf c; + in_string q false >>= fun () -> braces depth + | '`' -> + Buffer.add_char buf c; + in_template false >>= fun () -> braces depth + | '/' -> + Buffer.add_char buf c; + maybe_comment () >>= fun () -> braces depth + | c -> + Buffer.add_char buf c; + braces depth + and in_string q escaped = + any_char >>= fun c -> + Buffer.add_char buf c; + match c, escaped with + | '\\', false -> in_string q true + | c, false when Char.(c = q) -> return () + | _ -> in_string q false + and in_template escaped = + any_char >>= fun c -> + Buffer.add_char buf c; + match c, escaped with + | '\\', false -> in_template true + | '`', false -> return () + | '$', false -> ( + peek_char >>= function + | Some '{' -> + advance 1 >>= fun () -> + Buffer.add_char buf '{'; + braces 0 >>= fun () -> + Buffer.add_char buf '}'; + in_template false + | _ -> in_template false ) + | _ -> in_template false + and maybe_comment () = + peek_char >>= function + | Some '/' -> take_remaining >>| Buffer.add_string buf + | Some '*' -> + advance 1 >>= fun () -> + Buffer.add_char buf '*'; + in_block_comment () + | _ -> return () + and in_block_comment () = + any_char >>= fun c -> + Buffer.add_char buf c; + match c with + | '*' -> ( + peek_char >>= function + | Some '/' -> advance 1 >>| fun () -> Buffer.add_char buf '/' + | _ -> in_block_comment () ) + | _ -> in_block_comment () + in + + let braced_value = + char '{' >>= fun _ -> + Buffer.clear buf; + braces 0 >>| fun () -> Buffer.contents buf + in + let template_value = + char '`' >>= fun _ -> + Buffer.clear buf; + Buffer.add_char buf '`'; + in_template false >>| fun () -> Buffer.contents buf + in + let take_past stop = + let rec go () = + stop *> return () + >>| (fun () -> Buffer.contents buf) + <|> ( any_char >>= fun c -> + Buffer.add_char buf c; + go () ) + in + return () >>= fun () -> + Buffer.clear buf; + go () + in + let skip_past stop = + let rec go () = stop *> return () <|> (any_char >>= fun _ -> go ()) in + go () + in + + let tag_boundary = + peek_char >>= function + | Some c when is_identifier c -> fail "Not a tag boundary" + | _ -> return () + in + + (* Leniently consumes the remainder of an open tag up to and including its closing '>'. + Quoted attribute values are literal text in Astro and are skipped; braced and + template-literal values are captured as expressions. *) + let tag_rest = + let rec go exprs = + any_char >>= function + | '>' -> return (List.rev exprs, false) + | '/' -> ( + peek_char >>= function + | Some '>' -> advance 1 *> return (List.rev exprs, true) + | _ -> go exprs ) + | '"' -> take_till (Char.( = ) '"') *> advance 1 *> go exprs + | '\'' -> take_till (Char.( = ) '\'') *> advance 1 *> go exprs + | '`' -> + Buffer.clear buf; + Buffer.add_char buf '`'; + in_template false >>= fun () -> go (Buffer.contents buf :: exprs) + | '{' -> + Buffer.clear buf; + braces 0 >>= fun () -> go (Buffer.contents buf :: exprs) + | _ -> go exprs + in + go [] + in + + let script_block = + string " tag_boundary *> tag_rest >>= fun (_, self_closing) -> + match self_closing with + | true -> return [] + | false -> take_past (string " mlws *> char '>') >>| fun body -> [ Script body ] + in + let style_block = + string " tag_boundary *> tag_rest >>= fun (_, self_closing) -> + match self_closing with + | true -> return [] + | false -> skip_past (string " mlws *> char '>') *> return [] + in + + let attr_name = + take_while1 (fun c -> is_identifier c || Char.(c = ':') || Char.(c = '.') || Char.(c = '@')) + in + let plain_quoted q = char q *> take_till (Char.( = ) q) <* advance 1 in + let attr_value = + peek_char_fail >>= function + | '"' -> plain_quoted '"' >>| fun s -> `Literal s + | '\'' -> plain_quoted '\'' >>| fun s -> `Literal s + | '{' -> braced_value >>| fun s -> `Expr s + | '`' -> template_value >>| fun s -> `Expr s + | _ -> + take_while1 (fun c -> (not (is_mlws c)) && (not Char.(c = '>')) && not Char.(c = '/')) >>| fun s -> + `Literal s + in + let attr = lift2 Tuple2.create attr_name (maybe (mlws *> char '=' *> mlws *> attr_value)) in + + let i18n_block = + char '<' *> (string "I18n" <|> string "i18n") <* tag_boundary >>= fun tag -> + many (mlws1 *> attr) <* mlws >>= fun attrs -> + let is_raw = List.exists attrs ~f:(fun (name, _) -> String.(name = "is:raw")) in + let args = + List.find_map attrs ~f:(function + | "args", Some (`Expr s) -> Some s + | _ -> None ) + in + let extra_exprs = + List.filter_map attrs ~f:(function + | "args", _ -> None + | _, Some (`Expr s) -> Some (Expression s) + | _ -> None ) + in + string "/>" *> return (I18n { tag; is_raw; args; text = None } :: extra_exprs) + <|> ( char '>' *> take_past (string (" mlws *> char '>') >>| fun text -> + I18n { tag; is_raw; args; text = Some text } :: extra_exprs ) + in + + let generic_tag = + char '<' + *> choice + [ + char '/' *> take_till (Char.( = ) '>') *> advance 1 *> return []; + char '!' *> take_till (Char.( = ) '>') *> advance 1 *> return []; + (satisfy alphanum *> tag_rest >>| fun (exprs, _) -> List.map exprs ~f:(fun s -> Expression s)); + ] + in + + let element_chunk = + choice + [ + string "") *> return []; + script_block; + style_block; + i18n_block; + generic_tag; + (braced_value >>| fun s -> [ Expression s ]); + any_char *> return []; + ] + in + + let frontmatter = + let fence = string "---" in + let rec go acc = + fence *> return (String.concat ~sep:"\n" (List.rev acc)) + <|> ( take_remaining >>= fun line -> + end_of_line *> go (line :: acc) + <|> end_of_input *> return (String.concat ~sep:"\n" (List.rev (line :: acc))) ) + in + fence *> ws *> end_of_line *> go [] + in + + lift2 + (fun fm body -> + match fm with + | None -> body + | Some s -> Frontmatter s :: body) + (mlws *> maybe frontmatter) + (many element_chunk >>| List.concat) + <* end_of_input + +(* Tests *) + +let test_collect source = + let collector = Utils.Collector.create ~path:"test.astro" in + Basic.exec_parser + ~on_ok:(fun parsed -> collect collector parsed) + (parser ()) ~path:"test.astro" ~language_name:"Astro" source; + collector + +let%test_unit "astro: i18n slot text" = + let collector = test_collect "Logout" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Logout" ]; + [%test_eq: int] (Queue.length collector.warnings) 0 + +let%test_unit "astro: lowercase i18n tag" = + let collector = test_collect "Logout" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Logout" ] + +let%test_unit "astro: multiline slot text is stripped like the HTML path" = + let collector = test_collect "\n Hello {name}!\n" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Hello {name}!" ]; + [%test_eq: int] (Queue.length collector.warnings) 0 + +let%test_unit "astro: missing is:raw warning" = + let collector = test_collect "Hello {name}!" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Hello {name}!" ]; + [%test_eq: int] (Queue.length collector.warnings) 1 + +let%test_unit "astro: no warning without placeholders" = + let collector = test_collect "Hello!" in + [%test_eq: int] (Queue.length collector.warnings) 0 + +let%test_unit "astro: frontmatter" = + let collector = test_collect "---\nconst title = L('Create a group')\n---\n

ok

" in + [%test_eq: string list] + (Queue.to_list collector.possible_scripts) + [ "const title = L('Create a group')" ] + +let%test_unit "astro: body expression" = + let collector = test_collect "

{L('Welcome')}

" in + [%test_eq: string list] (Queue.to_list collector.possible_scripts) [ "(L('Welcome'))" ] + +let%test_unit "astro: attribute expression" = + let collector = test_collect "
x
" in + [%test_eq: string list] (Queue.to_list collector.possible_scripts) [ "(L('Hi'))" ] + +let%test_unit "astro: args object with spread and arrow function" = + let collector = + test_collect + " x, ...LTags(\"strong\") }}>Delete {name}" + in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Delete {name}" ]; + [%test_eq: string list] + (Queue.to_list collector.possible_scripts) + [ "({ name: groupName, f: () => x, ...LTags(\"strong\") })" ] + +let%test_unit "astro: braces inside template literals and strings" = + let collector = test_collect "{`a ${ { b: 1 }.b } c`}

{'}'}

" in + [%test_eq: string list] + (Queue.to_list collector.possible_scripts) + [ "(`a ${ { b: 1 }.b } c`)"; "('}')" ] + +let%test_unit "astro: script block" = + let collector = test_collect "" in + [%test_eq: string list] (Queue.to_list collector.possible_scripts) [ "\nconsole.log(L('Hey'))\n" ] + +let%test_unit "astro: style block is ignored" = + let collector = test_collect "" in + [%test_eq: string list] (Queue.to_list collector.possible_scripts) [] + +let%test_unit "astro: html comments are ignored" = + let collector = test_collect "Yes" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Yes" ] + +let%test_unit "astro: self-closing I18n" = + let collector = test_collect "" in + [%test_eq: string list] (Queue.to_list collector.strings) []; + [%test_eq: string list] (Queue.to_list collector.possible_scripts) [ "({ a: 1 })" ] + +let%test_unit "astro: full document" = + let source = + {|--- +import I18n from '../components/I18n.astro' +const title = L('Create a group') +--- + +

{L('Welcome to Group Income')}

+ +Logout + + + Yes, I want to {strong_}delete {name} permanently{_strong}. + +|} + in + let collector = test_collect source in + [%test_eq: string list] + (List.sort ~compare:String.compare (Queue.to_list collector.strings)) + [ "Logout"; "Yes, I want to {strong_}delete {name} permanently{_strong}." ]; + [%test_eq: int] (Queue.length collector.warnings) 0; + [%test_eq: bool] + (List.exists (Queue.to_list collector.possible_scripts) ~f:(fun s -> + String.is_substring s ~substring:"L('Create a group')" ) ) + true diff --git a/src/parsing/astro.mli b/src/parsing/astro.mli new file mode 100644 index 0000000..78f33b6 --- /dev/null +++ b/src/parsing/astro.mli @@ -0,0 +1,7 @@ +open! Core + +type t [@@deriving sexp_of] + +val collect : Utils.Collector.t -> t -> unit + +val parser : unit -> t Angstrom.t diff --git a/src/parsing/dune b/src/parsing/dune index 4217b2e..a94ef0a 100644 --- a/src/parsing/dune +++ b/src/parsing/dune @@ -1,5 +1,6 @@ (library (name parsing) + (inline_tests) (libraries core angstrom-lwt-unix From b916b29c3fdbbb19b083fc71c0d791146a2baeec Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 12:45:03 -0700 Subject: [PATCH 04/17] Dispatch .astro files in the CLI, always parsing their scripts as TypeScript --- src/cli/strings.ml | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/src/cli/strings.ml b/src/cli/strings.ml index 2cddb9f..bb2faf1 100644 --- a/src/cli/strings.ml +++ b/src/cli/strings.ml @@ -26,6 +26,7 @@ type counts = { html: int ref; js: int ref; ts: int ref; + astro: int ref; } type common_options = { @@ -47,7 +48,7 @@ type traversal = { counts: counts; } -let process_file traversal count filename ~f:get_collector = +let process_file ?template_script traversal count filename ~f:get_collector = Lwt_pool.use pool (fun () -> incr count; let* (collector : Utils.Collector.t) = @@ -68,7 +69,8 @@ let process_file traversal count filename ~f:get_collector = |> Option.value_map ~default:Lwt.return_unit ~f:Lwt_io.eprintl in Queue.iter collector.strings ~f:handler; - Vue.collect_from_possible_scripts collector traversal.template_script ~on_string:handler ) + let template_script = Option.value template_script ~default:traversal.template_script in + Vue.collect_from_possible_scripts collector template_script ~on_string:handler ) let rec process_dir traversal ~path = function | "node_modules" -> Lwt.return_unit @@ -112,6 +114,15 @@ let rec process_dir traversal ~path = function ~path ~language_name:"Pug" source in collector ) + | { st_kind = S_REG; _ }, _, _ when String.is_suffix filename ~suffix:".astro" -> + (* Astro frontmatter and expressions are TypeScript by definition *) + process_file ~template_script:Vue.TS traversal traversal.counts.astro path ~f:(fun ic -> + let collector = Utils.Collector.create ~path in + let+ source = Lwt_io.read ic in + let on_ok parsed = Parsing.Astro.collect collector parsed in + let on_error ~msg = Queue.enqueue collector.file_errors msg in + Parsing.(Basic.exec_parser ~on_ok ~on_error (Astro.parser ())) ~path ~language_name:"Astro" source; + collector ) | { st_kind = S_REG; _ }, _, _ when String.is_suffix filename ~suffix:".html" -> process_file traversal traversal.counts.html path ~f:(fun ic -> let collector = Utils.Collector.create ~path in @@ -283,7 +294,7 @@ let main options = function (* English *) let* english = let table = String.Table.create () in - let counts = { vue = ref 0; pug = ref 0; html = ref 0; js = ref 0; ts = ref 0 } in + let counts = { vue = ref 0; pug = ref 0; html = ref 0; js = ref 0; ts = ref 0; astro = ref 0 } in let time = Utils.Timing.start () in let* () = Lwt_list.iter_p @@ -306,9 +317,9 @@ let main options = function let f ext i = sprintf "%d %s file%s" i ext (plural i) in let time = Int63.(time `Stop - !Quickjs.init_time) in Lwt_io.printlf - !"✅ [%{Int63}ms] Processed %s, %s, %s, %s, and %s" + !"✅ [%{Int63}ms] Processed %s, %s, %s, %s, %s, and %s" time (f ".js" !(counts.js)) (f ".ts" !(counts.ts)) (f ".html" !(counts.html)) - (f ".vue" !(counts.vue)) (f ".pug" !(counts.pug)) + (f ".vue" !(counts.vue)) (f ".pug" !(counts.pug)) (f ".astro" !(counts.astro)) in let+ () = write_english ~outdir english in english From 27f310ad850a396c5d87ddc435363e4674e36c3b Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 12:54:48 -0700 Subject: [PATCH 05/17] Add --debug-astro flag for inspecting .astro file parsing --- src/cli/strings.ml | 14 +++++++++++++- src/cli/vue.ml | 25 ++++++++++++++++++++----- 2 files changed, 33 insertions(+), 6 deletions(-) diff --git a/src/cli/strings.ml b/src/cli/strings.ml index bb2faf1..fc963ff 100644 --- a/src/cli/strings.ml +++ b/src/cli/strings.ml @@ -273,6 +273,13 @@ let main options = function Vue.debug_template ~path [ Html { parsed; length = None } ] template_script lang in Parsing.Basic.exec_parser_lwt ~on_ok Parsing.Html.parser ~path ~language_name:"Pug" ic + | Astro, _ when String.is_suffix path ~suffix:".astro" -> + let* source = Lwt_io.read ic in + let on_ok parsed = + (* Astro scripts are always TypeScript *) + Vue.debug_template ~path [ Astro { parsed; length = None } ] Vue.TS lang + in + Parsing.(Basic.exec_parser ~on_ok (Astro.parser ())) ~path ~language_name:"Astro" source | _ -> Lwt_io.printlf "Nothing to do for file [%s]" path )) options.targets | Run -> @@ -377,7 +384,12 @@ let () = ~doc:"Debug html templates in .vue files" >>| Fn.flip Option.some_if (Debug Html) in - choose_one [ debug_pug; debug_html ] ~if_nothing_chosen:(Default_to Run) + let debug_astro = + flag "--debug-astro" ~aliases:[ "--da" ] ~full_flag_required:() no_arg + ~doc:"Debug .astro file parsing" + >>| Fn.flip Option.some_if (Debug Astro) + in + choose_one [ debug_pug; debug_html; debug_astro ] ~if_nothing_chosen:(Default_to Run) in let handle_system_failure = function diff --git a/src/cli/vue.ml b/src/cli/vue.ml index f3575e4..c3f8d39 100644 --- a/src/cli/vue.ml +++ b/src/cli/vue.ml @@ -26,6 +26,10 @@ module Language = struct collector: Utils.Collector.t; length: int; } + | Astro of { + parsed: Astro.t; + length: int option; + } | Css of int | Failed of string @@ -61,6 +65,7 @@ module Debug = struct type t = | Pug | Html + | Astro end let collect_from_possible_scripts Utils.Collector.{ possible_scripts; _ } template_script ~on_string = @@ -87,6 +92,9 @@ let collect_from_languages collector languages = | Pug { collector = src; length = _ } -> Utils.Collector.blit_transfer ~src ~dst:collector; Lwt.return_unit + | Astro { parsed; length = _ } -> + Astro.collect collector parsed; + Lwt.return_unit | Js source -> Js.extract_to_collector collector source; Lwt.return_unit @@ -133,11 +141,18 @@ let debug_template ~path languages template_script target = let* () = collect_from_languages collector [ lang ] in print_collector ~error_kind:"Pug" collector | Pug { collector; length = _ }, Pug -> print_collector ~error_kind:"Pug" collector - | Html { length = Some len; _ }, Pug -> Lwt_io.printlf "" len - | Html { length = None; _ }, Pug -> Lwt_io.printl "" - | Pug_native { length = Some len; _ }, Html -> Lwt_io.printlf "" len - | Pug_native { length = None; _ }, Html -> Lwt_io.printl "" - | Pug { length; _ }, Html -> Lwt_io.printlf "" length + | (Astro { parsed; length = _ } as lang), Astro -> + let* () = Lwt_io.printlf !"%{sexp#hum: Astro.t}" parsed in + let collector = Utils.Collector.create ~path in + let* () = collect_from_languages collector [ lang ] in + print_collector ~error_kind:"Astro" collector + | Astro { length = Some len; _ }, (Pug | Html) -> Lwt_io.printlf "" len + | Astro { length = None; _ }, (Pug | Html) -> Lwt_io.printl "" + | Html { length = Some len; _ }, (Pug | Astro) -> Lwt_io.printlf "" len + | Html { length = None; _ }, (Pug | Astro) -> Lwt_io.printl "" + | Pug_native { length = Some len; _ }, (Html | Astro) -> Lwt_io.printlf "" len + | Pug_native { length = None; _ }, (Html | Astro) -> Lwt_io.printl "" + | Pug { length; _ }, (Html | Astro) -> Lwt_io.printlf "" length | Failed msg, _ -> Lwt_io.printlf "❌ Parsing error in path: %s" msg) languages From 11028f44c8fff03fd2f1ae950b1014f476b71adb Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 12:56:37 -0700 Subject: [PATCH 06/17] Add unit and integration test coverage for .astro extraction --- tests/dune | 16 +++++++++ tests/fixtures/demo.astro | 15 +++++++++ tests/test_runner.ml | 68 ++++++++++++++++++++++++++++++++------- 3 files changed, 88 insertions(+), 11 deletions(-) create mode 100644 tests/fixtures/demo.astro diff --git a/tests/dune b/tests/dune index acdb36a..a6757e7 100644 --- a/tests/dune +++ b/tests/dune @@ -34,6 +34,22 @@ echo \"Error: Missing translation marker not found in .strings\" exit 1 fi + if ! grep -q 'Hello from Astro' $TMP_DIR/strings/english.strings; then + echo \"Error: Astro I18n string not found in english.strings\" + exit 1 + fi + if ! grep -q 'Hello from Astro Frontmatter' $TMP_DIR/strings/english.strings; then + echo \"Error: Astro frontmatter string not found in english.strings\" + exit 1 + fi + if ! grep -q 'Hello {strong_}from{_strong} Astro Args' $TMP_DIR/strings/english.strings; then + echo \"Error: Astro placeholder string not found in english.strings\" + exit 1 + fi + if ! grep -q \"MISSING TRANSLATION - demo.astro\" $TMP_DIR/strings/french.strings; then + echo \"Error: Missing translation marker for demo.astro not found in .strings\" + exit 1 + fi echo \"✅ French integration test passed\" rm -rf $TMP_DIR diff --git a/tests/fixtures/demo.astro b/tests/fixtures/demo.astro new file mode 100644 index 0000000..d11a45a --- /dev/null +++ b/tests/fixtures/demo.astro @@ -0,0 +1,15 @@ +--- +const title = L('Hello from Astro Frontmatter') +--- + +

{L('Hello from Astro Expression')}

+ +Hello from Astro + + + Hello {strong_}from{_strong} Astro Args + + + diff --git a/tests/test_runner.ml b/tests/test_runner.ml index f03d8d8..c5d6333 100644 --- a/tests/test_runner.ml +++ b/tests/test_runner.ml @@ -6,14 +6,16 @@ let%test_unit "js_extraction_basic" = let source = "L('Hello World'); L('Foo Bar');" in Js.extract_to_collector collector source; let strings = Queue.to_list collector.strings in - [%test_eq: string list] (List.sort strings ~compare:String.compare) (List.sort ["Hello World"; "Foo Bar"] ~compare:String.compare) + [%test_eq: string list] + (List.sort strings ~compare:String.compare) + (List.sort [ "Hello World"; "Foo Bar" ] ~compare:String.compare) let%test_unit "js_extraction_nested" = let collector = Utils.Collector.create ~path:"test.js" in let source = "function test() { if (true) { return L('Nested'); } }" in Js.extract_to_collector collector source; let strings = Queue.to_list collector.strings in - [%test_eq: string list] strings ["Nested"] + [%test_eq: string list] strings [ "Nested" ] let%test_unit "js_extraction_no_match" = let collector = Utils.Collector.create ~path:"test.js" in @@ -23,7 +25,8 @@ let%test_unit "js_extraction_no_match" = [%test_eq: string list] strings [] let%test_unit "strings_parsing" = - Lwt_main.run @@ ( + Lwt_main.run + @@ let path = "test.strings" in let content = {| /* Comment */ @@ -35,23 +38,26 @@ let%test_unit "strings_parsing" = let+ table = Strings.parse ~path ic in [%test_eq: string option] (Hashtbl.find table "Hello") (Some "Bonjour"); [%test_eq: string option] (Hashtbl.find table "World") (Some "Monde"); - [%test_eq: string option] (Hashtbl.find table "Missing") None) + [%test_eq: string option] (Hashtbl.find table "Missing") None let%test_unit "french_strings_parsing" = - Lwt_main.run @@ ( + Lwt_main.run + @@ let path = "french.strings" in - let content = {| + let content = + {| /* Accented characters */ "Logout" = "Déconnexion"; "You and {count} others" = "Vous et {count} autres"; "Settings" = "Paramètres"; -|} in +|} + in let ic = Lwt_io.of_bytes ~mode:Lwt_io.input @@ Lwt_bytes.of_string content in let open Lwt.Syntax in let+ table = Strings.parse ~path ic in [%test_eq: string option] (Hashtbl.find table "Logout") (Some "Déconnexion"); [%test_eq: string option] (Hashtbl.find table "You and {count} others") (Some "Vous et {count} autres"); - [%test_eq: string option] (Hashtbl.find table "Settings") (Some "Paramètres")) + [%test_eq: string option] (Hashtbl.find table "Settings") (Some "Paramètres") let%test_unit "html_extraction" = let collector = Utils.Collector.create ~path:"test.html" in @@ -59,13 +65,53 @@ let%test_unit "html_extraction" = let on_ok parsed = Parsing.Html.collect collector parsed in Parsing.Basic.exec_parser ~on_ok Parsing.Html.parser ~path:"test.html" ~language_name:"HTML" source; let strings = Queue.to_list collector.strings in - [%test_eq: string list] strings ["Hello HTML"] + [%test_eq: string list] strings [ "Hello HTML" ] let%test_unit "pug_extraction" = let collector = Utils.Collector.create ~path:"test.pug" in let source = "i18n Hello Pug" in let string_parsers = Parsing.Basic.make_string_parsers () in let on_ok parsed = Parsing.Pug.collect collector parsed in - Parsing.Basic.exec_parser ~on_ok (Parsing.Pug.parser string_parsers) ~path:"test.pug" ~language_name:"Pug" source; + Parsing.Basic.exec_parser ~on_ok (Parsing.Pug.parser string_parsers) ~path:"test.pug" + ~language_name:"Pug" source; let strings = Queue.to_list collector.strings in - [%test_eq: string list] strings ["Hello Pug"] + [%test_eq: string list] strings [ "Hello Pug" ] + +let extract_astro source = + let collector = Utils.Collector.create ~path:"test.astro" in + let on_ok parsed = Parsing.Astro.collect collector parsed in + Parsing.Basic.exec_parser ~on_ok (Parsing.Astro.parser ()) ~path:"test.astro" ~language_name:"Astro" + source; + collector + +let%test_unit "astro_i18n_extraction" = + let collector = extract_astro "Hello Astro" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Hello Astro" ]; + [%test_eq: int] (Queue.length collector.warnings) 0 + +let%test_unit "astro_lowercase_i18n_extraction" = + let collector = extract_astro "Hello Astro" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Hello Astro" ] + +let%test_unit "astro_missing_is_raw_warning" = + let collector = extract_astro "Hello {name}!" in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Hello {name}!" ]; + [%test_eq: int] (Queue.length collector.warnings) 1 + +let%test_unit "astro_frontmatter_and_expression_scripts" = + let collector = + extract_astro "---\nconst a = L('From Frontmatter')\n---\n

{L('From Expression')}

" + in + let scripts = Queue.to_list collector.possible_scripts in + [%test_eq: bool] (List.exists scripts ~f:(String.is_substring ~substring:"L('From Frontmatter')")) true; + [%test_eq: bool] (List.exists scripts ~f:(String.is_substring ~substring:"L('From Expression')")) true + +let%test_unit "astro_args_and_script_block" = + let collector = + extract_astro + "Hi" + in + [%test_eq: string list] (Queue.to_list collector.strings) [ "Hi" ]; + let scripts = Queue.to_list collector.possible_scripts in + [%test_eq: bool] (List.exists scripts ~f:(String.is_substring ~substring:"LTags(\"strong\")")) true; + [%test_eq: bool] (List.exists scripts ~f:(String.is_substring ~substring:"L('From Script')")) true From 6c429e4d633c9190d21e817c6ff3504ae35ea700 Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 12:59:15 -0700 Subject: [PATCH 07/17] Document .astro support in README, ARCHITECTURE, and AGENTS --- AGENTS.md | 15 ++++++++------- ARCHITECTURE.md | 7 +++++-- README.md | 37 +++++++++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7a22eb6..7501cae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Agent Information - String Extractor -This repository contains an OCaml-based internationalization (i18n) string extraction tool. It parses source files (JS, TS, Vue, Pug, HTML) and extracts strings for translation management. +This repository contains an OCaml-based internationalization (i18n) string extraction tool. It parses source files (JS, TS, Vue, Pug, HTML, Astro) and extracts strings for translation management. ## Documentation @@ -52,7 +52,7 @@ This runs both the inline unit tests (`tests/test_runner.ml`) and an integration - `--output DIR` / `-o`: change output directory (default `strings`). - `--ts`: treat scripts in HTML/Pug element attributes as TypeScript. - `--slow-pug` / `--sp`: use the official Pug parser via QuickJS instead of the fast native OCaml one. -- `--debug-pug` / `--dp` and `--debug-html` / `--dh`: debug template parsing in `.vue` files (mutually exclusive). +- `--debug-pug` / `--dp`, `--debug-html` / `--dh`, and `--debug-astro` / `--da`: debug template parsing (mutually exclusive). The first two target `.vue` files; `--debug-astro` targets `.astro` files. - There is **no** `--show-debugging` flag. ## Setup Gotchas (things that break builds) @@ -72,11 +72,12 @@ This runs both the inline unit tests (`tests/test_runner.ml`) and an integration - JavaScript uses `Flow_parser` and a custom AST walker in `src/parsing/js_ast.ml`. - TypeScript uses the official TS parser running inside QuickJS (`src/quickjs/`). 3. **Pug Parsing**: Has a "fast" OCaml implementation (`src/parsing/pug.ml`) and a "slow" official Pug implementation via QuickJS (enabled with `--slow-pug`). +4. **Astro Parsing**: Native Angstrom scanner (`src/parsing/astro.ml`) segments `.astro` files into frontmatter, ``/`` blocks, `{...}` expressions, and `" in + [%test_eq: string list] + (Queue.to_list collector.possible_scripts) + [ "alert(message)"; "({ message: L('Hello') })" ] + +let%test_unit "astro: define:vars expression on a style tag" = + let collector = + test_collect "" + in + [%test_eq: string list] (Queue.to_list collector.possible_scripts) [ "({ color: theme })" ] + +let%test_unit "astro: expression on a self-closing script tag" = + let collector = test_collect " +``` + +With a Pug template: + +```vue + +``` + +``` +/* Home.vue */ +"Good morning, {name}!" = "Good morning, {name}!"; + +/* Home.vue */ +"Logout" = "Logout"; + +/* Widget.vue */ +"You and {count} other members are contributing." = "You and {count} other members are contributing."; +``` + +Note how `{name}` and `{count}` placeholders are part of the extracted string: the translator moves them around freely, and the runtime `L()` / `i18n` implementation substitutes the values. + +Plain `.html` and `.pug` files are scanned the same way as the corresponding Vue template languages. + +### Astro The extractor also scans `.astro` files ([Astro framework](https://astro.build)). It extracts: @@ -50,11 +127,13 @@ Because Astro compiles `{...}` in element children as JavaScript expressions, ad ```astro --- -import I18n from '../components/I18n.astro' +import { I18n, L, LTags } from '' const title = L('Create a group') --- -

{L('Welcome to Group Income')}

+{title} + +Welcome to Group Income Logout @@ -65,16 +144,20 @@ const title = L('Create a group') If an `` element contains `{placeholders}` but is missing `is:raw`, the extractor prints a warning (the string is still extracted), because Astro would otherwise evaluate the placeholders as expressions. -A minimal `I18n.astro` component looks like this (note: the runtime lookup function is deliberately not named `L` — `L` is reserved for string literals, which is what the extractor scans for, and it is never called with a variable): +Running the extractor over the example above adds these entries to `strings/english.strings`: -```astro ---- -import { translate } from '../utils/translations' -const { args } = Astro.props -const text = await Astro.slots.render('default') ---- +``` +/* pages/example.astro */ +"Create a group" = "Create a group"; + +/* pages/example.astro */ +"Logout" = "Logout"; + +/* pages/example.astro */ +"Welcome to Group Income" = "Welcome to Group Income"; - +/* pages/example.astro */ +"Yes, I want to {strong_}delete {name} permanently{_strong}." = "Yes, I want to {strong_}delete {name} permanently{_strong}."; ``` ## Developers @@ -92,7 +175,7 @@ Simply run it before submitting a Pull Request! # Linux tar xzvf strings.linux.tar.gz # unzip -./strings.linux src/ +./strings src/ ``` - **MacOS**: Monterey or newer - **Linux** and **WSL** From 2bc8606d4b5823113f5ac0f750a3f1d8d1833289 Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 18:47:10 -0700 Subject: [PATCH 14/17] improve README --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 037bbcc..31d296c 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,8 @@ -# String Extractor +# The Strings Translation Utility -The String Extractor (`strings`) is a program that runs on your computer to help manage translations. +The `strings` utility extracts user-facing strings from source files to make it easy to both translate and dislay them. + +Unlike some i18n translation software, strings allows developers to write their strings directly in-place instead of having to manage some separate list of special "keys" to then reference wherever the string is supposed to show up. With strings, you just write your software like normal, and use special markers to indicate that this is a localized string. strings will then extract this string and place it into localization files for easy localization. You don't need special software, just Github and an LLM is all you need to translate your entire app. It works together with two client-side markers that your app defines: the **`L()` function** and the **``/`` tag**. These play a double role: From 2d384f80150adcda826613bfd10febecdac4c305 Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 18:51:24 -0700 Subject: [PATCH 15/17] Target master and harden the CI workflow Run tests on pushes and PRs to master (the default branch) instead of the old lwt/test-suite branches, restrict GITHUB_TOKEN to read-only contents, and pin actions to commit SHAs to guard against tag hijacking. --- .github/workflows/test.yml | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 5b6932c..c6e2542 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -2,19 +2,22 @@ name: Tests on: push: - branches: [ lwt, test-suite ] + branches: [ master ] pull_request: - branches: [ lwt ] + branches: [ master ] + +permissions: + contents: read jobs: test: runs-on: ubuntu-latest steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Set up OCaml - uses: ocaml/setup-ocaml@v3 + uses: ocaml/setup-ocaml@e32b06a3e831ff2fbc6f08cf35be2085e3918014 # v3.6.1 with: ocaml-compiler: 5.1.1 dune-cache: true From 5b9682056bfe101f252e0963546b5d2a277a1876 Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 19:01:46 -0700 Subject: [PATCH 16/17] feedback from Copilot (typos) --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 31d296c..05774ed 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # The Strings Translation Utility -The `strings` utility extracts user-facing strings from source files to make it easy to both translate and dislay them. +The `strings` utility extracts user-facing strings from source files to make it easy to both translate and display them. -Unlike some i18n translation software, strings allows developers to write their strings directly in-place instead of having to manage some separate list of special "keys" to then reference wherever the string is supposed to show up. With strings, you just write your software like normal, and use special markers to indicate that this is a localized string. strings will then extract this string and place it into localization files for easy localization. You don't need special software, just Github and an LLM is all you need to translate your entire app. +Unlike some i18n translation software, strings allows developers to write their strings directly in-place instead of having to manage some separate list of special "keys" to then reference wherever the string is supposed to show up. With strings, you just write your software like normal, and use special markers to indicate that this is a localized string. strings will then extract this string and place it into localization files for easy localization. You don't need special software, just GitHub and an LLM is all you need to translate your entire app. It works together with two client-side markers that your app defines: the **`L()` function** and the **``/`` tag**. These play a double role: From 38ad37e5dfeddbd1a50345d4325401aac943611f Mon Sep 17 00:00:00 2001 From: Greg Slepak Date: Thu, 11 Jun 2026 19:14:18 -0700 Subject: [PATCH 17/17] Cache QuickJS build and Flow clone in CI Skip the QuickJS download/compile and the Flow repository clone on cache hits to speed up test runs. Keys are pinned to the dependency versions so caches never go stale. --- .github/workflows/test.yml | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index c6e2542..f669829 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -31,16 +31,33 @@ jobs: opam install . --deps-only --update-invariant npm install --no-save typescript browserify pug-lexer pug-parser pug-walk + - name: Cache QuickJS + id: cache-quickjs + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: quickjs + key: quickjs-2021-03-27-${{ runner.os }} + - name: Install QuickJS + if: steps.cache-quickjs.outputs.cache-hit != 'true' run: | curl -fsSL https://bellard.org/quickjs/quickjs-2021-03-27.tar.xz -o quickjs.tar.xz tar xvf quickjs.tar.xz && rm quickjs.tar.xz mv quickjs-2021-03-27 quickjs cd quickjs && make + - name: Cache Flow + id: cache-flow + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 + with: + path: flow + key: flow-v0.183.1 + - name: Install Flow run: | - git clone --branch v0.183.1 --depth 1 https://github.com/facebook/flow.git flow + if [ ! -d flow ]; then + git clone --branch v0.183.1 --depth 1 https://github.com/facebook/flow.git flow + fi ln -s "$(pwd)/flow/src/parser" src/flow_parser ln -s "$(pwd)/flow/src/third-party/sedlex" src/sedlex ln -s "$(pwd)/flow/src/hack_forked/utils/collections" src/collections