From 09dc4ab6933bf75f88fd98998138bc6f11ebe996 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 20 Sep 2026 13:10:01 +0000 Subject: [PATCH 01/38] fix(snippets): refuse a primitive snippet parameter instead of resolving it as an entity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `create snippet Test.SNIPPET_Label (params: { $Label: string })` — the spelling `mxcli syntax snippet.create` printed in its own Syntax line — passed `mxcli check` and failed at exec with failed to build snippet: failed to resolve entity string: entity not found: string naming a type nobody spelled (mendixlabs/mxcli#1028). `snippetParameter`/`snippetParameterList` in MDLPage.g4 were a byte-identical duplicate of `pageParameter`/`pageParameterList` with their own visitor, which never called buildDataType — so a primitive type never reached the AST, and buildSnippetV3 (with no primitive branch either) took the source text for an entity name. The same duplication had produced the quoted-name bug fixed just before this one, and patching the copy that time left this half live and made it silent. The duplicate rule and its visitor are gone: a snippet's Params clause IS pageParameterList. The obvious repair is the wrong one. Storage would take a primitive — Forms$SnippetParameter's ParameterType is the polymorphic DataTypes$DataType, exactly as Forms$PageParameter's is — but mxbuild will not. Measured on 11.13.0 in one run, against a project holding nothing else: snippet params { $Label: string } -> CE0046 "Invalid data type 'String'." snippet params, all six primitives -> one CE0046 each snippet params { $Order: Mod.Order } -> 0 errors PAGE params, the same six primitives -> 0 errors So the restriction is on snippet parameters, not on primitives, and a primitive one is refused rather than written — by one rule (types.SnippetParameterTypeRule) that `mxcli check` reports as MDL087 and buildSnippetV3 refuses with, so a script cannot pass one and fail the other. Both name CE0046 and the caption mxbuild quotes, so the message matches a build log verbatim. Also fixed, found on the way: - pageParamBSONType returned "DataTypes$LongType", a $Type that exists in neither generated/metamodel nor modelsdk/gen (constant_write.go has carried the note, "storage has no LongType"). pageParamTypeToGen's default arm quietly rescued it into a String, so a `Long` PAGE parameter had been stored as String. It is now IntegerType, which is Studio Pro's single "Integer/Long" — verified at 0 errors on 11.13.0 and round-tripping through `describe` as Integer. - describe of a snippet parameter went through extractEntityQualifiedName, which answers "Unknown" for anything that is not an entity — MDL that re-executes as a reference to an entity called Unknown. - pageParamTypeToGen and snippetParameterToGen were two copies of one conversion; the snippet's could only ever produce an ObjectType. Now one builder. - `mxcli syntax snippet.create`, docs-site and the overview-pages skill said a snippet parameter may be a primitive. The reporter reached the bug by following that line. Control (recorded in the tests): with the pre-fix visitor restored, the new executor test fails with the reported message verbatim, and with snippetParameterToGen's unconditional ObjectType restored the write test reports `{"$Type": "DataTypes$ObjectType", "Entity": ""}`. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01McnNijpTzHtdEf6mtmz53r --- .../fix-issue/findings/mdl-grammar.jsonl | 1 + .claude/skills/mendix/overview-pages/SKILL.md | 7 +- cmd/mxcli/syntax/features_page.go | 10 +- .../src/reference/page/create-snippet.md | 10 +- .../1028-snippet-primitive-parameter.fail.mdl | 18 ++ .../1028-snippet-primitive-parameter.mdl | 72 ++++++++ mdl/backend/modelsdk/page.go | 12 +- mdl/backend/modelsdk/page_write.go | 24 ++- .../snippet_param_primitive_write_test.go | 81 +++++++++ mdl/backend/modelsdk/snippet_write.go | 14 +- mdl/executor/cmd_pages_builder_v3.go | 34 +++- mdl/executor/cmd_pages_describe.go | 64 +++++-- mdl/executor/snippet_param_primitive_test.go | 170 ++++++++++++++++++ mdl/executor/validate_program.go | 5 + mdl/executor/validate_snippet_parameters.go | 92 ++++++++++ .../validate_snippet_parameters_test.go | 113 ++++++++++++ mdl/grammar/domains/MDLPage.g4 | 14 +- mdl/types/snippet_parameter_types.go | 55 ++++++ mdl/visitor/visitor_page_v3.go | 51 +----- 19 files changed, 757 insertions(+), 90 deletions(-) create mode 100644 mdl-examples/bug-tests/1028-snippet-primitive-parameter.fail.mdl create mode 100644 mdl-examples/bug-tests/1028-snippet-primitive-parameter.mdl create mode 100644 mdl/backend/modelsdk/snippet_param_primitive_write_test.go create mode 100644 mdl/executor/snippet_param_primitive_test.go create mode 100644 mdl/executor/validate_snippet_parameters.go create mode 100644 mdl/executor/validate_snippet_parameters_test.go create mode 100644 mdl/types/snippet_parameter_types.go diff --git a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl index fd5e9c69d4..5e84353a88 100644 --- a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl @@ -57,3 +57,4 @@ {"area": "mdl/grammar", "date": "2026-09-08", "symptom": "Adding lexer tokens for a new feature broke an unrelated, previously-passing MDL example: `editable: never` on a list view stopped parsing after NEVER became a keyword", "cause": "A new lexer token steals every existing use of that word as an identifier or property value unless it is also added to the `keyword` rule in MDLSettings.g4. NEVER, ONLINE, SYNC and PRESERVE were added for offline sync; NEVER was already a real page property value", "file": "`mdl/grammar/domains/MDLSettings.g4` (keyword rule)", "insight": "Before adding a lexer token, grep the examples for that word as a value or name — the collision is with EXISTING scripts, so nothing in the new feature's own tests can find it. TestKeywordRuleCoverage catches the omission but only asserts the rule LISTS the token; add a test that the word still parses as an identifier, which is the property that actually matters. Here the two failures had one cause: the coverage test named the tokens and check-mdl named the victim file, and the file name (maint2-editable-never-create-page.mdl) said which word", "refs": ["PROPOSAL_offline_sync_configuration.md"]} {"area": "mdl/grammar", "date": "2026-09-15", "symptom": "Re-executing `describe workflow` output failed with `mismatched input 'boundary' expecting ';'` for any user task, call microflow or wait for notification that has two or more boundary events.", "cause": "formatBoundaryEvents emits `boundary event timer '…' { … }` per event (boundaryEventKeyword includes the prefix), and the syntax topic documents that per-clause form, but MDLWorkflow.g4 had `(BOUNDARY EVENT workflowBoundaryEventClause+)?` — one keyword, then clauses.", "fix": "All four sites accept `(BOUNDARY EVENT workflowBoundaryEventClause ((BOUNDARY EVENT)? workflowBoundaryEventClause)*)?`, so both the per-clause and the shared form parse; the visitor is unchanged.", "file": "mdl/grammar/domains/MDLWorkflow.g4", "insight": "A round trip that ends in `diff describe-1 describe-2` is vacuous when the exec in between fails: the second describe reads the unchanged document and matches. It reported IDENTICAL here while the exec had died on a parse error that a grep filter hid. Assert the exec itself — zero parse errors and a rewrite verb — before diffing. The integration round-trip tests had the same blind spot: they compare describe output but never feed it back to the parser, so a grammar/describer disagreement on a construct with more than one instance could not be seen. A test that re-parses describe output (TestWorkflowDescribe_TwoBoundaryEventsReparse) is the cheap guard."} {"area": "mdl/grammar", "date": "2026-09-18", "symptom": "Lint rule SEC005 reports \"strict mode is disabled\" and MDL has no statement that turns it on — the rule's own suggestion said \"not settable via MDL\". A lint rule with no remedy, recorded on the reporting project as the one finding left Open", "cause": "StrictMode was read everywhere and written nowhere: `security_read.go` reads it, `show security` prints it, the Starlark rule lints it, and `ProjectSecurity.SetStrictMode` existed in gen and was never called. `alterProjectSecurityStatement` had three variants (LEVEL, DEMO USERS, GUEST ACCESS) and no fourth", "file": "`mdl/grammar/MDLLexer.g4` + `domains/MDLSecurity.g4` + `domains/MDLSettings.g4` (keyword rule), `mdl/ast/ast_security.go`, `mdl/visitor/visitor_security.go`, `mdl/executor/cmd_security_write.go`, `mdl/backend/security.go`, `mdl/backend/modelsdk/security_write.go`, `mdl/backend/mock/mock_security.go`, `.claude/lint-rules/sec_strict_mode.star`", "insight": "**Writing a property gen merely offers is the trap; this is not one.** StrictMode is declared by BOTH generated sources and mxcli already reads it from real projects, which is the evidence that separates it from the Layout placeholder properties that make a document Studio Pro cannot open. **The AST field must be a POINTER** — a bare bool would disable strict mode on every DEMO USERS toggle, since \"said nothing\" and \"asked for off\" would be the same value (a test pins this). New tokens STRICT and MODE both go in the parser's `keyword` rule: `mode` is an entirely plausible attribute name and a keyword left out of that rule silently breaks every model already using the word (`TestKeywordRuleCoverage` catches it; a parse test pins it too). **Update the lint rule's suggestion in the same change** — a remedy that still says \"Studio Pro only\" leaves the finding exactly as unhelpful as before. No level-dependent refusal was added: the model stores StrictMode independently of SecurityLevel, and the rule already scopes its own advice to Production", "refs": ["#526"], "rules": ["SEC005"]} +{"area": "mdl/grammar", "date": "2026-09-20", "symptom": "`create snippet Test.SNIPPET_Label (params: { $Label: string }) { dynamictext dt (content: $Label) }` — the spelling `mxcli syntax snippet.create` printed in its own Syntax line — passed `mxcli check` and failed at exec with \"failed to build snippet: failed to resolve entity string: entity not found: string\", naming a type nobody spelled (mendixlabs/mxcli#1028).", "cause": "`snippetParameter`/`snippetParameterList` in MDLPage.g4 were a byte-identical duplicate of `pageParameter`/`pageParameterList` with their own visitor, buildSnippetParameterListAsPage, which never called buildDataType — so a primitive type never reached the AST and buildSnippetV3 (which had no primitive branch either) took the source text for an entity name. Collapsed: a snippet's Params clause IS pageParameterList, and buildPageParameters is the only conversion. The primitive is then REFUSED, not written: mxbuild rejects a primitive snippet parameter with CE0046, so writing one the way a page parameter writes one would have traded an unreadable exec error for a build failure.", "file": "`mdl/grammar/domains/MDLPage.g4` (duplicate rule deleted); `mdl/visitor/visitor_page_v3.go` (buildSnippetParameterListAsPage deleted); `mdl/types/snippet_parameter_types.go` (SnippetParameterTypeRule, the measurements); `mdl/executor/validate_snippet_parameters.go` (MDL087); `mdl/executor/cmd_pages_builder_v3.go` (buildSnippetV3 refusal, pageParamBSONType Long fix); `cmd/mxcli/syntax/features_page.go`; tests `mdl/executor/snippet_param_primitive_test.go`, `mdl/executor/validate_snippet_parameters_test.go`, `mdl-examples/bug-tests/1028-snippet-primitive-parameter{,.fail}.mdl`", "insight": "Two byte-identical grammar rules with two visitors is a bug generator, not a duplication smell: this clause produced TWO reported bugs from the same duplication in a fortnight (the quoted entity name, then this), and the first fix — patching the copy — left the second live and silent, turning a loud wrong error into a parameter with no type at all. When a fix is 'make X agree with Y' and X and Y are the same grammar, delete X. Second, and the reason step 6 of fix-issue is not optional: the obvious repair here (write the primitive the way a page parameter writes one) is supported by every source of truth in the repo — generated/metamodel declares Forms$SnippetParameter.ParameterType as the polymorphic DataTypes$DataType, exactly as Forms$PageParameter's, and the codec encodes it happily — and mxbuild rejects it with CE0046. A shape argument from the metamodel cannot see a validator rule. The control that made the rule crisp was putting the SAME six primitives on a PAGE in the SAME mxbuild run: six CE0046 on the snippet, 0 errors on the page, so the restriction is on snippet parameters and not on primitives, which is exactly what the error message now has to say. Third, a bug like this is a documentation bug as much as a code one — the reporter reached it by following `mxcli syntax snippet.create`, so a fix that leaves that line printing `$Label: String` re-creates the report. Aside found on the way: pageParamBSONType returned \"DataTypes$LongType\", a $Type that does not exist in gen OR generated/metamodel (constant_write.go had the note, 'storage has no LongType'), and pageParamTypeToGen's default arm quietly rescued it into a String — so a `Long` page parameter had been silently stored as String.", "ce": "CE0046", "rules": "MDL087", "refs": "mendixlabs/mxcli#1028; the sibling quoted-name fix in the same clause (mdl/visitor/snippet_param_quoted_entity_test.go); ADR-0005 guard-don't-drop"} diff --git a/.claude/skills/mendix/overview-pages/SKILL.md b/.claude/skills/mendix/overview-pages/SKILL.md index 664d7bcb4c..aa209a2a18 100644 --- a/.claude/skills/mendix/overview-pages/SKILL.md +++ b/.claude/skills/mendix/overview-pages/SKILL.md @@ -504,7 +504,12 @@ module/ ## Parameterized Snippets -Snippets can accept parameters to display context-specific data: +Snippets can accept parameters to display context-specific data. **A snippet +parameter must be an entity.** A primitive one (`params: { $Label: String }`) is +refused as **MDL087**, because Mendix rejects it with **CE0046** *"Invalid data +type 'String'."* — a *page* parameter may be a primitive, a snippet parameter may +not. To parameterise a snippet on a value, keep the primitive on the calling +page's parameters, or pass an object and read the member inside the snippet. ```sql -- Create a snippet with a parameter diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index c4609994a0..ae85a9dcf7 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -385,7 +385,7 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "snippet", "snippets", "reusable", "snippetcall", "page fragment", "component", }, - Syntax: "CREATE SNIPPET Module.Name\n [( Params: { $P: Module.Entity }, Folder: 'path' )]\n {\n -- widgets (same as page)\n }\n\n-- Embed in a page:\nSNIPPETCALL scName (Snippet: Module.SnippetName)", + Syntax: "CREATE SNIPPET Module.Name\n [( Params: { $P: Module.Entity }, Folder: 'path' )] -- parameters are entities only\n {\n -- widgets (same as page)\n }\n\n-- Embed in a page:\nSNIPPETCALL scName (Snippet: Module.SnippetName)", Example: "CREATE SNIPPET MyModule.CustomerInfo (\n Params: { $Customer: MyModule.Customer }\n)\n{\n DATAVIEW dv (DataSource: $Customer) {\n TEXTBOX txtName (Label: 'Name', Attribute: Name)\n TEXTBOX txtEmail (Label: 'Email', Attribute: Email)\n }\n}", SeeAlso: []string{"snippet.create", "snippet.alter", "page"}, }) @@ -397,7 +397,13 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "create snippet", "new snippet", "snippet parameters", "snippet variables", }, - Syntax: "CREATE SNIPPET Module.Name\n [( Params: { $P: Module.Entity, $Label: String } )]\n [( Variables: { $isEditable: Boolean = 'true' } )]\n [( Folder: 'Snippets/Common' )]\n {\n -- widgets\n }", + // Every snippet parameter is an entity. This line used to print + // "$Label: String", which is not executable: mxbuild rejects a + // primitive snippet parameter with CE0046 "Invalid data type 'String'." + // (a PAGE parameter may be primitive; a snippet parameter may not), and + // the reporter of mendixlabs/mxcli#1028 reached the bug by following + // this line. A primitive is now refused as MDL087. + Syntax: "CREATE SNIPPET Module.Name\n [( Params: { $P: Module.Entity } )] -- entities only; a primitive is CE0046\n [( Variables: { $isEditable: Boolean = 'true' } )]\n [( Folder: 'Snippets/Common' )]\n {\n -- widgets\n }\n\n-- To parameterise a snippet on a primitive, keep the primitive on the\n-- calling PAGE and pass an object, or read the value off an entity member.", Example: "CREATE SNIPPET MyModule.NavigationMenu\n{\n NAVIGATIONLIST navMenu {\n ITEM itemCustomers (Action: SHOW_PAGE MyModule.CustomerOverview) {\n DYNAMICTEXT txtCustomers (Content: 'Customers')\n }\n }\n}", SeeAlso: []string{"snippet", "snippet.alter", "page.widgets"}, }) diff --git a/docs-site/src/reference/page/create-snippet.md b/docs-site/src/reference/page/create-snippet.md index 191bfc5335..72dbe5229d 100644 --- a/docs-site/src/reference/page/create-snippet.md +++ b/docs-site/src/reference/page/create-snippet.md @@ -35,7 +35,15 @@ The optional `Folder` property places the snippet in a subfolder within the modu : The qualified name of the snippet (`Module.SnippetName`). The module must already exist. `Params: { ... }` -: Optional snippet parameters. Each parameter has a `$`-prefixed name and a type (entity or primitive). +: Optional snippet parameters. Each parameter has a `$`-prefixed name and an + **entity** type (`Module.Entity`). + + A snippet parameter cannot be a primitive. Mendix rejects one with + **CE0046** *"Invalid data type 'String'."* — a *page* parameter may be a + primitive, a snippet parameter may not — so `mxcli check` refuses it as + **MDL087** rather than letting it reach a build. To parameterise a snippet + on a value, keep the primitive on the calling page's parameters, or pass an + object and read the member inside the snippet. `Folder: 'path'` : Optional folder path within the module. diff --git a/mdl-examples/bug-tests/1028-snippet-primitive-parameter.fail.mdl b/mdl-examples/bug-tests/1028-snippet-primitive-parameter.fail.mdl new file mode 100644 index 0000000000..761aa98a3e --- /dev/null +++ b/mdl-examples/bug-tests/1028-snippet-primitive-parameter.fail.mdl @@ -0,0 +1,18 @@ +-- ============================================================================ +-- Upstream #1028 (the refusal): a primitive SNIPPET parameter is MDL087 +-- ============================================================================ +-- +-- The reported script, unchanged. `mxcli check` must refuse it rather than +-- pass it on to an exec that says "entity not found: string" — or to a build +-- that says CE0046 "Invalid data type 'String'." See the companion +-- 1028-snippet-primitive-parameter.mdl for the measurements and the form that +-- works. +-- +-- This file is expected to FAIL `mxcli check`. +-- ============================================================================ + +create module F1028; + +create snippet F1028.SNIPPET_Label ( params: { $Label: string } ) { + dynamictext dt (content: $Label) +} diff --git a/mdl-examples/bug-tests/1028-snippet-primitive-parameter.mdl b/mdl-examples/bug-tests/1028-snippet-primitive-parameter.mdl new file mode 100644 index 0000000000..4a09f14a17 --- /dev/null +++ b/mdl-examples/bug-tests/1028-snippet-primitive-parameter.mdl @@ -0,0 +1,72 @@ +-- ============================================================================ +-- Upstream #1028: a primitive-typed SNIPPET parameter was taken for an entity +-- ============================================================================ +-- +-- create snippet Test.SNIPPET_Label ( params: { $Label: string } ) +-- { dynamictext dt (content: $Label) }; +-- +-- passed `mxcli check` and failed at exec with +-- +-- Error: failed to build snippet: failed to resolve entity string: +-- entity not found: string +-- +-- naming a type nobody spelled. `snippetParameter` was a byte-identical +-- duplicate of the `pageParameter` grammar rule with its own, lesser visitor: +-- it never called buildDataType, so a primitive never reached the AST and the +-- executor took the source text for an entity name. (The same duplication had +-- produced the quoted-name bug fixed just before this one.) The duplicate rule +-- is gone — a snippet's Params clause IS pageParameterList. +-- +-- The obvious repair is the wrong one. Storage would take a primitive: +-- Forms$SnippetParameter's ParameterType is the polymorphic DataTypes$DataType, +-- exactly as Forms$PageParameter's is. mxbuild will not. Measured on 11.13.0: +-- +-- create snippet S.Label ( params: { $Label: string } ) +-- -> [error] [CE0046] "Invalid data type 'String'." at Snippet 'S.Label' +-- ... one CE0046 per primitive parameter, quoting Studio Pro's caption +-- ('String', 'Integer/Long', 'Decimal', 'Boolean', 'Date and time') +-- +-- and the control, in the same run and the same project: a PAGE declaring all +-- six of those primitives as parameters builds at 0 ERRORS. The restriction is +-- on snippet parameters, not on primitives. +-- +-- So a primitive snippet parameter is refused, by one rule that `mxcli check` +-- (MDL087) and `exec` both call. This script is the form that works. +-- +-- Verified on Mendix 11.13.0: 0 errors. +-- ============================================================================ + +create module S1028; +create persistent entity S1028.Order (Code: String(20), Label: String(100)); + +-- The reported snippet, in the shape Mendix accepts: the value travels on an +-- object. `create snippet S1028.SNIPPET_Label ( params: { $Label: string } )` +-- is refused as MDL087 — see 1028-snippet-primitive-parameter.fail.mdl. +create snippet S1028.SNIPPET_Label ( params: { $Order: S1028.Order } ) { + layoutgrid gl { row rl { column cl (desktopwidth: autofill) { + dataview dvLabel (datasource: $Order) { + textbox tbLabel (attribute: Label, label: 'Label') + } + } } } +} + +-- The primitives belong on the calling PAGE, which accepts every one of them. +-- Long is stored as DataTypes$IntegerType (storage has no LongType; Studio +-- Pro's type is the single "Integer/Long"), so `describe` re-emits it as +-- Integer — which is why this page is here rather than only in a unit test. +create page S1028.PagePrimitives ( + params: { + $Order: S1028.Order, + $Label: String, + $Count: Long, + $Rank: Integer, + $Amount: Decimal, + $Active: Boolean, + $Due: DateTime + }, + title: 'Primitives', layout: Atlas_Core.Atlas_Default +) { + layoutgrid g { row r { column c (desktopwidth: 12) { + snippetcall scLabel (snippet: S1028.SNIPPET_Label, params: {Order: $Order}) + } } } +} diff --git a/mdl/backend/modelsdk/page.go b/mdl/backend/modelsdk/page.go index e2e49a4613..54148bddfa 100644 --- a/mdl/backend/modelsdk/page.go +++ b/mdl/backend/modelsdk/page.go @@ -102,8 +102,16 @@ func (b *Backend) ListSnippets() ([]*pages.Snippet, error) { } p := &pages.SnippetParameter{Name: sp.Name()} p.ID = model.ID(sp.ID()) - if ot, ok := sp.ParameterType().(*genDT.ObjectType); ok { - p.EntityName = ot.EntityQualifiedName() + // Entity or primitive — a snippet parameter's ParameterType is the + // polymorphic DataTypes$DataType, the same as a page parameter's. + // Reading only the ObjectType arm made every primitive parameter + // read back as an untyped one (mendixlabs/mxcli#1028). + if pt := sp.ParameterType(); pt != nil { + if ot, ok := pt.(*genDT.ObjectType); ok { + p.EntityName = ot.EntityQualifiedName() + } else { + p.Type = pt.TypeName() + } } s.Parameters = append(s.Parameters, p) } diff --git a/mdl/backend/modelsdk/page_write.go b/mdl/backend/modelsdk/page_write.go index dad3067752..15c5320f81 100644 --- a/mdl/backend/modelsdk/page_write.go +++ b/mdl/backend/modelsdk/page_write.go @@ -256,15 +256,31 @@ func pageParameterToGen(p *pages.PageParameter, pv *types.ProjectVersion) *genPg // entity parameter, or the named primitive DataTypes type. p.TypeName carries the // primitive's BSON $Type (e.g. "DataTypes$StringType") when set. func pageParamTypeToGen(p *pages.PageParameter) element.Element { - if p.TypeName == "" { + return paramTypeToGen(p.TypeName, p.EntityName) +} + +// paramTypeToGen builds the ParameterType child shared by Forms$PageParameter and +// Forms$SnippetParameter — both declare it as the polymorphic DataTypes$DataType, +// so one builder serves both. bsonType empty means "entity", and entityName is +// then the qualified name the DataTypes$ObjectType points at. +// +// Keeping the two on one builder is the point: a snippet parameter used to have +// its own, entity-only copy, so `Params: { $Label: String }` wrote a snippet +// parameter with no type at all (mendixlabs/mxcli#1028). +func paramTypeToGen(bsonType, entityName string) element.Element { + if bsonType == "" { t := genDT.NewObjectType() assignID(t) - t.SetEntityQualifiedName(p.EntityName) + t.SetEntityQualifiedName(entityName) return t } var t element.Element - switch p.TypeName { - case "DataTypes$IntegerType": + switch bsonType { + case "DataTypes$IntegerType", "DataTypes$LongType": + // gen has no Long: Mendix models both as the same data type, as + // localVarTypeToGen notes for the sibling case. pageParamBSONType no + // longer produces DataTypes$LongType, but a caller that sets TypeName + // itself must not fall through to the String default the way one did. t = genDT.NewIntegerType() case "DataTypes$BooleanType": t = genDT.NewBooleanType() diff --git a/mdl/backend/modelsdk/snippet_param_primitive_write_test.go b/mdl/backend/modelsdk/snippet_param_primitive_write_test.go new file mode 100644 index 0000000000..4a8a8aebd3 --- /dev/null +++ b/mdl/backend/modelsdk/snippet_param_primitive_write_test.go @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "testing" + + "github.com/mendixlabs/mxcli/modelsdk/codec" + "github.com/mendixlabs/mxcli/sdk/pages" + "go.mongodb.org/mongo-driver/bson" +) + +// mendixlabs/mxcli#1028 — shape guard on the one builder that now serves both +// Forms$PageParameter and Forms$SnippetParameter. Both declare ParameterType as +// the polymorphic DataTypes$DataType (generated/metamodel: +// PagesSnippetParameter.ParameterType *DataTypesDataType), so one builder is +// correct for both, and snippetParameterToGen no longer keeps a second copy +// that could only ever produce an ObjectType. +// +// A primitive snippet parameter is refused upstream (MDL087 — mxbuild rejects +// it with CE0046, measured), so these rows are not a supported authoring path. +// They pin the builder's contract: whatever type it is handed, it writes THAT +// type, never an ObjectType pointing at an entity named "" — which is what the +// duplicate copy produced, and the reason a wrong parameter type used to show +// up as a dangling reference rather than as anything an author could read. +func snippetParamType(t *testing.T, p *pages.SnippetParameter) bson.Raw { + t.Helper() + b, err := (&codec.Encoder{}).Encode(snippetParameterToGen(p)) + if err != nil { + t.Fatalf("encode: %v", err) + } + child, err := bson.Raw(b).LookupErr("ParameterType") + if err != nil { + t.Fatalf("no ParameterType in %v", bson.Raw(b)) + } + doc, ok := child.DocumentOK() + if !ok { + t.Fatalf("ParameterType is not a document: %v", child) + } + return doc +} + +func TestSnippetParameterToGen_PrimitiveWritesItsOwnDataType(t *testing.T) { + for _, tc := range []struct{ stored, wantType string }{ + {"DataTypes$StringType", "DataTypes$StringType"}, + {"DataTypes$IntegerType", "DataTypes$IntegerType"}, + {"DataTypes$DecimalType", "DataTypes$DecimalType"}, + {"DataTypes$BooleanType", "DataTypes$BooleanType"}, + {"DataTypes$DateTimeType", "DataTypes$DateTimeType"}, + } { + t.Run(tc.stored, func(t *testing.T) { + doc := snippetParamType(t, &pages.SnippetParameter{Name: "P", Type: tc.stored}) + got, err := doc.LookupErr("$Type") + if err != nil { + t.Fatalf("no $Type: %v", doc) + } + if got.StringValue() != tc.wantType { + t.Errorf("$Type = %q, want %q", got.StringValue(), tc.wantType) + } + // A primitive type has no Entity — writing one would be a key the + // type does not declare. + if _, err := doc.LookupErr("Entity"); err == nil { + t.Errorf("primitive ParameterType carries an Entity key: %v", doc) + } + }) + } +} + +// CONTROL: an entity parameter is unchanged — still a DataTypes$ObjectType +// naming the entity. Without this the fix could be "always write a StringType". +func TestSnippetParameterToGen_EntityStillWritesObjectType(t *testing.T) { + doc := snippetParamType(t, &pages.SnippetParameter{Name: "C", EntityName: "Sales.Customer"}) + ty, err := doc.LookupErr("$Type") + if err != nil || ty.StringValue() != "DataTypes$ObjectType" { + t.Fatalf("$Type = %v, want DataTypes$ObjectType", ty) + } + ent, err := doc.LookupErr("Entity") + if err != nil || ent.StringValue() != "Sales.Customer" { + t.Errorf("Entity = %v, want Sales.Customer", ent) + } +} diff --git a/mdl/backend/modelsdk/snippet_write.go b/mdl/backend/modelsdk/snippet_write.go index cd2c99a9ae..6a6671ef84 100644 --- a/mdl/backend/modelsdk/snippet_write.go +++ b/mdl/backend/modelsdk/snippet_write.go @@ -8,7 +8,6 @@ import ( "github.com/mendixlabs/mxcli/model" "github.com/mendixlabs/mxcli/modelsdk/codec" "github.com/mendixlabs/mxcli/modelsdk/element" - genDT "github.com/mendixlabs/mxcli/modelsdk/gen/datatypes" genPg "github.com/mendixlabs/mxcli/modelsdk/gen/pages" mmpr "github.com/mendixlabs/mxcli/modelsdk/mpr" "github.com/mendixlabs/mxcli/sdk/pages" @@ -117,7 +116,13 @@ func snippetToGen(s *pages.Snippet) (*genPg.Snippet, error) { return out, nil } -// snippetParameterToGen builds a Forms$SnippetParameter (entity-typed). +// snippetParameterToGen builds a Forms$SnippetParameter. Its ParameterType is +// the same polymorphic DataTypes$DataType a page parameter carries, so it goes +// through the same builder: p.Type holds a primitive's BSON $Type when the +// parameter is primitive, and is empty for an entity parameter. +// +// It used to build a DataTypes$ObjectType unconditionally, so a primitive-typed +// parameter was written pointing at an entity named "" (mendixlabs/mxcli#1028). func snippetParameterToGen(p *pages.SnippetParameter) *genPg.SnippetParameter { gp := genPg.NewSnippetParameter() if p.ID != "" { @@ -125,9 +130,6 @@ func snippetParameterToGen(p *pages.SnippetParameter) *genPg.SnippetParameter { } assignID(gp) gp.SetName(p.Name) - t := genDT.NewObjectType() - assignID(t) - t.SetEntityQualifiedName(p.EntityName) - gp.SetParameterType(t) + gp.SetParameterType(paramTypeToGen(p.Type, p.EntityName)) return gp } diff --git a/mdl/executor/cmd_pages_builder_v3.go b/mdl/executor/cmd_pages_builder_v3.go index 21bdc1e358..705d4591e2 100644 --- a/mdl/executor/cmd_pages_builder_v3.go +++ b/mdl/executor/cmd_pages_builder_v3.go @@ -252,7 +252,20 @@ func (pb *pageBuilder) buildSnippetV3(s *ast.CreateSnippetStmtV3) (*pages.Snippe Name: param.Name, } - // Resolve entity type + // A snippet parameter must name an entity. A primitive one is refused + // rather than resolved as an entity name — the reported symptom was + // "entity not found: string", for a type nobody spelled — and rather + // than written, which storage would allow and mxbuild would not + // (CE0046). Same rule check applies, so a script cannot pass one and + // fail the other (mendixlabs/mxcli#1028). + if caption := types.SnippetParameterTypeRule(pageParamBSONType(param.Type)); caption != "" { + return nil, mdlerrors.NewValidationf( + "snippet '%s' declares parameter $%s with the primitive type %s — a snippet "+ + "parameter must be an entity, and mxbuild rejects a primitive one with "+ + "CE0046 (\"Invalid data type '%s'.\"). Pass the value on an object, or "+ + "keep the primitive on the calling page's parameters.", + s.Name.String(), param.Name, paramTypeSourceName(param.Type), caption) + } if param.EntityType.Name != "" { entityID, err := pb.resolveEntity(param.EntityType) if err != nil { @@ -261,6 +274,8 @@ func (pb *pageBuilder) buildSnippetV3(s *ast.CreateSnippetStmtV3) (*pages.Snippe entityName := param.EntityType.String() snippetParam.EntityID = entityID snippetParam.EntityName = entityName + // Only entity-typed parameters enter paramScope — it maps a name to + // an entity ID, and a primitive has none. Same as the page path. pb.paramScope[param.Name] = entityID pb.paramEntityNames[param.Name] = entityName } @@ -1692,16 +1707,23 @@ func (pb *pageBuilder) getEntityNameByID(entityID model.ID) (string, error) { return "", mdlerrors.NewNotFound("entity", string(entityID)) } -// pageParamBSONType maps a DataType to the BSON $Type string for primitive page parameters. -// Returns empty string for entity/enum types (which use DataTypes$ObjectType instead). +// pageParamBSONType maps a DataType to the BSON $Type string for a primitive +// page or snippet parameter. Returns empty string for entity/enum types (which +// use DataTypes$ObjectType instead), which is the signal the callers branch on. +// +// Long maps to DataTypes$IntegerType because storage has no LongType: neither +// generated/metamodel (the 11.6.0 arbiter) nor modelsdk/gen declares one, and +// Studio Pro's own parameter type is the single "Integer/Long". This used to +// return "DataTypes$LongType", a $Type Mendix does not have — the CLAUDE.md +// "never invent a key" case, which on the way to disk was quietly rescued into +// a String by pageParamTypeToGen's default arm. constant_write.go has carried +// the same note ("storage has no LongType") all along. func pageParamBSONType(dt ast.DataType) string { switch dt.Kind { case ast.TypeString: return "DataTypes$StringType" - case ast.TypeInteger: + case ast.TypeInteger, ast.TypeLong: return "DataTypes$IntegerType" - case ast.TypeLong: - return "DataTypes$LongType" case ast.TypeDecimal: return "DataTypes$DecimalType" case ast.TypeBoolean: diff --git a/mdl/executor/cmd_pages_describe.go b/mdl/executor/cmd_pages_describe.go index 8049e35a3f..f04a3358e4 100644 --- a/mdl/executor/cmd_pages_describe.go +++ b/mdl/executor/cmd_pages_describe.go @@ -275,8 +275,7 @@ func describeSnippet(ctx *ExecContext, name ast.QualifiedName) error { paramParts := []string{} for _, p := range params { paramName, _ := p["Name"].(string) - entityName := extractEntityQualifiedName(p["ParameterType"]) - paramParts = append(paramParts, fmt.Sprintf("$%s: %s", paramName, entityName)) + paramParts = append(paramParts, fmt.Sprintf("$%s: %s", paramName, snippetParamTypeMDL(p["ParameterType"]))) } snippetProps = append(snippetProps, fmt.Sprintf("Params: { %s }", strings.Join(paramParts, ", "))) } @@ -456,6 +455,26 @@ func extractEntityQualifiedName(paramType any) string { return "Unknown" } +// snippetParamTypeMDL renders a snippet parameter's stored ParameterType as the +// MDL type that produces it — an entity's qualified name, or a primitive keyword. +// +// It used to be extractEntityQualifiedName alone, which answers "Unknown" for +// anything that is not an entity: a primitive-typed parameter described as +// `$Label: Unknown`, MDL that re-executes as a reference to an entity called +// Unknown (mendixlabs/mxcli#1028). +func snippetParamTypeMDL(paramType any) string { + ptMap, ok := paramType.(map[string]any) + if !ok { + return "Unknown" + } + bsonType, _ := ptMap["$Type"].(string) + switch bsonType { + case "", "Pages$EntityType", "Forms$EntityType", "DataTypes$ObjectType": + return extractEntityQualifiedName(paramType) + } + return primitiveParamTypeMDL(bsonType) +} + // getBsonArrayMaps extracts []map[string]interface{} from BSON array types. func getBsonArrayMaps(v any) []map[string]any { if v == nil { @@ -991,25 +1010,34 @@ func wrapStringLiteralExpression(value string) string { // Primitive params return "String", "Integer", etc.; entity params return the qualified name. func pageParamTypeMDL(p *pages.PageParameter) string { if p.TypeName != "" { - switch p.TypeName { - case "DataTypes$StringType": - return "String" - case "DataTypes$IntegerType": - return "Integer" - case "DataTypes$LongType": - return "Long" - case "DataTypes$DecimalType": - return "Decimal" - case "DataTypes$BooleanType": - return "Boolean" - case "DataTypes$DateTimeType": - return "DateTime" - default: - return p.TypeName - } + return primitiveParamTypeMDL(p.TypeName) } if p.EntityName != "" { return p.EntityName } return string(p.EntityID) } + +// primitiveParamTypeMDL maps a parameter's primitive BSON $Type back to the MDL +// keyword that produces it, so `describe` re-emits something `exec` accepts. +// An unrecognised $Type is returned as-is rather than guessed at — it will not +// re-parse, which is the visible symptom a silent "String" would hide. +// +// Long has no entry: storage has no DataTypes$LongType (see pageParamBSONType), +// so `Long` and `Integer` are one stored type and describe says Integer. +func primitiveParamTypeMDL(bsonType string) string { + switch bsonType { + case "DataTypes$StringType": + return "String" + case "DataTypes$IntegerType": + return "Integer" + case "DataTypes$DecimalType": + return "Decimal" + case "DataTypes$BooleanType": + return "Boolean" + case "DataTypes$DateTimeType": + return "DateTime" + default: + return bsonType + } +} diff --git a/mdl/executor/snippet_param_primitive_test.go b/mdl/executor/snippet_param_primitive_test.go new file mode 100644 index 0000000000..4b956309b0 --- /dev/null +++ b/mdl/executor/snippet_param_primitive_test.go @@ -0,0 +1,170 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/visitor" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// mendixlabs/mxcli#1028. A primitive-typed snippet parameter — the spelling +// `mxcli syntax snippet.create` printed in its own Syntax line — +// +// CREATE SNIPPET Test.SNIPPET_Label ( Params: { $Label: string } ) +// { dynamictext dt (content: $Label) }; +// +// passed `mxcli check` and then failed at exec with +// +// Error: failed to build snippet: failed to resolve entity string: +// entity not found: string +// +// naming a type nobody spelled. `snippetParameter` was a byte-identical +// duplicate of the `pageParameter` grammar rule with its own, lesser visitor: +// it never called buildDataType, so a primitive type never reached the AST at +// all and the executor took the source text for an entity name. (The same +// duplication had produced the quoted-name bug fixed just before this one — +// see visitor.TestSnippetParameter_QuotedEntityNameIsUnquoted.) +// +// The repair is NOT to write the primitive the way a page parameter writes one. +// Storage would take it — Forms$SnippetParameter's ParameterType is the same +// polymorphic DataTypes$DataType — but mxbuild rejects every primitive snippet +// parameter with CE0046, measured in types.SnippetParameterTypeRule. So the +// statement is refused, by the rule `mxcli check` also applies (MDL087). + +// buildSnippetFromMDL parses one CREATE SNIPPET statement and builds it against +// a backend that knows no entities, so an attempt to resolve a primitive as an +// entity surfaces rather than being masked by a lucky name collision. +func buildSnippetFromMDL(t *testing.T, src string) (*pages.Snippet, error) { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs[0]) + } + var stmt *ast.CreateSnippetStmtV3 + for _, s := range prog.Statements { + if cs, ok := s.(*ast.CreateSnippetStmtV3); ok { + stmt = cs + break + } + } + if stmt == nil { + t.Fatal("no CREATE SNIPPET statement built") + } + + mod := mkModule("Test") + h := mkHierarchy(mod) + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListSnippetsFunc: func() ([]*pages.Snippet, error) { return nil, nil }, + } + return newPageBuilder(mb, h, "Test").buildSnippetV3(stmt) +} + +// The reported repro. The refusal must name the CE the author would otherwise +// meet a whole build later, and must not be the old "entity not found" — which +// pointed at a lower-cased word that appears nowhere in the script and offered +// no way to act. +func TestBuildSnippetV3_PrimitiveParameterIsRefusedNotMistakenForAnEntity(t *testing.T) { + _, err := buildSnippetFromMDL(t, `CREATE SNIPPET Test.SNIPPET_Label ( + Params: { $Label: string } +) { + DYNAMICTEXT dt (Content: $Label) +}`) + if err == nil { + t.Fatal("a primitive snippet parameter was accepted; mxbuild rejects it with CE0046") + } + msg := err.Error() + for _, want := range []string{"$Label", "String", "CE0046", "must be an entity"} { + if !strings.Contains(msg, want) { + t.Errorf("error does not mention %q: %s", want, msg) + } + } + if strings.Contains(msg, "entity not found") { + t.Errorf("still the #1028 message, which names a type nobody spelled: %s", msg) + } +} + +// Every primitive, in the reporter's own casing and in the documented casing — +// MDL type keywords are case-insensitive and the old visitor passed the source +// text through, which is why the report said "string" and the docs said +// "String". The refusal must not depend on either. +func TestBuildSnippetV3_EveryPrimitiveParameterIsRefused(t *testing.T) { + for _, spelling := range []string{ + "string", "String", "STRING", + "Integer", "Long", "Decimal", "Boolean", "DateTime", + } { + t.Run(spelling, func(t *testing.T) { + _, err := buildSnippetFromMDL(t, `CREATE SNIPPET Test.S ( + Params: { $P: `+spelling+` } +) { DYNAMICTEXT dt (Content: 'x') }`) + if err == nil { + t.Fatalf("%q accepted as a snippet parameter type", spelling) + } + if !strings.Contains(err.Error(), "CE0046") { + t.Errorf("%q: error = %v, want it to name CE0046", spelling, err) + } + }) + } +} + +// CONTROL: an entity-typed parameter is still built, still resolved, and still +// fails loudly when the entity does not exist. Without this the refusal above +// is satisfied by "refuse every snippet parameter". +func TestBuildSnippetV3_EntityParameterStillResolves(t *testing.T) { + _, err := buildSnippetFromMDL(t, `CREATE SNIPPET Test.S ( + Params: { $C: Test.NoSuchEntity } +) { DYNAMICTEXT dt (Content: 'x') }`) + if err == nil { + t.Fatal("an unknown entity-typed parameter was accepted; want a resolve failure") + } + if !strings.Contains(err.Error(), "Test.NoSuchEntity") { + t.Errorf("error = %v, want it to name Test.NoSuchEntity", err) + } + if strings.Contains(err.Error(), "CE0046") { + t.Errorf("entity parameter hit the primitive refusal: %v", err) + } +} + +// CONTROL: a PAGE parameter of the same primitive is accepted and typed — the +// restriction is on snippet parameters, not on primitives. Both halves were +// measured in one mxbuild 11.13.0 run: the page below builds at 0 errors while +// the identical clause on a snippet produces one CE0046 per parameter. +func TestBuildPageV3_PrimitiveParameterStillAccepted(t *testing.T) { + // No widgets and no layout: this asserts what the parameter clause builds, + // and a layout the mock backend does not have would fail the page for an + // unrelated reason. + prog, errs := visitor.Build(`CREATE PAGE Test.P ( + Title: 'P', Params: { $Label: String, $Count: Long } +) { }`) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs[0]) + } + var stmt *ast.CreatePageStmtV3 + for _, s := range prog.Statements { + if cp, ok := s.(*ast.CreatePageStmtV3); ok { + stmt = cp + break + } + } + if stmt == nil { + t.Fatal("no CREATE PAGE statement built") + } + mod := mkModule("Test") + pb := newPageBuilder(&mock.MockBackend{IsConnectedFunc: func() bool { return true }}, + mkHierarchy(mod), "Test") + page, err := pb.buildPageV3(stmt) + if err != nil { + t.Fatalf("buildPageV3: %v", err) + } + want := []string{"DataTypes$StringType", "DataTypes$IntegerType"} + for i, p := range page.Parameters { + if p.TypeName != want[i] { + t.Errorf("page param %s = %q, want %q", p.Name, p.TypeName, want[i]) + } + } +} diff --git a/mdl/executor/validate_program.go b/mdl/executor/validate_program.go index 22150fc32a..d6f36ac4f8 100644 --- a/mdl/executor/validate_program.go +++ b/mdl/executor/validate_program.go @@ -63,6 +63,11 @@ func ValidateProgram(prog *ast.Program, projectPath string) []linter.Violation { // script passed check AND exec and failed a build later // (mendixlabs/mxcli#1063). violations = append(violations, validateLayoutPlaceholders(stmt)...) + // A snippet parameter must be an entity; mxbuild rejects a primitive one + // with CE0046 (MDL087). The documented spelling used the primitive form, + // so this was reachable straight from `mxcli syntax snippet.create` + // (mendixlabs/mxcli#1028). + violations = append(violations, validateSnippetParameters(stmt)...) // A microflow's URL / export level / concurrency clauses, against the // same rules the writer applies (MDL-MF01..MF04). violations = append(violations, validateMicroflowDocumentProperties(stmt)...) diff --git a/mdl/executor/validate_snippet_parameters.go b/mdl/executor/validate_snippet_parameters.go new file mode 100644 index 0000000000..5a566c622c --- /dev/null +++ b/mdl/executor/validate_snippet_parameters.go @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/types" +) + +// validateSnippetParameters (MDL087) refuses a primitive-typed snippet +// parameter. A snippet parameter must name an entity; mxbuild rejects every +// primitive with CE0046 "Invalid data type ''." — the measurements, +// and the page control that makes this specific to snippets, are in +// types.SnippetParameterTypeRule. +// +// # Why this is a check and not a write +// +// mendixlabs/mxcli#1028 reported the documented spelling failing at exec: +// +// create snippet Test.SNIPPET_Label ( params: { $Label: string } ) +// → Error: failed to build snippet: failed to resolve entity string: +// entity not found: string +// +// — a type nobody spelled, because the snippet's Params clause had its own +// visitor that handed the resolver the source text verbatim. The obvious repair +// is to write the primitive the way a page parameter writes one; the storage +// even allows it. mxbuild does not, so that repair trades an incomprehensible +// refusal for a document that fails a build later. The statement is refused +// here instead, in the project-free pass, naming the CE the author would +// otherwise meet at the far end of a build. +// +// Reported per parameter, so a clause with several bad ones names them all +// rather than one per run. +func validateSnippetParameters(stmt ast.Statement) []linter.Violation { + snippet, ok := stmt.(*ast.CreateSnippetStmtV3) + if !ok { + return nil + } + + var out []linter.Violation + for _, p := range snippet.Parameters { + caption := types.SnippetParameterTypeRule(pageParamBSONType(p.Type)) + if caption == "" { + continue + } + out = append(out, linter.Violation{ + RuleID: "MDL087", + Severity: linter.SeverityError, + Location: linter.Location{ + Module: snippet.Name.Module, + DocumentType: "snippet", + DocumentName: snippet.Name.Name, + }, + Message: fmt.Sprintf( + "snippet '%s' declares parameter $%s with the primitive type %s. "+ + "A snippet parameter must be an entity — mxbuild rejects a primitive one "+ + "with CE0046 (\"Invalid data type '%s'.\"). A page parameter may be primitive; "+ + "a snippet parameter may not.", + snippet.Name.String(), p.Name, paramTypeSourceName(p.Type), caption), + Suggestion: fmt.Sprintf( + "pass the value on an object: declare `$%s: .` and read the "+ + "member inside the snippet, or move the primitive to the calling PAGE's "+ + "parameters and keep it out of the snippet.", p.Name), + }) + } + return out +} + +// paramTypeSourceName names a primitive parameter type the way MDL spells it, +// for the message. It is deliberately not the CE0046 caption: the author is +// looking for the word in their own script. +func paramTypeSourceName(dt ast.DataType) string { + switch dt.Kind { + case ast.TypeString: + return "String" + case ast.TypeInteger: + return "Integer" + case ast.TypeLong: + return "Long" + case ast.TypeDecimal: + return "Decimal" + case ast.TypeBoolean: + return "Boolean" + case ast.TypeDateTime: + return "DateTime" + default: + return "a primitive" + } +} diff --git a/mdl/executor/validate_snippet_parameters_test.go b/mdl/executor/validate_snippet_parameters_test.go new file mode 100644 index 0000000000..dbe0d042a7 --- /dev/null +++ b/mdl/executor/validate_snippet_parameters_test.go @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// MDL087 — mendixlabs/mxcli#1028. `mxcli check` has to carry this because +// nothing between the author and mxbuild types a snippet parameter: the +// reported script passed `check`, and the documented spelling is the one that +// fails. The rule and its measurements live in types.SnippetParameterTypeRule; +// this asserts the check reports them, on the same function exec refuses with. +func checkSnippetSource(t *testing.T, src string) []string { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse: %v", errs[0]) + } + var msgs []string + for _, stmt := range prog.Statements { + for _, v := range validateSnippetParameters(stmt) { + if v.RuleID != "MDL087" { + t.Errorf("unexpected rule %s", v.RuleID) + } + msgs = append(msgs, v.Message) + } + } + return msgs +} + +func TestValidateSnippetParameters(t *testing.T) { + cases := []struct { + name string + params string + want int + phrases []string + }{{ + // The reported repro, in the reporter's casing. + name: "the reported String parameter", + params: "$Label: string", + want: 1, + phrases: []string{"$Label", "CE0046", "Invalid data type 'String'"}, + }, { + // Long and Integer are one stored type, so both quote Studio Pro's + // combined caption — the message is meant to match a build log verbatim. + name: "Long reports the Integer/Long caption", + params: "$Count: Long", + want: 1, + phrases: []string{"Integer/Long"}, + }, { + name: "DateTime reports Mendix's own wording", + params: "$Due: DateTime", + want: 1, + phrases: []string{"Date and time"}, + }, { + // One violation per bad parameter: a clause with six of them names all + // six rather than one per run. + name: "every primitive in one clause", + params: "$Label: String, $Count: Long, $Rank: Integer, " + + "$Amount: Decimal, $Active: Boolean, $Due: DateTime", + want: 6, + }, { + // CONTROL: the form Mendix accepts. Measured at 0 errors on 11.13.0. + name: "an entity parameter is clean", + params: "$Order: Sales.Order", + want: 0, + }, { + // CONTROL: a quoted entity name is still an entity — the clause's other + // bug (visitor.TestSnippetParameter_QuotedEntityNameIsUnquoted) must not + // come back as a spurious MDL087. + name: "a quoted entity parameter is clean", + params: `$Order: Sales."Order"`, + want: 0, + }, { + name: "no parameters at all", + params: "", + want: 0, + }} + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + header := "" + if tc.params != "" { + header = "( Params: { " + tc.params + " } )" + } + msgs := checkSnippetSource(t, "CREATE SNIPPET M.S "+header+ + " { DYNAMICTEXT dt (Content: 'x') }") + if len(msgs) != tc.want { + t.Fatalf("got %d violations, want %d: %v", len(msgs), tc.want, msgs) + } + for _, phrase := range tc.phrases { + if !strings.Contains(msgs[0], phrase) { + t.Errorf("message does not contain %q: %s", phrase, msgs[0]) + } + } + }) + } +} + +// CONTROL: a PAGE with the same parameters produces nothing. MDL087 is about +// snippet parameters, and a rule that also fired on pages would break the +// documented, measured-valid page form. +func TestValidateSnippetParameters_IgnoresPages(t *testing.T) { + if msgs := checkSnippetSource(t, `CREATE PAGE M.P ( + Title: 'P', Params: { $Label: String, $Count: Long } +) { }`); len(msgs) != 0 { + t.Errorf("page flagged by MDL087: %v", msgs) + } +} diff --git a/mdl/grammar/domains/MDLPage.g4 b/mdl/grammar/domains/MDLPage.g4 index bdb8a6e406..573657ef0c 100644 --- a/mdl/grammar/domains/MDLPage.g4 +++ b/mdl/grammar/domains/MDLPage.g4 @@ -59,13 +59,11 @@ pageParameter : (IDENTIFIER | VARIABLE | QUOTED_IDENTIFIER) COLON dataType ; -snippetParameterList - : snippetParameter (COMMA snippetParameter)* - ; - -snippetParameter - : (IDENTIFIER | VARIABLE | QUOTED_IDENTIFIER) COLON dataType - ; +// A snippet parameter is a page parameter. There used to be a byte-identical +// `snippetParameterList` rule here with its own visitor, and the two drifted +// twice from the same clause: a quoted entity name reached the resolver with +// its quotes, and a primitive type was taken for an entity +// (mendixlabs/mxcli#1028). One rule, one conversion. variableDeclarationList : variableDeclaration (COMMA variableDeclaration)* @@ -246,7 +244,7 @@ snippetHeaderV3 ; snippetHeaderPropertyV3 - : PARAMS COLON LBRACE snippetParameterList RBRACE // Params: { $Customer: Entity } + : PARAMS COLON LBRACE pageParameterList RBRACE // Params: { $Customer: Module.Entity } — entities only (MDL087) | VARIABLES_KW COLON LBRACE variableDeclarationList RBRACE // Variables: { $show: Boolean = 'true' } | FOLDER COLON STRING_LITERAL // Folder: 'Snippets/Common' ; diff --git a/mdl/types/snippet_parameter_types.go b/mdl/types/snippet_parameter_types.go new file mode 100644 index 0000000000..f23012c76f --- /dev/null +++ b/mdl/types/snippet_parameter_types.go @@ -0,0 +1,55 @@ +// SPDX-License-Identifier: Apache-2.0 + +package types + +// SnippetParameterTypeRule answers the one question both `mxcli check` and the +// snippet writer have to answer about a snippet parameter's declared type: +// may it be a primitive? +// +// It may not. Measured on Mendix 11.13.0 against a project whose only content +// was the snippets below (so nothing else could be the cause): +// +// create snippet S.Label ( params: { $Label: string } ) +// → [error] [CE0046] "Invalid data type 'String'." at Snippet 'S.Label' +// create snippet S.All ( params: { $Label: String, $Count: Long, +// $Rank: Integer, $Amount: Decimal, +// $Active: Boolean, $Due: DateTime } ) +// → one CE0046 per parameter, naming Studio Pro's caption for each +// ('String', 'Integer/Long' twice, 'Decimal', 'Boolean', 'Date and time') +// create snippet S.Order ( params: { $Order: S.Order } ) +// → 0 errors +// +// And in the same run, the control that makes this a rule about SNIPPET +// parameters rather than about primitives: a PAGE declaring all six of those +// primitives as parameters builds at 0 errors. +// +// Storage does not encode the restriction — Forms$SnippetParameter's +// ParameterType is the polymorphic DataTypes$DataType, exactly as +// Forms$PageParameter's is (generated/metamodel: PagesSnippetParameter +// .ParameterType *DataTypesDataType), so a primitive serializes perfectly well. +// Only mxbuild's validator says no. That is why this has to live somewhere +// mxcli can consult it, and why a shape argument from the metamodel was not +// enough to settle it. +// +// Returned string is the Studio Pro caption mxbuild quotes in CE0046, so the +// message can be checked against a build log verbatim; "" means allowed. +func SnippetParameterTypeRule(bsonType string) (ce0046Caption string) { + switch bsonType { + case "DataTypes$StringType": + return "String" + case "DataTypes$IntegerType": + // Studio Pro's single type for Integer and Long; storage has no LongType. + return "Integer/Long" + case "DataTypes$DecimalType": + return "Decimal" + case "DataTypes$BooleanType": + return "Boolean" + case "DataTypes$DateTimeType": + return "Date and time" + default: + // "" is the entity case (no primitive $Type was resolved). An unmeasured + // primitive is not guessed at — it falls through to mxbuild, which is + // where the rule is actually enforced. + return "" + } +} diff --git a/mdl/visitor/visitor_page_v3.go b/mdl/visitor/visitor_page_v3.go index 86f6aa649c..9c7594e8b6 100644 --- a/mdl/visitor/visitor_page_v3.go +++ b/mdl/visitor/visitor_page_v3.go @@ -248,8 +248,8 @@ func (b *Builder) parseSnippetHeaderV3(ctx parser.ISnippetHeaderV3Context, stmt if prop.PARAMS() != nil { // Params: { $Customer: Entity, ... } - if paramList := prop.SnippetParameterList(); paramList != nil { - stmt.Parameters = buildSnippetParameterListAsPage(paramList) + if paramList := prop.PageParameterList(); paramList != nil { + stmt.Parameters = buildPageParameters(paramList) } } else if prop.VARIABLES_KW() != nil { // Variables: { $showStock: Boolean = 'true', ... } @@ -265,46 +265,13 @@ func (b *Builder) parseSnippetHeaderV3(ctx parser.ISnippetHeaderV3Context, stmt } } -// buildSnippetParameterListAsPage converts snippet parameters to page parameters. -func buildSnippetParameterListAsPage(ctx parser.ISnippetParameterListContext) []ast.PageParameter { - if ctx == nil { - return nil - } - listCtx := ctx.(*parser.SnippetParameterListContext) - var params []ast.PageParameter - - for _, sp := range listCtx.AllSnippetParameter() { - spCtx := sp.(*parser.SnippetParameterContext) - param := ast.PageParameter{} - - if id := spCtx.IDENTIFIER(); id != nil { - param.Name = id.GetText() - } else if v := spCtx.VARIABLE(); v != nil { - // VARIABLE token is $name, strip the $ prefix - param.Name = strings.TrimPrefix(v.GetText(), "$") - } else if qid := spCtx.QUOTED_IDENTIFIER(); qid != nil { - // Quoted name for reserved-keyword params, e.g. "List". See issue #114. - param.Name = unquoteIdentifier(qid.GetText()) - } - - // Walk the parse tree rather than re-splitting its TEXT. GetText() hands - // back the source verbatim, so a quoted entity name arrived as - // `Pd."Thing"` and exec failed with `entity not found: Pd."Thing"` — - // while the identical quoted form in a PAGE parameter resolved, because - // that path has always used buildQualifiedName (ako/CapTrackV4 019). The - // project convention is to quote every identifier, so this was reached by - // following the house style. - if dt := spCtx.DataType(); dt != nil { - if qn := dt.(*parser.DataTypeContext).QualifiedName(); qn != nil { - param.EntityType = buildQualifiedName(qn) - } - } - - params = append(params, param) - } - - return params -} +// A snippet's Params clause is the page's pageParameterList rule (they were +// byte-identical), so one conversion serves both. buildSnippetParameterListAsPage +// used to live here and drifted from buildPageParameters twice out of the same +// clause: it re-split the parse node's TEXT, so a quoted entity name reached +// the resolver as `Pd."Thing"`, and it never called buildDataType, so a +// primitive type was left for the executor to take for an entity name +// (mendixlabs/mxcli#1028 — "entity not found: string"). // buildVariableDeclarations builds variable declarations from the parse context. func buildVariableDeclarations(ctx parser.IVariableDeclarationListContext) []ast.PageVariable { From 7c3b2eedeff9ccc2a3e0c3a4e3c6b928104adb89 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 07:19:18 +0000 Subject: [PATCH 02/38] fix: accept CREATE WORKFLOW clauses in any order (ako/mxcli#586) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ON CREATED MICROFLOW` written in the wrong position failed to parse with `mismatched input 'ON' expecting ';'`, which names neither the clause nor the rule — and cascaded into up to six more errors, including a bogus `extraneous input 'END'`. The reporter worked the required order out empirically and wrote it into their notes. `createWorkflowStatement` and `workflowUserTaskStmt` were a fixed sequence of optional groups, so every clause was optional but its position was not. They are now a set: any order, each at most once, which is how the rest of MDL reads (ADR-0003). Relaxing the grammar alone would have mis-assigned every clause, because the visitor read qualified names by counting them (`names[1]` or `names[2]` for the overview page depending on whether PARAMETER was present) and strings by index. Each clause is now read off its own clause context. Two things the clause set had to keep from the sequence: - At most once. `page M.A page M.B` would otherwise parse with the second silently winning — a worse failure than the parse error it replaces. The rule is enforced in the visitor rather than the grammar, so the message can name the clause: `duplicate PAGE clause on user task Review (already given on line 12)`. The list-valued clauses — `outcomes`, `boundary event`, and the header's event handlers — still accumulate. - The multi-user vocabulary. The MULTI alternative keeps its own clause rule, so a single user task still refuses `participants`, `decide by` and `await all users`. `targeting microflow` and `targeting xpath` now count as one clause. A user task stores one UserSource, and the sequence grammar accepted both and let whichever was written last silently win. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qPaSqkSeaM4Ziuex4nxSG --- .../fix-issue/findings/mdl-grammar.jsonl | 1 + .../skills/mendix/write-workflows/SKILL.md | 31 +- cmd/mxcli/syntax/features_workflow.go | 27 +- docs/01-project/MDL_QUICK_REFERENCE.md | 13 +- .../workflow-586-clause-order-canonical.mdl | 121 +++++++ .../bug-tests/workflow-586-clause-order.mdl | 145 ++++++++ .../workflow-586-duplicate-clause.fail.mdl | 52 +++ mdl/grammar/domains/MDLWorkflow.g4 | 94 ++++-- mdl/visitor/visitor_workflow.go | 314 ++++++++--------- .../visitor_workflow_clause_order_test.go | 318 ++++++++++++++++++ mdl/visitor/visitor_workflow_clauses.go | 154 +++++++++ 11 files changed, 1072 insertions(+), 198 deletions(-) create mode 100644 mdl-examples/bug-tests/workflow-586-clause-order-canonical.mdl create mode 100644 mdl-examples/bug-tests/workflow-586-clause-order.mdl create mode 100644 mdl-examples/bug-tests/workflow-586-duplicate-clause.fail.mdl create mode 100644 mdl/visitor/visitor_workflow_clause_order_test.go create mode 100644 mdl/visitor/visitor_workflow_clauses.go diff --git a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl index 4c4b92ad02..e8b8412bd8 100644 --- a/.claude/skills/fix-issue/findings/mdl-grammar.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-grammar.jsonl @@ -58,3 +58,4 @@ {"area": "mdl/grammar", "date": "2026-09-15", "symptom": "Re-executing `describe workflow` output failed with `mismatched input 'boundary' expecting ';'` for any user task, call microflow or wait for notification that has two or more boundary events.", "cause": "formatBoundaryEvents emits `boundary event timer '…' { … }` per event (boundaryEventKeyword includes the prefix), and the syntax topic documents that per-clause form, but MDLWorkflow.g4 had `(BOUNDARY EVENT workflowBoundaryEventClause+)?` — one keyword, then clauses.", "fix": "All four sites accept `(BOUNDARY EVENT workflowBoundaryEventClause ((BOUNDARY EVENT)? workflowBoundaryEventClause)*)?`, so both the per-clause and the shared form parse; the visitor is unchanged.", "file": "mdl/grammar/domains/MDLWorkflow.g4", "insight": "A round trip that ends in `diff describe-1 describe-2` is vacuous when the exec in between fails: the second describe reads the unchanged document and matches. It reported IDENTICAL here while the exec had died on a parse error that a grep filter hid. Assert the exec itself — zero parse errors and a rewrite verb — before diffing. The integration round-trip tests had the same blind spot: they compare describe output but never feed it back to the parser, so a grammar/describer disagreement on a construct with more than one instance could not be seen. A test that re-parses describe output (TestWorkflowDescribe_TwoBoundaryEventsReparse) is the cheap guard."} {"area": "mdl/grammar", "date": "2026-09-18", "symptom": "Lint rule SEC005 reports \"strict mode is disabled\" and MDL has no statement that turns it on — the rule's own suggestion said \"not settable via MDL\". A lint rule with no remedy, recorded on the reporting project as the one finding left Open", "cause": "StrictMode was read everywhere and written nowhere: `security_read.go` reads it, `show security` prints it, the Starlark rule lints it, and `ProjectSecurity.SetStrictMode` existed in gen and was never called. `alterProjectSecurityStatement` had three variants (LEVEL, DEMO USERS, GUEST ACCESS) and no fourth", "file": "`mdl/grammar/MDLLexer.g4` + `domains/MDLSecurity.g4` + `domains/MDLSettings.g4` (keyword rule), `mdl/ast/ast_security.go`, `mdl/visitor/visitor_security.go`, `mdl/executor/cmd_security_write.go`, `mdl/backend/security.go`, `mdl/backend/modelsdk/security_write.go`, `mdl/backend/mock/mock_security.go`, `.claude/lint-rules/sec_strict_mode.star`", "insight": "**Writing a property gen merely offers is the trap; this is not one.** StrictMode is declared by BOTH generated sources and mxcli already reads it from real projects, which is the evidence that separates it from the Layout placeholder properties that make a document Studio Pro cannot open. **The AST field must be a POINTER** — a bare bool would disable strict mode on every DEMO USERS toggle, since \"said nothing\" and \"asked for off\" would be the same value (a test pins this). New tokens STRICT and MODE both go in the parser's `keyword` rule: `mode` is an entirely plausible attribute name and a keyword left out of that rule silently breaks every model already using the word (`TestKeywordRuleCoverage` catches it; a parse test pins it too). **Update the lint rule's suggestion in the same change** — a remedy that still says \"Studio Pro only\" leaves the finding exactly as unhelpful as before. No level-dependent refusal was added: the model stores StrictMode independently of SecurityLevel, and the rule already scopes its own advice to Production", "refs": ["#526"], "rules": ["SEC005"]} {"area": "mdl/grammar", "date": "2026-09-21", "symptom": "A new settings option list keyed on `IDENTIFIER` makes the feature's ONLY option a parse error: `alter settings workflows add group 'Approvers' (Description: '\u2026')` \u2192 \"mismatched input 'Description' expecting IDENTIFIER\"", "cause": "`Description` is an MDL lexer keyword (DESCRIPTION, from the security statements), so it never matches IDENTIFIER. The rule was copied from `languageOption`, whose keys (CheckCompleteness, CustomDateFormat\u2026) all happen to be plain identifiers \u2014 so the pattern looked safe and was not", "file": "`mdl/grammar/domains/MDLSettings.g4` (`settingsItemOption`) + `mdl/visitor/visitor_settings.go` (`collectSettingsItemOptions`)", "insight": "Any `( key: value )` option list must key on `identifierOrKeyword`, not IDENTIFIER, and the visitor must read it with `unquoteIdentifier(ctx.IdentifierOrKeyword().GetText())`. Before writing one, grep MDLLexer.g4 for each key you intend to accept \u2014 the check costs seconds and the failure lands on the single statement the feature exists for. Copying an existing option rule proves nothing about your key set. Control: reverting the rule to IDENTIFIER fails TestAlterSettings_WorkflowGroup with exactly that message. mendixlabs/mxcli#272", "refs": ["mendixlabs/mxcli#272"]} +{"area": "mdl/grammar", "date": "2026-09-22", "symptom": "A `create workflow` clause written in the \"wrong\" position is a parse error — `on created microflow` anywhere but between the targeting clauses and `entity` gives `line 6:4 mismatched input 'ON' expecting ';'`, and a header clause out of place gives `mismatched input 'DISPLAY' expecting {ON, BEGIN, EXPORT, DUE, OVERVIEW}`. Neither names the clause or the rule, and one misplaced clause cascades into 3–7 more errors including a bogus `extraneous input 'END'`. The reporter reverse-engineered the order empirically and wrote it into their notes", "cause": "`createWorkflowStatement` and `workflowUserTaskStmt` were a fixed SEQUENCE of optional groups — each clause optional, its POSITION not — and the VISITOR depended on that: it read qualified names by COUNTING (`names[1]` or `names[2]` for the overview page depending on whether PARAMETER was present; `nameIdx` walked page → targeting → on-created → entity) and strings by index off `AllSTRING_LITERAL()`. So the grammar could not simply be relaxed", "file": "`mdl/grammar/domains/MDLWorkflow.g4` (new `workflowHeaderClause`, `workflowUserTaskClause`, `workflowMultiUserTaskClause`), `mdl/visitor/visitor_workflow.go` (`applyWorkflowUserTaskClause`), `mdl/visitor/visitor_workflow_clauses.go` (`checkWorkflowClausesAtMostOnce`)", "insight": "**Positional reading is what makes a clause order load-bearing, so the grammar fix is a visitor fix.** The tell is `names[idx++]` in an exit-listener: the rule already carried a comment warning that reading strings by position had nearly mis-assigned FOLDER, and the same hazard had simply been left standing for qualified names. **A clause set must re-add the at-most-once rule the sequence gave for free**, or `page M.A page M.B` starts parsing with the second silently winning — a worse failure than the parse error it replaces. Enforce it in the visitor, not the grammar: only there can the message say `duplicate PAGE clause on user task Review (already given on line 12)`. **Two spellings that fill one model slot are ONE clause**: `targeting microflow` + `targeting xpath` were both accepted and the LAST one won, though a user task stores one UserSource — order-dependence in its most damaging form, and now a duplicate. **Keep the MULTI alternative's own clause rule rather than collapsing to `MULTI?`** — relaxing the order must not relax the vocabulary, or a single user task starts accepting `decide by`. **Control that settles it**: build a `bin/mxcli` from HEAD in a `git worktree`, exec the canonical-order script with it, and compare the written `.mxunit` against the fixed binary's output for BOTH orders — 6,038 bytes each, identical in every string ≥8 chars, differing only in the randomly minted element `$ID`s. AST `reflect.DeepEqual` between the two orders is the unit-level version of the same claim; both-parse is not enough, since a relaxed grammar over a positional visitor parses and mis-assigns. Found in passing and NOT fixed here: `create workflow … overview page X` writes nothing (`mdl/backend/modelsdk/workflow_write.go` has no `OverviewPage`), while `alter workflow … set overview page` does. Tests `mdl/visitor/visitor_workflow_clause_order_test.go`; repro `mdl-examples/bug-tests/workflow-586-clause-order.mdl` with its `-canonical.mdl` control and `-duplicate-clause.fail.mdl` sibling", "refs": ["ako/mxcli#586"]} diff --git a/.claude/skills/mendix/write-workflows/SKILL.md b/.claude/skills/mendix/write-workflows/SKILL.md index da0f9d7e1c..2e3327e610 100644 --- a/.claude/skills/mendix/write-workflows/SKILL.md +++ b/.claude/skills/mendix/write-workflows/SKILL.md @@ -24,9 +24,11 @@ the `LeaveRequest` being reviewed). User tasks render a page bound to ## Syntax — CREATE WORKFLOW -The header options are **order-sensitive** (parameter → display → description → -export level → overview page → due date → event handlers), and the body **must** -close with `END WORKFLOW`. +The header options may be written in **any order** — each at most once — and the +body **must** close with `END WORKFLOW`. (They used to be order-sensitive, in +exactly the sequence below; a clause written out of place failed with +`mismatched input 'DISPLAY' expecting {ON, BEGIN, EXPORT, DUE, OVERVIEW}`, which +named neither the clause nor the rule. See `ako/mxcli#586`.) ```sql create workflow Module.ApprovalFlow @@ -43,6 +45,22 @@ begin end workflow; ``` +**Clause order does not matter, but repetition is refused.** A workflow's header +clauses and a user task's clauses are a **set**: any order, each **at most +once**. Writing one twice is reported by name — + +``` +line 5:2: duplicate DISPLAY clause on workflow Module.ApprovalFlow + (already given on line 4) — each clause may appear at most once, in any order +``` + +Three clauses are list-valued and accumulate instead: the header's +`on workflow event(s)` handlers, and a task's `outcomes` and `boundary event`. +The two `targeting` spellings count as **one** clause — a user task stores one +user source — so `targeting microflow …` and `targeting xpath …` on the same +task is a duplicate, not two clauses. It used to be accepted, with the one +written **last** silently winning. + **Two gotchas that trip up first attempts:** - `PARAMETER` takes a **`$`-variable then a context entity**: `parameter $Context: @@ -419,10 +437,9 @@ values. The full list and the System **entities** are in `system-module`. An outcome left **empty** does not stop anything — it rejoins the main flow. `comment '…'` sets the End's caption, as on every workflow activity. -- **A multi-user task says who must respond and how their outcomes decide**, - in this clause order before `outcomes`: - `participants all | | percent`, then `decide by …`, then - `await all users`. The rules (`decide by`): +- **A multi-user task says who must respond and how their outcomes decide**: + `participants all | | percent`, `decide by …` and `await all users`, + in any order (see the clause-order note below). The rules (`decide by`): `consensus fallback ''`, `majority more than half fallback '…'`, `majority most chosen fallback '…'`, `threshold percent|votes fallback '…'`, `veto ''`, `microflow Module.Decide`. Omitted means all participants, diff --git a/cmd/mxcli/syntax/features_workflow.go b/cmd/mxcli/syntax/features_workflow.go index 74b08961fe..699660f841 100644 --- a/cmd/mxcli/syntax/features_workflow.go +++ b/cmd/mxcli/syntax/features_workflow.go @@ -32,7 +32,23 @@ func init() { "create workflow", "new workflow", "define workflow", "parameter", "overview page", "due date", }, - Syntax: "CREATE [OR MODIFY] WORKFLOW Module.Name\n [FOLDER 'path']\n PARAMETER $Context: Module.Entity\n [OVERVIEW PAGE Module.OverviewPage]\n [DUE DATE '']\n [ON WORKFLOW EVENTS (, ...) MICROFLOW Module.Handler [AS '']]...\n [ON ANY WORKFLOW EVENT MICROFLOW Module.Handler [AS '']]...\nBEGIN\n \nEND WORKFLOW;", + Syntax: "CREATE [OR MODIFY] WORKFLOW Module.Name\n" + + " [FOLDER 'path']\n" + + " PARAMETER $Context: Module.Entity\n" + + " [DISPLAY '']\n" + + " [DESCRIPTION '']\n" + + " [EXPORT LEVEL Hidden | API]\n" + + " [OVERVIEW PAGE Module.OverviewPage]\n" + + " [DUE DATE '']\n" + + " [ON WORKFLOW EVENTS (, ...) MICROFLOW Module.Handler [AS '']]...\n" + + " [ON ANY WORKFLOW EVENT MICROFLOW Module.Handler [AS '']]...\n" + + "BEGIN\n \nEND WORKFLOW;\n\n" + + "-- The header clauses are a SET: write them in ANY order, each at most\n" + + "-- once. (Before ako/mxcli#586 the order above was mandatory and writing\n" + + "-- one out of place was a token error naming neither the clause nor the\n" + + "-- rule.) The event handlers are the exception and may repeat.\n" + + "-- A clause written twice is reported by name, e.g.\n" + + "-- duplicate DISPLAY clause on workflow M.W (already given on line 3)", Example: "CREATE WORKFLOW Module.ApprovalFlow\n PARAMETER $Context: Module.Request\n OVERVIEW PAGE Module.WF_Overview\nBEGIN\n USER TASK ReviewTask 'Review the request'\n PAGE Module.ReviewPage\n OUTCOMES 'Approve' { } 'Reject' { };\nEND WORKFLOW;", SeeAlso: []string{"workflow.user-task", "workflow.event-handlers", "workflow.decision", "workflow.drop"}, }) @@ -109,7 +125,11 @@ func init() { " [ENTITY Module.Entity]\n" + " [DUE DATE '']\n" + " [DESCRIPTION '']\n" + - " OUTCOMES '' { } '' { };\n\n" + + " OUTCOMES '' { } '' { };\n\n" + "-- The clauses are a SET: write them in ANY order, each at most once\n" + + "-- (ako/mxcli#586). The two TARGETING spellings are ONE clause — a task\n" + + "-- stores one user source — so writing both is refused rather than\n" + + "-- letting the second silently win. OUTCOMES and BOUNDARY EVENT are\n" + + "-- list-valued and may repeat.\n\n" + "-- The task page is opened with the TASK, not with the workflow's context\n" + "-- object, so it must take a System.WorkflowUserTask parameter:\n" + "-- page with no parameters -> CE7410\n" + @@ -187,7 +207,8 @@ func init() { "-- veto needs its outcome (CE1867); both must name one of the task's outcomes\n" + "-- (MDL-WF13). Omitted: all participants, consensus on the first outcome, not\n" + "-- waiting. The build does not range-check thresholds or participant counts.\n" + - "-- Same page and targeting rules as USER TASK.", + "-- Same page and targeting rules as USER TASK, and the same clause rule:\n" + + "-- ANY order, each at most once (ako/mxcli#586).", Example: "MULTI USER TASK ExpertAssessment 'Expert assessment'\n PAGE MOC.AssessmentPage\n TARGETING MICROFLOW MOC.GetAssessors\n PARTICIPANTS 80 PERCENT\n DECIDE BY THRESHOLD 60 PERCENT FALLBACK 'Reject'\n AWAIT ALL USERS\n OUTCOMES 'Approve' { } 'Reject' { };", SeeAlso: []string{"workflow.user-task", "workflow.user-task.targeting"}, }) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index b3ab7b267c..ef2bdd0167 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -673,8 +673,19 @@ Nested folders use `/` separator: `'Parent/Child/Grandchild'`. Missing folders a | Create workflow | `create [or modify] workflow Module.Name [folder 'path'] parameter $Ctx: Module.Entity [on workflow events (, ...) microflow Mod.MF [as '']] [on any workflow event microflow Mod.MF [as '']] begin ... end workflow;` | See activity types and event handlers below | | Drop workflow | `drop workflow Module.Name;` | | +**Clause order does not matter.** A workflow's header clauses and a user task's +clauses are a **set**: write them in any order, each **at most once**. A clause +written twice is reported by name (`duplicate PAGE clause on user task Review +(already given on line 12)`). The exceptions are the list-valued ones, which +accumulate: the header's `on workflow event(s)` handlers, and a task's +`outcomes` and `boundary event`. The two `targeting` spellings are **one** +clause — a task stores one user source — so writing both is refused rather than +letting the second silently win. Before `ako/mxcli#586` the order below was +mandatory and a misplaced clause failed with a token error +(`mismatched input 'ON' expecting ';'`) that named neither the clause nor the rule. + **Workflow Activity Types:** -- `[multi] user task '' [page Mod.Page] [targeting [users|groups] microflow Mod.MF] [targeting [users|groups] xpath ''] [on created microflow Mod.MF] [participants all|| percent] [decide by ] [await all users] [outcomes '' { } ...];` +- `[multi] user task '' [page Mod.Page] [targeting [users|groups] microflow Mod.MF] [targeting [users|groups] xpath ''] [on created microflow Mod.MF] [entity Mod.Entity] [due date ''] [description ''] [participants all|| percent] [decide by ] [await all users] [outcomes '' { } ...] [boundary event …];` - **Multi-user only:** `decide by consensus|majority more than half|majority most chosen|threshold percent|votes fallback ''`, `decide by veto ''`, `decide by microflow Mod.MF`. A fallback is required for consensus, majority and threshold (CE1866), a veto needs its outcome (CE1867), and a decision microflow returns String (CE5012) — all `MDL-WF13` / check. Omitted: all participants, consensus on the first outcome, not waiting. - The **task page** must take a `System.WorkflowUserTask` parameter — none at all is CE7410, none of that type is CE7412; extra parameters are allowed. - A **targeting microflow** takes exactly `System.Workflow` + the context entity (or a generalization of it), in either order — anything else is CE6677. Users targeting returns a list of `System.User`, groups a list of `System.WorkflowGroup`. diff --git a/mdl-examples/bug-tests/workflow-586-clause-order-canonical.mdl b/mdl-examples/bug-tests/workflow-586-clause-order-canonical.mdl new file mode 100644 index 0000000000..caaa7e8322 --- /dev/null +++ b/mdl-examples/bug-tests/workflow-586-clause-order-canonical.mdl @@ -0,0 +1,121 @@ +-- ako/mxcli#586 — the CONTROL for `workflow-586-clause-order.mdl`. +-- +-- Same two workflows, every clause in the order the old sequence grammar +-- demanded: parameter -> display -> description -> export level -> overview +-- page -> due date -> events, and within a user task page -> targeting -> on +-- created -> entity -> due date -> description -> outcomes -> boundary event. +-- +-- This file parsed BEFORE the fix and its sibling did not. Executing the two +-- against the same project must produce the same model: run one, then the +-- other, and the second reports `Unchanged workflow: WF586.Review` — the +-- write-elision of ADR-0008 used as the comparator. A shuffled order that +-- landed a qualified name on the wrong field would report `Replaced` instead. + +create module WF586; + +create entity WF586.Request ( + Status : string(200) +); + +create page WF586.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: { $WorkflowUserTask: System.WorkflowUserTask } +) { + layoutgrid g1 { + row r1 { + column c1 (desktopwidth: 12) { + dynamictext txt1 (content: 'Review the request', rendermode: H2) + } + } + } +} +/ + +create page WF586.Overview ( + title: 'Overview', + layout: Atlas_Core.Atlas_Default +) { + layoutgrid g1 { + row r1 { + column c1 (desktopwidth: 12) { + dynamictext txt1 (content: 'Requests', rendermode: H2) + } + } + } +} +/ + +-- Targeting microflow: (System.Workflow, context) -> List of System.User +create microflow WF586.ACT_GetUsers ( + $Workflow : System.Workflow, + $Context : WF586.Request +) +returns list of System.User as $Users +begin + @position(200,200) + retrieve $Users from System.User; + @position(400,200) return $Users; +end; +/ + +create microflow WF586.ACT_OnCreated ( + $WorkflowUserTask : System.WorkflowUserTask, + $WorkflowContext : WF586.Request +) +begin + @position(200,200) declare $Message String = 'task created'; +end; +/ + +create microflow WF586.ACT_OnEvent ( + $WorkflowEvent : System.WorkflowEvent, + $WorkflowRecord : System.WorkflowRecord, + $WorkflowActivityRecord : System.WorkflowActivityRecord +) +begin + @position(200,200) declare $Message String = 'workflow event'; +end; +/ + +create microflow WF586.ACT_Audit ( + $WorkflowContext : WF586.Request +) +begin + @position(200,200) declare $Message String = 'audited'; +end; +/ + +create or replace workflow WF586.Review + folder 'Flows' + parameter $WorkflowContext: WF586.Request + display 'Request review' + description 'Reviews a request' + export level Hidden + overview page WF586.Overview + due date '${P2D}' + on any workflow event microflow WF586.ACT_OnEvent as 'all events' +begin + user task Review 'Review the request' + page WF586.TaskPage + targeting microflow WF586.ACT_GetUsers + on created microflow WF586.ACT_OnCreated + entity WF586.Request + due date '${PT4H}' + description 'Check the request and decide' + outcomes + 'Approve' { call microflow WF586.ACT_Audit; } + 'Reject' { } + boundary event interrupting timer '${PT8H}'; + + multi user task Confirm 'Confirm the decision' + page WF586.TaskPage + targeting groups xpath '[Name = ''Administrator'']' + due date '${PT2H}' + participants 60 percent + decide by majority more than half fallback 'Confirmed' + await all users + outcomes + 'Confirmed' { }; +end workflow; +/ diff --git a/mdl-examples/bug-tests/workflow-586-clause-order.mdl b/mdl-examples/bug-tests/workflow-586-clause-order.mdl new file mode 100644 index 0000000000..a931f66b58 --- /dev/null +++ b/mdl-examples/bug-tests/workflow-586-clause-order.mdl @@ -0,0 +1,145 @@ +-- ako/mxcli#586 — CREATE WORKFLOW clause order was load-bearing, and the parse +-- error did not say so. +-- +-- The grammar was a fixed SEQUENCE of optional clauses, so every clause was +-- optional but its POSITION was not. `on created microflow` written anywhere +-- but between the targeting clauses and `entity` failed with +-- +-- line 6:4 mismatched input 'ON' expecting ';' +-- +-- which names neither the clause nor the rule. The reporter worked the order +-- out empirically and wrote it into their notes — "parameter -> display -> +-- description -> export level -> overview page -> due date -> events, and +-- within a user task: page -> targeting -> on created -> description -> +-- outcomes". That is not something a language should ask of its users, and it +-- sat oddly beside the rest of MDL, where properties go in an order-free block +-- (ADR-0003). +-- +-- Both the header clauses and a user task's clauses are now a SET: any order, +-- each at most once. The at-most-once half is enforced by the visitor rather +-- than the grammar, so the message can name the clause +-- (`duplicate PAGE clause on user task Review …`) instead of pointing at a +-- token. The two `targeting` spellings count as ONE clause, because a user task +-- stores one UserSource — under the old grammar both were accepted and the one +-- written last silently won. +-- +-- Verified: this script and `workflow-586-clause-order-canonical.mdl` (the same +-- workflows with every clause in the old mandatory order) produce the SAME +-- model — re-running either over the other's result reports `Unchanged +-- workflow`, which is the idempotence machinery of ADR-0008 used as the +-- comparator. The `.fail.mdl` sibling holds the duplicate-clause cases. + +create module WF586; + +create entity WF586.Request ( + Status : string(200) +); + +create page WF586.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: { $WorkflowUserTask: System.WorkflowUserTask } +) { + layoutgrid g1 { + row r1 { + column c1 (desktopwidth: 12) { + dynamictext txt1 (content: 'Review the request', rendermode: H2) + } + } + } +} +/ + +create page WF586.Overview ( + title: 'Overview', + layout: Atlas_Core.Atlas_Default +) { + layoutgrid g1 { + row r1 { + column c1 (desktopwidth: 12) { + dynamictext txt1 (content: 'Requests', rendermode: H2) + } + } + } +} +/ + +-- Targeting microflow: (System.Workflow, context) -> List of System.User +create microflow WF586.ACT_GetUsers ( + $Workflow : System.Workflow, + $Context : WF586.Request +) +returns list of System.User as $Users +begin + @position(200,200) + retrieve $Users from System.User; + @position(400,200) return $Users; +end; +/ + +create microflow WF586.ACT_OnCreated ( + $WorkflowUserTask : System.WorkflowUserTask, + $WorkflowContext : WF586.Request +) +begin + @position(200,200) declare $Message String = 'task created'; +end; +/ + +create microflow WF586.ACT_OnEvent ( + $WorkflowEvent : System.WorkflowEvent, + $WorkflowRecord : System.WorkflowRecord, + $WorkflowActivityRecord : System.WorkflowActivityRecord +) +begin + @position(200,200) declare $Message String = 'workflow event'; +end; +/ + +create microflow WF586.ACT_Audit ( + $WorkflowContext : WF586.Request +) +begin + @position(200,200) declare $Message String = 'audited'; +end; +/ + +-- The header clauses in a deliberately different order from the one the old +-- grammar demanded: the event handler first, `folder` last. +create or replace workflow WF586.Review + on any workflow event microflow WF586.ACT_OnEvent as 'all events' + due date '${P2D}' + description 'Reviews a request' + overview page WF586.Overview + display 'Request review' + parameter $WorkflowContext: WF586.Request + export level Hidden + folder 'Flows' +begin + -- A user task with `on created microflow` FIRST — the position that used to + -- be a parse error. + user task Review 'Review the request' + on created microflow WF586.ACT_OnCreated + description 'Check the request and decide' + boundary event interrupting timer '${PT8H}' + entity WF586.Request + outcomes + 'Approve' { call microflow WF586.ACT_Audit; } + 'Reject' { } + due date '${PT4H}' + page WF586.TaskPage + targeting microflow WF586.ACT_GetUsers; + + -- The multi-user clauses are order-free too, including the three that only a + -- multi user task has. + multi user task Confirm 'Confirm the decision' + await all users + outcomes + 'Confirmed' { } + decide by majority more than half fallback 'Confirmed' + page WF586.TaskPage + participants 60 percent + targeting groups xpath '[Name = ''Administrator'']' + due date '${PT2H}'; +end workflow; +/ diff --git a/mdl-examples/bug-tests/workflow-586-duplicate-clause.fail.mdl b/mdl-examples/bug-tests/workflow-586-duplicate-clause.fail.mdl new file mode 100644 index 0000000000..d2e5d36c6c --- /dev/null +++ b/mdl-examples/bug-tests/workflow-586-duplicate-clause.fail.mdl @@ -0,0 +1,52 @@ +-- ako/mxcli#586 — the negative half of `workflow-586-clause-order.mdl`. +-- +-- Making the clauses order-free must not also make them repeatable. The old +-- sequence grammar allowed each clause at most once; a clause list that took +-- them in any order would accept `page M.A page M.B` and let the second one +-- silently win, which is a worse failure than the parse error it replaced. +-- +-- So the at-most-once rule is enforced in the visitor, where the message can +-- name the clause and the line the first one was on, rather than by the +-- grammar, which can only point at a token. Every statement below must be +-- refused by `mxcli check`. +-- +-- Three clauses are deliberately exempt because they are list-valued and +-- accumulate: `outcomes`, `boundary event`, and the header's workflow event +-- handlers. + +create module WF586Dup; + +-- A header clause written twice. +create workflow WF586Dup.HeaderTwice + display 'A' + display 'B' +begin + user task T 'c' outcomes 'Done' { }; +end workflow; +/ + +-- A user task clause written twice, with another clause between them, so the +-- report cannot be a matter of the two sitting next to each other. +create workflow WF586Dup.TaskTwice +begin + user task T 'c' + on created microflow WF586Dup.ACT_A + description 'between' + on created microflow WF586Dup.ACT_B + outcomes 'Done' { }; +end workflow; +/ + +-- `targeting microflow` and `targeting xpath` fill the SAME slot — a user task +-- stores one UserSource — so writing both is a duplicate, not two clauses. +-- Under the old grammar both were accepted and the one written LAST won, which +-- is the order-dependence of #586 in its most damaging form: the task ends up +-- targeted by whichever clause the author happened to put second. +create workflow WF586Dup.TwoTargetings +begin + user task T 'c' + targeting microflow WF586Dup.ACT_A + targeting xpath '[true()]' + outcomes 'Done' { }; +end workflow; +/ diff --git a/mdl/grammar/domains/MDLWorkflow.g4 b/mdl/grammar/domains/MDLWorkflow.g4 index e911213c59..a03f3e8336 100644 --- a/mdl/grammar/domains/MDLWorkflow.g4 +++ b/mdl/grammar/domains/MDLWorkflow.g4 @@ -14,22 +14,37 @@ options { tokenVocab = MDLLexer; } */ createWorkflowStatement : WORKFLOW qualifiedName - (FOLDER folder=STRING_LITERAL)? - (PARAMETER VARIABLE COLON qualifiedName)? - (DISPLAY display=STRING_LITERAL)? - (DESCRIPTION description=STRING_LITERAL)? - // HIDDEN_KW is listed beside IDENTIFIER because `Hidden` used to lex as an - // identifier and stopped when the microflow clauses made it a keyword. - // Anything matching a bare IDENTIFIER here is one token away from the same - // break — the hazard `identifierOrKeyword` exists to absorb, which this - // rule bypasses by taking IDENTIFIER directly. - (EXPORT LEVEL (IDENTIFIER | API | HIDDEN_KW))? - (OVERVIEW PAGE qualifiedName)? - (DUE DATE_TYPE dueDate=STRING_LITERAL)? - workflowEventHandlerClause* + workflowHeaderClause* BEGIN workflowMainBody workflowEventSubProcess* END WORKFLOW SEMICOLON? SLASH? ; +/** + * One header clause. The clauses used to be a fixed SEQUENCE of optional + * groups, so each was optional but its POSITION was not: `display` after + * `description` failed with `mismatched input 'DISPLAY' expecting {ON, BEGIN, + * EXPORT, DUE, OVERVIEW}`, which names neither the clause nor the rule, and the + * author had to reverse-engineer the order from the failure. They are now a + * set — any order, each at most once, which is how the rest of MDL reads + * (ADR-0003). The at-most-once half is NOT in the grammar: a repeated clause is + * reported by `checkWorkflowClausesAtMostOnce` in the visitor, which can name + * the clause instead of pointing at a token. See ako/mxcli#586. + */ +workflowHeaderClause + : FOLDER folder=STRING_LITERAL + | PARAMETER VARIABLE COLON qualifiedName + | DISPLAY display=STRING_LITERAL + | DESCRIPTION description=STRING_LITERAL + // HIDDEN_KW is listed beside IDENTIFIER because `Hidden` used to lex as an + // identifier and stopped when the microflow clauses made it a keyword. + // Anything matching a bare IDENTIFIER here is one token away from the same + // break — the hazard `identifierOrKeyword` exists to absorb, which this + // rule bypasses by taking IDENTIFIER directly. + | EXPORT LEVEL (IDENTIFIER | API | HIDDEN_KW) + | OVERVIEW PAGE qualifiedName + | DUE DATE_TYPE dueDate=STRING_LITERAL + | workflowEventHandlerClause + ; + /** * An event sub-process: a flow outside the main flow that its own start event * triggers while the workflow runs — a `notify workflow … target `, or a @@ -128,30 +143,41 @@ workflowActivityName | QUOTED_IDENTIFIER ; +/** + * A user task. Its clauses are a SET, not a sequence — see + * `workflowHeaderClause` for why, and `checkWorkflowClausesAtMostOnce` for the + * half of the old rule the grammar no longer carries. + * + * The two alternatives keep their own clause rules rather than collapsing into + * `MULTI?`, so a single-user task still refuses `participants`, `decide by` and + * `await all users` — relaxing the ORDER must not also relax the vocabulary. + */ workflowUserTaskStmt : USER TASK (IDENTIFIER | QUOTED_IDENTIFIER) STRING_LITERAL - (PAGE qualifiedName)? - (TARGETING (USERS | GROUPS)? MICROFLOW qualifiedName)? - (TARGETING (USERS | GROUPS)? XPATH STRING_LITERAL)? - (ON CREATED MICROFLOW qualifiedName)? - (ENTITY qualifiedName)? - (DUE DATE_TYPE STRING_LITERAL)? - (DESCRIPTION STRING_LITERAL)? - (OUTCOMES workflowUserTaskOutcome+)? - (BOUNDARY EVENT workflowBoundaryEventClause ((BOUNDARY EVENT)? workflowBoundaryEventClause)*)? + workflowUserTaskClause* | MULTI USER TASK (IDENTIFIER | QUOTED_IDENTIFIER) STRING_LITERAL - (PAGE qualifiedName)? - (TARGETING (USERS | GROUPS)? MICROFLOW qualifiedName)? - (TARGETING (USERS | GROUPS)? XPATH STRING_LITERAL)? - (ON CREATED MICROFLOW qualifiedName)? - (ENTITY qualifiedName)? - (DUE DATE_TYPE STRING_LITERAL)? - (DESCRIPTION STRING_LITERAL)? - workflowParticipantsClause? - workflowCompletionClause? - (AWAIT ALL USERS)? - (OUTCOMES workflowUserTaskOutcome+)? - (BOUNDARY EVENT workflowBoundaryEventClause ((BOUNDARY EVENT)? workflowBoundaryEventClause)*)? + workflowMultiUserTaskClause* + ; + +/** A clause every user task accepts. */ +workflowUserTaskClause + : PAGE qualifiedName + | TARGETING (USERS | GROUPS)? MICROFLOW qualifiedName + | TARGETING (USERS | GROUPS)? XPATH STRING_LITERAL + | ON CREATED MICROFLOW qualifiedName + | ENTITY qualifiedName + | DUE DATE_TYPE STRING_LITERAL + | DESCRIPTION STRING_LITERAL + | OUTCOMES workflowUserTaskOutcome+ + | BOUNDARY EVENT workflowBoundaryEventClause ((BOUNDARY EVENT)? workflowBoundaryEventClause)* + ; + +/** The above, plus the three clauses only a multi user task has. */ +workflowMultiUserTaskClause + : workflowUserTaskClause + | workflowParticipantsClause + | workflowCompletionClause + | AWAIT ALL USERS ; /** diff --git a/mdl/visitor/visitor_workflow.go b/mdl/visitor/visitor_workflow.go index 4c568f2ec6..8d688f3981 100644 --- a/mdl/visitor/visitor_workflow.go +++ b/mdl/visitor/visitor_workflow.go @@ -14,92 +14,86 @@ import ( // ExitCreateWorkflowStatement handles CREATE WORKFLOW statements. func (b *Builder) ExitCreateWorkflowStatement(ctx *parser.CreateWorkflowStatementContext) { - names := ctx.AllQualifiedName() - if len(names) == 0 { + name := ctx.QualifiedName() + if name == nil { return } stmt := &ast.CreateWorkflowStmt{ - Name: buildQualifiedName(names[0]), - } - - // Parse PARAMETER $Var: Entity - if ctx.PARAMETER() != nil && ctx.VARIABLE() != nil { - stmt.ParameterVar = ctx.VARIABLE().GetText() - // The parameter entity is the second qualified name - if len(names) > 1 { - stmt.ParameterEntity = buildQualifiedName(names[1]) + Name: buildQualifiedName(name), + } + + // Header clauses are a SET, not a sequence (ako/mxcli#586): each one is read + // off its own clause context, so nothing here depends on the order they were + // written in, and no clause's qualified name or string can be mistaken for + // another's. The rule this replaces read `names[1]` or `names[2]` for the + // overview page depending on whether PARAMETER was present — which is the + // shape of reading that made the order load-bearing in the first place. + b.checkWorkflowClausesAtMostOnce(ctx) + for _, clause := range ctx.AllWorkflowHeaderClause() { + hc, ok := clause.(*parser.WorkflowHeaderClauseContext) + if !ok { + continue } - } - - // Each optional string clause is read by its grammar LABEL, not by counting - // STRING_LITERALs. The positional version worked only while the clauses - // happened to be the rule's only strings: adding the FOLDER clause would - // have made allStrings[0] the folder path whenever one was given, so a - // foldered workflow would have silently taken its display name from it. - if tok := ctx.GetFolder(); tok != nil { - stmt.Folder = unquoteString(tok.GetText()) - } - if tok := ctx.GetDisplay(); tok != nil { - stmt.DisplayName = unquoteString(tok.GetText()) - } - if tok := ctx.GetDescription(); tok != nil { - stmt.Description = unquoteString(tok.GetText()) - } - - // EXPORT LEVEL (Identifier | API | Hidden) - // - // HIDDEN_KW is read alongside IDENTIFIER because `Hidden` was an ordinary - // identifier here until the microflow header clauses made it a keyword — at - // which point this read silently produced "" and three tests caught it. Any - // rule taking a bare IDENTIFIER for a fixed vocabulary has the same fragility. - if ctx.EXPORT() != nil && ctx.LEVEL() != nil { switch { - case ctx.IDENTIFIER() != nil: - stmt.ExportLevel = ctx.IDENTIFIER().GetText() - case ctx.HIDDEN_KW() != nil: - stmt.ExportLevel = ctx.HIDDEN_KW().GetText() - case ctx.API() != nil: - stmt.ExportLevel = "API" - } - } - - // Parse OVERVIEW PAGE QualifiedName - overviewPageIdx := -1 - if ctx.OVERVIEW() != nil && ctx.PAGE() != nil { - // Find the overview page qualified name - // It's either names[1] or names[2] depending on whether PARAMETER was present - startIdx := 1 - if ctx.PARAMETER() != nil { - startIdx = 2 - } - if len(names) > startIdx { - stmt.OverviewPage = buildQualifiedName(names[startIdx]) - overviewPageIdx = startIdx - } - } - _ = overviewPageIdx - - // Parse DUE DATE 'expression' - if tok := ctx.GetDueDate(); tok != nil { - stmt.DueDate = unquoteString(tok.GetText()) - } - - // Workflow event handlers: each clause carries its own qualified name, so - // they do not shift the header's name indices above. - for _, hc := range ctx.AllWorkflowEventHandlerClause() { - h := hc.(*parser.WorkflowEventHandlerClauseContext) - node := ast.WorkflowEventHandlerNode{AnyEvent: h.ANY() != nil} - if qn := h.QualifiedName(); qn != nil { - node.Microflow = buildQualifiedName(qn) - } - for _, id := range h.AllIDENTIFIER() { - node.EventTypes = append(node.EventTypes, id.GetText()) - } - if s := h.STRING_LITERAL(); s != nil { - node.Description = unquoteString(s.GetText()) + case hc.FOLDER() != nil: + if tok := hc.GetFolder(); tok != nil { + stmt.Folder = unquoteString(tok.GetText()) + } + case hc.PARAMETER() != nil: + if v := hc.VARIABLE(); v != nil { + stmt.ParameterVar = v.GetText() + } + if qn := hc.QualifiedName(); qn != nil { + stmt.ParameterEntity = buildQualifiedName(qn) + } + case hc.DISPLAY() != nil: + if tok := hc.GetDisplay(); tok != nil { + stmt.DisplayName = unquoteString(tok.GetText()) + } + case hc.DESCRIPTION() != nil: + if tok := hc.GetDescription(); tok != nil { + stmt.Description = unquoteString(tok.GetText()) + } + case hc.EXPORT() != nil && hc.LEVEL() != nil: + // HIDDEN_KW is read alongside IDENTIFIER because `Hidden` was an + // ordinary identifier here until the microflow header clauses made it + // a keyword — at which point this read silently produced "" and three + // tests caught it. Any rule taking a bare IDENTIFIER for a fixed + // vocabulary has the same fragility. + switch { + case hc.IDENTIFIER() != nil: + stmt.ExportLevel = hc.IDENTIFIER().GetText() + case hc.HIDDEN_KW() != nil: + stmt.ExportLevel = hc.HIDDEN_KW().GetText() + case hc.API() != nil: + stmt.ExportLevel = "API" + } + case hc.OVERVIEW() != nil && hc.PAGE() != nil: + if qn := hc.QualifiedName(); qn != nil { + stmt.OverviewPage = buildQualifiedName(qn) + } + case hc.DUE() != nil && hc.DATE_TYPE() != nil: + if tok := hc.GetDueDate(); tok != nil { + stmt.DueDate = unquoteString(tok.GetText()) + } + case hc.WorkflowEventHandlerClause() != nil: + h, ok := hc.WorkflowEventHandlerClause().(*parser.WorkflowEventHandlerClauseContext) + if !ok { + continue + } + node := ast.WorkflowEventHandlerNode{AnyEvent: h.ANY() != nil} + if qn := h.QualifiedName(); qn != nil { + node.Microflow = buildQualifiedName(qn) + } + for _, id := range h.AllIDENTIFIER() { + node.EventTypes = append(node.EventTypes, id.GetText()) + } + if str := h.STRING_LITERAL(); str != nil { + node.Description = unquoteString(str.GetText()) + } + stmt.EventHandlers = append(stmt.EventHandlers, node) } - stmt.EventHandlers = append(stmt.EventHandlers, node) } // Parse CREATE OR MODIFY @@ -133,6 +127,10 @@ func (b *Builder) exitAlterWorkflowStatement(ctx *parser.AlterStatementContext) Name: buildQualifiedName(qn), } + // ALTER's INSERT/REPLACE ACTIVITY take a whole workflow activity, user tasks + // included, so the at-most-once rule has to be applied here as well. + b.checkWorkflowClausesAtMostOnce(ctx) + for _, actionCtx := range ctx.AllAlterWorkflowAction() { op := buildAlterWorkflowAction(actionCtx.(*parser.AlterWorkflowActionContext)) if op != nil { @@ -516,6 +514,12 @@ func buildWorkflowActivityStmt(ctx parser.IWorkflowActivityStmtContext) ast.Work } // buildWorkflowUserTask builds a WorkflowUserTaskNode from the grammar context. +// +// Clauses are a SET (ako/mxcli#586), so each is read off its own clause context +// in source order rather than by counting the statement's qualified names and +// string literals. The counting version is what made the order load-bearing: +// `on created microflow` had to sit between the targeting clauses and `entity` +// or its qualified name landed on a different field. func buildWorkflowUserTask(ctx parser.IWorkflowUserTaskStmtContext) *ast.WorkflowUserTaskNode { utCtx := ctx.(*parser.WorkflowUserTaskStmtContext) @@ -529,97 +533,101 @@ func buildWorkflowUserTask(ctx parser.IWorkflowUserTaskStmtContext) *ast.Workflo } node := &ast.WorkflowUserTaskNode{ - Name: taskName, - IsMultiUser: utCtx.MULTI() != nil, - AwaitAllUsers: utCtx.AWAIT() != nil, - } - if pc, ok := utCtx.WorkflowParticipantsClause().(*parser.WorkflowParticipantsClauseContext); ok && pc != nil { - node.Participants = buildWorkflowParticipants(pc) - } - if cc, ok := utCtx.WorkflowCompletionClause().(*parser.WorkflowCompletionClauseContext); ok && cc != nil { - node.Completion = buildWorkflowCompletionRule(cc) + Name: taskName, + IsMultiUser: utCtx.MULTI() != nil, } - // Caption is the first STRING_LITERAL - allStrings := utCtx.AllSTRING_LITERAL() - if len(allStrings) > 0 { - node.Caption = unquoteString(allStrings[0].GetText()) + // The caption is the statement's own string literal; every other string now + // belongs to a clause. + if caption := utCtx.STRING_LITERAL(); caption != nil { + node.Caption = unquoteString(caption.GetText()) } - // Qualified names: PAGE, TARGETING MICROFLOW, ENTITY (in order) - names := utCtx.AllQualifiedName() - nameIdx := 0 - - if utCtx.PAGE() != nil && nameIdx < len(names) { - node.Page = buildQualifiedName(names[nameIdx]) - nameIdx++ + for _, clause := range utCtx.AllWorkflowUserTaskClause() { + applyWorkflowUserTaskClause(node, clause) + } + for _, clause := range utCtx.AllWorkflowMultiUserTaskClause() { + mc, ok := clause.(*parser.WorkflowMultiUserTaskClauseContext) + if !ok { + continue + } + switch { + case mc.WorkflowUserTaskClause() != nil: + applyWorkflowUserTaskClause(node, mc.WorkflowUserTaskClause()) + case mc.WorkflowParticipantsClause() != nil: + if pc, ok := mc.WorkflowParticipantsClause().(*parser.WorkflowParticipantsClauseContext); ok { + node.Participants = buildWorkflowParticipants(pc) + } + case mc.WorkflowCompletionClause() != nil: + if cc, ok := mc.WorkflowCompletionClause().(*parser.WorkflowCompletionClauseContext); ok { + node.Completion = buildWorkflowCompletionRule(cc) + } + case mc.AWAIT() != nil: + node.AwaitAllUsers = true + } } - // Determine if group targeting (TARGETING GROUPS vs TARGETING [USERS]) - isGroupTargeting := len(utCtx.AllGROUPS()) > 0 + return node +} - // MICROFLOW appears in both TARGETING … MICROFLOW and ON CREATED MICROFLOW, - // so targeting is present when a MICROFLOW token is left over after the - // on-created one. Qualified names come in clause order: page, targeting, - // on-created, entity. - onCreated := utCtx.CREATED() != nil - targetingMicroflows := len(utCtx.AllMICROFLOW()) - if onCreated { - targetingMicroflows-- +// applyWorkflowUserTaskClause folds one clause into the task node. Reading +// TARGETING's USERS/GROUPS off the clause that carries it — rather than asking +// whether the whole statement mentions GROUPS anywhere — is what lets the two +// targeting clauses appear in either order without one borrowing the other's +// audience. +func applyWorkflowUserTaskClause(node *ast.WorkflowUserTaskNode, clause parser.IWorkflowUserTaskClauseContext) { + c, ok := clause.(*parser.WorkflowUserTaskClauseContext) + if !ok { + return } - - if targetingMicroflows > 0 && nameIdx < len(names) { - if isGroupTargeting { + switch { + case c.PAGE() != nil: + if qn := c.QualifiedName(); qn != nil { + node.Page = buildQualifiedName(qn) + } + case c.TARGETING() != nil && c.MICROFLOW() != nil: + if c.GROUPS() != nil { node.Targeting.Kind = "group_microflow" } else { node.Targeting.Kind = "microflow" } - node.Targeting.Microflow = buildQualifiedName(names[nameIdx]) - nameIdx++ - } - - stringIdx := 1 // allStrings[0] is the caption - if utCtx.XPATH() != nil && stringIdx < len(allStrings) { - if isGroupTargeting { + if qn := c.QualifiedName(); qn != nil { + node.Targeting.Microflow = buildQualifiedName(qn) + } + case c.TARGETING() != nil && c.XPATH() != nil: + if c.GROUPS() != nil { node.Targeting.Kind = "group_xpath" } else { node.Targeting.Kind = "xpath" } - node.Targeting.XPath = unquoteString(allStrings[stringIdx].GetText()) - stringIdx++ - } - - if onCreated && nameIdx < len(names) { - node.OnCreated = buildQualifiedName(names[nameIdx]) - nameIdx++ - } - - if utCtx.ENTITY() != nil && nameIdx < len(names) { - node.Entity = buildQualifiedName(names[nameIdx]) - } - - if utCtx.DUE() != nil && utCtx.DATE_TYPE() != nil && stringIdx < len(allStrings) { - node.DueDate = unquoteString(allStrings[stringIdx].GetText()) - stringIdx++ - } - - if utCtx.DESCRIPTION() != nil && stringIdx < len(allStrings) { - node.TaskDescription = unquoteString(allStrings[stringIdx].GetText()) - stringIdx++ - } - - // Outcomes - for _, outcomeCtx := range utCtx.AllWorkflowUserTaskOutcome() { - outcome := buildWorkflowUserTaskOutcome(outcomeCtx) - node.Outcomes = append(node.Outcomes, outcome) - } - - // BoundaryEvents (Issue #7) - for _, beCtx := range utCtx.AllWorkflowBoundaryEventClause() { - node.BoundaryEvents = append(node.BoundaryEvents, buildBoundaryEventNode(beCtx)) + if str := c.STRING_LITERAL(); str != nil { + node.Targeting.XPath = unquoteString(str.GetText()) + } + case c.ON() != nil && c.CREATED() != nil: + if qn := c.QualifiedName(); qn != nil { + node.OnCreated = buildQualifiedName(qn) + } + case c.ENTITY() != nil: + if qn := c.QualifiedName(); qn != nil { + node.Entity = buildQualifiedName(qn) + } + case c.DUE() != nil && c.DATE_TYPE() != nil: + if str := c.STRING_LITERAL(); str != nil { + node.DueDate = unquoteString(str.GetText()) + } + case c.DESCRIPTION() != nil: + if str := c.STRING_LITERAL(); str != nil { + node.TaskDescription = unquoteString(str.GetText()) + } + case c.OUTCOMES() != nil: + for _, outcomeCtx := range c.AllWorkflowUserTaskOutcome() { + node.Outcomes = append(node.Outcomes, buildWorkflowUserTaskOutcome(outcomeCtx)) + } + case len(c.AllBOUNDARY()) > 0: + for _, beCtx := range c.AllWorkflowBoundaryEventClause() { + node.BoundaryEvents = append(node.BoundaryEvents, buildBoundaryEventNode(beCtx)) + } } - - return node } // buildWorkflowUserTaskOutcome builds a WorkflowUserTaskOutcomeNode. diff --git a/mdl/visitor/visitor_workflow_clause_order_test.go b/mdl/visitor/visitor_workflow_clause_order_test.go new file mode 100644 index 0000000000..4467273020 --- /dev/null +++ b/mdl/visitor/visitor_workflow_clause_order_test.go @@ -0,0 +1,318 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "reflect" + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" +) + +// ako/mxcli#586: a CREATE WORKFLOW clause written in the "wrong" position was a +// parse error — the grammar was a fixed sequence of optional clauses, so each +// clause was optional but its position was not. The reported symptom is a token +// error naming neither the clause nor the rule: +// +// line 5:4 mismatched input 'ON' expecting ';' +// +// These tests hold the two properties the fix has to have: a shuffled clause +// order parses, and it produces the SAME AST as the canonical order. Both +// halves matter — a grammar that merely accepts the tokens while the visitor +// still reads them positionally would mis-assign every qualified name. + +func TestWorkflowUserTaskClauseOrderIsFree(t *testing.T) { + canonical := `CREATE WORKFLOW M.WF +BEGIN + USER TASK ut1 'Review' + PAGE M.TaskPage + TARGETING GROUPS XPATH '[Name = ''Admin'']' + ON CREATED MICROFLOW M.OnCreated + ENTITY M.Order + DUE DATE '${PT4H}' + DESCRIPTION 'Review the order' + OUTCOMES + 'Approve' { } + 'Reject' { } + BOUNDARY EVENT INTERRUPTING TIMER '${PT1H}'; +END WORKFLOW;` + + // The order the reporter reached for: on-created first, page later. + shuffled := `CREATE WORKFLOW M.WF +BEGIN + USER TASK ut1 'Review' + ON CREATED MICROFLOW M.OnCreated + DESCRIPTION 'Review the order' + BOUNDARY EVENT INTERRUPTING TIMER '${PT1H}' + ENTITY M.Order + OUTCOMES + 'Approve' { } + 'Reject' { } + TARGETING GROUPS XPATH '[Name = ''Admin'']' + DUE DATE '${PT4H}' + PAGE M.TaskPage; +END WORKFLOW;` + + want := buildWorkflowStmt(t, canonical) + got := buildWorkflowStmt(t, shuffled) + + if !reflect.DeepEqual(got.Activities, want.Activities) { + t.Errorf("shuffled clause order built a different user task\n got: %#v\nwant: %#v", + got.Activities[0], want.Activities[0]) + } + + ut, ok := want.Activities[0].(*ast.WorkflowUserTaskNode) + if !ok { + t.Fatalf("expected *ast.WorkflowUserTaskNode, got %T", want.Activities[0]) + } + // Spot-check the clauses that the positional reader used to assign by + // counting qualified names, so a DeepEqual of two equally-wrong trees + // cannot pass this test. + if ut.Page.String() != "M.TaskPage" { + t.Errorf("Page = %q, want M.TaskPage", ut.Page.String()) + } + if ut.OnCreated.String() != "M.OnCreated" { + t.Errorf("OnCreated = %q, want M.OnCreated", ut.OnCreated.String()) + } + if ut.Entity.String() != "M.Order" { + t.Errorf("Entity = %q, want M.Order", ut.Entity.String()) + } + if ut.Targeting.Kind != "group_xpath" || ut.Targeting.XPath != "[Name = 'Admin']" { + t.Errorf("Targeting = %q %q, want group_xpath [Name = 'Admin']", ut.Targeting.Kind, ut.Targeting.XPath) + } + if ut.DueDate != "${PT4H}" { + t.Errorf("DueDate = %q", ut.DueDate) + } + if ut.TaskDescription != "Review the order" { + t.Errorf("TaskDescription = %q", ut.TaskDescription) + } + if ut.Caption != "Review" { + t.Errorf("Caption = %q", ut.Caption) + } +} + +func TestWorkflowMultiUserTaskClauseOrderIsFree(t *testing.T) { + canonical := `CREATE WORKFLOW M.WF +BEGIN + MULTI USER TASK ut1 'Review' + PAGE M.TaskPage + TARGETING MICROFLOW M.PickUsers + ON CREATED MICROFLOW M.OnCreated + ENTITY M.Order + DUE DATE '${PT4H}' + DESCRIPTION 'Review the order' + PARTICIPANTS 60 PERCENT + DECIDE BY MAJORITY MORE THAN HALF FALLBACK 'Reject' + AWAIT ALL USERS + OUTCOMES + 'Approve' { } + 'Reject' { }; +END WORKFLOW;` + + shuffled := `CREATE WORKFLOW M.WF +BEGIN + MULTI USER TASK ut1 'Review' + AWAIT ALL USERS + OUTCOMES + 'Approve' { } + 'Reject' { } + ON CREATED MICROFLOW M.OnCreated + DECIDE BY MAJORITY MORE THAN HALF FALLBACK 'Reject' + DESCRIPTION 'Review the order' + PAGE M.TaskPage + PARTICIPANTS 60 PERCENT + ENTITY M.Order + TARGETING MICROFLOW M.PickUsers + DUE DATE '${PT4H}'; +END WORKFLOW;` + + want := buildWorkflowStmt(t, canonical) + got := buildWorkflowStmt(t, shuffled) + + if !reflect.DeepEqual(got.Activities, want.Activities) { + t.Errorf("shuffled clause order built a different multi user task\n got: %#v\nwant: %#v", + got.Activities[0], want.Activities[0]) + } + + ut, ok := want.Activities[0].(*ast.WorkflowUserTaskNode) + if !ok { + t.Fatalf("expected *ast.WorkflowUserTaskNode, got %T", want.Activities[0]) + } + if !ut.IsMultiUser || !ut.AwaitAllUsers { + t.Errorf("IsMultiUser=%v AwaitAllUsers=%v, want both true", ut.IsMultiUser, ut.AwaitAllUsers) + } + if ut.OnCreated.String() != "M.OnCreated" { + t.Errorf("OnCreated = %q, want M.OnCreated", ut.OnCreated.String()) + } + if ut.Entity.String() != "M.Order" { + t.Errorf("Entity = %q, want M.Order", ut.Entity.String()) + } +} + +func TestWorkflowHeaderClauseOrderIsFree(t *testing.T) { + canonical := `CREATE WORKFLOW M.WF + FOLDER 'Flows' + PARAMETER $Order: M.Order + DISPLAY 'Order review' + DESCRIPTION 'Reviews an order' + EXPORT LEVEL Hidden + OVERVIEW PAGE M.Overview + DUE DATE '${P1D}' + ON ANY WORKFLOW EVENT MICROFLOW M.OnEvent AS 'all' +BEGIN + ANNOTATION 'body'; +END WORKFLOW;` + + // The reporter's note gives the order they worked out empirically; this is + // a different one, which must mean the same thing. + shuffled := `CREATE WORKFLOW M.WF + ON ANY WORKFLOW EVENT MICROFLOW M.OnEvent AS 'all' + DUE DATE '${P1D}' + DESCRIPTION 'Reviews an order' + OVERVIEW PAGE M.Overview + DISPLAY 'Order review' + PARAMETER $Order: M.Order + EXPORT LEVEL Hidden + FOLDER 'Flows' +BEGIN + ANNOTATION 'body'; +END WORKFLOW;` + + want := buildWorkflowStmt(t, canonical) + got := buildWorkflowStmt(t, shuffled) + + if !reflect.DeepEqual(got, want) { + t.Errorf("shuffled header order built a different workflow\n got: %#v\nwant: %#v", got, want) + } + + if want.Folder != "Flows" { + t.Errorf("Folder = %q", want.Folder) + } + if want.ParameterVar != "$Order" || want.ParameterEntity.String() != "M.Order" { + t.Errorf("Parameter = %q %q", want.ParameterVar, want.ParameterEntity.String()) + } + if want.DisplayName != "Order review" { + t.Errorf("DisplayName = %q", want.DisplayName) + } + if want.Description != "Reviews an order" { + t.Errorf("Description = %q", want.Description) + } + if want.ExportLevel != "Hidden" { + t.Errorf("ExportLevel = %q", want.ExportLevel) + } + if want.OverviewPage.String() != "M.Overview" { + t.Errorf("OverviewPage = %q", want.OverviewPage.String()) + } + if want.DueDate != "${P1D}" { + t.Errorf("DueDate = %q", want.DueDate) + } + if len(want.EventHandlers) != 1 || !want.EventHandlers[0].AnyEvent { + t.Errorf("EventHandlers = %#v", want.EventHandlers) + } +} + +// Order-freedom must not become "write it twice and the last one wins": the old +// grammar allowed each clause at most once, and a repeated-clause list silently +// overwriting is a worse failure than the parse error it replaces. +func TestWorkflowDuplicateClauseIsReported(t *testing.T) { + tests := []struct { + name string + input string + want string + }{ + { + name: "user task page twice", + input: `CREATE WORKFLOW M.WF +BEGIN + USER TASK ut1 'Review' + PAGE M.A + PAGE M.B; +END WORKFLOW;`, + want: "PAGE", + }, + { + name: "user task on created twice", + input: `CREATE WORKFLOW M.WF +BEGIN + USER TASK ut1 'Review' + ON CREATED MICROFLOW M.A + DESCRIPTION 'd' + ON CREATED MICROFLOW M.B; +END WORKFLOW;`, + want: "ON CREATED MICROFLOW", + }, + { + name: "header display twice", + input: `CREATE WORKFLOW M.WF + DISPLAY 'A' + DISPLAY 'B' +BEGIN +END WORKFLOW;`, + want: "DISPLAY", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, errs := Build(tt.input) + if len(errs) == 0 { + t.Fatalf("expected a duplicate-clause error, got none") + } + joined := joinErrors(errs) + if !strings.Contains(joined, tt.want) { + t.Errorf("error does not name the duplicated clause %q:\n%s", tt.want, joined) + } + if !strings.Contains(strings.ToLower(joined), "at most once") { + t.Errorf("error does not say the clause may appear at most once:\n%s", joined) + } + }) + } +} + +func joinErrors(errs []error) string { + parts := make([]string, 0, len(errs)) + for _, err := range errs { + parts = append(parts, err.Error()) + } + return strings.Join(parts, "\n") +} + +// `targeting microflow` and `targeting xpath` fill the same slot — a user task +// stores one UserSource — so writing both is a duplicate, not two clauses. The +// sequence grammar accepted both and let the LAST one win, which is the +// order-dependence of #586 in its most damaging form. +func TestWorkflowTwoTargetingClausesAreADuplicate(t *testing.T) { + input := `CREATE WORKFLOW M.WF +BEGIN + USER TASK ut1 'Review' + TARGETING MICROFLOW M.PickUsers + TARGETING XPATH '[true()]'; +END WORKFLOW;` + + _, errs := Build(input) + if len(errs) == 0 { + t.Fatal("expected a duplicate TARGETING error, got none") + } + if joined := joinErrors(errs); !strings.Contains(joined, "TARGETING") { + t.Errorf("error does not name TARGETING:\n%s", joined) + } +} + +// Relaxing the clause ORDER must not relax the clause VOCABULARY: the +// multi-user-only clauses stay refused on a single user task. +func TestSingleUserTaskStillRefusesMultiUserClauses(t *testing.T) { + for _, clause := range []string{ + "PARTICIPANTS 60 PERCENT", + "DECIDE BY CONSENSUS", + "AWAIT ALL USERS", + } { + t.Run(clause, func(t *testing.T) { + input := "CREATE WORKFLOW M.WF\nBEGIN\n USER TASK ut1 'Review'\n " + + clause + "\n OUTCOMES 'Done' { };\nEND WORKFLOW;" + if _, errs := Build(input); len(errs) == 0 { + t.Errorf("%s was accepted on a single user task", clause) + } + }) + } +} diff --git a/mdl/visitor/visitor_workflow_clauses.go b/mdl/visitor/visitor_workflow_clauses.go new file mode 100644 index 0000000000..f1b7d57889 --- /dev/null +++ b/mdl/visitor/visitor_workflow_clauses.go @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: Apache-2.0 + +package visitor + +import ( + "fmt" + + "github.com/antlr4-go/antlr/v4" + + "github.com/mendixlabs/mxcli/mdl/grammar/parser" +) + +// A workflow's header clauses and a user task's clauses used to be a fixed +// SEQUENCE of optional groups in the grammar, which made each clause optional +// but its POSITION mandatory — `on created microflow` written anywhere but +// between the targeting clauses and `entity` failed with `mismatched input 'ON' +// expecting ';'`, naming neither the clause nor the rule (ako/mxcli#586). +// +// The grammar now takes them as a set. That alone would have RELAXED the +// language in a second way nobody asked for: `page M.A page M.B` would parse +// and the second would silently win. A clause written twice is a better error +// than a clause written out of order, not a licence to accept it, so the +// at-most-once half of the old rule is enforced here — where the message can +// name the clause, which is what the token error could not do. +// +// Three clauses are deliberately exempt because they are list-valued and +// accumulate: `outcomes`, `boundary event` and the header's workflow event +// handlers. A task may carry several boundary events, and under the old grammar +// they were already spelled as a repeated `boundary event` inside one clause. +func (b *Builder) checkWorkflowClausesAtMostOnce(node antlr.Tree) { + if node == nil { + return + } + owner := workflowClauseOwnerDescription(node) + seen := make(map[string]int) + for i := 0; i < node.GetChildCount(); i++ { + child := node.GetChild(i) + kind, tok := workflowClauseKind(child) + if kind != "" && tok != nil { + if first, dup := seen[kind]; dup { + b.addError(fmt.Errorf( + "line %d:%d: duplicate %s clause on %s (already given on line %d) — "+ + "each clause may appear at most once, in any order", + tok.GetLine(), tok.GetColumn(), kind, owner, first)) + } else { + seen[kind] = tok.GetLine() + } + } + b.checkWorkflowClausesAtMostOnce(child) + } +} + +// workflowClauseKind names the clause a node spells, in the words the author +// wrote, or "" when the node is not a single-valued workflow clause. +func workflowClauseKind(node antlr.Tree) (string, antlr.Token) { + switch c := node.(type) { + case *parser.WorkflowHeaderClauseContext: + return workflowHeaderClauseKind(c), c.GetStart() + case *parser.WorkflowUserTaskClauseContext: + return workflowUserTaskClauseKind(c), c.GetStart() + case *parser.WorkflowMultiUserTaskClauseContext: + // The common clauses are wrapped one level deeper here, so unwrap rather + // than grouping them under the wrapper — otherwise every multi user task + // clause would sit alone in its own group and no duplicate could be seen. + switch { + case c.WorkflowUserTaskClause() != nil: + if inner, ok := c.WorkflowUserTaskClause().(*parser.WorkflowUserTaskClauseContext); ok { + return workflowUserTaskClauseKind(inner), c.GetStart() + } + case c.WorkflowParticipantsClause() != nil: + return "PARTICIPANTS", c.GetStart() + case c.WorkflowCompletionClause() != nil: + return "DECIDE BY", c.GetStart() + case c.AWAIT() != nil: + return "AWAIT ALL USERS", c.GetStart() + } + } + return "", nil +} + +func workflowHeaderClauseKind(c *parser.WorkflowHeaderClauseContext) string { + switch { + case c.FOLDER() != nil: + return "FOLDER" + case c.PARAMETER() != nil: + return "PARAMETER" + case c.DISPLAY() != nil: + return "DISPLAY" + case c.DESCRIPTION() != nil: + return "DESCRIPTION" + case c.EXPORT() != nil: + return "EXPORT LEVEL" + case c.OVERVIEW() != nil: + return "OVERVIEW PAGE" + case c.DUE() != nil: + return "DUE DATE" + } + // Event handlers accumulate. + return "" +} + +func workflowUserTaskClauseKind(c *parser.WorkflowUserTaskClauseContext) string { + switch { + case c.PAGE() != nil: + return "PAGE" + case c.TARGETING() != nil: + // Both spellings fill the SAME slot — a user task stores one UserSource + // — so `targeting microflow … targeting xpath …` is a duplicate, not two + // clauses. Under the sequence grammar both were accepted and the second + // silently won, which is the order-dependence of #586 in its most + // damaging form: a task targeted by the clause the author wrote last. + return "TARGETING" + case c.ON() != nil && c.CREATED() != nil: + return "ON CREATED MICROFLOW" + case c.ENTITY() != nil: + return "ENTITY" + case c.DUE() != nil: + return "DUE DATE" + case c.DESCRIPTION() != nil: + return "DESCRIPTION" + case c.OUTCOMES() != nil: + return "OUTCOMES" + } + // Boundary events accumulate. + return "" +} + +// workflowClauseOwnerDescription names the thing the clauses belong to, so the +// error reads "on user task ut1" rather than pointing at a rule name. +func workflowClauseOwnerDescription(node antlr.Tree) string { + switch c := node.(type) { + case *parser.CreateWorkflowStatementContext: + if qn := c.QualifiedName(); qn != nil { + return "workflow " + qn.GetText() + } + return "the workflow" + case *parser.WorkflowUserTaskStmtContext: + name := "" + if id := c.IDENTIFIER(); id != nil { + name = id.GetText() + } else if qid := c.QUOTED_IDENTIFIER(); qid != nil { + name = unquoteIdentifier(qid.GetText()) + } + kind := "user task" + if c.MULTI() != nil { + kind = "multi user task" + } + if name == "" { + return kind + } + return kind + " " + name + } + return "this statement" +} From e5a2ac5a385bd2181dc4666074543fe40faff7a4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 08:54:03 +0000 Subject: [PATCH 03/38] ci: stream go test output instead of capturing it (ako/mxcli#594) Two jobs captured their test command into a variable and echoed it on the next line: out=$(go test -v -count=1 -run '...' ./cmd/mxcli/docker/) echo "$out" The runner's shell is `bash --noprofile --norc -e -o pipefail`, so a non-zero `go test` aborts the step AT THE ASSIGNMENT and the echo never runs. The capture prints on success, where nobody reads it, and prints nothing on failure, where it is the only thing anyone wants. Measured on ako/mxcli#594: `windows-process-regression` went red with a log holding one line of substance -- ##[error]Process completed with exit code 1 -- no test name, no failure message. The only evidence of WHICH test had tripped was the runner's own cleanup line, `Terminate orphan process: pid (2760) (PING)`, which the green run on main does not have: the `ping` grandchild that TestKillProcessGroup_ReapsGrandchildAndUnblocksWait spawns, still alive because the test had hit one of its deadlines. Replaced with `2>&1 | tee go-test-output.txt`, `status=${PIPESTATUS[0]}`, then grep the file. Output streams as it is produced, the vacuous-`-run` guard the capture existed for still counts its `--- PASS:` lines, and a real failure exits with go test's own status. Verified without CI, by extracting each step's `run:` block straight out of the YAML (yaml.safe_load) and running it under the runner's own shell flags with a stub `go` on PATH. Six cases, all correct: win job fail -> full output + "go test exited 1" exit 1 win job pass -> full output + "executed: 9" exit 0 win job vacuous -> "executed: 0" + the renamed-tests msg exit 1 seam job fail -> full output + "go test exited 1" exit 1 seam job pass -> full output + "executed: 4" exit 0 seam job vacuous -> "executed: 0" + the !linux-stub msg exit 1 Control: the OLD body, same stub, prints zero lines and exits 1 -- the CI log reproduced exactly. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DXYNJwiutu5AjxLmG7Fgsu --- .claude/skills/fix-issue/findings/other.jsonl | 1 + .github/workflows/push-test.yml | 37 ++++++++++++++++--- .gitignore | 4 ++ 3 files changed, 36 insertions(+), 6 deletions(-) diff --git a/.claude/skills/fix-issue/findings/other.jsonl b/.claude/skills/fix-issue/findings/other.jsonl index 51edfa9882..c70fd11eca 100644 --- a/.claude/skills/fix-issue/findings/other.jsonl +++ b/.claude/skills/fix-issue/findings/other.jsonl @@ -16,3 +16,4 @@ {"area": "web/dist", "date": "2026-08-30", "raw": "| After `mxcli test … --local`, an app another `mxcli run --local` is serving goes blank while still answering HTTP 200 (~1.7 KB, the Mendix SPA shell); the runtime log shows `Connector: 404 - file not found for file: dist%2Findex.js` and `deployment/web/dist` is gone | `cmd/mxcli/testrunner/localapp_options.go`, `cmd/mxcli/testrunner/runner_local.go` (`localTestDeployDir`), `cmd/mxcli/testrunner/runner.go` (`checkScratchDeploymentExists`) | A local test run already used its own ports and its own `_test` database — the code comment says why, verbatim — but shared the **deployment directory**, which is the one the *browser* reads. A headless test boot does not bundle the web client, so its build left the running app serving the shell over a 404: tests pass, run keeps running, app is blank, nothing reported at either end. **Detection was not the fix**: the two processes use different ports by design, so no port check can see it, and a lock file would only turn a silent blanking into a refusal. The test boot now builds into `/.mxcli/deployment-test/` — gitignored, already where the test runtime log lives — which makes the collision impossible. Note booting a runtime against the shared directory damages it even **without** a rebuild (the packaging step removes the bundle — FINDINGS §35, `ReportLostWebClientBundle`), so \"reuse the dev loop's tree read-only\" is not an alternative. Consequence to wire: `--skip-build` used to mean \"reuse deployment/\" and now has nothing until tests have run once, so it is refused with the reason rather than failing inside the runtime boot against a path the user never chose. Reported as mxcli-formula1 FINDINGS §62 |"} {"area": ".claude/skills/mendix/record-narrated-demo", "date": "2026-09-13", "symptom": "In a narrated demo recorded with CSS `zoom` (take.js's fix for a fixed-width Mendix page), narrate.js's `point()` highlight ring is drawn around the wrong control or off the edge of the frame, and the caption plate is the wrong height and sits outside the film's caption band. Nothing in the take, the beat assertions or the contact sheet reports anything.", "cause": "Under `html{zoom:z}` Chromium reports `getBoundingClientRect()` in ZOOM-ADJUSTED (video) pixels but `getComputedStyle()` and `style.*` in CSS pixels. `point()` read a rect and assigned it straight to `style.left/top/width/height`, so the ring landed at position x z (measured at z=1.6842: a target at (168,202) ringed at (274,330)). The plate had the mirror-image problem: its geometry was declared in CSS pixels, so a 96px bar reached the file as 96 x z = 162 video px against a 184px caption band.", "file": ".claude/skills/mendix/record-narrated-demo/narrate.js (`point`, `css`, `checkOverlay`)", "insight": "The overlay lives in the page's coordinate space and the film is specified in the frame's, and `zoom` is the only conversion between them - so every overlay number is now stated in VIDEO pixels and divided by a zoom passed to `configure()`. The trap is that the conversion runs in opposite directions depending on which API you read it back with, which is why the fix came with `checkOverlay()`: it measures the installed plate against the band and refuses the take, with the control being one line (build the overlay without telling it the zoom -> 'caption plate is 310 video px tall, the band is 184'). A design rule that can be measured in the page should be a check that throws at record time, not a note in a skill - the same argument PRODUCTION.md sec 12 makes for compositions.", "refs": ["ako/mxcli-intro-video video-system/DESIGN-LANGUAGE.md"]} {"area":".claude/skills/packs","date":"2026-09-21","symptom":"A URL-fed Vega-Lite chart in the mendix-vega-charts pack drew its axes and a FULL legend with zero data points, no console error and no Vega warning. The skill stated that \"same-origin requests carry the session cookie, so an endpoint authenticated by session is reachable ... without any token handling\".","cause":"Mendix refuses a session-authenticated request without the session's CSRF token on READS too, not just writes and not just /xas/. The cookie is sent; it is not sufficient. Vega's loader read the 401 body as an empty dataset, so the failure surfaced as a plausible-looking empty chart rather than as an error.","file":".claude/skills/packs/mendix-vega-charts/widget/src/csrf.ts, .../SKILL.md, cmd/mxcli/skillpacks_test.go","insight":"THE LEGEND IS THE TELL: it is built from the spec's scales, not from rows, so a chart with a complete legend and no marks has had its DATA refused, while a chart with a broken legend has a spec problem. That one distinction separates the two hypotheses before any measurement. Two things then send the diagnosis the wrong way and cost the time: document.cookie shows only originURI=/login.html (XASSESSIONID and xasid are httpOnly, so the browser IS sending them and JavaScript cannot see them) and basic auth on the same URL returns the data, which reads as proof the endpoint is fine and the chart is broken. Isolate on the HEADER, not the URL: two requests, one added header, everything else equal -- 401 vs 200, measured on a fresh 11.14.0 app. Skip the plausible wrong turn of adding the header unconditionally: the issue's own suggested loader tests the URI with /^[a-z][a-z0-9+.-]*:\\/\\//i, which passes //elsewhere.example/rows.json (no scheme, another host) and hands that host a working session credential. Resolve with new URL(uri, base) and compare origins instead -- it also gets the converse right, an absolute URL naming the app's own origin IS the app. End-to-end control through vega's real loader against the running app: 0 marks / 3 axes / no error without the token, 1 mark with it, which reproduces the reported symptom exactly.","refs":["ako/mxcli#574"],"ce":[],"mendix":"11.14.0"} +{"area": "ci", "date": "2026-09-22", "symptom": "A failing CI job's log held nothing but `##[error]Process completed with exit code 1` — no test name, no failure message. On `windows-process-regression` that made a red check impossible to diagnose from the log alone: the only evidence of WHICH test had failed was the runner's own `Terminate orphan process: pid (2760) (PING)` cleanup line, absent from the green run on main.", "cause": "The step captured the command into a variable — `out=$(go test -v -run '…' ./cmd/mxcli/docker/)` — and echoed it on the NEXT line. The runner's shell is `bash --noprofile --norc -e -o pipefail`, so a non-zero `go test` aborts the step AT THE ASSIGNMENT and the `echo \"$out\"` never runs. The capture existed only to count `--- PASS:` lines (a `-run` filter passes vacuously if the tests are renamed away). Replaced with `go test … 2>&1 | tee go-test-output.txt`, `status=${PIPESTATUS[0]}`, then grep the file — output streams as it is produced, the vacuous-run guard still counts, and a real failure is reported with its own exit status. Same pattern was in `tunnel-seam-cross-platform`; both fixed.", "file": "`.github/workflows/push-test.yml` (tunnel-seam-cross-platform, windows-process-regression)", "insight": "**A CI step that captures output to echo it later loses exactly the runs you need it for** — it prints on success, where nobody reads it, and prints nothing on failure. `set -e` is what makes it silent, so it looks fine in local testing without `-e`. Grep workflows for `=$(` around a build/test command before trusting a bare exit code. The measurement that settles it costs a minute and needs no CI: extract the step's `run:` block straight out of the YAML (`yaml.safe_load`), put a stub `go` on PATH that exits 1 with realistic output, and run the block under `bash --noprofile --norc -e -o pipefail` — the old body prints zero lines. Exercise the vacuous-`-run` case too, or the fix quietly disables the guard the capture was there for.", "refs": ["ako/mxcli#594"]} diff --git a/.github/workflows/push-test.yml b/.github/workflows/push-test.yml index fa5734cd21..262c5d23f3 100644 --- a/.github/workflows/push-test.yml +++ b/.github/workflows/push-test.yml @@ -39,11 +39,24 @@ jobs: # # -run can pass vacuously if the tests are renamed or deleted, so assert that # the expected number actually ran. + # + # `tee` rather than a command substitution: the runner's shell is + # `bash -e -o pipefail`, so capturing into `out=$(go test ...)` aborts the + # step AT THE ASSIGNMENT when go test fails, and the `echo "$out"` below + # never runs. The log then holds nothing but "Process completed with exit + # code 1" — no test name, no failure message, no way to tell a real break + # from a flake. Write the output as it is produced, then judge it. run: | - out=$(go test -v -count=1 -run 'Unsupported' ./cmd/mxcli/docker/... ./cmd/mxcli/tunnelhub/...) - echo "$out" - n=$(printf '%s\n' "$out" | grep -c '^--- PASS: Test.*Unsupported' || true) + set +e + go test -v -count=1 -run 'Unsupported' ./cmd/mxcli/docker/... ./cmd/mxcli/tunnelhub/... 2>&1 | tee go-test-output.txt + status=${PIPESTATUS[0]} + set -e + n=$(grep -c '^--- PASS: Test.*Unsupported' go-test-output.txt || true) echo "seam tests executed: $n" + if [ "$status" -ne 0 ]; then + echo "FAIL: go test exited $status. Its output is above." + exit "$status" + fi if [ "$n" -lt 4 ]; then echo "FAIL: expected at least 4 tunnel-seam tests to run, -run matched $n." echo " The !linux stubs in cmd/mxcli/docker and cmd/mxcli/tunnelhub" @@ -75,11 +88,23 @@ jobs: # # -run can pass vacuously if the tests are renamed or deleted, so assert # that the expected number actually ran. + # + # `tee` rather than a command substitution — see the tunnel-seam job above. + # This job is where it bit: a run of these tests failed with the log holding + # only "Process completed with exit code 1", so the failing test could not be + # named and the leftover `ping` in the runner's orphan-process cleanup was + # the only evidence of WHICH test had tripped (ako/mxcli#594). run: | - out=$(go test -v -count=1 -run 'TestProcessAlive|TestKillProcessGroup|TestServeServer_AliveTracksProcess|TestLocalRuntime_AliveTracksProcess' ./cmd/mxcli/docker/) - echo "$out" - n=$(printf '%s\n' "$out" | grep -c '^--- PASS: Test' || true) + set +e + go test -v -count=1 -run 'TestProcessAlive|TestKillProcessGroup|TestServeServer_AliveTracksProcess|TestLocalRuntime_AliveTracksProcess' ./cmd/mxcli/docker/ 2>&1 | tee go-test-output.txt + status=${PIPESTATUS[0]} + set -e + n=$(grep -c '^--- PASS: Test' go-test-output.txt || true) echo "windows process tests executed: $n" + if [ "$status" -ne 0 ]; then + echo "FAIL: go test exited $status. Its output is above." + exit "$status" + fi if [ "$n" -lt 5 ]; then echo "FAIL: expected at least 5 Windows process tests to run, -run matched $n." echo " procgroup_windows_test.go must keep its processAlive / tree-kill /" diff --git a/.gitignore b/.gitignore index 5d64b99f15..4f7c87398d 100644 --- a/.gitignore +++ b/.gitignore @@ -67,3 +67,7 @@ snap-bson # Anchored: an unanchored "mprsnapshot" also matches the scripts/mprsnapshot/ # source directory, which silently excludes it from `git add`. /mprsnapshot + +# CI test-output capture (.github/workflows/push-test.yml); also created if you +# run those step bodies locally. +go-test-output.txt From 72d36eeaeee99e854254cff26bcee4669203c8a9 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 09:13:46 +0000 Subject: [PATCH 04/38] fix: store and read a workflow's overview page (ako/mxcli#586) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `create workflow … overview page X` reported `Created workflow` and exit 0 and stored nothing. The written unit carried no page reference and not even the page's qualified name as a string; `mx check` passed, because a workflow with no overview page is valid, and `describe workflow` omitted the clause, so nothing revealed the loss. Two fields for one concept, never joined. The executor set the semantic `Workflow.OverviewPage`; the backend only ever wrote `Workflow.AdminPage`, which nothing set. The read half was wrong in the mirror direction — the reader took `g.OverviewPageQualifiedName()` — so even `alter workflow … set overview page`, which writes the right key and always has, read back empty. Which name is the stored one is settled by the Model SDK's own StructureVersionInfo (mendixmodelsdk 4.115.0, src/gen/workflows.js): overviewPage: { deleted: "9.11.0" } adminPage: { introduced: "9.11.0" } and generated/metamodel, the arbiter where it and modelsdk/gen disagree, declares AdminPage and no OverviewPage at all. modelsdk/gen declares both, which is how a reader and a writer in one package ended up on opposite sides of a 9.11 rename. The two semantic fields collapse into one. OverviewPage is the word the MDL clause, the describer and the catalog use; the storage name stays in the storage adapter, per ADR-0005. Only AdminPage is written. The version branch CLAUDE.md's overlay rule asks for on a CREATE would be dead code here, and that is a measurement rather than an assumption: workflowToGen writes WorkflowV2 unconditionally, and that property was introduced in 11.1.0, so no reachable project wants the pre-9.11 key. It is still read, as a fallback — a read fallback invents nothing. Measured on mxbuild 11.6.6, same script and project, only the write suppressed: control fixed workflow unit 2,192 bytes 2,284 bytes page in the document no yes describe workflow no clause overview page WF586B.Overview mx check, valid overview page 0 errors 0 errors mx check, page without a param 0 errors CE7410 The last row is the evidence. mxbuild can only validate a page it can see, so CE7410 firing only with the fix proves the property reaches the platform; the row above it is why the unit test asserts on the raw document instead of on a build. That rule is now documented too: an overview page takes System.Workflow, while a user task's page takes System.WorkflowUserTask. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015qPaSqkSeaM4Ziuex4nxSG --- .../fix-issue/findings/mdl-backend.jsonl | 1 + .../skills/mendix/write-workflows/SKILL.md | 7 +- cmd/mxcli/syntax/features_workflow.go | 5 +- docs/01-project/MDL_QUICK_REFERENCE.md | 6 + .../workflow-586b-overview-page-dropped.mdl | 94 +++++++++++ .../modelsdk/workflow_overview_page_test.go | 151 ++++++++++++++++++ mdl/backend/modelsdk/workflow_read.go | 19 ++- mdl/backend/modelsdk/workflow_write.go | 10 +- sdk/workflows/workflow.go | 20 ++- 9 files changed, 305 insertions(+), 8 deletions(-) create mode 100644 mdl-examples/bug-tests/workflow-586b-overview-page-dropped.mdl create mode 100644 mdl/backend/modelsdk/workflow_overview_page_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index e081d80995..ba0abbd24f 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -123,3 +123,4 @@ {"area": "mdl/backend", "date": "2026-09-20", "symptom": "`DROP ENTITY` left every CROSS-MODULE association pointing at the deleted entity in place. Dropping the local BY-ID (FROM) end made mxbuild 11.14.0 unable to LOAD the project: `System.AggregateException \u2026 (The given key '' was not present in the dictionary.)` at `StreamingBsonUnitReader.ResolvePostponedProperties()` \u2014 no CE code, no document named, so the obvious reading is 'the project is corrupt, restore from git'. Dropping the BY-NAME (TO) end is milder and still wrong: CE1613 at the cross-module association. `show associations` shows a raw GUID where the parent entity should be.", "cause": "`removeAssocsReferencing` swept `dm.AssociationsItems()` and asserted `*genDm.Association` per item, so the SEPARATE `CrossAssociations` collection was never looked at. Fixed with `removeCrossAssocsReferencing`, matching BOTH ends because a cross-module association addresses them differently \u2014 FROM by element id (local), TO by qualified name (another module) \u2014 called in DeleteEntity locally and in its cascade over the other domain models.", "file": "`mdl/backend/modelsdk/domainmodel_alter.go` (removeCrossAssocsReferencing, DeleteEntity)", "insight": "**Reported against a view entity; nothing about it was view-entity specific.** The reporter met it dropping view entities (whose associations are DERIVED from OQL, so there is no CREATE ASSOCIATION to undo) and filed it that way. The first probe \u2014 a view entity and its source entity in the SAME module \u2014 did not reproduce at all, and that negative is the useful one: it says the variable is cross-module, not view-ness. A plain `create association A.X from A.X to B.Y` plus `drop entity A.X` reproduces the identical crash. Two lessons: when a repro fails, vary the dimension the report did not mention before doubting the report, and treat a collection-typed `.(*T)` assertion in a cascade as a place where a sibling type hides. mxbuild's diagnostic distinguishes the two ends for free \u2014 a dangling 16-byte pointer is a LOAD crash, a dangling qualified name is CE1613 \u2014 so testing only one end proves half the fix.", "refs": ["#553", "#556"]} {"area": "mdl/backend", "date": "2026-09-21", "symptom": "`alter settings workflows add group 'Auditors'` reports \"Added workflow group: Auditors (3 group(s))\" and writes nothing \u2014 `show workflow groups` still lists 2, and `mx check` is 0 errors either way", "cause": "`UpdateProjectSettings` overlays the workflows part field by field onto the PRESERVED raw part, so a child LIST that nothing rebuilds is carried through from disk unchanged. Adding `Groups` to the semantic model and to the read path is not enough; the write needs `settingsoverlay.WorkflowGroups(ws, rawPart)`. Identical shape to the enabled-language list the same function already documents", "file": "`mdl/backend/modelsdk/settings_write.go` + `mdl/settingsoverlay/settingsoverlay.go` (`WorkflowGroups`)", "insight": "For anything under Settings$ProjectSettings, the executor's success message proves NOTHING \u2014 it reports the in-memory model, and the overlay is where a list quietly fails to land. Assert on the re-read document, not the handler's output. Two more things a reference project settles in one dump and a guess gets wrong: the `Groups` typed-array marker is 2, not the 3 every other settings child list uses (`ArrayMarker` preserves a stored one, but the fallback matters on a fresh list), and the element's `$ID` is the RUNTIME's identity \u2014 a booted 11.13.0 app keys `system$workflowgroup.modelguid` on it, byte-identical once the .NET GUID field order is undone, so re-minting it on a description edit would orphan every group membership with a perfectly valid model. Control: deleting the one overlay call reproduces the symptom verbatim. mendixlabs/mxcli#272", "refs": ["mendixlabs/mxcli#272"]} {"area": "mdl/backend", "date": "2026-09-21", "symptom": "`CREATE OR MODIFY VIEW ENTITY` that changed ONLY the OQL printed `Unchanged view entity: \u2026` while `describe entity` showed the new query stored. Changing the attribute list as well reported `Modified` correctly, which is why it hid. Also: the OQL document's unit was replaced under a FRESH GUID on every run, even a byte-identical one, so an MDL-generated project could never come back clean in git (one of the four units #556 measured).", "cause": "A view entity's OQL lives in a separate `DomainModels$ViewEntitySourceDocument` unit, and the executor DELETED it and INSERTED a fresh one on every write. `ReportMutation` downgrades the verb when writes were offered and none landed, but the counters are incremented only at the update choke points (`writer_core.go` reconcileWithStored / MoveUnit) \u2014 `InsertUnit` is not counted at all. So the domain-model unit was offered and correctly elided, the OQL write was invisible, and the report believed the half it could see. Fixed with `WriteViewEntitySourceDocument`, which keeps the stored unit's id and goes through `UpdateRawUnit` \u2192 reconcile: an identical query is elided, a changed one lands and is counted, duplicates are still cleared.", "file": "`mdl/backend/modelsdk/move_view_write.go` (WriteViewEntitySourceDocument, encodeViewEntitySourceDocument), `mdl/executor/cmd_entities.go`", "insight": "**The first fix that comes to mind \u2014 count InsertUnit \u2014 would have swapped a false \"Unchanged\" for a false \"Modified\".** Measuring before changing is what caught it: re-running a BYTE-IDENTICAL script still re-minted the source document's unit id, so counting inserts would have made every view-entity statement report Modified forever. The right fix was the one ADR-0008 already mandates (wire the write path to canon.Reconcile), and it fixes the churn and the verb together. Generalisation worth remembering: any content that reaches storage through `InsertUnit` is invisible to the elision check, so a statement whose only landing write is a NEW unit can still be mis-reported \u2014 `MoveUnit` has a comment explaining it was counted for exactly this reason, and insert/delete were missed. Control the fix on the identical re-run, not just the changed one.", "refs": ["#583", "#556", "#910"]} +{"area": "mdl/backend", "date": "2026-09-22", "symptom": "`create workflow … overview page X` reports `Created workflow` and exit 0 and stores NOTHING — the written unit carries no page reference and not even the page's qualified name as a string. `mx check` passes (a workflow with no overview page is valid) and `describe workflow` omits the clause, so nothing reveals the loss. Running `alter workflow … set overview page X` afterwards DOES write it, which is what makes the split visible", "cause": "Two fields for one concept, never joined: the executor set semantic `Workflow.OverviewPage` (`cmd_workflows_write.go:170`) and `workflowToGen` only ever read `Workflow.AdminPage`, which nothing set. The READ half was wrong in the mirror direction — `workflowFromGen` took `g.OverviewPageQualifiedName()`, so even the correctly-written ALTER read back empty and the catalog's overview-page reference edge never fired", "file": "`sdk/workflows/workflow.go` (the two fields collapsed to one), `mdl/backend/modelsdk/workflow_write.go` (`workflowToGen`), `mdl/backend/modelsdk/workflow_read.go` (`workflowOverviewPageName`)", "insight": "**The Model SDK's StructureVersionInfo settles which of two rival property names is real, in one grep**: `npm pack mendixmodelsdk` then `src/gen/workflows.js` gives `overviewPage: {deleted: \"9.11.0\"}` and `adminPage: {introduced: \"9.11.0\"}` — so AdminPage (a `Workflows$PageReference` CHILD, not a by-name string) is the stored property, and `generated/metamodel` agrees by declaring AdminPage and no OverviewPage. `modelsdk/gen` declares BOTH, which is how a reader and a writer ended up on opposite sides of a 9.11 rename inside one package. **The version branch CLAUDE.md's overlay rule would demand is dead here, and that is a measurement not an assumption**: `workflowToGen` writes `WorkflowV2`, introduced in 11.1.0, unconditionally — so no reachable project wants the pre-9.11 key. Write one spelling, READ both (a read fallback invents nothing). **The differential that proves it on a real build**: same script, same project, only the write suppressed — control 0 errors, fixed `CE7410 \"The selected page 'Overview' should accept a parameter of type 'Workflow'\"` on mxbuild 11.6.6. mxbuild can only validate a page it can see, so the error IS the evidence; with a valid overview page both variants are 0 errors, which is the usual weak-signal trap. Useful side-finding: an overview page takes **System.Workflow**, while a user task's page takes **System.WorkflowUserTask** — two pages, two parameters. NOT fixed: no check rule for CE7410 yet, and `WorkflowV2` being written unconditionally is questionable for a 10.x project. Same shape as the `create … comment 'text'` bug (findings/mdl-grammar.jsonl 2026-08-25): grep for `stmt.X = …` / `wf.X = …` with no matching read. Tests `mdl/backend/modelsdk/workflow_overview_page_test.go`; repro `mdl-examples/bug-tests/workflow-586b-overview-page-dropped.mdl`", "refs": ["ako/mxcli#586"], "ce": ["CE7410"]} diff --git a/.claude/skills/mendix/write-workflows/SKILL.md b/.claude/skills/mendix/write-workflows/SKILL.md index 2e3327e610..f3cf88f8d6 100644 --- a/.claude/skills/mendix/write-workflows/SKILL.md +++ b/.claude/skills/mendix/write-workflows/SKILL.md @@ -36,7 +36,7 @@ create workflow Module.ApprovalFlow display 'Request Approval' -- optional human-readable name description 'Approves incoming requests' -- optional export level Hidden -- optional: Hidden | API (default Hidden) - overview page Module.WF_Overview -- optional admin overview page + overview page Module.WF_Overview -- optional; takes a System.Workflow param on workflow events (UserTaskStarted, UserTaskEnded) -- optional, repeatable microflow Module.ACT_AuditTask as 'Task audit' on any workflow event microflow Module.ACT_LogEvent -- every type this Mendix version has @@ -68,6 +68,11 @@ written **last** silently winning. both fail (`expecting VARIABLE`). - The body closer is `end workflow`, **not** `end`. `end;` fails (`missing WORKFLOW`). +- The **overview page takes a `System.Workflow` parameter**, not the workflow's + context object. Measured on mxbuild 11.6.6: a page without one is + `CE7410 "The selected page 'Overview' should accept a parameter of type + 'Workflow'"`. (The **task** page takes `System.WorkflowUserTask` instead — + two different pages, two different parameters.) **The context is always stored as `WorkflowContext`.** Whatever you name the variable in the header, mxcli writes the parameter as `WorkflowContext`, so diff --git a/cmd/mxcli/syntax/features_workflow.go b/cmd/mxcli/syntax/features_workflow.go index 699660f841..14223800cc 100644 --- a/cmd/mxcli/syntax/features_workflow.go +++ b/cmd/mxcli/syntax/features_workflow.go @@ -48,7 +48,10 @@ func init() { "-- one out of place was a token error naming neither the clause nor the\n" + "-- rule.) The event handlers are the exception and may repeat.\n" + "-- A clause written twice is reported by name, e.g.\n" + - "-- duplicate DISPLAY clause on workflow M.W (already given on line 3)", + "-- duplicate DISPLAY clause on workflow M.W (already given on line 3)\n\n" + + "-- The OVERVIEW PAGE must accept a System.Workflow parameter, or the\n" + + "-- build fails CE7410 \"The selected page should accept a parameter of\n" + + "-- type 'Workflow'\" (measured on mxbuild 11.6.6).", Example: "CREATE WORKFLOW Module.ApprovalFlow\n PARAMETER $Context: Module.Request\n OVERVIEW PAGE Module.WF_Overview\nBEGIN\n USER TASK ReviewTask 'Review the request'\n PAGE Module.ReviewPage\n OUTCOMES 'Approve' { } 'Reject' { };\nEND WORKFLOW;", SeeAlso: []string{"workflow.user-task", "workflow.event-handlers", "workflow.decision", "workflow.drop"}, }) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index ef2bdd0167..aa7cd38680 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -673,6 +673,12 @@ Nested folders use `/` separator: `'Parent/Child/Grandchild'`. Missing folders a | Create workflow | `create [or modify] workflow Module.Name [folder 'path'] parameter $Ctx: Module.Entity [on workflow events (, ...) microflow Mod.MF [as '']] [on any workflow event microflow Mod.MF [as '']] begin ... end workflow;` | See activity types and event handlers below | | Drop workflow | `drop workflow Module.Name;` | | +The **overview page** must accept a `System.Workflow` parameter — the build +fails `CE7410 "The selected page … should accept a parameter of type +'Workflow'"` otherwise (measured on mxbuild 11.6.6). It is stored under the +`AdminPage` key: Mendix deleted the `overviewPage` property in 9.11.0 and +introduced `adminPage` in the same release. + **Clause order does not matter.** A workflow's header clauses and a user task's clauses are a **set**: write them in any order, each **at most once**. A clause written twice is reported by name (`duplicate PAGE clause on user task Review diff --git a/mdl-examples/bug-tests/workflow-586b-overview-page-dropped.mdl b/mdl-examples/bug-tests/workflow-586b-overview-page-dropped.mdl new file mode 100644 index 0000000000..c86f9a8281 --- /dev/null +++ b/mdl-examples/bug-tests/workflow-586b-overview-page-dropped.mdl @@ -0,0 +1,94 @@ +-- ako/mxcli#586 (follow-up) — `create workflow … overview page X` reported +-- success and stored NOTHING. +-- +-- `exec` printed `Created workflow: …` and exit 0, the written `.mxunit` +-- contained no page reference and no `M.Overview` string anywhere, `mx check` +-- passed (a workflow with no overview page is valid) and `describe workflow` +-- omitted the clause — so nothing revealed the loss. Running +-- `alter workflow … set overview page X` immediately afterwards DID write it, +-- which is what made the split visible. +-- +-- Two halves, both settled by the Model SDK's own StructureVersionInfo +-- (mendixmodelsdk 4.115.0, `src/gen/workflows.js`): +-- +-- overviewPage: { deleted: "9.11.0" } +-- adminPage: { introduced: "9.11.0" } +-- +-- so the stored property is **AdminPage**, a `Workflows$PageReference` child — +-- and `generated/metamodel`, the arbiter when it and `modelsdk/gen` disagree, +-- declares `AdminPage` and no `OverviewPage` at all. +-- +-- 1. WRITE: the executor set the semantic `Workflow.OverviewPage`; the +-- backend only ever wrote `Workflow.AdminPage`. Two fields for one +-- concept, never joined — the same shape as the `create … comment 'text'` +-- bug (see findings/mdl-grammar.jsonl, 2026-08-25). +-- 2. READ: the reader took `g.OverviewPageQualifiedName()`, the pre-9.11 +-- property, so even the correctly-written ALTER read back empty. +-- +-- Only ONE spelling is written. mxcli's own workflows already carry +-- `WorkflowV2`, introduced in 11.1.0, so no reachable project wants the +-- pre-9.11 key, and writing both as a hedge is what CLAUDE.md's overlay rule +-- forbids. The pre-9.11 key is still READ, as a fallback. +-- +-- Verified on Mendix 11.6.6: the workflow unit goes from 6,038 to 6,129 bytes, +-- the 91 added bytes being the `AdminPage` page reference, `OverviewPage` is +-- absent from the document, and `describe workflow` now emits the clause, so +-- describe -> exec round-trips it. + +create module WF586B; + +create entity WF586B.Request ( + Status : string(200) +); + +-- An overview page takes a System.Workflow parameter. mxbuild enforces it +-- (CE7410 "The selected page 'Overview' should accept a parameter of type +-- 'Workflow'") — and that error is itself the proof the fix works: before it, +-- the page was never stored, so there was nothing for the build to check. +create page WF586B.Overview ( + title: 'Requests', + layout: Atlas_Core.Atlas_Default, + params: { $Workflow: System.Workflow } +) { + layoutgrid g1 { + row r1 { + column c1 (desktopwidth: 12) { + dynamictext txt1 (content: 'All requests', rendermode: H2) + } + } + } +} +/ + +create page WF586B.TaskPage ( + title: 'Task', + layout: Atlas_Core.Atlas_Default, + params: { $WorkflowUserTask: System.WorkflowUserTask } +) { + layoutgrid g1 { + row r1 { + column c1 (desktopwidth: 12) { + dynamictext txt1 (content: 'Handle the request', rendermode: H2) + } + } + } +} +/ + +-- The clause under test. Before the fix this workflow was written with no +-- overview page at all. +create or replace workflow WF586B.Review + parameter $WorkflowContext: WF586B.Request + overview page WF586B.Overview +begin + user task ReviewTask 'Review the request' + page WF586B.TaskPage + outcomes + 'Approve' { } + 'Reject' { }; +end workflow; +/ + +-- The ALTER path has always written AdminPage correctly; it is here because +-- until the READ was fixed it, too, described back as absent. +alter workflow WF586B.Review set overview page WF586B.Overview; diff --git a/mdl/backend/modelsdk/workflow_overview_page_test.go b/mdl/backend/modelsdk/workflow_overview_page_test.go new file mode 100644 index 0000000000..db2d1bf413 --- /dev/null +++ b/mdl/backend/modelsdk/workflow_overview_page_test.go @@ -0,0 +1,151 @@ +// SPDX-License-Identifier: Apache-2.0 + +package modelsdkbackend + +import ( + "sort" + "testing" + + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/workflows" +) + +// ako/mxcli#586 (follow-up): `create workflow … overview page X` reported +// success and stored NOTHING — the written unit carried no page reference and +// no "MyFirstModule.Overview" string anywhere. `mx check` passes, because a +// workflow with no overview page is valid, and `describe workflow` did not show +// it either, so nothing revealed the loss. +// +// Two halves, both measured against the Model SDK's own StructureVersionInfo +// (mendixmodelsdk 4.115.0, src/gen/workflows.js): `overviewPage` was DELETED in +// 9.11.0 and `adminPage` INTRODUCED in 9.11.0, so `AdminPage` — a +// Workflows$PageReference child — is the stored property, and +// generated/metamodel (the arbiter, an 11.6.0 snapshot) declares AdminPage and +// no OverviewPage at all. +// +// 1. WRITE: the executor set the semantic `Workflow.OverviewPage`, and the +// backend only ever wrote `Workflow.AdminPage`. Two fields for one concept, +// never joined — nothing copied one to the other. +// 2. READ: the reader took `g.OverviewPageQualifiedName()`, the pre-9.11 +// property, so even the `alter workflow … set overview page` path — which +// writes AdminPage correctly — read back empty. + +func createWorkflowWithOverviewPage(t *testing.T, b *Backend, containerID model.ID, name, page string) { + t.Helper() + wf := &workflows.Workflow{ + ContainerID: containerID, + Name: name, + WorkflowName: name, + OverviewPage: page, + Parameter: &workflows.WorkflowParameter{EntityRef: "MyFirstModule.Ctx"}, + Flow: &workflows.Flow{ + Activities: []workflows.WorkflowActivity{ + &workflows.StartWorkflowActivity{BaseWorkflowActivity: workflows.BaseWorkflowActivity{Name: "Start"}}, + &workflows.EndWorkflowActivity{BaseWorkflowActivity: workflows.BaseWorkflowActivity{Name: "End"}}, + }, + }, + } + if err := b.CreateWorkflow(wf); err != nil { + t.Fatalf("CreateWorkflow: %v", err) + } +} + +// The written document must carry the page under AdminPage, as a +// Workflows$PageReference. Asserted on the raw unit rather than on a read-back, +// so a reader that is wrong in the same direction cannot make this pass. +func TestCreateWorkflowStoresOverviewPageAsAdminPage(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + createWorkflowWithOverviewPage(t, b, mod.ID, "ZzOverview", "MyFirstModule.Overview") + + wfs, err := b.ListWorkflows() + if err != nil { + t.Fatalf("ListWorkflows: %v", err) + } + var id model.ID + for _, w := range wfs { + if w.Name == "ZzOverview" { + id = w.ID + } + } + if id == "" { + t.Fatal("workflow ZzOverview not found after create") + } + + raw, err := b.GetRawUnit(id) + if err != nil { + t.Fatalf("GetRawUnit: %v", err) + } + admin, ok := raw["AdminPage"].(map[string]any) + if !ok { + t.Fatalf("AdminPage is %T, want a Workflows$PageReference document; keys = %v", + raw["AdminPage"], sortedKeys(raw)) + } + if got := admin["$Type"]; got != "Workflows$PageReference" { + t.Errorf("AdminPage.$Type = %v, want Workflows$PageReference", got) + } + if got := admin["Page"]; got != "MyFirstModule.Overview" { + t.Errorf("AdminPage.Page = %v, want MyFirstModule.Overview", got) + } + // The pre-9.11 spelling must NOT also be written — writing both as a hedge + // is what CLAUDE.md's overlay rule forbids, and Studio Pro resolves every + // stored property against the type's property list. + if _, present := raw["OverviewPage"]; present { + t.Errorf("OverviewPage was written as well as AdminPage: %v", raw["OverviewPage"]) + } +} + +// And it must read back, so describe → exec round-trips the clause. +func TestReadWorkflowOverviewPageFromAdminPage(t *testing.T) { + proj := copyFixture(t) + b := New() + if err := b.Connect(proj); err != nil { + t.Fatalf("connect: %v", err) + } + t.Cleanup(func() { _ = b.Disconnect() }) + + mod, err := b.GetModuleByName("MyFirstModule") + if err != nil || mod == nil { + t.Fatalf("GetModuleByName: %v", err) + } + createWorkflowWithOverviewPage(t, b, mod.ID, "ZzOverviewRead", "MyFirstModule.Overview") + + b2 := New() + if err := b2.Connect(proj); err != nil { + t.Fatalf("reconnect: %v", err) + } + t.Cleanup(func() { _ = b2.Disconnect() }) + + wfs, err := b2.ListWorkflows() + if err != nil { + t.Fatalf("ListWorkflows: %v", err) + } + for _, w := range wfs { + if w.Name != "ZzOverviewRead" { + continue + } + if w.OverviewPage != "MyFirstModule.Overview" { + t.Fatalf("OverviewPage read back as %q, want MyFirstModule.Overview", w.OverviewPage) + } + return + } + t.Fatal("workflow ZzOverviewRead not found") +} + +func sortedKeys(m map[string]any) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} diff --git a/mdl/backend/modelsdk/workflow_read.go b/mdl/backend/modelsdk/workflow_read.go index 3c8681f10d..44b995c5e1 100644 --- a/mdl/backend/modelsdk/workflow_read.go +++ b/mdl/backend/modelsdk/workflow_read.go @@ -50,7 +50,7 @@ func workflowFromGen(g *genWf.Workflow, containerID model.ID) *workflows.Workflo Documentation: g.Documentation(), ExportLevel: g.ExportLevel(), Excluded: g.Excluded(), - OverviewPage: g.OverviewPageQualifiedName(), + OverviewPage: workflowOverviewPageName(g), DueDate: g.DueDate(), WorkflowName: workflowTemplateText(g.WorkflowName()), WorkflowDescription: workflowTemplateText(g.WorkflowDescription()), @@ -466,6 +466,23 @@ func microflowEventName(el element.Element) string { return "" } +// workflowOverviewPageName reads the workflow's overview page. +// +// It is stored under AdminPage, as a Workflows$PageReference child — Mendix +// deleted the `overviewPage` property in 9.11.0 and introduced `adminPage` in +// the same release. Reading only the old one is what made `describe workflow` +// silent about a page that `alter workflow … set overview page` had written +// correctly all along. +// +// The pre-9.11 key is still read as a fallback. Reading both costs nothing and +// invents nothing; WRITING both would be the mistake. +func workflowOverviewPageName(g *genWf.Workflow) string { + if name := taskPageName(g.AdminPage()); name != "" { + return name + } + return g.OverviewPageQualifiedName() +} + // taskPageName extracts the page qualified name from a TaskPage part // (Workflows$PageReference). func taskPageName(el element.Element) string { diff --git a/mdl/backend/modelsdk/workflow_write.go b/mdl/backend/modelsdk/workflow_write.go index 16e74d2de6..e3587917b0 100644 --- a/mdl/backend/modelsdk/workflow_write.go +++ b/mdl/backend/modelsdk/workflow_write.go @@ -200,8 +200,14 @@ func (b *Backend) DeleteWorkflow(id model.ID) error { func workflowToGen(wf *workflows.Workflow) element.Element { g := newElem("Workflows$Workflow", string(wf.ID)) - if wf.AdminPage != "" { - addPart(g, "AdminPage", pageReferenceElem(wf.AdminPage)) + // The overview page is stored under AdminPage (Mendix renamed the property in + // 9.11.0 — see workflows.Workflow.OverviewPage). Only this spelling is + // written: a mxcli-authored workflow already carries WorkflowV2, introduced + // in 11.1.0, so there is no reachable project for which the pre-9.11 + // OverviewPage key would be right, and writing both as a hedge is what the + // overlay rule in CLAUDE.md forbids. + if wf.OverviewPage != "" { + addPart(g, "AdminPage", pageReferenceElem(wf.OverviewPage)) } if wf.Annotation != "" { addPart(g, "Annotation", annotationElem(wf.Annotation)) diff --git a/sdk/workflows/workflow.go b/sdk/workflows/workflow.go index 63023bc132..a0e9259663 100644 --- a/sdk/workflows/workflow.go +++ b/sdk/workflows/workflow.go @@ -19,9 +19,23 @@ type Workflow struct { Excluded bool `json:"excluded"` WorkflowName string `json:"workflowName,omitempty"` // Template string for display name WorkflowDescription string `json:"workflowDescription,omitempty"` // Template string for description - OverviewPage string `json:"overviewPage,omitempty"` // Qualified name of overview page - DueDate string `json:"dueDate,omitempty"` // Due date expression - AdminPage string `json:"adminPage,omitempty"` // Qualified name of admin page + + // OverviewPage is the qualified name of the workflow's overview page — the + // `overview page` clause. It is stored under the BSON key **AdminPage**, as a + // Workflows$PageReference child: Mendix DELETED the `overviewPage` property + // in 9.11.0 and INTRODUCED `adminPage` in the same release (measured in the + // Model SDK's own StructureVersionInfo, mendixmodelsdk 4.115.0 + // src/gen/workflows.js; generated/metamodel declares AdminPage and no + // OverviewPage at all). + // + // There used to be a second field, AdminPage, holding the same thing under + // the storage name. Nothing ever set it and nothing ever read this one back, + // so `create workflow … overview page X` reported success and stored + // nothing. One concept gets one field; the storage name stays in the storage + // adapter, per ADR-0005. + OverviewPage string `json:"overviewPage,omitempty"` + + DueDate string `json:"dueDate,omitempty"` // Due date expression // Annotation Annotation string `json:"annotation,omitempty"` // Annotation description text From 0a5b9a363b62f7e67340a922a0a4caf517ae31b3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 09:30:28 +0000 Subject: [PATCH 05/38] Correct the System-module ceiling advice with a measurement (#587) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both skills offered a microflow data source as the way past the System-module ceiling. It is not one, and the way it fails is silent: a workflow inbox built that way drew the right number of cards and every card was blank, with mxcli check, lint, report and docker check all at 0 errors. The rule, measured in a browser on 11.14.0 rather than reasoned about: a microflow data source moves the ROWS, not the MEMBERS. A microflow does not apply entity access, so its retrieve returns every row — but the runtime re-applies entity access when it serializes those objects to the client, XPath constraint included, so a row the role may not read arrives with every member empty. The probe is one page with TWO microflow-sourced lists over the SAME System.User retrieve, opened by an Administrator and by a plain User: list Administrator plain User A the System.User objects admin, viewer (blank), viewer B a module-owned copy, Name read admin, viewer admin, viewer INSIDE the microflow All four cells hold two rows, which is what proves the microflow did carry the rows past entity access and isolates the loss to serialization. Only A loses the values, and it loses them per OBJECT: System.User's rule grants read where [id = '[%CurrentUser%]'], which is why a user picker shows you yourself and nobody else while a workflow inbox — no such escape hatch for an ordinary role — comes out entirely blank. The control is the ROLE, not a before/after build: same binary, same model, two logins. - manage-security gains the rule, the measurement and a second remedy (read the member inside the microflow, return an object your module owns), replacing the advice that caused the rebuild. - system-module states the rule where it repeats the ceiling, rather than linking to it. - mdl-examples/bug-tests/security-587-system-member-access.mdl is the probe, so the measurement can be repeated rather than taken on trust. Verified from a blank project: mxcli check --references clean, exec clean, mx check 0 errors. One thing found on the way, not fixed here: `grant on System.User` PASSES `mxcli check --references` and is refused only by `exec`, so a script carrying one checks clean and then stops part-way. Noted in the skill and in the finding. Closes #587 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01L2aYb3zDscezm874CDTH6R --- .claude/skills/fix-issue/findings/other.jsonl | 1 + .../skills/mendix/manage-security/SKILL.md | 60 +++++++-- .claude/skills/mendix/system-module/SKILL.md | 15 ++- .../security-587-system-member-access.mdl | 115 ++++++++++++++++++ 4 files changed, 174 insertions(+), 17 deletions(-) create mode 100644 mdl-examples/bug-tests/security-587-system-member-access.mdl diff --git a/.claude/skills/fix-issue/findings/other.jsonl b/.claude/skills/fix-issue/findings/other.jsonl index 51edfa9882..99322a1abc 100644 --- a/.claude/skills/fix-issue/findings/other.jsonl +++ b/.claude/skills/fix-issue/findings/other.jsonl @@ -16,3 +16,4 @@ {"area": "web/dist", "date": "2026-08-30", "raw": "| After `mxcli test … --local`, an app another `mxcli run --local` is serving goes blank while still answering HTTP 200 (~1.7 KB, the Mendix SPA shell); the runtime log shows `Connector: 404 - file not found for file: dist%2Findex.js` and `deployment/web/dist` is gone | `cmd/mxcli/testrunner/localapp_options.go`, `cmd/mxcli/testrunner/runner_local.go` (`localTestDeployDir`), `cmd/mxcli/testrunner/runner.go` (`checkScratchDeploymentExists`) | A local test run already used its own ports and its own `_test` database — the code comment says why, verbatim — but shared the **deployment directory**, which is the one the *browser* reads. A headless test boot does not bundle the web client, so its build left the running app serving the shell over a 404: tests pass, run keeps running, app is blank, nothing reported at either end. **Detection was not the fix**: the two processes use different ports by design, so no port check can see it, and a lock file would only turn a silent blanking into a refusal. The test boot now builds into `/.mxcli/deployment-test/` — gitignored, already where the test runtime log lives — which makes the collision impossible. Note booting a runtime against the shared directory damages it even **without** a rebuild (the packaging step removes the bundle — FINDINGS §35, `ReportLostWebClientBundle`), so \"reuse the dev loop's tree read-only\" is not an alternative. Consequence to wire: `--skip-build` used to mean \"reuse deployment/\" and now has nothing until tests have run once, so it is refused with the reason rather than failing inside the runtime boot against a path the user never chose. Reported as mxcli-formula1 FINDINGS §62 |"} {"area": ".claude/skills/mendix/record-narrated-demo", "date": "2026-09-13", "symptom": "In a narrated demo recorded with CSS `zoom` (take.js's fix for a fixed-width Mendix page), narrate.js's `point()` highlight ring is drawn around the wrong control or off the edge of the frame, and the caption plate is the wrong height and sits outside the film's caption band. Nothing in the take, the beat assertions or the contact sheet reports anything.", "cause": "Under `html{zoom:z}` Chromium reports `getBoundingClientRect()` in ZOOM-ADJUSTED (video) pixels but `getComputedStyle()` and `style.*` in CSS pixels. `point()` read a rect and assigned it straight to `style.left/top/width/height`, so the ring landed at position x z (measured at z=1.6842: a target at (168,202) ringed at (274,330)). The plate had the mirror-image problem: its geometry was declared in CSS pixels, so a 96px bar reached the file as 96 x z = 162 video px against a 184px caption band.", "file": ".claude/skills/mendix/record-narrated-demo/narrate.js (`point`, `css`, `checkOverlay`)", "insight": "The overlay lives in the page's coordinate space and the film is specified in the frame's, and `zoom` is the only conversion between them - so every overlay number is now stated in VIDEO pixels and divided by a zoom passed to `configure()`. The trap is that the conversion runs in opposite directions depending on which API you read it back with, which is why the fix came with `checkOverlay()`: it measures the installed plate against the band and refuses the take, with the control being one line (build the overlay without telling it the zoom -> 'caption plate is 310 video px tall, the band is 184'). A design rule that can be measured in the page should be a check that throws at record time, not a note in a skill - the same argument PRODUCTION.md sec 12 makes for compositions.", "refs": ["ako/mxcli-intro-video video-system/DESIGN-LANGUAGE.md"]} {"area":".claude/skills/packs","date":"2026-09-21","symptom":"A URL-fed Vega-Lite chart in the mendix-vega-charts pack drew its axes and a FULL legend with zero data points, no console error and no Vega warning. The skill stated that \"same-origin requests carry the session cookie, so an endpoint authenticated by session is reachable ... without any token handling\".","cause":"Mendix refuses a session-authenticated request without the session's CSRF token on READS too, not just writes and not just /xas/. The cookie is sent; it is not sufficient. Vega's loader read the 401 body as an empty dataset, so the failure surfaced as a plausible-looking empty chart rather than as an error.","file":".claude/skills/packs/mendix-vega-charts/widget/src/csrf.ts, .../SKILL.md, cmd/mxcli/skillpacks_test.go","insight":"THE LEGEND IS THE TELL: it is built from the spec's scales, not from rows, so a chart with a complete legend and no marks has had its DATA refused, while a chart with a broken legend has a spec problem. That one distinction separates the two hypotheses before any measurement. Two things then send the diagnosis the wrong way and cost the time: document.cookie shows only originURI=/login.html (XASSESSIONID and xasid are httpOnly, so the browser IS sending them and JavaScript cannot see them) and basic auth on the same URL returns the data, which reads as proof the endpoint is fine and the chart is broken. Isolate on the HEADER, not the URL: two requests, one added header, everything else equal -- 401 vs 200, measured on a fresh 11.14.0 app. Skip the plausible wrong turn of adding the header unconditionally: the issue's own suggested loader tests the URI with /^[a-z][a-z0-9+.-]*:\\/\\//i, which passes //elsewhere.example/rows.json (no scheme, another host) and hands that host a working session credential. Resolve with new URL(uri, base) and compare origins instead -- it also gets the converse right, an absolute URL naming the app's own origin IS the app. End-to-end control through vega's real loader against the running app: 0 marks / 3 axes / no error without the token, 1 mark with it, which reproduces the reported symptom exactly.","refs":["ako/mxcli#574"],"ce":[],"mendix":"11.14.0"} +{"area": "skills", "date": "2026-09-22", "symptom": "A workflow inbox over System.WorkflowUserTask, re-sourced from a MICROFLOW to get past the System-module ceiling, drew the right number of cards and every card was COMPLETELY BLANK — no CE code, no console warning, and mxcli check, lint, report and docker check all at 0 errors. The manage-security and system-module skills had offered exactly that microflow data source as the way past the ceiling (ako/ChipCoV4, Mendix 11.14.0; ako/mxcli#587).", "cause": "Not an mxcli defect — a Mendix rule both skills stated as a workaround without having measured it. A microflow does not apply entity access, so its retrieve returns every row, but the runtime RE-APPLIES entity access when it serializes those objects to the client, XPath constraint included: a row the role may not read arrives with every member empty. Both skills now state the rule (\"a microflow data source moves the ROWS, not the MEMBERS\") with the measurement, and point at reading the member inside the microflow and returning a module-owned object.", "file": "`.claude/skills/mendix/manage-security/SKILL.md` (The System-module ceiling); `.claude/skills/mendix/system-module/SKILL.md`; measurement `mdl-examples/bug-tests/security-587-system-member-access.mdl`", "insight": "The probe that settles this in one page: TWO microflow-sourced lists over the SAME retrieve — one over the System objects, one over a module-owned copy whose attribute was read inside the microflow — opened by an Administrator and by a plain User. The row COUNTS are the discriminator and they are equal in all four cells (2 and 2), which is what proves the microflow moved the rows and isolates the loss to serialization; only the System list loses values, and only for the non-admin. Use System.User rather than System.WorkflowUserTask for the probe: it needs no workflow and it shows the mechanism MORE sharply, because System.User's own rule reads [id = '[%CurrentUser%]'] so the non-admin sees exactly one populated row and one blank — per-OBJECT blanking, not per-attribute. That also explains why a user picker 'lists the current user only' and a workflow inbox is entirely blank: same rule, different constraint. The control here is the ROLE, not a before/after build — same binary, same model, two logins — so no A/B rebuild is needed. Two traps in the probe itself: dynamictext content is a static template and renders '[%Name%]' literally, so bind the attribute with a TEXTBOX; and `grant on System.User` PASSES `mxcli check --references` and is refused only by `exec`, so a script carrying one checks clean and then stops part-way."} diff --git a/.claude/skills/mendix/manage-security/SKILL.md b/.claude/skills/mendix/manage-security/SKILL.md index 5f06eadaa5..466bffd7c1 100644 --- a/.claude/skills/mendix/manage-security/SKILL.md +++ b/.claude/skills/mendix/manage-security/SKILL.md @@ -33,27 +33,63 @@ constraint, not a detail: it decides what your screens can be, and finding it la means rebuilding them (ako/mxcli-maintenance-2 designed a technician picker, built it, tested it, and tore it out). -Both consequences are **silent** — the page renders, the data is simply missing, and -`mx check` and `mxcli lint` both pass: +Every consequence is **silent** — the page renders, the data is simply missing, and +`mx check`, `mxcli lint` and `mxcli report` all pass: | What you build | What a non-Administrator sees | |---|---| | A combo box over `System.User` (e.g. "pick a technician") | **The current user only** | | A grid over `System.Workflow` / `System.WorkflowUserTask` | **Empty** | - -Three ways around it, in order of preference: - -1. **Record, don't pick.** Target the task at a *role*, let whoever opens it do the - work, and stamp who acted on completion — a plain association plus a denormalised - name your own module owns, which every role can then read. -2. **A microflow data source.** Microflows bypass entity access by default, so a page - can show data the role cannot read directly. +| Either of those, re-sourced from a **microflow** | **Every row present, every field blank** | + +### The rule: a microflow data source moves the ROWS, not the MEMBERS + +Learn this as a rule rather than as a symptom, because the obvious workaround only +*looks* like it worked, and the same trap is waiting on the next screen. + +A microflow does not apply entity access, so its retrieve returns every row. But the +runtime **re-applies entity access when it serializes those objects to the client, +XPath constraint included** — so a row the role may not read arrives with every +member empty. The list comes out the right length and the cards come out blank. + +Measured in a browser on Mendix 11.14.0 — one page, two microflow-sourced lists over +the *same* `System.User` retrieve, opened by two users: + +| list | as Administrator | as a plain User | +|---|---|---| +| A — the `System.User` objects themselves | `probe_admin`, `probe_viewer` | **(blank)**, `probe_viewer` | +| B — a module-owned copy, `Name` read *inside* the microflow | `probe_admin`, `probe_viewer` | `probe_admin`, `probe_viewer` | + +Both lists hold two rows for both users, so the microflow really did carry the rows +past entity access. Only A loses the values, and it loses them **per object**: +`System.User`'s own rule grants read where `[id = '[%CurrentUser%]']`, which is why +a user picker shows you yourself and nobody else rather than showing nothing. + +`System.WorkflowUserTask` has no such escape hatch for an ordinary role, so a +workflow inbox built this way comes out *entirely* blank — the reported case +(ako/mxcli#587): the inbox drew the right number of cards, every one of them empty, +with `mxcli check`, `lint`, `report` and `docker check` all at 0 errors. + +### What to do instead + +1. **Record, don't pick.** Have the workflow stamp itself against a row your own + module owns — an `ON CREATED MICROFLOW` writing an association plus a plain status + string — and list *those* objects. Target a task at a *role*, let whoever opens it + do the work, and record who acted when they act. +2. **Read it inside a microflow, return your OWN object.** Entity access does not + apply to the retrieve *or* to the member read, only to what crosses to the client + — so copy the values you need onto an entity your module owns (persistent or + non-persistent) and bind the page to that. This is list B above, and it is the + same shape the reporter arrived at independently for user pickers: `Engineer` is a + module-owned entity rather than `System.User`. 3. **Split the page by role.** Keep raw System grids on an Administrator-only page. Mendix hides a button to a page the user may not view, so the link simply does not appear. -Related: `grant … on System.User` is refused outright — the System module's domain -model is not stored in the project, so it has no access rules to add to. +Related: `grant … on System.User` is refused by `exec` — the System module's domain +model is not stored in the project, so it has no access rules to add to. The refusal +is at execution, not at `mxcli check`, so a script carrying one passes `check` and +then stops part-way through. ## Syntax Reference diff --git a/.claude/skills/mendix/system-module/SKILL.md b/.claude/skills/mendix/system-module/SKILL.md index 709c004c6d..eee046dae9 100644 --- a/.claude/skills/mendix/system-module/SKILL.md +++ b/.claude/skills/mendix/system-module/SKILL.md @@ -12,13 +12,18 @@ The `System` module is a built-in Mendix module present in every application. It **No project module can widen access to `System.User`, `System.Workflow` or `System.WorkflowUserTask`.** Their access comes from the System module's own roles, and a `grant` in your module cannot raise it — so any UI over them is -Administrator-only unless the data is denormalised into your own entities or reached -through a microflow data source (microflows bypass entity access by default). +Administrator-only unless the data is denormalised into entities your own module owns. + +**A microflow data source is not the way out.** It moves the ROWS, not the MEMBERS: +the retrieve is unconstrained, but the runtime re-applies entity access when it +serializes those objects to the client, so the list is the right length and every +field is blank (measured on 11.14.0, ako/mxcli#587). Read the members *inside* the +microflow and return an object your module owns. It fails **silently**: a combo box over `System.User` lists the current user only, a -grid over `System.Workflow` renders empty, and both `mx check` and `mxcli lint` pass. -See the System-module ceiling section in [manage-security](../manage-security/SKILL.md) -for the three ways around it. +grid over `System.Workflow` renders empty, and `mx check`, `mxcli lint` and +`mxcli report` all pass. See the System-module ceiling section in +[manage-security](../manage-security/SKILL.md) for the measurement and the remedies. ## Reference files diff --git a/mdl-examples/bug-tests/security-587-system-member-access.mdl b/mdl-examples/bug-tests/security-587-system-member-access.mdl new file mode 100644 index 0000000000..2411d5661e --- /dev/null +++ b/mdl-examples/bug-tests/security-587-system-member-access.mdl @@ -0,0 +1,115 @@ +-- ako/mxcli#587 — the manage-security skill offered a microflow data source as +-- the way past the System-module ceiling. It is not one: a microflow data +-- source moves the ROWS, not the MEMBERS. +-- +-- A microflow does not apply entity access, so its retrieve returns every row. +-- The runtime then RE-APPLIES entity access when it serializes those objects to +-- the client, XPath constraint included — so a row the role may not read arrives +-- with every member empty. The list is the right length and the cards are blank, +-- with `mxcli check`, `lint`, `report` and `mx check` all at 0 errors. +-- +-- THIS FILE IS THE MEASUREMENT, not a unit test. The symptom is a property of +-- the RUNNING APP (.claude/skills/verify-in-runtime.md), so nothing below the +-- browser can see it. To reproduce, on a blank project: +-- +-- mxcli new SecProbe --version 11.14.0 --theme none --layout none --skip-init \ +-- --skip-build --output-dir /root/sec +-- mxcli exec mdl-examples/bug-tests/security-587-system-member-access.mdl \ +-- -p /root/sec/SecProbe.mpr +-- mxcli run --local -p /root/sec/SecProbe.mpr --setup --ensure-db +-- mxcli run --local -p /root/sec/SecProbe.mpr +-- # open http://127.0.0.1:8080/ and sign in as each demo user below +-- +-- Measured on Mendix 11.14.0. Both lists hold TWO rows for BOTH users — the +-- microflow really did carry the rows past entity access — and only list A loses +-- the values: +-- +-- list as probe_admin as probe_viewer +-- A the System.User objects probe_admin, (blank), +-- probe_viewer probe_viewer +-- B a module-owned copy, Name read probe_admin, probe_admin, +-- INSIDE the microflow probe_viewer probe_viewer +-- +-- A loses them PER OBJECT: System.User's own rule grants read where +-- [id = '[%CurrentUser%]'], which is why a user picker shows you yourself and +-- nobody else rather than showing nothing. System.WorkflowUserTask has no such +-- escape hatch for an ordinary role, so the reporter's workflow inbox came out +-- entirely blank. + +create module Sec; + +-- A row the project owns, so every role can be granted it. +create non-persistent entity Sec.UserRow ( + DisplayName: String(200) +); + +create module role Sec.Viewer; +create module role Sec.Admin; + +-- A: rows straight out of System.User. +create microflow Sec.GetSystemUsers () +returns list of System.User as $Users +begin + retrieve $Users from System.User; + return $Users; +end; + +-- B: the same rows, with Name read inside the microflow and copied onto an +-- entity this module owns. Entity access applies to neither the retrieve nor the +-- member read — only to what crosses to the client. +create microflow Sec.GetUserRows () +returns list of Sec.UserRow as $Rows +begin + $Rows = create list of Sec.UserRow; + retrieve $Users from System.User; + loop $u in $Users begin + $row = create Sec.UserRow (DisplayName = $u/Name); + add $row to $Rows; + end loop; + return $Rows; +end; + +create or replace page Sec.Home ( + title: 'System vs owned', + Layout: Atlas_Core.Atlas_Default +) { + container cA { + dynamictext hA (content: 'A: System.User via microflow datasource') + listview lvSystem (DataSource: MICROFLOW Sec.GetSystemUsers) { + textbox tbSysName (Label: 'SysName', Attribute: Name) + } + dynamictext hB (content: 'B: Sec.UserRow via microflow datasource') + listview lvRows (DataSource: MICROFLOW Sec.GetUserRows) { + textbox tbRowName (Label: 'RowName', Attribute: DisplayName) + } + } +} +/ +alter project security level production; + +-- Sec.Viewer may read everything this MODULE owns and may run both microflows +-- and open the page — so anything blank is the System module's doing, not a +-- grant we forgot. +alter user role User add module roles (Sec.Viewer); +alter user role Administrator add module roles (Sec.Admin); + +grant Sec.Viewer on Sec.UserRow (read *, write *, create, delete); +grant Sec.Admin on Sec.UserRow (read *, write *, create, delete); + +grant execute on microflow Sec.GetSystemUsers to Sec.Viewer, Sec.Admin; +grant execute on microflow Sec.GetUserRows to Sec.Viewer, Sec.Admin; +grant view on page Sec.Home to Sec.Viewer, Sec.Admin; + +create or replace navigation Responsive + home page Sec.Home + menu ( + menu item 'Home' page Sec.Home icon glyph 57377; + ) +; + +-- The control is the ROLE, not the build: Administrator sees both names in list +-- A, so a blank there is entity access and not a broken page. +drop demo user if exists 'demo_user'; +drop demo user if exists 'demo_administrator'; +create demo user 'probe_admin' password 'Passw0rdProbe1' (Administrator); +create demo user 'probe_viewer' password 'Passw0rdProbe1' (User); From 761c3c5e13eaa7c61a14b55cb3debceaf62f8d2c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:10:28 +0000 Subject: [PATCH 06/38] Alias ProjectVersion instead of duplicating it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit modelsdk/mpr/version declared its own ProjectVersion struct with the same seven fields as mdl/types.ProjectVersion rather than aliasing it, so the two were unrelated Go types that print under the same name: a value could not cross the mdl/ <-> modelsdk/ boundary without a conversion. The sdk/mpr copy deleted in the legacy-engine retirement aliased the canonical type; this one did not, and CLAUDE.md's shared-types rule asks for the alias. A same-shape duplicate is invisible to every signal except an assignment across the boundary. It compiles, the tests pass, and the error it eventually produces names the same type on both sides of "want" — the shape that cost a session once already on widget BSON, where a delegation handed back a v2 bson.D to a caller asserting the v1 one and the failure read "widget type is bson.D, want bson.D". So the guard is a COMPILE-TIME assertion rather than a test body: var _ *types.ProjectVersion = (*version.ProjectVersion)(nil) which builds only under an alias. Written first, it failed to compile with three errors showing the two types non-interchangeable in both directions — that failure is the reproduction. Two measurements made the cleanup safe rather than brave. The four methods this package redeclared (IsAtLeast, IsAtLeastFull, String, IsMPRv2) were diffed against types' BEFORE assuming they were redundant: identical behaviour, IsAtLeast differing only in early-return style. And the two that could not survive as methods on an aliased type, IsSupported and SupportsFeature, were counted first — zero callers anywhere, in this package or any other — so they go with Feature, MinVersion, featureVersions and SupportedVersionRange. That map described itself as "the fallback when the YAML registry is unavailable"; the live registry is sdk/versions/mendix-{9,10,11}.yaml read through checkFeature, so this was a second hand-maintained copy with nothing reading it. CLAUDE.md cited this as an open cautionary case; it now states the rule and points at the compile-time guard. Gates: build, vet (incl. -tags integration), go test ./... (exit 0), check-mdl (611), check-findings (1,164). Four cmd/mxcli skill-reading tests failed once in a full run interleaved with make check-mdl, which runs sync-skills (rsync --delete into cmd/mxcli/skills/); they pass in isolation, on clean main, and in an uninterleaved full run, so they are not this change. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../fix-issue/findings/mdl-backend.jsonl | 1 + CLAUDE.md | 2 +- modelsdk/mpr/version/version.go | 136 ++++-------------- modelsdk/mpr/version/version_alias_test.go | 73 ++++++++++ 4 files changed, 100 insertions(+), 112 deletions(-) create mode 100644 modelsdk/mpr/version/version_alias_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-backend.jsonl b/.claude/skills/fix-issue/findings/mdl-backend.jsonl index e081d80995..5ceb0ecc22 100644 --- a/.claude/skills/fix-issue/findings/mdl-backend.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-backend.jsonl @@ -123,3 +123,4 @@ {"area": "mdl/backend", "date": "2026-09-20", "symptom": "`DROP ENTITY` left every CROSS-MODULE association pointing at the deleted entity in place. Dropping the local BY-ID (FROM) end made mxbuild 11.14.0 unable to LOAD the project: `System.AggregateException \u2026 (The given key '' was not present in the dictionary.)` at `StreamingBsonUnitReader.ResolvePostponedProperties()` \u2014 no CE code, no document named, so the obvious reading is 'the project is corrupt, restore from git'. Dropping the BY-NAME (TO) end is milder and still wrong: CE1613 at the cross-module association. `show associations` shows a raw GUID where the parent entity should be.", "cause": "`removeAssocsReferencing` swept `dm.AssociationsItems()` and asserted `*genDm.Association` per item, so the SEPARATE `CrossAssociations` collection was never looked at. Fixed with `removeCrossAssocsReferencing`, matching BOTH ends because a cross-module association addresses them differently \u2014 FROM by element id (local), TO by qualified name (another module) \u2014 called in DeleteEntity locally and in its cascade over the other domain models.", "file": "`mdl/backend/modelsdk/domainmodel_alter.go` (removeCrossAssocsReferencing, DeleteEntity)", "insight": "**Reported against a view entity; nothing about it was view-entity specific.** The reporter met it dropping view entities (whose associations are DERIVED from OQL, so there is no CREATE ASSOCIATION to undo) and filed it that way. The first probe \u2014 a view entity and its source entity in the SAME module \u2014 did not reproduce at all, and that negative is the useful one: it says the variable is cross-module, not view-ness. A plain `create association A.X from A.X to B.Y` plus `drop entity A.X` reproduces the identical crash. Two lessons: when a repro fails, vary the dimension the report did not mention before doubting the report, and treat a collection-typed `.(*T)` assertion in a cascade as a place where a sibling type hides. mxbuild's diagnostic distinguishes the two ends for free \u2014 a dangling 16-byte pointer is a LOAD crash, a dangling qualified name is CE1613 \u2014 so testing only one end proves half the fix.", "refs": ["#553", "#556"]} {"area": "mdl/backend", "date": "2026-09-21", "symptom": "`alter settings workflows add group 'Auditors'` reports \"Added workflow group: Auditors (3 group(s))\" and writes nothing \u2014 `show workflow groups` still lists 2, and `mx check` is 0 errors either way", "cause": "`UpdateProjectSettings` overlays the workflows part field by field onto the PRESERVED raw part, so a child LIST that nothing rebuilds is carried through from disk unchanged. Adding `Groups` to the semantic model and to the read path is not enough; the write needs `settingsoverlay.WorkflowGroups(ws, rawPart)`. Identical shape to the enabled-language list the same function already documents", "file": "`mdl/backend/modelsdk/settings_write.go` + `mdl/settingsoverlay/settingsoverlay.go` (`WorkflowGroups`)", "insight": "For anything under Settings$ProjectSettings, the executor's success message proves NOTHING \u2014 it reports the in-memory model, and the overlay is where a list quietly fails to land. Assert on the re-read document, not the handler's output. Two more things a reference project settles in one dump and a guess gets wrong: the `Groups` typed-array marker is 2, not the 3 every other settings child list uses (`ArrayMarker` preserves a stored one, but the fallback matters on a fresh list), and the element's `$ID` is the RUNTIME's identity \u2014 a booted 11.13.0 app keys `system$workflowgroup.modelguid` on it, byte-identical once the .NET GUID field order is undone, so re-minting it on a description edit would orphan every group membership with a perfectly valid model. Control: deleting the one overlay call reproduces the symptom verbatim. mendixlabs/mxcli#272", "refs": ["mendixlabs/mxcli#272"]} {"area": "mdl/backend", "date": "2026-09-21", "symptom": "`CREATE OR MODIFY VIEW ENTITY` that changed ONLY the OQL printed `Unchanged view entity: \u2026` while `describe entity` showed the new query stored. Changing the attribute list as well reported `Modified` correctly, which is why it hid. Also: the OQL document's unit was replaced under a FRESH GUID on every run, even a byte-identical one, so an MDL-generated project could never come back clean in git (one of the four units #556 measured).", "cause": "A view entity's OQL lives in a separate `DomainModels$ViewEntitySourceDocument` unit, and the executor DELETED it and INSERTED a fresh one on every write. `ReportMutation` downgrades the verb when writes were offered and none landed, but the counters are incremented only at the update choke points (`writer_core.go` reconcileWithStored / MoveUnit) \u2014 `InsertUnit` is not counted at all. So the domain-model unit was offered and correctly elided, the OQL write was invisible, and the report believed the half it could see. Fixed with `WriteViewEntitySourceDocument`, which keeps the stored unit's id and goes through `UpdateRawUnit` \u2192 reconcile: an identical query is elided, a changed one lands and is counted, duplicates are still cleared.", "file": "`mdl/backend/modelsdk/move_view_write.go` (WriteViewEntitySourceDocument, encodeViewEntitySourceDocument), `mdl/executor/cmd_entities.go`", "insight": "**The first fix that comes to mind \u2014 count InsertUnit \u2014 would have swapped a false \"Unchanged\" for a false \"Modified\".** Measuring before changing is what caught it: re-running a BYTE-IDENTICAL script still re-minted the source document's unit id, so counting inserts would have made every view-entity statement report Modified forever. The right fix was the one ADR-0008 already mandates (wire the write path to canon.Reconcile), and it fixes the churn and the verb together. Generalisation worth remembering: any content that reaches storage through `InsertUnit` is invisible to the elision check, so a statement whose only landing write is a NEW unit can still be mis-reported \u2014 `MoveUnit` has a comment explaining it was counted for exactly this reason, and insert/delete were missed. Control the fix on the identical re-run, not just the changed one.", "refs": ["#583", "#556", "#910"]} +{"area":"mdl/backend","date":"2026-09-22","symptom":"modelsdk/mpr/version.ProjectVersion declared its own struct with the same seven fields as mdl/types.ProjectVersion instead of aliasing it, so a *version.ProjectVersion could not be passed where a *types.ProjectVersion was wanted and vice versa — two unrelated Go types that both print as 'ProjectVersion'.","cause":"The deleted sdk/mpr/version aliased the canonical type (`type ProjectVersion = types.ProjectVersion`); this copy declared a duplicate. CLAUDE.md's shared-types rule asks for the alias, and nothing enforced it. The duplication survived the legacy-engine retirement because it compiles perfectly — the two declarations are field-for-field identical, so only an assignment ACROSS the boundary reveals them as different types.","file":"modelsdk/mpr/version/version.go","fix":"Made it an alias. The four methods it redeclared (IsAtLeast, IsAtLeastFull, String, IsMPRv2) were verified semantically identical to types' first — IsAtLeast differed only in early-return style, same truth table — and now come from types. IsSupported/SupportsFeature could not survive as methods on an aliased type and had ZERO callers anywhere (measured), so they went with Feature, MinVersion, featureVersions and SupportedVersionRange; that map called itself 'the fallback when the YAML registry is unavailable' and the live registry is sdk/versions/mendix-{9,10,11}.yaml via checkFeature.","insight":"A same-shape duplicate type is invisible to every signal except an assignment across the package boundary: it compiles, tests pass, and the error it eventually produces names the same type on both sides of 'want'. So the guard is a COMPILE-TIME assertion, not a runtime test — `var _ *types.ProjectVersion = (*version.ProjectVersion)(nil)` builds only under an alias and fails to build under a duplicate, which is strictly stronger than anything a test body can assert. Write it before the fix and watch it fail to compile; that failure IS the reproduction. Two measurements that made the cleanup safe rather than brave: diff the method BODIES before assuming the redeclarations are redundant (identical behaviour, different style, is the common case and the dangerous one is the near-miss), and count callers of anything the alias forces you to drop — here six exported symbols had zero. Unrelated trap hit while verifying: four cmd/mxcli tests that read skill files failed once in a full `go test ./...` interleaved with `make check-mdl`, which runs sync-skills (rsync --delete into cmd/mxcli/skills/). They pass in isolation, on clean main, and in an uninterleaved full run — do not attribute a skills-reading test failure to your change without re-running it alone."} diff --git a/CLAUDE.md b/CLAUDE.md index 780027be91..e20dd0bef9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -639,7 +639,7 @@ All executor code must go through the backend abstraction layer. **`sdk/mpr` no - [ ] **Mock stub in `mdl/backend/mock/`** — every new backend method has a `Func`-field stub with a descriptive `"MockBackend.X not configured"` error default (not `nil, nil`) - [ ] **Compile-time interface check** — new backend implementations have `var _ backend.SomeInterface = (*impl)(nil)` - [ ] **ALTER operations use mutator pattern** — page/workflow mutations go through `ctx.Backend.OpenPageForMutation()` / `OpenWorkflowForMutation()`, not inline BSON construction -- [ ] **New shared types in `mdl/types/`** — a type used by more than one layer goes in `mdl/types/` and the others alias it (`type Foo = types.Foo`), never as duplicate definitions. `modelsdk/mpr/version.ProjectVersion` is the cautionary case: it *duplicates* `types.ProjectVersion` instead of aliasing it, so the two are unrelated Go types that print under the same name +- [ ] **New shared types in `mdl/types/`** — a type used by more than one layer goes in `mdl/types/` and the others alias it (`type Foo = types.Foo`), never as duplicate definitions. A same-shape duplicate compiles and tests green; it shows up only as an assignment failure *across* the boundary, naming the same type on both sides of "want". `modelsdk/mpr/version.ProjectVersion` was that case and is now an alias — the guard is a compile-time assertion (`var _ *types.ProjectVersion = (*version.ProjectVersion)(nil)`, `version_alias_test.go`), which builds only under an alias and so is stronger than anything a test body can assert - [ ] **Map iteration is deterministic** — any map iterated for serialization output must sort keys first (`sort.Strings(keys)` pattern); non-deterministic output causes flaky diffs and BSON instability - [ ] **Pluggable widgets via WidgetEngine** — new pluggable widget support uses `.def.json` + `WidgetRegistry`; no hardcoded BSON widget builders in the executor diff --git a/modelsdk/mpr/version/version.go b/modelsdk/mpr/version/version.go index ba387b6ad6..2b5a737075 100644 --- a/modelsdk/mpr/version/version.go +++ b/modelsdk/mpr/version/version.go @@ -8,31 +8,23 @@ import ( "fmt" "strconv" "strings" -) - -// ProjectVersion contains version information for a Mendix project. -type ProjectVersion struct { - // ProductVersion is the full Mendix version string (e.g., "10.18.0", "11.6.0") - ProductVersion string - - // BuildVersion is the build version, usually same as ProductVersion - BuildVersion string - - // FormatVersion is the MPR format version (1 for legacy, 2 for mprcontents) - FormatVersion int - - // SchemaHash is the SHA256 hash of the metamodel schema - SchemaHash string - // MajorVersion is the major version number (e.g., 10, 11) - MajorVersion int - - // MinorVersion is the minor version number (e.g., 18, 6) - MinorVersion int + "github.com/mendixlabs/mxcli/mdl/types" +) - // PatchVersion is the patch version number (e.g., 0, 1) - PatchVersion int -} +// ProjectVersion is an alias for types.ProjectVersion, the canonical +// declaration. All its methods — IsAtLeast, IsAtLeastFull, String, IsMPRv2 — +// are defined there. +// +// It is an ALIAS and not a struct of its own on purpose. This package used to +// declare a duplicate whose fields matched types.ProjectVersion exactly, which +// made the two unrelated Go types that print under the same name: a value could +// not cross the mdl/ ↔ modelsdk/ boundary without a conversion, and a mismatch +// reported itself as a tautology rather than as a type error anyone could read. +// The sdk/mpr copy deleted in the legacy-engine retirement aliased the canonical +// type; this one did not, and that asymmetry is what CLAUDE.md's shared-types +// rule exists to prevent. +type ProjectVersion = types.ProjectVersion // DefaultVersion returns the default version (11.6.0) used when detection fails. func DefaultVersion() *ProjectVersion { @@ -100,91 +92,13 @@ func parseVersion(version string) (major, minor, patch int) { return } -// String returns the product version string. -func (v *ProjectVersion) String() string { - return v.ProductVersion -} - -// IsMPRv2 returns true if the project uses MPR v2 format (mprcontents folder). -func (v *ProjectVersion) IsMPRv2() bool { - return v.FormatVersion >= 2 -} - -// IsAtLeast returns true if this version is at least the specified major.minor version. -func (v *ProjectVersion) IsAtLeast(major, minor int) bool { - if v.MajorVersion > major { - return true - } - if v.MajorVersion == major && v.MinorVersion >= minor { - return true - } - return false -} - -// IsAtLeastFull returns true if this version is at least the specified major.minor.patch version. -func (v *ProjectVersion) IsAtLeastFull(major, minor, patch int) bool { - if v.MajorVersion > major { - return true - } - if v.MajorVersion == major && v.MinorVersion > minor { - return true - } - if v.MajorVersion == major && v.MinorVersion == minor && v.PatchVersion >= patch { - return true - } - return false -} - -// SupportedVersionRange defines the range of Mendix versions supported for read/write. -var SupportedVersionRange = struct { - MinMajor int - MaxMajor int -}{ - MinMajor: 9, - MaxMajor: 11, -} - -// IsSupported returns true if this version is within the supported range for writing. -func (v *ProjectVersion) IsSupported() bool { - return v.MajorVersion >= SupportedVersionRange.MinMajor && - v.MajorVersion <= SupportedVersionRange.MaxMajor -} - -// SupportsFeature checks if a specific feature is available in this version. -func (v *ProjectVersion) SupportsFeature(feature Feature) bool { - minVersion, ok := featureVersions[feature] - if !ok { - return false - } - return v.IsAtLeast(minVersion.Major, minVersion.Minor) -} - -// Feature represents a Mendix feature that may or may not be available. -type Feature string - -// Known features with version requirements -const ( - FeatureViewEntities Feature = "ViewEntities" - FeatureAssociationStorage Feature = "AssociationStorageFormat" - FeatureMPRv2 Feature = "MPRv2Format" - FeatureBusinessEvents Feature = "BusinessEvents" - FeatureWorkflows Feature = "Workflows" - FeaturePortableApp Feature = "PortableApp" -) - -// MinVersion represents a minimum version requirement. -type MinVersion struct { - Major int - Minor int -} - -// featureVersions maps features to their minimum required versions. -// This is the fallback when the YAML registry is unavailable. -var featureVersions = map[Feature]MinVersion{ - FeatureViewEntities: {Major: 10, Minor: 18}, - FeatureAssociationStorage: {Major: 11, Minor: 0}, - FeatureMPRv2: {Major: 10, Minor: 18}, - FeatureBusinessEvents: {Major: 10, Minor: 0}, - FeatureWorkflows: {Major: 9, Minor: 0}, - FeaturePortableApp: {Major: 11, Minor: 6}, -} +// Removed with the alias: IsSupported, SupportsFeature, Feature and its +// constants, MinVersion, featureVersions and SupportedVersionRange. +// +// They could not survive as methods on an aliased type, and nothing called +// them — measured, zero references outside their own declarations, in this +// package or any other. They were also a second, hand-maintained copy of the +// feature registry: the live one is sdk/versions/mendix-{9,10,11}.yaml, read +// through checkFeature in mdl/executor/cmd_features.go, and featureVersions +// described itself as "the fallback when the YAML registry is unavailable" — +// a fallback with no caller is a list that can only drift. diff --git a/modelsdk/mpr/version/version_alias_test.go b/modelsdk/mpr/version/version_alias_test.go new file mode 100644 index 0000000000..7d57fa2c9b --- /dev/null +++ b/modelsdk/mpr/version/version_alias_test.go @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 + +package version_test + +// ProjectVersion must be an ALIAS of types.ProjectVersion, not a struct that +// happens to match it field for field. +// +// The distinction is invisible in an error message and that is the whole point: +// two identically-named types from different packages both print as +// "version.ProjectVersion", so a mismatch reads as a tautology. The same shape +// cost a session once already, on widget BSON, where a delegation handed back a +// v2 bson.D to a caller asserting the v1 one and the failure read +// "widget type is bson.D, want bson.D". +// +// The deleted sdk/mpr/version aliased the canonical type; this copy declared its +// own, which is what made the two engines' version values non-interchangeable. + +import ( + "testing" + + "github.com/mendixlabs/mxcli/mdl/types" + "github.com/mendixlabs/mxcli/modelsdk/mpr/version" +) + +// Compile-time: only an alias satisfies this. A duplicate struct — even one +// whose fields match exactly — fails to build here. +var _ *types.ProjectVersion = (*version.ProjectVersion)(nil) + +func TestProjectVersionIsTheCanonicalType(t *testing.T) { + // Assignable in both directions without conversion, which is what callers + // crossing the mdl/ ↔ modelsdk/ boundary actually need. + var fromTypes *types.ProjectVersion = version.DefaultVersion() + if fromTypes == nil { + t.Fatal("DefaultVersion returned nil") + } + var back *version.ProjectVersion = fromTypes + if back.ProductVersion != fromTypes.ProductVersion { + t.Fatalf("round trip changed the value: %q vs %q", back.ProductVersion, fromTypes.ProductVersion) + } +} + +// The behaviour the alias must not change. These ran against the local struct +// before the alias and must give the same answers after it, or the two +// declarations were not equivalent after all. +func TestProjectVersionBehaviourIsUnchanged(t *testing.T) { + v := &version.ProjectVersion{MajorVersion: 11, MinorVersion: 6, PatchVersion: 2, FormatVersion: 2, ProductVersion: "11.6.2"} + + for _, c := range []struct { + name string + got bool + want bool + }{ + {"IsAtLeast lower major", v.IsAtLeast(10, 0), true}, + {"IsAtLeast same major lower minor", v.IsAtLeast(11, 5), true}, + {"IsAtLeast same major same minor", v.IsAtLeast(11, 6), true}, + {"IsAtLeast same major higher minor", v.IsAtLeast(11, 7), false}, + {"IsAtLeast higher major", v.IsAtLeast(12, 0), false}, + {"IsAtLeastFull same patch", v.IsAtLeastFull(11, 6, 2), true}, + {"IsAtLeastFull higher patch", v.IsAtLeastFull(11, 6, 3), false}, + {"IsAtLeastFull lower minor", v.IsAtLeastFull(11, 5, 9), true}, + {"IsMPRv2", v.IsMPRv2(), true}, + } { + if c.got != c.want { + t.Errorf("%s = %v, want %v", c.name, c.got, c.want) + } + } + if v.String() != "11.6.2" { + t.Errorf("String() = %q, want 11.6.2", v.String()) + } + if v1 := (&version.ProjectVersion{FormatVersion: 1}); v1.IsMPRv2() { + t.Error("FormatVersion 1 reported as MPRv2") + } +} From 79f49c86c8016841b633c5c4fbd3ef4f4fef60b8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:27:58 +0000 Subject: [PATCH 07/38] fix(widgets): UPDATE WIDGETS must not report success after writing nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `update widgets` counted widgets it FOUND, not assignments that SUCCEEDED, so a run where everything was refused still claimed success --- immediately after warning about each refusal. Measured on a blank 11.12.2 project with two Data grid 2 widgets: Found 2 widget(s) in 2 container(s) matching the criteria Warning: Failed to set 'Compact' on dgA: pluggable property … not found Warning: Failed to set 'Striped' on dgA: pluggable property … not found …same for dgB… Updated 2 widget(s) Note: Run 'refresh catalog full force' to update the catalog with changes. Four failures out of four, an instruction to pick up changes that did not exist, and exit 0. `updated++` sat outside the assignment loop and was unconditional, so the counter meant "this widget was found". The same counter gated `mutator.Save()`, so a container whose every assignment failed was saved anyway. The outcome is now the three things that actually happen --- changed, matched but unwritable, listed in the catalog but absent from the document --- because rounding the third into either of the others is how a stale catalog reads as success. Save is gated on something having changed, the catalog note only prints when it is true, and a run that matched widgets and wrote none of them exits non-zero with a message naming the likely cause. Updated 0 widget(s) 2 widget(s) matched but had no property that could be set Error: no widget was updated: 4 assignment(s) could not be applied. A design property (Atlas styling) is not a pluggable widget property and cannot be set this way — see `mxcli syntax page.styling` exit 1 DRY RUN had the same defect one step earlier, and is the worse half because the syntax help says to run it first: it printed `Would set …` without attempting anything. It now applies the assignments to a discardable copy (`pagemutator.Probe`, the seam `mxcli check` already uses for ALTER PAGE SET) and reports `Cannot set` / `Would update 0`. Best-effort: a mutator without a probe keeps the optimistic preview, which is no worse than before. Severity was bounded, and the commit says so rather than letting this read as corruption: the rebuilt document was semantically identical, so ADR-0008 elision skipped the write --- no mprcontents/ unit changed mtime, `mx check` stayed at 0 errors. A reporting defect, not a data one. Controls, all passing: a real pluggable property still counts, saves, and round-trips (`PageSize: 25` via `describe page`, `mx check` 0 errors); a partial run reports both halves; a widget missing from the document is neither changed nor a property failure. Not fixed here: `Compact`/`Striped` are Atlas DESIGN properties in Appearance.DesignProperties, which SetWidgetProperty does not reach. That capability gap is ako/mxcli#515. Fixes ako/mxcli#520 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LYaTtzjmMCUjo8XAVN1sQx --- .../fix-issue/findings/mdl-executor.jsonl | 1 + .../widgets-520-bulk-update-false-success.mdl | 75 +++++++++ mdl/executor/cmd_widgets.go | 141 +++++++++++++--- .../widget_bulk_update_outcome_test.go | 156 ++++++++++++++++++ 4 files changed, 351 insertions(+), 22 deletions(-) create mode 100644 mdl-examples/bug-tests/widgets-520-bulk-update-false-success.mdl create mode 100644 mdl/executor/widget_bulk_update_outcome_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 3a491f5698..7d0073f414 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -670,3 +670,4 @@ {"area": "mdl/executor", "date": "2026-09-21", "symptom": "`create or modify entity` drops an attribute a LATER script added, silently. Reported shape: entity created in 01-domain-core.mdl, a calculated attribute added in 03-logic.mdl (its microflow does not exist until then); re-running slice 01 ALONE rebuilt the entity from its own statement and removed the attribute, with `Modified entity: ServiceCore.LithoSystem` as the only output. It surfaced two slices later as `[CE1613] \"The selected attribute 'ServiceCore.LithoSystem.OpenRequestCount' no longer exists.\" at Text 'dtOpen'` — an error naming the PAGE, never the script that removed the attribute. `mxcli check … -p app.mpr --references` said \"Check passed!\".", "ce": "CE1613", "rules": ["MDL087"], "cause": "Half the ask was already shipped and half was not, and the report could not tell them apart. exec's warning (droppedEntityMembers, findings #24, landed 320a304 two weeks before the report) DOES fire — measured on a real 11.6.6 project re-running the reporter's slice 01, it prints the attribute by name — so the reporter was on an older binary. What genuinely did not exist was the issue's second ask: `check` had no project-aware pass for member loss at all, so the one command that runs BEFORE anything is written was the silent one. Added CheckEntityMemberDrops (MDL087, warning) to cmd_check.go's catalog-backed tier, and refactored droppedEntityMembers to share its comparison.", "file": "`mdl/executor/validate_entity_member_drops.go` (new: entityMemberSet, droppedMembers, CheckEntityMemberDrops), `mdl/executor/cmd_entities.go` (droppedEntityMembers now delegates), `cmd/mxcli/cmd_check.go` (projectViolations)", "insight": "**Reproduce before theorising when the report predates a fix in the same area** — exec already printed the exact line the issue asks for, so reading the issue text alone leads either to 'already fixed, close it' or to reimplementing the shipped half. Running the reporter's own sequence against a real project separated the two halves in one command each, and the isolated-slice check printing `Check passed!` is what identified the actual gap. **A check-time twin of an exec-time warning must NOT be the same computation.** exec is per-statement because it is applying statements; check sees the whole script, so it has to be the NET effect — a script that rebuilds an entity and then `alter entity … add attribute`s the members back loses nothing, and that is the IDIOMATIC full-script order, so a per-statement port would warn on every correct script and be switched off within a day. **Intent has to be tracked, not inferred from the outcome**: `drop attribute` / `rename attribute` / `drop entity` produce the same before/after diff as the accident, and a pure diff cannot separate them. Both of those are separate controls, and the naive implementation fails each one specifically (measured: stubbing the net/intent logic fails TestMDL087_ExplicitRemovalIsSilent on 3 of 4 spellings while the positive test still passes — so the positive test alone proves nothing). **One comparison, two layers**: the audit system fields and an omitted `extends` were reported by exec and would have been missed by a second hand-written diff, which is why droppedEntityMembers was refactored onto the shared entityMemberSet rather than copied. An audit pseudo-type (`AutoOwner`) is a FLAG, not an attribute — exec `continue`s past it — so counting it as one makes a faithful restatement read as a drop.", "refs": ["ako/mxcli#562", "findings #24", "findings #13"]} {"area": "mdl/executor", "date": "2026-09-21", "symptom": "`retrieve $AccountList from Administration.Account sort by System.Language.Code asc;` — MDL that `mxcli describe` had just emitted — passed `mxcli check` and was refused by `mxcli exec`: \"sort by attribute 'System.Language.Code' does not belong to entity 'Administration.Account'\". Reported as a check/exec inconsistency (mendixlabs/mxcli#1152); the real defect is that the round trip cannot replay its own output for any sort over an association reached from an ANCESTOR.", "cause": "inferSortEntityRefSteps searched ONE domain model — the retrieved entity's own module — for associations whose parent was the retrieved entity ITSELF, and qualified the association it found with the retrieved entity's module. All three assumptions hold only when the hop starts on the retrieved entity in its own module. Administration.Account reaches System.Language through System.User_Language, declared on System.User and stored in the System module: parent is an ancestor, the domain model is another module's, and the qualified name carries THAT module. Rewritten as a generalization-chain walk that looks each ancestor up in its own module and qualifies the association with the module storing it; the destination end is matched with entityIsSubtypeOf rather than by equality, since an association may point at a specialization of the entity that declares the attribute.", "file": "`mdl/executor/cmd_microflows_builder_actions.go` (inferSortEntityRefSteps); tests `mdl/executor/cmd_microflows_sort_association_test.go`, `mdl/backend/modelsdk/microflow_retrievesort_test.go`; example `mdl-examples/bug-tests/microflow-1152-sort-over-association.mdl`", "insight": "**The second control is the one that pays.** Reverting the fix reproduces the refusal, which only proves the test fires. The control that taught something was building a binary that DERIVES the hop and does not WRITE it — exec succeeds and mxbuild 11.12.3 answers CE7247 \"Cannot sort on attribute 'System.Language.Code'. Attribute 'System.Language.Code' is not an attribute of entity 'Administration.Account'\" — the executor's refusal message almost word for word, from the other end of the pipeline. That is what fixes the qualified name as load-bearing: the stored EntityRefStep must read System.User_Language, and the pre-existing code would have written Administration.User_Language had it found anything at all. **Skip the theory that check is missing a rule**: check has no sort-attribute rule at all and resolves no hops, so it was never going to disagree with exec here — the inconsistency in the report is a symptom of the false refusal, not a second defect. **Known residue, stated because the round trip rests on it**: DESCRIBE emits only the attribute's qualified name, so where several associations reach one entity the replay picks the nearest ancestor's first and can silently land on the other hop. Spelling the hop needs grammar (sortColumn is qualifiedName|IDENTIFIER, no `/` path) and is a language change, not a fix."} {"area": "mdl/executor", "date": "2026-09-21", "symptom": "Follow-up to the sort-hop inference fix: with the hop derivable but not SAYABLE, `describe → exec` still silently changed the program wherever two associations reach the same entity. Measured on 11.12.3 with Order_ShipTo and Order_BillTo (both Order -> Address): a microflow sorting by the BILLING address came back sorting by the SHIPPING one, `mx check` 0 errors on both sides. Same for a page datasource's sort bar.", "cause": "DESCRIBE emitted only the sort attribute's qualified name and the reader never looked at the hop at all — `sortItemsFromRaw` read AttributeRef.Attribute and skipped AttributeRef.EntityRef, so the association was written and never read back. MDL had no spelling for it either (`sortColumn : (qualifiedName | IDENTIFIER)`). Closed end to end: sortColumn takes `qualifiedName (SLASH qualifiedName)*` (the shape MDLCatalog.g4 already uses for Association/Entity), SortColumnDef/OrderByItemV3 carry the hops, the executor resolves the NAMED association instead of inferring, both readers reconstruct EntityRef.Steps, both describers emit `Assoc/.../Attr`, and the page writers moved from attributeRefToGen to inputAttributeRefToGen. Inference stays as the fallback, so every script written before still works.", "file": "`mdl/grammar/domains/MDLPage.g4` (sortColumn) + `mdl/ast/ast_page.go`/`ast_page_v3.go` + `mdl/visitor/visitor_microflow_statements.go` (sortColumnHops) + `visitor_page_v3.go` + `mdl/executor/cmd_microflows_builder_actions.go` (resolveSortAssociationPath, lookupSortHop, entityChainModules) + `cmd_microflows_format_action.go` + `cmd_pages_builder_v3.go` (resolveAssociationAttributePathForEntity) + `cmd_pages_describe_datasource.go` (sortAttributeHops, sortColumnPath) + `mdl/backend/modelsdk/microflow_read_actions.go` (entityRefStepsFromRaw) + `widget_write.go` + `sdk/pages/pages_datasources.go` (GridSort.AttributeRefSteps)", "insight": "**The measurement that decides whether a lossy describer is worth a language change is a CONSTRUCTED one.** The corpus agrees with the inference rule by construction — every document mxcli itself wrote stores the association inference would have picked, so the round trip is a fixed point on everything to hand and looks faithful. The case that matters had to be built: two associations to one entity, then the stored hop edited to the one inference does NOT pick. Byte-patching the .mxunit is enough and takes a minute — `Order_ShipTo` and `Order_BillTo` are the same length, so a `sed` on the BSON needs no resize — and the replay flipped it back immediately. **Control on a binary that drops the hop, not just on one that reverts the fix**: reverting only proves the test fires, while dropping the hop gets mxbuild to say CE7247 \"Cannot sort on attribute … is not an attribute of entity …\" — the executor's own refusal message from the other end of the pipeline, which is what proves the EntityRef load-bearing rather than cosmetic. **Two reads were missing, not one**: the microflow reader and the page reader each drop the hop separately, and fixing only the half named in the report would have shipped a describer that emits the path for microflows and silently drops it for pages. **The strongest round-trip evidence is 'Unchanged'** — with identity preservation and write elision, replaying DESCRIBE output on a correct implementation elides the write entirely, so `Unchanged microflow: …` is a stronger result than any byte comparison."} +{"area":"mdl/executor","date":"2026-09-22","symptom":"`UPDATE WIDGETS` prints a per-property `Warning: Failed to set …` for every assignment and then reports `Updated 2 widget(s)`, plus `Note: Run 'refresh catalog full force' to update the catalog with changes`, and exits 0. `describe styling` afterwards shows nothing was written","cause":"`updated++` sat OUTSIDE the assignment loop and was unconditional, so the counter meant \"this widget was found\" and was reported as \"Updated\". The same counter gated `mutator.Save()`, so a container whose every assignment failed was still saved","file":"`mdl/executor/cmd_widgets.go` (`updateOutcome`, `updateWidgetsInContainer`, `execUpdateWidgets` summary)","insight":"**A success counter incremented in the wrong loop is invisible to every test that only checks the happy path** — the failures were already being printed correctly one line above the lie. Split the outcome into the three things that actually happen (changed / matched-but-unwritable / in-catalog-but-not-in-document) rather than adding a boolean: rounding the third into either of the others is how a stale catalog reads as success. **Bound the severity before writing it up**: the rebuilt document was semantically identical, so ADR-0008 elision skipped the write — measured, no `mprcontents/` unit changed mtime and `mx check` stayed at 0 errors, making this a reporting defect and not a data one. Worth saying, because \"claims success after failing\" otherwise reads as corruption. **The DRY RUN had the same defect one step earlier and is the worse half**, since the syntax help tells you to run it first: it printed `Would set …` without attempting anything. Fixed by running the assignments against `pagemutator.Probe()` — the discardable copy `mxcli check` already uses for ALTER PAGE SET — so the preview reports `Cannot set`. Reuse that seam rather than re-deriving what a setter accepts; a preview that re-implements the rule drifts from it in exactly the direction that hurts","refs":["ako/mxcli#520","ako/mxcli#515"]} diff --git a/mdl-examples/bug-tests/widgets-520-bulk-update-false-success.mdl b/mdl-examples/bug-tests/widgets-520-bulk-update-false-success.mdl new file mode 100644 index 0000000000..281163a908 --- /dev/null +++ b/mdl-examples/bug-tests/widgets-520-bulk-update-false-success.mdl @@ -0,0 +1,75 @@ +-- ako/mxcli#520 — UPDATE WIDGETS reported success after every assignment failed. +-- +-- It counted widgets it FOUND, not assignments that SUCCEEDED. Measured on a +-- blank Mendix 11.12.2 project with two Data grid 2 widgets: +-- +-- Found 2 widget(s) in 2 container(s) matching the criteria +-- Warning: Failed to set 'Compact' on dgA: pluggable property "Compact" not found +-- Warning: Failed to set 'Striped' on dgA: pluggable property "Striped" not found +-- …same for dgB… +-- Updated 2 widget(s) +-- Note: Run 'refresh catalog full force' to update the catalog with changes. +-- +-- Four failures out of four, then "Updated 2" and an instruction to pick up +-- changes that did not exist. `describe styling` afterwards: "No styled widgets +-- found". Exit code 0. +-- +-- After the fix: +-- +-- Updated 0 widget(s) +-- 2 widget(s) matched but had no property that could be set +-- Error: no widget was updated: 4 assignment(s) could not be applied. A design +-- property (Atlas styling) is not a pluggable widget property and cannot be set +-- this way — see `mxcli syntax page.styling` +-- exit 1 +-- +-- The DRY RUN told the same lie one step earlier, which matters more because the +-- syntax help says to run it first. It now attempts the assignments against a +-- discardable copy of the document (pagemutator.Probe, the same mechanism +-- `mxcli check` uses for ALTER PAGE SET) and reports "Cannot set" / "Would +-- update 0". +-- +-- Severity was bounded and worth recording: nothing was corrupted. The rebuilt +-- document was semantically identical, so idempotent-write elision (ADR-0008) +-- skipped it — no mprcontents/ unit changed mtime and `mx check` stayed at 0 +-- errors. A reporting defect, not a data one. +-- +-- Why 'Compact'/'Striped' cannot be set this way: they are Data grid 2's Atlas +-- DESIGN properties, stored in Appearance.DesignProperties, and +-- SetWidgetProperty reaches only the pluggable property bag. That capability gap +-- is ako/mxcli#515. This issue was only about reporting the outcome truthfully. + +create persistent entity MyFirstModule.Vehicle ( + Brand: string(100), + Model: string(100) +); + +create or replace page MyFirstModule.GridA ( + title: 'A', layout: 'Atlas_Core.Atlas_Default' +) { + datagrid dgA (DataSource: database from MyFirstModule.Vehicle) { + column colBrand (Attribute: Brand, Caption: 'Brand') + } +}; + +-- The control: a property the widget really has. Reports `Updated 1 widget(s)`, +-- writes, and `describe page` reads back `PageSize: 25`. +update widgets + set 'pageSize' = 25 + where WidgetType like '%datagrid.Datagrid%' + in MyFirstModule; + +-- The bug, kept commented because `make check-mdl` runs `check` with no project +-- and this one needs a catalog: +-- +-- update widgets +-- set 'Compact' = true, 'Striped' = true +-- where WidgetType like '%datagrid.Datagrid%' +-- in MyFirstModule; +-- -> Updated 0 widget(s) / 2 matched but had no property that could be set / exit 1 +-- +-- Adjacent, and not a bug: `like '%datagrid%'` (without `.Datagrid`) matches 20 +-- widgets in 6 containers on a blank project, because it sweeps in +-- DatagridTextFilter, DatagridDateFilter and DatagridDropdownFilter. That is the +-- predicate doing what it was asked, and is why #515 proposes selecting by the +-- MDL keyword instead. diff --git a/mdl/executor/cmd_widgets.go b/mdl/executor/cmd_widgets.go index 8add4d0194..151bc1c6ca 100644 --- a/mdl/executor/cmd_widgets.go +++ b/mdl/executor/cmd_widgets.go @@ -8,6 +8,7 @@ import ( "strings" "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" "github.com/mendixlabs/mxcli/model" ) @@ -113,23 +114,51 @@ func execUpdateWidgets(ctx *ExecContext, s *ast.UpdateWidgetsStmt) error { } // Process each container - totalUpdated := 0 + var total updateOutcome for containerID, widgetRefs := range containers { - updated, err := updateWidgetsInContainer(ctx, containerID, widgetRefs, s.Assignments, s.DryRun) + outcome, err := updateWidgetsInContainer(ctx, containerID, widgetRefs, s.Assignments, s.DryRun) if err != nil { fmt.Fprintf(ctx.Output, "Warning: Failed to update widgets in %s: %v\n", containerID, err) continue } - totalUpdated += updated + total.add(outcome) + } + + verb := "Updated" + if s.DryRun { + verb = "[dry run] Would update" + } + fmt.Fprintf(ctx.Output, "\n%s %d widget(s)\n", verb, total.WidgetsChanged) + // Name the two ways a matched widget is not an updated one, so a run that + // changed less than it matched says so rather than rounding to the headline. + if total.WidgetsUnchanged > 0 { + fmt.Fprintf(ctx.Output, "%d widget(s) matched but had no property that could be set\n", + total.WidgetsUnchanged) + } + if total.WidgetsMissing > 0 { + fmt.Fprintf(ctx.Output, "%d widget(s) are in the catalog but not in the document — "+ + "run 'refresh catalog full force' and try again\n", total.WidgetsMissing) } if s.DryRun { - fmt.Fprintf(ctx.Output, "\n[dry run] Would update %d widget(s)\n", totalUpdated) fmt.Fprintln(ctx.Output, "\nRun without dry run to apply changes.") - } else { - fmt.Fprintf(ctx.Output, "\nUpdated %d widget(s)\n", totalUpdated) + return nil + } + // The catalog note is only true when something changed; printing it after a + // run that wrote nothing told the reader there were changes to pick up. + if total.WidgetsChanged > 0 { fmt.Fprintln(ctx.Output, "\nNote: Run 'refresh catalog full force' to update the catalog with changes.") } + // A statement that matched widgets and wrote none of them has not succeeded. + // Reporting that as an error is what stops a script silently doing nothing — + // the failure mode ako/mxcli#520 was filed for. + if total.changedNothing() { + return mdlerrors.NewValidation(fmt.Sprintf( + "no widget was updated: %d assignment(s) could not be applied. "+ + "A design property (Atlas styling) is not a pluggable widget property and "+ + "cannot be set this way — see `mxcli syntax page.styling`", + len(total.Failures))) + } return nil } @@ -195,11 +224,40 @@ func groupWidgetsByContainer(widgets []widgetRef) map[string][]widgetRef { return containers } +// updateOutcome is what a run actually did, which is three numbers and not one. +// +// It used to be a single `updated` count that meant "widgets found", and was +// reported as "Updated N widget(s)" — so a run where every assignment was +// refused still claimed success, right after warning about each refusal +// (ako/mxcli#520). Splitting the outcome is the fix: a widget nothing could be +// written to is not updated, and a widget the document does not carry is neither +// updated nor a property failure. +type updateOutcome struct { + WidgetsChanged int // at least one assignment landed + WidgetsUnchanged int // found in the document, nothing could be set + WidgetsMissing int // listed by the catalog, absent from the document + Failures []string // one per refused assignment, "'prop' on widget" +} + +func (o *updateOutcome) add(other updateOutcome) { + o.WidgetsChanged += other.WidgetsChanged + o.WidgetsUnchanged += other.WidgetsUnchanged + o.WidgetsMissing += other.WidgetsMissing + o.Failures = append(o.Failures, other.Failures...) +} + +// changedNothing reports a run that matched widgets and wrote none of them. +// Distinct from an empty match, which is reported before we get here. +func (o *updateOutcome) changedNothing() bool { + return o.WidgetsChanged == 0 && (o.WidgetsUnchanged > 0 || o.WidgetsMissing > 0) +} + // updateWidgetsInContainer updates widgets within a single page or snippet // using the PageMutator backend (no direct BSON manipulation). -func updateWidgetsInContainer(ctx *ExecContext, containerID string, widgetRefs []widgetRef, assignments []ast.WidgetPropertyAssignment, dryRun bool) (int, error) { +func updateWidgetsInContainer(ctx *ExecContext, containerID string, widgetRefs []widgetRef, assignments []ast.WidgetPropertyAssignment, dryRun bool) (updateOutcome, error) { + var out updateOutcome if len(widgetRefs) == 0 { - return 0, nil + return out, nil } containerName := widgetRefs[0].ContainerName @@ -207,43 +265,82 @@ func updateWidgetsInContainer(ctx *ExecContext, containerID string, widgetRefs [ // Open the container (page, layout, or snippet) through the backend mutator. mutator, err := ctx.Backend.OpenPageForMutation(model.ID(containerID)) if err != nil { - return 0, mdlerrors.NewBackend(fmt.Sprintf("open %s for mutation", containerName), err) + return out, mdlerrors.NewBackend(fmt.Sprintf("open %s for mutation", containerName), err) } if mutator == nil { - return 0, mdlerrors.NewBackend(fmt.Sprintf("open %s for mutation", containerName), + return out, mdlerrors.NewBackend(fmt.Sprintf("open %s for mutation", containerName), fmt.Errorf("backend returned nil mutator for %s", containerID)) } - updated := 0 + // A dry run attempts the same assignments against a DISCARDABLE COPY of the + // document, so the preview reports what would actually happen rather than + // assuming every assignment lands. Without this the preview told the same + // lie one step earlier — and the syntax help says to run it first, which is + // exactly when a user is relying on it (ako/mxcli#520). + // + // Best-effort: a backend whose mutator offers no probe keeps the optimistic + // preview it had, which is no worse than before. + target := mutator + if dryRun { + if p, ok := mutator.(interface { + Probe() (backend.PageMutator, error) + }); ok { + if probe, perr := p.Probe(); perr == nil && probe != nil { + target = probe + } + } + } + for _, ref := range widgetRefs { // Verify the widget exists before attempting assignments. - if !mutator.FindWidget(ref.Name) { + if !target.FindWidget(ref.Name) { fmt.Fprintf(ctx.Output, " Warning: Widget %q not found in %s %s\n", ref.Name, mutator.ContainerType(), containerName) + out.WidgetsMissing++ continue } + landed := 0 for _, assignment := range assignments { + err := target.SetWidgetProperty(ref.Name, assignment.PropertyPath, assignment.Value) + if err != nil { + out.Failures = append(out.Failures, + fmt.Sprintf("'%s' on %s (%s) in %s: %v", + assignment.PropertyPath, ref.Name, ref.WidgetType, containerName, err)) + verb := "Failed to set" + if dryRun { + verb = "Cannot set" + } + fmt.Fprintf(ctx.Output, " Warning: %s '%s' on %s: %v\n", + verb, assignment.PropertyPath, ref.Name, err) + continue + } + landed++ if dryRun { fmt.Fprintf(ctx.Output, " Would set '%s' = %v on %s (%s) in %s\n", assignment.PropertyPath, assignment.Value, ref.Name, ref.WidgetType, containerName) - } else { - if err := mutator.SetWidgetProperty(ref.Name, assignment.PropertyPath, assignment.Value); err != nil { - fmt.Fprintf(ctx.Output, " Warning: Failed to set '%s' on %s: %v\n", - assignment.PropertyPath, ref.Name, err) - } } } - updated++ + // A widget nothing could be written to is not an updated widget. That + // distinction is the whole of ako/mxcli#520. + if landed > 0 { + out.WidgetsChanged++ + } else { + out.WidgetsUnchanged++ + } } - // Persist changes via the mutator. - if !dryRun && updated > 0 { + // Persist only when something actually changed. Gating on the found-count + // offered a write for a container whose every assignment was refused; + // idempotent-write elision (ADR-0008) discarded the bytes, so the damage was + // confined to the summary — but offering it at all is what produced the + // summary. + if !dryRun && out.WidgetsChanged > 0 { if err := mutator.Save(); err != nil { - return updated, mdlerrors.NewBackend(fmt.Sprintf("save %s", containerName), err) + return out, mdlerrors.NewBackend(fmt.Sprintf("save %s", containerName), err) } } - return updated, nil + return out, nil } // mapWidgetFilterField maps user-facing field names to catalog column names. diff --git a/mdl/executor/widget_bulk_update_outcome_test.go b/mdl/executor/widget_bulk_update_outcome_test.go new file mode 100644 index 0000000000..59b95dc5e6 --- /dev/null +++ b/mdl/executor/widget_bulk_update_outcome_test.go @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "bytes" + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/mdl/backend/pagemutator" + "github.com/mendixlabs/mxcli/model" +) + +// bulkUpdateCtx wires a context whose only page is the stored document passed +// in, opened through the real BSON mutator — so the assignments below succeed or +// fail for the reasons they would in a project, not because a stub said so. +func bulkUpdateCtx(t *testing.T, stored bson.D) (*ExecContext, *bytes.Buffer, *countingDeps) { + t.Helper() + deps := &countingDeps{} + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + OpenPageForMutationFunc: func(unitID model.ID) (backend.PageMutator, error) { + return pagemutator.New(stored, unitID, deps), nil + }, + } + ctx, _ := newMockCtx(t, withBackend(mb)) + buf := &bytes.Buffer{} + ctx.Output = buf + return ctx, buf, deps +} + +func bulkRefs(names ...string) []widgetRef { + refs := make([]widgetRef, 0, len(names)) + for _, n := range names { + refs = append(refs, widgetRef{ + Name: n, WidgetType: "com.mendix.widget.web.datagrid.Datagrid", + ContainerID: "c1", ContainerName: "MyModule.P", ContainerType: "page", + }) + } + return refs +} + +func bulkAssign(pairs ...string) []ast.WidgetPropertyAssignment { + out := make([]ast.WidgetPropertyAssignment, 0, len(pairs)) + for _, p := range pairs { + out = append(out, ast.WidgetPropertyAssignment{PropertyPath: p, Value: true}) + } + return out +} + +// ako/mxcli#520. +// +// UPDATE WIDGETS counted widgets it FOUND, not assignments that SUCCEEDED, so a +// run where nothing could be written reported success — after warning about +// every failure. Measured on a blank 11.12.2 project, setting two of Data grid +// 2's real Atlas design properties (they live in Appearance.DesignProperties, +// which SetWidgetProperty does not reach — the capability gap is ako/mxcli#515): +// +// Found 2 widget(s) in 2 container(s) matching the criteria +// Warning: Failed to set 'Compact' on dgA: pluggable property "Compact" not found +// Warning: Failed to set 'Striped' on dgA: pluggable property "Striped" not found +// … same for dgB … +// Updated 2 widget(s) +// +// Four failures out of four, then "Updated 2". `describe styling` afterwards +// reported "No styled widgets found". +func TestUpdateWidgets_NoAssignmentSucceedsIsNotAnUpdate(t *testing.T) { + ctx, _, deps := bulkUpdateCtx(t, storedGridPage()) + + out, err := updateWidgetsInContainer(ctx, "c1", + bulkRefs("dgProducts"), bulkAssign("Compact", "Striped"), false) + if err != nil { + t.Fatalf("update: %v", err) + } + if out.WidgetsChanged != 0 { + t.Errorf("WidgetsChanged = %d, want 0 — every assignment was refused", out.WidgetsChanged) + } + if out.WidgetsUnchanged != 1 { + t.Errorf("WidgetsUnchanged = %d, want 1", out.WidgetsUnchanged) + } + if len(out.Failures) != 2 { + t.Errorf("Failures = %v, want both properties named", out.Failures) + } + // The container must not be saved when nothing changed. Elision would skip + // the bytes anyway, but offering the write is how "Updated 2" got printed. + if deps.saves != 0 { + t.Errorf("saved the container %d time(s) with nothing changed", deps.saves) + } +} + +// The control that stops the fix being "always report zero": a property the +// widget really has must still count, and still save. +func TestUpdateWidgets_SuccessfulAssignmentCounts(t *testing.T) { + ctx, _, deps := bulkUpdateCtx(t, storedGridPage()) + + out, err := updateWidgetsInContainer(ctx, "c1", + bulkRefs("dgProducts"), bulkAssign("pageSize"), false) + if err != nil { + t.Fatalf("update: %v", err) + } + if out.WidgetsChanged != 1 { + t.Errorf("WidgetsChanged = %d, want 1 — pageSize is a real template key", out.WidgetsChanged) + } + if len(out.Failures) != 0 { + t.Errorf("unexpected failures: %v", out.Failures) + } + if deps.saves != 1 { + t.Errorf("saved %d time(s), want 1", deps.saves) + } +} + +// A partial run must report BOTH halves rather than rounding to success or to +// failure — the shape a real sweep produces, where a property exists on some of +// the matched widgets and not others. +func TestUpdateWidgets_PartialRunReportsBothHalves(t *testing.T) { + ctx, _, deps := bulkUpdateCtx(t, storedGridPage()) + + out, err := updateWidgetsInContainer(ctx, "c1", + bulkRefs("dgProducts"), bulkAssign("pageSize", "Compact"), false) + if err != nil { + t.Fatalf("update: %v", err) + } + if out.WidgetsChanged != 1 { + t.Errorf("WidgetsChanged = %d, want 1 — one assignment landed", out.WidgetsChanged) + } + if len(out.Failures) != 1 || !strings.Contains(out.Failures[0], "Compact") { + t.Errorf("Failures = %v, want the one refused property named", out.Failures) + } + if deps.saves != 1 { + t.Errorf("saved %d time(s), want 1 — something did change", deps.saves) + } +} + +// A widget the catalog lists but the document does not carry is neither changed +// nor a property failure; it is its own outcome, and rounding it into either +// one is how a stale catalog reads as success. +func TestUpdateWidgets_MissingWidgetIsItsOwnOutcome(t *testing.T) { + ctx, _, deps := bulkUpdateCtx(t, storedGridPage()) + + out, err := updateWidgetsInContainer(ctx, "c1", + bulkRefs("ghostWidget"), bulkAssign("pageSize"), false) + if err != nil { + t.Fatalf("update: %v", err) + } + if out.WidgetsChanged != 0 || out.WidgetsMissing != 1 { + t.Errorf("got changed=%d missing=%d, want 0 and 1", out.WidgetsChanged, out.WidgetsMissing) + } + if deps.saves != 0 { + t.Errorf("saved the container for a widget that is not in it") + } +} From 3b771f3a87f7bfb4cc387843b7f189f457bf9624 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:31:27 +0000 Subject: [PATCH 08/38] fix(layout): rewrite the stored unit instead of replacing it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `CREATE OR REPLACE LAYOUT` was DeleteLayout + CreateLayout with a freshly built layout, and a create goes through InsertUnit under a newly minted id. So an identical re-run replaced the layout's unit under a new GUID every time. Measured on a blank 11.14.0 project, three runs of one idempotent statement: run 1 Replaced layout … D mprcontents/8a/a3/8aa37ee1-….mxunit ?? .../fd/6e/ run 2 Replaced layout … D mprcontents/fd/6e/fd6e9c96-….mxunit ?? .../6d/2a/ run 3 Replaced layout … 3 changed files The whole .mxunit is renamed each run — a delete plus an untracked add, not churned bytes in a stable file — so `git status` never comes back clean and an MDL-generated project is not reviewable in version control. The storage-layer net for delete+insert recreates cannot reach this. It keys on the unit ID and this path re-mints it, so there is nothing to reconcile the re-insert against; the decision has to be made at the statement. execCreateLayout now keeps the stored layout's unit and rewrites it through a new UpdateLayout, which goes to UpdateRawUnit and so reaches canon.Reconcile: an identical rewrite is elided outright and a real one keeps the stored element $IDs. Duplicates of the same name are still deleted; a layout that does not exist yet still goes through the insert. The verb is reported through ReportMutation, so an elided rewrite says Unchanged rather than claiming a replacement that did not happen. Measured after: three identical re-runs report `Unchanged layout …` with 0 changed files, and a real edit reports `Replaced layout …` as a modification to the SAME .mxunit file. 0 errors on mxbuild 11.14.0 with a page bound to it. A second defect falls out, and only the control found it: a layout MOVEd into a folder was filed back into the module root on every rewrite. There is no FOLDER clause on CREATE LAYOUT, so the rebuild always names the module root and the insert applied it to the unit's row — the defect #932 fixed for REST clients. Measured, `show layouts` Folder column: Layouts -> (empty) on the old build, Layouts -> Layouts on this one. An in-place write does not touch the row. Neither symptom is visible to a build. With a real page bound to the churned layout, mxbuild reports 0 errors on both variants, because pages resolve layouts by qualified name and not by unit GUID — which also corrects ako/mxcli#556's reading of this as the same class as #553, where the project stopped loading. Refs: ako/mxcli#600, ako/mxcli#556 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QWZjWZQhk3cNQCcy1Z2xzH --- mdl/backend/mcp/unsupported_gen.go | 15 +- mdl/backend/mock/backend.go | 1 + mdl/backend/mock/mock_page.go | 7 + mdl/backend/modelsdk/layout_write.go | 51 +++++- mdl/backend/page.go | 8 + mdl/executor/cmd_pages_layout_replace_test.go | 172 ++++++++++++++++++ mdl/executor/cmd_pages_layout_v3.go | 57 ++++-- 7 files changed, 288 insertions(+), 23 deletions(-) create mode 100644 mdl/executor/cmd_pages_layout_replace_test.go diff --git a/mdl/backend/mcp/unsupported_gen.go b/mdl/backend/mcp/unsupported_gen.go index e8e5fc6522..5cbd5c6d3c 100644 --- a/mdl/backend/mcp/unsupported_gen.go +++ b/mdl/backend/mcp/unsupported_gen.go @@ -264,11 +264,6 @@ func (unsupportedBackend) CreateViewEntitySourceDocument(_ model.ID, _ string, _ return } -func (unsupportedBackend) WriteViewEntitySourceDocument(_ model.ID, _ string, _ string, _ string, _ string) (r0 model.ID, err1 error) { - err1 = errUnsupported("WriteViewEntitySourceDocument") - return -} - func (unsupportedBackend) CreateWorkflow(_ *workflows.Workflow) (err0 error) { err0 = errUnsupported("CreateWorkflow") return @@ -1252,6 +1247,11 @@ func (unsupportedBackend) UpdateJsonStructure(_ *types.JsonStructure) (err0 erro return } +func (unsupportedBackend) UpdateLayout(_ *pages.Layout) (err0 error) { + err0 = errUnsupported("UpdateLayout") + return +} + func (unsupportedBackend) UpdateMenuDocument(_ *types.MenuDocument) (err0 error) { err0 = errUnsupported("UpdateMenuDocument") return @@ -1375,3 +1375,8 @@ func (unsupportedBackend) WriteJavaSourceFile(_ string, _ string, _ string, _ [] err0 = errUnsupported("WriteJavaSourceFile") return } + +func (unsupportedBackend) WriteViewEntitySourceDocument(_ model.ID, _ string, _ string, _ string, _ string) (r0 model.ID, err1 error) { + err1 = errUnsupported("WriteViewEntitySourceDocument") + return +} diff --git a/mdl/backend/mock/backend.go b/mdl/backend/mock/backend.go index 971df19442..da89235572 100644 --- a/mdl/backend/mock/backend.go +++ b/mdl/backend/mock/backend.go @@ -121,6 +121,7 @@ type MockBackend struct { PageLayoutNameFunc func(id model.ID) (string, error) GetLayoutFunc func(id model.ID) (*pages.Layout, error) CreateLayoutFunc func(layout *pages.Layout) error + UpdateLayoutFunc func(layout *pages.Layout) error DeleteLayoutFunc func(id model.ID) error ListSnippetsFunc func() ([]*pages.Snippet, error) CreateSnippetFunc func(snippet *pages.Snippet) error diff --git a/mdl/backend/mock/mock_page.go b/mdl/backend/mock/mock_page.go index 469380ec84..e4dc92e799 100644 --- a/mdl/backend/mock/mock_page.go +++ b/mdl/backend/mock/mock_page.go @@ -72,6 +72,13 @@ func (m *MockBackend) CreateLayout(layout *pages.Layout) error { return nil } +func (m *MockBackend) UpdateLayout(layout *pages.Layout) error { + if m.UpdateLayoutFunc != nil { + return m.UpdateLayoutFunc(layout) + } + return errors.New("MockBackend.UpdateLayout not configured") +} + func (m *MockBackend) DeleteLayout(id model.ID) error { if m.DeleteLayoutFunc != nil { return m.DeleteLayoutFunc(id) diff --git a/mdl/backend/modelsdk/layout_write.go b/mdl/backend/modelsdk/layout_write.go index 7e42f53d52..6b25dd7d3b 100644 --- a/mdl/backend/modelsdk/layout_write.go +++ b/mdl/backend/modelsdk/layout_write.go @@ -221,8 +221,55 @@ func (b *Backend) CreateLayout(layout *pages.Layout) error { return nil } -// DeleteLayout removes a Forms$Layout unit. CREATE OR REPLACE LAYOUT is a -// delete followed by a create, so this is on the write path, not a convenience. +// UpdateLayout rewrites an existing Forms$Layout unit in place. +// +// CREATE OR REPLACE LAYOUT used to be DeleteLayout + CreateLayout, and a create +// goes through InsertUnit under a freshly minted id. So an identical re-run +// replaced the unit under a new GUID every time: measured on 11.14.0, three +// runs of one idempotent statement produced three different .mxunit files, a +// delete plus an untracked add each run, and `git status` never cleared +// (ako/mxcli#600). +// +// The storage layer's own net for delete+insert recreates (#556, +// carryIdentityFromRemovedUnit) cannot help there, because it keys on the unit +// id and that path re-mints it. Going through UpdateRawUnit instead reaches +// canon.Reconcile, so an identical rewrite is elided outright and a real one +// keeps the stored element $IDs. +// +// It also leaves the unit's ROW alone, which is what keeps a foldered layout in +// its folder: there is no FOLDER clause on CREATE LAYOUT, so the rebuilt layout +// always carries the module root as its container. +// +// The same encoder as CreateLayout, deliberately: two encoders for one document +// is how the two drift into writing different BSON for the same layout. +func (b *Backend) UpdateLayout(layout *pages.Layout) error { + if layout == nil { + return fmt.Errorf("UpdateLayout: nil layout") + } + if b.writer == nil { + return fmt.Errorf("UpdateLayout: not connected for writing") + } + if layout.ID == "" { + return fmt.Errorf("UpdateLayout: layout %q has no id — an in-place rewrite needs the stored unit", layout.Name) + } + g, err := layoutToGen(layout) + if err != nil { + return err + } + g.SetID(element.ID(layout.ID)) + contents, err := (&codec.Encoder{}).Encode(g) + if err != nil { + return fmt.Errorf("UpdateLayout: encode: %w", err) + } + if err := b.writer.UpdateRawUnit(string(layout.ID), contents); err != nil { + return fmt.Errorf("UpdateLayout: update: %w", err) + } + return nil +} + +// DeleteLayout removes a Forms$Layout unit. Still on the write path rather than +// a convenience: DROP LAYOUT uses it, and so does a rewrite that has to clear a +// duplicate or move the layout to another container. func (b *Backend) DeleteLayout(id model.ID) error { if b.writer == nil { return fmt.Errorf("DeleteLayout: not connected for writing") diff --git a/mdl/backend/page.go b/mdl/backend/page.go index 79f0f09661..bb730461ab 100644 --- a/mdl/backend/page.go +++ b/mdl/backend/page.go @@ -21,6 +21,14 @@ type PageBackend interface { ListLayouts() ([]*pages.Layout, error) GetLayout(id model.ID) (*pages.Layout, error) CreateLayout(layout *pages.Layout) error + // UpdateLayout rewrites an existing layout's unit IN PLACE, keeping its unit + // id and its row (so a foldered layout stays in its folder). + // + // CREATE OR REPLACE LAYOUT used to be a delete followed by a create, which + // replaced the unit under a fresh GUID on every run — the .mxunit was + // renamed each time and the tree never came back clean (ako/mxcli#600). + // An update reaches canon.Reconcile, so an identical rewrite is elided. + UpdateLayout(layout *pages.Layout) error DeleteLayout(id model.ID) error // PageLayoutName returns the qualified name of the layout a page renders diff --git a/mdl/executor/cmd_pages_layout_replace_test.go b/mdl/executor/cmd_pages_layout_replace_test.go new file mode 100644 index 0000000000..78651e0cdd --- /dev/null +++ b/mdl/executor/cmd_pages_layout_replace_test.go @@ -0,0 +1,172 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend/mock" + "github.com/mendixlabs/mxcli/model" + "github.com/mendixlabs/mxcli/sdk/pages" +) + +// ako/mxcli#600, the last open piece of #556. `CREATE OR REPLACE LAYOUT` was a +// DELETE followed by a CREATE with a freshly built layout, so the unit was +// replaced under a NEW GUID on every run. +// +// MEASURED on a blank 11.14.0 project, one idempotent statement, three runs: +// +// run 1 Replaced layout … D mprcontents/8a/a3/8aa37ee1-….mxunit ?? mprcontents/fd/6e/ +// run 2 Replaced layout … D mprcontents/fd/6e/fd6e9c96-….mxunit ?? mprcontents/6d/2a/ +// run 3 Replaced layout … 3 changed files +// +// The whole .mxunit is renamed each run — a delete plus an untracked add, not +// churned bytes in a stable file — so `git status` never comes back clean. +// +// The storage-layer net from #556 cannot catch this: carryIdentityFromRemovedUnit +// keys on the unit ID and this path mints a fresh one, so there is nothing to +// reconcile the re-insert against. The fix has to be at the statement. +// +// Not a correctness bug, and the issue it came from says otherwise: #556 ties +// it to #553 ("the project stopped loading"). Measured with a real page bound +// to the churned layout, `mx check` reports 0 errors — pages resolve layouts by +// qualified NAME, not by unit GUID. The cost is reviewability. + +func replaceLayoutCtx(t *testing.T, stored []*pages.Layout) (*ExecContext, *[]model.ID, *[]*pages.Layout, *[]*pages.Layout) { + t.Helper() + mod := &model.Module{ + BaseElement: model.BaseElement{ID: model.ID("mod-own")}, + Name: "M", + } + h := mkHierarchy(mod) + // A layout inside a folder of that module, so the foldered case resolves to + // the module the way a real project's does. + withContainer(h, model.ID("folder-7"), mod.ID) + var deleted []model.ID + var created, updated []*pages.Layout + mb := &mock.MockBackend{ + IsConnectedFunc: func() bool { return true }, + ListModulesFunc: func() ([]*model.Module, error) { return []*model.Module{mod}, nil }, + ListLayoutsFunc: func() ([]*pages.Layout, error) { return stored, nil }, + DeleteLayoutFunc: func(id model.ID) error { deleted = append(deleted, id); return nil }, + CreateLayoutFunc: func(l *pages.Layout) error { created = append(created, l); return nil }, + UpdateLayoutFunc: func(l *pages.Layout) error { updated = append(updated, l); return nil }, + } + ctx, _ := newMockCtx(t, withBackend(mb), withHierarchy(h)) + ctx.Output = &strings.Builder{} + return ctx, &deleted, &created, &updated +} + +func storedLayout(id, container string) *pages.Layout { + l := &pages.Layout{Name: "App_Default"} + l.ID = model.ID(id) + l.ContainerID = model.ID(container) + return l +} + +func replaceStmt() *ast.CreateLayoutStmt { + s := layoutStmt(map[string]any{"layouttype": "Responsive"}, scrollWithMain()) + s.IsReplace = true + return s +} + +func TestExecCreateLayout_RewritesTheStoredUnitInsteadOfReplacingIt(t *testing.T) { + ctx, deleted, created, updated := replaceLayoutCtx(t, []*pages.Layout{storedLayout("lay-1", "mod-own")}) + + if err := execCreateLayout(ctx, replaceStmt()); err != nil { + t.Fatalf("execCreateLayout: %v", err) + } + + if len(*deleted) != 0 { + t.Errorf("the stored layout was deleted (%v) — a same-module rewrite must update the unit in place, "+ + "or it comes back under a new GUID and `git status` never clears (ako/mxcli#600)", *deleted) + } + if len(*created) != 0 { + t.Errorf("CreateLayout was called for a layout that already exists (%d times)", len(*created)) + } + if len(*updated) != 1 { + t.Fatalf("UpdateLayout called %d times, want 1", len(*updated)) + } + if got := (*updated)[0].ID; got != model.ID("lay-1") { + t.Errorf("the rewrite wrote unit %q, want the stored lay-1 — a fresh id IS the bug", got) + } +} + +// The stored unit's container is preserved too. There is no FOLDER clause on +// CREATE LAYOUT, so the rebuild always sets ContainerID to the module root — +// which files a foldered layout back into the root on every rewrite, the same +// defect #932 fixed for REST clients. An in-place write does not touch the +// unit's row at all, so this comes for free; the assertion is here so it cannot +// regress if the write ever goes back through an insert. +func TestExecCreateLayout_KeepsAFolderedLayoutInItsFolder(t *testing.T) { + ctx, _, _, updated := replaceLayoutCtx(t, []*pages.Layout{storedLayout("lay-1", "folder-7")}) + + if err := execCreateLayout(ctx, replaceStmt()); err != nil { + t.Fatalf("execCreateLayout: %v", err) + } + if len(*updated) != 1 { + t.Fatalf("UpdateLayout called %d times, want 1", len(*updated)) + } + if got := (*updated)[0].ContainerID; got != model.ID("folder-7") { + t.Errorf("container = %q, want folder-7 — the rewrite moved the layout to the module root", got) + } +} + +// CONTROL: a layout that does NOT exist yet is still created, through the +// insert path. Without this, a fix broken into "always update" would pass the +// test above and write nothing at all for a new layout. +func TestExecCreateLayout_StillCreatesANewLayout(t *testing.T) { + ctx, deleted, created, updated := replaceLayoutCtx(t, nil) + + if err := execCreateLayout(ctx, replaceStmt()); err != nil { + t.Fatalf("execCreateLayout: %v", err) + } + if len(*created) != 1 { + t.Fatalf("CreateLayout called %d times, want 1", len(*created)) + } + if len(*updated) != 0 || len(*deleted) != 0 { + t.Errorf("a brand-new layout took the rewrite path (updated=%d deleted=%d)", len(*updated), len(*deleted)) + } + if out := ctx.Output.(*strings.Builder).String(); !strings.Contains(out, "Created layout M.App_Default") { + t.Errorf("output %q, want it to report a create", out) + } +} + +// CONTROL: duplicates are still removed. A project that already holds two +// layouts of one name (the old delete+create could leave one behind on a failed +// run) must end up with one — the first is rewritten, the rest deleted. +func TestExecCreateLayout_RemovesDuplicatesAndKeepsTheFirst(t *testing.T) { + ctx, deleted, created, updated := replaceLayoutCtx(t, []*pages.Layout{ + storedLayout("lay-1", "mod-own"), + storedLayout("lay-2", "mod-own"), + }) + + if err := execCreateLayout(ctx, replaceStmt()); err != nil { + t.Fatalf("execCreateLayout: %v", err) + } + if len(*updated) != 1 || (*updated)[0].ID != model.ID("lay-1") { + t.Fatalf("updated = %v, want the first stored layout rewritten", *updated) + } + if len(*deleted) != 1 || (*deleted)[0] != model.ID("lay-2") { + t.Errorf("deleted = %v, want only the duplicate lay-2", *deleted) + } + if len(*created) != 0 { + t.Errorf("CreateLayout was called %d times", len(*created)) + } +} + +// Without OR REPLACE / OR MODIFY an existing layout is still refused, and +// nothing is written. The rewrite path must not turn CREATE into an upsert. +func TestExecCreateLayout_StillRefusesAPlainCreateOverAnExistingLayout(t *testing.T) { + ctx, deleted, created, updated := replaceLayoutCtx(t, []*pages.Layout{storedLayout("lay-1", "mod-own")}) + + s := layoutStmt(map[string]any{"layouttype": "Responsive"}, scrollWithMain()) + if err := execCreateLayout(ctx, s); err == nil { + t.Fatal("a plain CREATE over an existing layout must be refused") + } + if len(*deleted)+len(*created)+len(*updated) != 0 { + t.Errorf("the refused statement still wrote something") + } +} diff --git a/mdl/executor/cmd_pages_layout_v3.go b/mdl/executor/cmd_pages_layout_v3.go index cee03814c0..c07f5b29a3 100644 --- a/mdl/executor/cmd_pages_layout_v3.go +++ b/mdl/executor/cmd_pages_layout_v3.go @@ -111,10 +111,11 @@ func execCreateLayout(ctx *ExecContext, s *ast.CreateLayoutStmt) error { s.Name.String(), s.Name.Module)) } + // The stored layout this statement rewrites, if there is one, plus any + // duplicates of the same name to clear out. existing, _ := ctx.Backend.ListLayouts() - var toDelete []model.ID - var existingLayoutDoc string - haveExistingLayout := false + var stored *pages.Layout + var duplicates []model.ID for _, l := range existing { modName := getModuleName(ctx, getModuleID(ctx, l.ContainerID)) if modName != s.Name.Module || l.Name != s.Name.Name { @@ -123,11 +124,11 @@ func execCreateLayout(ctx *ExecContext, s *ast.CreateLayoutStmt) error { if !s.IsReplace && !s.IsModify { return mdlerrors.NewAlreadyExists("layout", s.Name.String()) } - if len(toDelete) == 0 { - existingLayoutDoc = l.Documentation - haveExistingLayout = true + if stored == nil { + stored = l + continue } - toDelete = append(toDelete, l.ID) + duplicates = append(duplicates, l.ID) } pb := &pageBuilder{ @@ -155,26 +156,50 @@ func execCreateLayout(ctx *ExecContext, s *ast.CreateLayoutStmt) error { } // A rewrite that carried no doc comment keeps the stored one (#1018). - if haveExistingLayout { - layout.Documentation = carriedDocumentation(s.DocumentationSet, s.Documentation, existingLayoutDoc) + if stored != nil { + layout.Documentation = carriedDocumentation(s.DocumentationSet, s.Documentation, stored.Documentation) } - for _, id := range toDelete { + // A duplicate of the same name is cleared; the FIRST stored layout is + // rewritten rather than deleted (see below), so it is not in this list. + for _, id := range duplicates { if err := ctx.Backend.DeleteLayout(id); err != nil { return mdlerrors.NewBackend("delete existing layout", err) } } - if err := ctx.Backend.CreateLayout(layout); err != nil { + + verb := "Created" + if stored != nil { + verb = "Replaced" + // REWRITE THE STORED UNIT, do not replace it. A delete followed by a + // create goes through InsertUnit under a freshly minted id, so an + // identical re-run replaced the layout's unit under a NEW GUID every + // time — measured on 11.14.0, three runs produced three different + // .mxunit files, each run a delete plus an untracked add, and the tree + // never came back clean (ako/mxcli#600). The storage layer's net for + // delete+insert recreates (#556) keys on the unit id and so cannot + // catch a path that re-mints it; this has to be decided here. + // + // Carrying the stored container is the other half: there is no FOLDER + // clause on CREATE LAYOUT, so the rebuilt layout always names the + // module root, and an insert would file a foldered layout back into it + // (the defect #932 fixed for REST clients). UpdateLayout does not touch + // the unit's row at all. + layout.ID = stored.ID + layout.ContainerID = stored.ContainerID + if err := ctx.Backend.UpdateLayout(layout); err != nil { + return mdlerrors.NewBackend("update layout", err) + } + } else if err := ctx.Backend.CreateLayout(layout); err != nil { return mdlerrors.NewBackend("create layout", err) } invalidateHierarchy(ctx) - verb := "Created" - if len(toDelete) > 0 { - verb = "Replaced" - } - fmt.Fprintf(ctx.Output, "%s layout %s\n", verb, s.Name.String()) + // Through ReportMutation: the rewrite now reaches canon.Reconcile, so a + // statement that matches what is stored is elided and must say so rather + // than claiming a replacement that did not happen. + ctx.ReportMutation(verb, "layout %s", s.Name.String()) return nil } From 8884eae98773d968c0aa593f815c456a6242fb25 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:31:27 +0000 Subject: [PATCH 09/38] test: regression case and the delete+insert tell for layout rewrites MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bug test is idempotent: running it twice must leave the project byte-identical. Control on a binary built without the fix — `Replaced layout …` and 3 changed files on every re-run, against `Unchanged layout …` and a clean tree with it. CLAUDE.md gains the rule the third instance in a week earned. The #556 carry keys on the unit ID, so it does not reach a handler that re-mints one, and three did: REST client (#556), view entity OQL document (#583), layout (#600). The cheap tell is to `ls` the .mxunit filenames across two identical runs — a changed filename is delete+insert and the handler is wrong, a same filename with different bytes is the codec or a missing carry and canon is where to look. Refs: ako/mxcli#600 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01QWZjWZQhk3cNQCcy1Z2xzH --- .../fix-issue/findings/mdl-executor.jsonl | 1 + CLAUDE.md | 12 +++++ .../layout-600-replace-rewrites-in-place.mdl | 52 +++++++++++++++++++ 3 files changed, 65 insertions(+) create mode 100644 mdl-examples/bug-tests/layout-600-replace-rewrites-in-place.mdl diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 3a491f5698..88a9af2a90 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -670,3 +670,4 @@ {"area": "mdl/executor", "date": "2026-09-21", "symptom": "`create or modify entity` drops an attribute a LATER script added, silently. Reported shape: entity created in 01-domain-core.mdl, a calculated attribute added in 03-logic.mdl (its microflow does not exist until then); re-running slice 01 ALONE rebuilt the entity from its own statement and removed the attribute, with `Modified entity: ServiceCore.LithoSystem` as the only output. It surfaced two slices later as `[CE1613] \"The selected attribute 'ServiceCore.LithoSystem.OpenRequestCount' no longer exists.\" at Text 'dtOpen'` — an error naming the PAGE, never the script that removed the attribute. `mxcli check … -p app.mpr --references` said \"Check passed!\".", "ce": "CE1613", "rules": ["MDL087"], "cause": "Half the ask was already shipped and half was not, and the report could not tell them apart. exec's warning (droppedEntityMembers, findings #24, landed 320a304 two weeks before the report) DOES fire — measured on a real 11.6.6 project re-running the reporter's slice 01, it prints the attribute by name — so the reporter was on an older binary. What genuinely did not exist was the issue's second ask: `check` had no project-aware pass for member loss at all, so the one command that runs BEFORE anything is written was the silent one. Added CheckEntityMemberDrops (MDL087, warning) to cmd_check.go's catalog-backed tier, and refactored droppedEntityMembers to share its comparison.", "file": "`mdl/executor/validate_entity_member_drops.go` (new: entityMemberSet, droppedMembers, CheckEntityMemberDrops), `mdl/executor/cmd_entities.go` (droppedEntityMembers now delegates), `cmd/mxcli/cmd_check.go` (projectViolations)", "insight": "**Reproduce before theorising when the report predates a fix in the same area** — exec already printed the exact line the issue asks for, so reading the issue text alone leads either to 'already fixed, close it' or to reimplementing the shipped half. Running the reporter's own sequence against a real project separated the two halves in one command each, and the isolated-slice check printing `Check passed!` is what identified the actual gap. **A check-time twin of an exec-time warning must NOT be the same computation.** exec is per-statement because it is applying statements; check sees the whole script, so it has to be the NET effect — a script that rebuilds an entity and then `alter entity … add attribute`s the members back loses nothing, and that is the IDIOMATIC full-script order, so a per-statement port would warn on every correct script and be switched off within a day. **Intent has to be tracked, not inferred from the outcome**: `drop attribute` / `rename attribute` / `drop entity` produce the same before/after diff as the accident, and a pure diff cannot separate them. Both of those are separate controls, and the naive implementation fails each one specifically (measured: stubbing the net/intent logic fails TestMDL087_ExplicitRemovalIsSilent on 3 of 4 spellings while the positive test still passes — so the positive test alone proves nothing). **One comparison, two layers**: the audit system fields and an omitted `extends` were reported by exec and would have been missed by a second hand-written diff, which is why droppedEntityMembers was refactored onto the shared entityMemberSet rather than copied. An audit pseudo-type (`AutoOwner`) is a FLAG, not an attribute — exec `continue`s past it — so counting it as one makes a faithful restatement read as a drop.", "refs": ["ako/mxcli#562", "findings #24", "findings #13"]} {"area": "mdl/executor", "date": "2026-09-21", "symptom": "`retrieve $AccountList from Administration.Account sort by System.Language.Code asc;` — MDL that `mxcli describe` had just emitted — passed `mxcli check` and was refused by `mxcli exec`: \"sort by attribute 'System.Language.Code' does not belong to entity 'Administration.Account'\". Reported as a check/exec inconsistency (mendixlabs/mxcli#1152); the real defect is that the round trip cannot replay its own output for any sort over an association reached from an ANCESTOR.", "cause": "inferSortEntityRefSteps searched ONE domain model — the retrieved entity's own module — for associations whose parent was the retrieved entity ITSELF, and qualified the association it found with the retrieved entity's module. All three assumptions hold only when the hop starts on the retrieved entity in its own module. Administration.Account reaches System.Language through System.User_Language, declared on System.User and stored in the System module: parent is an ancestor, the domain model is another module's, and the qualified name carries THAT module. Rewritten as a generalization-chain walk that looks each ancestor up in its own module and qualifies the association with the module storing it; the destination end is matched with entityIsSubtypeOf rather than by equality, since an association may point at a specialization of the entity that declares the attribute.", "file": "`mdl/executor/cmd_microflows_builder_actions.go` (inferSortEntityRefSteps); tests `mdl/executor/cmd_microflows_sort_association_test.go`, `mdl/backend/modelsdk/microflow_retrievesort_test.go`; example `mdl-examples/bug-tests/microflow-1152-sort-over-association.mdl`", "insight": "**The second control is the one that pays.** Reverting the fix reproduces the refusal, which only proves the test fires. The control that taught something was building a binary that DERIVES the hop and does not WRITE it — exec succeeds and mxbuild 11.12.3 answers CE7247 \"Cannot sort on attribute 'System.Language.Code'. Attribute 'System.Language.Code' is not an attribute of entity 'Administration.Account'\" — the executor's refusal message almost word for word, from the other end of the pipeline. That is what fixes the qualified name as load-bearing: the stored EntityRefStep must read System.User_Language, and the pre-existing code would have written Administration.User_Language had it found anything at all. **Skip the theory that check is missing a rule**: check has no sort-attribute rule at all and resolves no hops, so it was never going to disagree with exec here — the inconsistency in the report is a symptom of the false refusal, not a second defect. **Known residue, stated because the round trip rests on it**: DESCRIBE emits only the attribute's qualified name, so where several associations reach one entity the replay picks the nearest ancestor's first and can silently land on the other hop. Spelling the hop needs grammar (sortColumn is qualifiedName|IDENTIFIER, no `/` path) and is a language change, not a fix."} {"area": "mdl/executor", "date": "2026-09-21", "symptom": "Follow-up to the sort-hop inference fix: with the hop derivable but not SAYABLE, `describe → exec` still silently changed the program wherever two associations reach the same entity. Measured on 11.12.3 with Order_ShipTo and Order_BillTo (both Order -> Address): a microflow sorting by the BILLING address came back sorting by the SHIPPING one, `mx check` 0 errors on both sides. Same for a page datasource's sort bar.", "cause": "DESCRIBE emitted only the sort attribute's qualified name and the reader never looked at the hop at all — `sortItemsFromRaw` read AttributeRef.Attribute and skipped AttributeRef.EntityRef, so the association was written and never read back. MDL had no spelling for it either (`sortColumn : (qualifiedName | IDENTIFIER)`). Closed end to end: sortColumn takes `qualifiedName (SLASH qualifiedName)*` (the shape MDLCatalog.g4 already uses for Association/Entity), SortColumnDef/OrderByItemV3 carry the hops, the executor resolves the NAMED association instead of inferring, both readers reconstruct EntityRef.Steps, both describers emit `Assoc/.../Attr`, and the page writers moved from attributeRefToGen to inputAttributeRefToGen. Inference stays as the fallback, so every script written before still works.", "file": "`mdl/grammar/domains/MDLPage.g4` (sortColumn) + `mdl/ast/ast_page.go`/`ast_page_v3.go` + `mdl/visitor/visitor_microflow_statements.go` (sortColumnHops) + `visitor_page_v3.go` + `mdl/executor/cmd_microflows_builder_actions.go` (resolveSortAssociationPath, lookupSortHop, entityChainModules) + `cmd_microflows_format_action.go` + `cmd_pages_builder_v3.go` (resolveAssociationAttributePathForEntity) + `cmd_pages_describe_datasource.go` (sortAttributeHops, sortColumnPath) + `mdl/backend/modelsdk/microflow_read_actions.go` (entityRefStepsFromRaw) + `widget_write.go` + `sdk/pages/pages_datasources.go` (GridSort.AttributeRefSteps)", "insight": "**The measurement that decides whether a lossy describer is worth a language change is a CONSTRUCTED one.** The corpus agrees with the inference rule by construction — every document mxcli itself wrote stores the association inference would have picked, so the round trip is a fixed point on everything to hand and looks faithful. The case that matters had to be built: two associations to one entity, then the stored hop edited to the one inference does NOT pick. Byte-patching the .mxunit is enough and takes a minute — `Order_ShipTo` and `Order_BillTo` are the same length, so a `sed` on the BSON needs no resize — and the replay flipped it back immediately. **Control on a binary that drops the hop, not just on one that reverts the fix**: reverting only proves the test fires, while dropping the hop gets mxbuild to say CE7247 \"Cannot sort on attribute … is not an attribute of entity …\" — the executor's own refusal message from the other end of the pipeline, which is what proves the EntityRef load-bearing rather than cosmetic. **Two reads were missing, not one**: the microflow reader and the page reader each drop the hop separately, and fixing only the half named in the report would have shipped a describer that emits the path for microflows and silently drops it for pages. **The strongest round-trip evidence is 'Unchanged'** — with identity preservation and write elision, replaying DESCRIBE output on a correct implementation elides the write entirely, so `Unchanged microflow: …` is a stronger result than any byte comparison."} +{"area": "mdl/executor", "date": "2026-09-22", "symptom": "`CREATE OR REPLACE LAYOUT` re-run with an identical statement reported `Replaced layout …` and dirtied 3 files EVERY time. Measured on a blank 11.14.0 project, three runs produced three different .mxunit filenames (8aa37ee1… -> fd6e9c96… -> 6d2a…): the unit was deleted and re-inserted under a fresh GUID, so git shows a delete plus an untracked add rather than a modified file. Second, unreported symptom found by the control: a layout MOVEd into a folder was filed back into the module root on every rewrite (`show layouts` Folder column Layouts -> empty).", "cause": "execCreateLayout collected the stored layout's id into `toDelete`, deleted it, and called CreateLayout with a freshly built layout — CreateLayout goes through InsertUnit under a newly minted id, and InsertUnit is not a canon.Reconcile choke point. The #556 net (carryIdentityFromRemovedUnit) cannot cover it: that keys on the unit ID and this path re-mints it, so there is nothing to reconcile the re-insert against. The folder half has the same single cause: there is no FOLDER clause on CREATE LAYOUT, so buildLayoutV3 always sets ContainerID to the module root, and only an INSERT applies that to the unit's row.", "file": "`mdl/executor/cmd_pages_layout_v3.go` (execCreateLayout), `mdl/backend/modelsdk/layout_write.go` (UpdateLayout), `mdl/backend/page.go` + `mdl/backend/mock/`", "insight": "**When a `create or modify` handler churns, look for delete+create before looking at the codec.** This is the third instance in one week — REST client (#556), view entity OQL document (#583), layout (#600) — and all three were the same shape and took the same fix: rewrite the stored unit through UpdateRawUnit instead of replacing it. The tell is cheap: `ls` the .mxunit filenames across two runs. A CHANGED filename means delete+insert (fix the handler); a same filename with different bytes means the codec or a carry (fix canon). **A storage-layer net that keys on the unit ID cannot cover a path that re-mints the ID** — worth stating because #556's fix reads like it generalised, and it does not reach here. **The folder defect is the one the tests would not have found**: it only appears once a layout has been moved, which no unit test set up and no reported symptom mentioned; it surfaced from running the faulted binary through a MOVE, which is why the control is worth running on more than the reported case. **Do not trust the issue's severity**: #556 ties this to #553 (project unloadable). Measured with a real page bound to the churned layout, mxbuild reports 0 errors on both variants, because pages resolve layouts by qualified name and not by unit GUID — so `mx check` is not a control for this class at all and the version-control diff is the only signal.", "refs": ["ako/mxcli#600", "ako/mxcli#556", "ako/mxcli#583", "ako/mxcli#932", "mendixlabs/mxcli#1063"]} diff --git a/CLAUDE.md b/CLAUDE.md index 780027be91..1460e66579 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -354,6 +354,18 @@ gone quiet. Prefer an in-place update where the handler can do one: the REST client's own fix is to call `UpdateConsumedRestService` and keep delete+create only for a folder move, which lives in the unit's row rather than its contents. +**The carry keys on the unit ID, so it does not reach a handler that re-mints +one** — and three handlers did, in one week: the REST client (#556), the view +entity's OQL document (#583) and the layout (ako/mxcli#600). All three took the +same fix, an in-place `UpdateRawUnit` rather than a replacement. The tell is +cheap and worth reaching for first: `ls` the `.mxunit` filenames across two +identical runs. A **changed filename** is delete+insert and the handler is +wrong; a **same filename with different bytes** is the codec or a missing carry +and `canon` is where to look. A replacement also silently reverts the unit's +ROW, which is how `create or replace layout` moved a foldered layout back to the +module root on every rewrite — there is no `FOLDER` clause on the statement, so +the rebuild always names the module root and only an insert applies it. + When something *has* changed, `Reconcile` still does not let the rebuild's fresh `$ID`s reach disk: `canon.TransplantIDs` matches the incoming document against the stored one element by element (by `$Type` and shape, by `Name` where there is one, diff --git a/mdl-examples/bug-tests/layout-600-replace-rewrites-in-place.mdl b/mdl-examples/bug-tests/layout-600-replace-rewrites-in-place.mdl new file mode 100644 index 0000000000..0551f7a91a --- /dev/null +++ b/mdl-examples/bug-tests/layout-600-replace-rewrites-in-place.mdl @@ -0,0 +1,52 @@ +-- ============================================================================ +-- ako/mxcli#600 — CREATE OR REPLACE LAYOUT replaced the unit under a new GUID +-- ============================================================================ +-- +-- The last open piece of ako/mxcli#556. Running this file twice must leave the +-- project byte-identical, so `git status` comes back clean. +-- +-- `CREATE OR REPLACE LAYOUT` was DeleteLayout + CreateLayout with a freshly +-- built layout, and a create goes through InsertUnit under a newly minted id. +-- Measured on a blank Mendix 11.14.0 project, three identical re-runs of the +-- statement below: +-- +-- run 1 Replaced layout … D mprcontents/8a/a3/8aa37ee1-….mxunit ?? .../fd/6e/ +-- run 2 Replaced layout … D mprcontents/fd/6e/fd6e9c96-….mxunit ?? .../6d/2a/ +-- run 3 Replaced layout … 3 changed files +-- +-- The whole .mxunit is renamed each run — a delete plus an untracked add, not +-- churned bytes — so an MDL-generated project is not reviewable in version +-- control. #556's storage-layer net cannot catch this: it keys on the unit id +-- and this path re-mints it. +-- +-- TWO things the control establishes, neither of which a build reports: +-- +-- 1. mxbuild does not care. With a real page bound to the churned layout, +-- `mxcli docker check` is 0 errors on BOTH the churning and the fixed +-- project. Pages resolve layouts by qualified NAME, not by unit GUID, so +-- nothing dangles — the cost is reviewability, not correctness. (#556 says +-- otherwise, tying this to #553 where the project stopped loading.) +-- 2. A foldered layout was moved back to the module root on every rewrite. +-- There is no FOLDER clause on CREATE LAYOUT, so the rebuild always named +-- the module root, and the insert applied it — the same defect #932 fixed +-- for REST clients. Measured: `show layouts` Folder column goes Layouts -> +-- (empty) on the unfixed build and stays Layouts on the fixed one. An +-- in-place rewrite does not touch the unit's row, so it cannot happen. +-- +-- After the fix, an identical re-run reports `Unchanged layout …` and a real +-- edit reports `Replaced layout …` against the SAME .mxunit file. +-- ============================================================================ + +CREATE MODULE BugLayoutRewrite; + +CREATE OR REPLACE LAYOUT BugLayoutRewrite.Probe_Layout ( + layouttype: 'Responsive' +) { + scrollcontainer scrollContainer1 { + region top (Class: 'region-topbar') { + } + region center (Class: 'region-content') { + placeholder Main + } + } +} From 992dc05b08d4a42ddfe2f6f15f75b85cc94e17ea Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:50:04 +0000 Subject: [PATCH 10/38] feat(pages): ALTER PAGE SET writes Atlas design properties MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `alter page … set '' = on ` dead-ended: `set` reaches a handful of first-class properties and the stored widget's pluggable property bag, while a design property lives in Appearance.DesignProperties. The only spelling that worked was ALTER STYLING --- a second statement for the same operation. The same operation, measured rather than assumed. ALTER STYLING's grammar is ALTER STYLING ON (PAGE|SNIPPET) qualifiedName WIDGET IDENTIFIER … with BOTH operands mandatory (omitting either is a parse error), and there is no bulk form. So it is the one-widget-on-one-page operation ALTER PAGE SET already performs, which "One Way to Do Each Thing" rules out having twice. alter page MyFirstModule.ThingList { set 'Row size' = 'Small' on lvThings; set 'Hover style' = on on lvThings; }; Measured on a blank 11.12.2 project: describe styling reads back `['Row size': 'Small', 'Hover style': on]`, mx check 0 errors --- and writing the same thing via ALTER STYLING first makes this run rewrite ZERO units, so elision found the two documents semantically equal. (`Altered page` is ALTER PAGE's fixed verb, not the elision verb, so it proves nothing on its own.) The resolver this needed already existed with no callers. bsonTypeToDesignPropsKey maps $Type → theme key and had never been referenced, hence never validated; ako/mxcli#509 deliberately avoided standing up a third consumer of the concept before something needed it, and this is that something. The mutator returns raw storage facts (WidgetStorageType: $Type plus, for a pluggable widget, Type.WidgetId) and the executor owns the mapping, so the theme concern stays in one place. Routing happens only on a POSITIVE declaration for this widget's type. Anything else falls through to the pluggable setter and keeps that error --- routing on "the theme says nothing, so it must be a design property" is how a typo becomes a silently-written design property. Two shapes are refused, inheriting existing refusals rather than forking the vocabulary: a flat value on a multi-select property (CE6084, ako/mxcli#511) and a compound one. Both name the inline `DesignProperties: [...]` spelling, because a SET assignment carries one scalar --- exactly as a StylingAssignment does. TestBsonTypeAndKeywordDesignPropsKeysAgree pins both maps. It does NOT assert they agree, because they do not, and both directions have measured reasons: $Type-only DataGrid/Gallery are the native widgets whose MDL keywords now resolve to pluggable ids, so the stored path resolves more than the inline one; keyword-only Header/Footer are wrong, since MDL builds both as Forms$DivContainer and Atlas declares no such groups --- so inline design-property validation for a header silently skips the widget. Left uncorrected: it changes what existing pages validate against. Knock-on: the #1135 error message named ALTER STYLING as the route, which is stale now that `set` has it. It now says the key is neither a property of the widget nor a design property the theme declares for it, and points at `show design properties`. The test asserting the old wording is inverted. Part of ako/mxcli#515; the plural ALTER PAGES form is still to come. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LYaTtzjmMCUjo8XAVN1sQx --- .../fix-issue/findings/mdl-executor.jsonl | 1 + cmd/mxcli/syntax/features_page.go | 2 +- docs/01-project/MDL_QUICK_REFERENCE.md | 1 + ...yling-515-alter-page-design-properties.mdl | 82 ++++ mdl/backend/pagemutator/mutator.go | 25 +- .../pagemutator/native_set_diagnostic_test.go | 39 +- mdl/backend/pagemutator/probe.go | 24 ++ mdl/executor/cmd_alter_page.go | 15 + mdl/executor/design_property_routing.go | 158 ++++++++ mdl/executor/design_property_routing_test.go | 352 ++++++++++++++++++ 10 files changed, 670 insertions(+), 29 deletions(-) create mode 100644 mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl create mode 100644 mdl/executor/design_property_routing.go create mode 100644 mdl/executor/design_property_routing_test.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index 7d0073f414..e465152c8a 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -671,3 +671,4 @@ {"area": "mdl/executor", "date": "2026-09-21", "symptom": "`retrieve $AccountList from Administration.Account sort by System.Language.Code asc;` — MDL that `mxcli describe` had just emitted — passed `mxcli check` and was refused by `mxcli exec`: \"sort by attribute 'System.Language.Code' does not belong to entity 'Administration.Account'\". Reported as a check/exec inconsistency (mendixlabs/mxcli#1152); the real defect is that the round trip cannot replay its own output for any sort over an association reached from an ANCESTOR.", "cause": "inferSortEntityRefSteps searched ONE domain model — the retrieved entity's own module — for associations whose parent was the retrieved entity ITSELF, and qualified the association it found with the retrieved entity's module. All three assumptions hold only when the hop starts on the retrieved entity in its own module. Administration.Account reaches System.Language through System.User_Language, declared on System.User and stored in the System module: parent is an ancestor, the domain model is another module's, and the qualified name carries THAT module. Rewritten as a generalization-chain walk that looks each ancestor up in its own module and qualifies the association with the module storing it; the destination end is matched with entityIsSubtypeOf rather than by equality, since an association may point at a specialization of the entity that declares the attribute.", "file": "`mdl/executor/cmd_microflows_builder_actions.go` (inferSortEntityRefSteps); tests `mdl/executor/cmd_microflows_sort_association_test.go`, `mdl/backend/modelsdk/microflow_retrievesort_test.go`; example `mdl-examples/bug-tests/microflow-1152-sort-over-association.mdl`", "insight": "**The second control is the one that pays.** Reverting the fix reproduces the refusal, which only proves the test fires. The control that taught something was building a binary that DERIVES the hop and does not WRITE it — exec succeeds and mxbuild 11.12.3 answers CE7247 \"Cannot sort on attribute 'System.Language.Code'. Attribute 'System.Language.Code' is not an attribute of entity 'Administration.Account'\" — the executor's refusal message almost word for word, from the other end of the pipeline. That is what fixes the qualified name as load-bearing: the stored EntityRefStep must read System.User_Language, and the pre-existing code would have written Administration.User_Language had it found anything at all. **Skip the theory that check is missing a rule**: check has no sort-attribute rule at all and resolves no hops, so it was never going to disagree with exec here — the inconsistency in the report is a symptom of the false refusal, not a second defect. **Known residue, stated because the round trip rests on it**: DESCRIBE emits only the attribute's qualified name, so where several associations reach one entity the replay picks the nearest ancestor's first and can silently land on the other hop. Spelling the hop needs grammar (sortColumn is qualifiedName|IDENTIFIER, no `/` path) and is a language change, not a fix."} {"area": "mdl/executor", "date": "2026-09-21", "symptom": "Follow-up to the sort-hop inference fix: with the hop derivable but not SAYABLE, `describe → exec` still silently changed the program wherever two associations reach the same entity. Measured on 11.12.3 with Order_ShipTo and Order_BillTo (both Order -> Address): a microflow sorting by the BILLING address came back sorting by the SHIPPING one, `mx check` 0 errors on both sides. Same for a page datasource's sort bar.", "cause": "DESCRIBE emitted only the sort attribute's qualified name and the reader never looked at the hop at all — `sortItemsFromRaw` read AttributeRef.Attribute and skipped AttributeRef.EntityRef, so the association was written and never read back. MDL had no spelling for it either (`sortColumn : (qualifiedName | IDENTIFIER)`). Closed end to end: sortColumn takes `qualifiedName (SLASH qualifiedName)*` (the shape MDLCatalog.g4 already uses for Association/Entity), SortColumnDef/OrderByItemV3 carry the hops, the executor resolves the NAMED association instead of inferring, both readers reconstruct EntityRef.Steps, both describers emit `Assoc/.../Attr`, and the page writers moved from attributeRefToGen to inputAttributeRefToGen. Inference stays as the fallback, so every script written before still works.", "file": "`mdl/grammar/domains/MDLPage.g4` (sortColumn) + `mdl/ast/ast_page.go`/`ast_page_v3.go` + `mdl/visitor/visitor_microflow_statements.go` (sortColumnHops) + `visitor_page_v3.go` + `mdl/executor/cmd_microflows_builder_actions.go` (resolveSortAssociationPath, lookupSortHop, entityChainModules) + `cmd_microflows_format_action.go` + `cmd_pages_builder_v3.go` (resolveAssociationAttributePathForEntity) + `cmd_pages_describe_datasource.go` (sortAttributeHops, sortColumnPath) + `mdl/backend/modelsdk/microflow_read_actions.go` (entityRefStepsFromRaw) + `widget_write.go` + `sdk/pages/pages_datasources.go` (GridSort.AttributeRefSteps)", "insight": "**The measurement that decides whether a lossy describer is worth a language change is a CONSTRUCTED one.** The corpus agrees with the inference rule by construction — every document mxcli itself wrote stores the association inference would have picked, so the round trip is a fixed point on everything to hand and looks faithful. The case that matters had to be built: two associations to one entity, then the stored hop edited to the one inference does NOT pick. Byte-patching the .mxunit is enough and takes a minute — `Order_ShipTo` and `Order_BillTo` are the same length, so a `sed` on the BSON needs no resize — and the replay flipped it back immediately. **Control on a binary that drops the hop, not just on one that reverts the fix**: reverting only proves the test fires, while dropping the hop gets mxbuild to say CE7247 \"Cannot sort on attribute … is not an attribute of entity …\" — the executor's own refusal message from the other end of the pipeline, which is what proves the EntityRef load-bearing rather than cosmetic. **Two reads were missing, not one**: the microflow reader and the page reader each drop the hop separately, and fixing only the half named in the report would have shipped a describer that emits the path for microflows and silently drops it for pages. **The strongest round-trip evidence is 'Unchanged'** — with identity preservation and write elision, replaying DESCRIBE output on a correct implementation elides the write entirely, so `Unchanged microflow: …` is a stronger result than any byte comparison."} {"area":"mdl/executor","date":"2026-09-22","symptom":"`UPDATE WIDGETS` prints a per-property `Warning: Failed to set …` for every assignment and then reports `Updated 2 widget(s)`, plus `Note: Run 'refresh catalog full force' to update the catalog with changes`, and exits 0. `describe styling` afterwards shows nothing was written","cause":"`updated++` sat OUTSIDE the assignment loop and was unconditional, so the counter meant \"this widget was found\" and was reported as \"Updated\". The same counter gated `mutator.Save()`, so a container whose every assignment failed was still saved","file":"`mdl/executor/cmd_widgets.go` (`updateOutcome`, `updateWidgetsInContainer`, `execUpdateWidgets` summary)","insight":"**A success counter incremented in the wrong loop is invisible to every test that only checks the happy path** — the failures were already being printed correctly one line above the lie. Split the outcome into the three things that actually happen (changed / matched-but-unwritable / in-catalog-but-not-in-document) rather than adding a boolean: rounding the third into either of the others is how a stale catalog reads as success. **Bound the severity before writing it up**: the rebuilt document was semantically identical, so ADR-0008 elision skipped the write — measured, no `mprcontents/` unit changed mtime and `mx check` stayed at 0 errors, making this a reporting defect and not a data one. Worth saying, because \"claims success after failing\" otherwise reads as corruption. **The DRY RUN had the same defect one step earlier and is the worse half**, since the syntax help tells you to run it first: it printed `Would set …` without attempting anything. Fixed by running the assignments against `pagemutator.Probe()` — the discardable copy `mxcli check` already uses for ALTER PAGE SET — so the preview reports `Cannot set`. Reuse that seam rather than re-deriving what a setter accepts; a preview that re-implements the rule drifts from it in exactly the direction that hurts","refs":["ako/mxcli#520","ako/mxcli#515"]} +{"area":"mdl/executor","date":"2026-09-22","symptom":"`alter page … set '' = on ` dead-ended — `set` reaches first-class properties and the stored widget's PLUGGABLE property bag, and a design property lives in `Appearance.DesignProperties`. The only spelling that worked was `alter styling`, a second statement for the same operation","cause":"No resolution from a STORED widget to its theme-registry key, so `set` could not tell a design property from a mistyped pluggable one and had to assume the latter","file":"`mdl/backend/pagemutator/probe.go` (`WidgetStorageType`); `mdl/executor/design_property_routing.go` (new); `cmd_alter_page.go` (`applySetPropertyMutator`); `mdl/backend/pagemutator/mutator.go` (the now-stale error message)","insight":"**The resolver the routing needed already existed with zero callers.** `bsonTypeToDesignPropsKey` ($Type → theme key) had never been referenced, so it had never been validated against anything; ako/mxcli#509 deliberately avoided standing up a third consumer of the concept before something needed it, and this was that something. **Do not assert the two key maps are consistent — they are not, and both directions have measured reasons.** $Type-only: `DataGrid`/`Gallery` are the NATIVE widgets, which the MDL keywords no longer produce (`datagrid`→Data grid 2's id via pluggableKeywordIDs), so the stored path resolves MORE than the inline one. Keyword-only: `header`/`footer` map to \"Header\"/\"Footer\" but MDL builds BOTH as `Forms$DivContainer`, and Atlas declares no such groups — so the inline design-property validation for a header widget misses and skips the widget silently, the same shape pluggableKeywordIDs records for combobox/gallery/image. A test that pins both exclusive SETS with their reasons is the useful shape; a consistency assertion fails on correct code. **Route only on a positive theme declaration for THIS widget's type** — routing on \"the theme says nothing, so it must be a design property\" turns a typo into a silently-written design property. **Prove the two statements are the same operation on bytes, not on reasoning**: write via `alter styling`, then run the `alter page` form and count rewritten units — 0 means elision found them semantically equal. `Altered page` is ALTER PAGE's fixed verb and is NOT the elision verb, so it proves nothing. Knock-on: the #1135 error message named `alter styling` as the route, which became stale the moment `set` learned the route — and a test asserted that wording, so it had to be inverted like the others","refs":["ako/mxcli#515","ako/mxcli#509","ako/mxcli#511","mendixlabs/mxcli#1135"]} diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index ffc4f7c8b3..e98e911458 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -284,7 +284,7 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "popup width", "popup height", "popup resizable", "drop template", "insert template", "list view template", }, - Syntax: "ALTER PAGE Module.Name {\n SET property = value ON widgetName; -- widget property names: any casing\n SET Action = MICROFLOW Module.MF ON btnSave; -- any CREATE PAGE action form\n SET DataSource = $Param ON dvOrder; -- parameter/microflow/nanoflow/selection;\n -- DATABASE and association are REPLACE-only,\n -- and a data view takes no database source\n SET (prop1 = val1, prop2 = val2) ON widgetName;\n SET Title = 'New Title'; -- page-level (case-sensitive)\n SET Documentation = 'What this page is for.';\n SET Class = 'css-class'; -- page-level CSS class / style\n SET Style = 'css: rule';\n SET PopupWidth = 800; -- page-level pop-up dimensions\n SET PopupHeight = 480;\n SET PopupResizable = true;\n INSERT AFTER widgetName { };\n INSERT BEFORE widgetName { };\n INSERT INTO containerName { };\n DROP WIDGET name1, name2;\n DROP TEMPLATE FOR Module.Specialization IN listViewName;\n REPLACE widgetName WITH { };\n};", + Syntax: "ALTER PAGE Module.Name {\n SET property = value ON widgetName; -- widget property names: any casing\n SET 'Row size' = 'Small' ON lvOrders; -- an Atlas DESIGN property of that widget's\n -- type; quoted and case-sensitive.\n -- `show design properties for ` lists\n -- them. ON/OFF for a toggle, where OFF\n -- REMOVES the entry.\n -- A multi-select ('Hide on') or compound\n -- ('Spacing') one needs the inline\n -- DesignProperties: [...] form, because a\n -- SET assignment carries one value.\n SET Action = MICROFLOW Module.MF ON btnSave; -- any CREATE PAGE action form\n SET DataSource = $Param ON dvOrder; -- parameter/microflow/nanoflow/selection;\n -- DATABASE and association are REPLACE-only,\n -- and a data view takes no database source\n SET (prop1 = val1, prop2 = val2) ON widgetName;\n SET Title = 'New Title'; -- page-level (case-sensitive)\n SET Documentation = 'What this page is for.';\n SET Class = 'css-class'; -- page-level CSS class / style\n SET Style = 'css: rule';\n SET PopupWidth = 800; -- page-level pop-up dimensions\n SET PopupHeight = 480;\n SET PopupResizable = true;\n INSERT AFTER widgetName { };\n INSERT BEFORE widgetName { };\n INSERT INTO containerName { };\n DROP WIDGET name1, name2;\n DROP TEMPLATE FOR Module.Specialization IN listViewName;\n REPLACE widgetName WITH { };\n};", Example: "ALTER PAGE Module.EditPage {\n SET (Caption = 'Save & Close', ButtonStyle = Success) ON btnSave;\n INSERT AFTER txtName {\n TEXTBOX txtMiddleName (Label: 'Middle Name', Attribute: MiddleName)\n };\n DROP WIDGET txtUnused;\n};", SeeAlso: []string{"page.create", "page.show", "snippet.alter"}, }) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index aa7cd38680..d18a639799 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1470,6 +1470,7 @@ MDL uses explicit property declarations for pages: | Drop layout | `drop layout Module.Name;` | Pages still bound to it are named in a warning and the drop proceeds; left dropped they fail **CE1613**, which names the *page* | | Declare a placeholder | `placeholder Main` | **No body.** Exactly one must be named `Main` — mxbuild enforces it (**CE0848**/**CE0849**), and names must be unique (**CE0495**). `placeholder X { … }` is the page-side form and declares nothing (MDL083) | | Alter layout | `alter layout Module.Name { };` | Edits the stored document, so widgets MDL cannot spell survive. Refused for a Marketplace target | +| Set a design property | `alter page Module.Page { set 'Row size' = 'Small' on lvOrders; };` | An Atlas design property of that widget's **type** — quoted, case-sensitive; `show design properties for ` lists them. `on`/`off` for a toggle, where `off` removes the entry. Same document `alter styling` writes. A **multi-select** (`Hide on`) or **compound** (`Spacing`) property needs the inline `DesignProperties: [...]` form, since a `set` assignment carries one value | | Repoint one page | `alter page Module.Page { set Layout = Module.Layout [map (Old as New, …)]; };` | Rewrites the layout reference **and** every placeholder binding | | Repoint many pages | `alter pages [in ] set layout = Module.Layout [map (…)] [where layout = Module.Old];` | The migration form. Marketplace pages are skipped and named. A `where layout` that names no real layout is an error, not a 0-page success | diff --git a/mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl b/mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl new file mode 100644 index 0000000000..64e34ad5cd --- /dev/null +++ b/mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl @@ -0,0 +1,82 @@ +-- ako/mxcli#515 — `alter page … set '' = on ` +-- could not write a design property. The only spelling that worked was +-- ALTER STYLING, and the two have identical scope. +-- +-- Before: the statement dead-ended, because `set` reaches a handful of +-- first-class properties and the stored widget's PLUGGABLE property bag, while a +-- design property lives in Appearance.DesignProperties. +-- +-- Error: property "Row size" is not a property of this built-in widget — … +-- for an Atlas design property use `alter styling …` instead +-- +-- The scope measurement that makes this one operation and not two (see #515): +-- ALTER STYLING's grammar is +-- ALTER STYLING ON (PAGE|SNIPPET) qualifiedName WIDGET IDENTIFIER … +-- with BOTH operands mandatory — omitting either is a parse error. There is no +-- bulk form. So it is the same one-widget-on-one-page operation ALTER PAGE SET +-- performs, which "One Way to Do Each Thing" rules out having twice. +-- +-- Measured on a blank Mendix 11.12.2 project: +-- +-- alter page … { set 'Row size' = 'Small' on lvThings; } +-- -> Altered page +-- describe styling -> DesignProperties: ['Row size': 'Small', 'Hover style': on] +-- mx check -> The app contains: 0 errors. +-- +-- And the two statements produce the SAME document: writing via ALTER STYLING +-- and then running the ALTER PAGE form rewrote 0 units — idempotent-write +-- elision (ADR-0008) skipped it, which is only possible if the documents are +-- semantically equal. +-- +-- How a design property is told from a mistyped pluggable one: the key is +-- resolved against the STORED widget's own type, via the $Type → theme-registry +-- mapping that already existed (`bsonTypeToDesignPropsKey`) but had no callers. +-- ako/mxcli#509 deliberately avoided standing that up before something needed +-- it; this is that something. +-- +-- Two shapes `set` still refuses, both because a SET assignment carries ONE +-- scalar — exactly as a StylingAssignment does: +-- +-- set 'Hide on' = 'Phone' on lvThings; +-- -> design property "Hide on" takes a SET of options, not one value — write it +-- inline instead: `DesignProperties: ['Hide on': ['Phone': on]]` … A single +-- value is stored as an Option and mxbuild refuses it with CE6084 +-- (that is ako/mxcli#511, and this path inherits its refusal rather than +-- forking the vocabulary) +-- +-- a compound such as 'Spacing' — same reason, same remedy. + +create entity MyFirstModule.Thing ( Name: string(200) ); + +create or replace page MyFirstModule.ThingList ( + title: 'Things', layout: 'Atlas_Core.Atlas_Default' +) { + listview lvThings (DataSource: database from MyFirstModule.Thing) { + dynamictext dtName (Content: 'x') + } +}; + +-- A type-specific design property, and an inherited one (from Atlas's `Widget` +-- group, which applies to every widget — resolving only the type-specific list +-- is the mistake this area keeps producing). +alter page MyFirstModule.ThingList { + set 'Row size' = 'Small' on lvThings; + set 'Align self' = 'Right' on lvThings; +}; + +-- A toggle. OFF is a REMOVAL, not a stored false: Mendix represents the off +-- state as the absence of the entry, matching what `alter styling … = off` does. +alter page MyFirstModule.ThingList { + set 'Hover style' = on on lvThings; +}; + +-- The control that keeps the routing narrow. A key the theme does not declare +-- for this widget stays on the pluggable path and keeps that path's error: +-- +-- alter page MyFirstModule.ThingList { set 'Rowsize' = 'Small' on lvThings; }; +-- -> property "Rowsize" is not a property of this built-in widget, and not an +-- Atlas design property your theme declares for it — … Run +-- `mxcli show design properties for ` to see which those are +-- +-- That message used to name ALTER STYLING as the route. It no longer does: +-- naming a second statement for something `set` now writes is stale advice. diff --git a/mdl/backend/pagemutator/mutator.go b/mdl/backend/pagemutator/mutator.go index d1fa60de20..0c8983e9f3 100644 --- a/mdl/backend/pagemutator/mutator.go +++ b/mdl/backend/pagemutator/mutator.go @@ -2900,11 +2900,15 @@ func setPluggableWidgetPropertyMut(widget bson.D, propName string, value any) er // // The message used to be "property %q not found (widget has no pluggable // Object)". That is true and unusable: "pluggable Object" is not something the -// author wrote, and it is not the whole truth either. An Atlas design property -// on a built-in widget — "Remove empty text" on a List View, the case reported -// as mendixlabs/mxcli#1135 — IS writable, through ALTER STYLING, which the old -// message never mentioned. A dead end that names its exit is a one-line fix for -// the reader; one that does not is a bug report. +// author wrote, and it was not the whole truth either — an Atlas design property +// on a built-in widget (the case reported as mendixlabs/mxcli#1135) is writable. +// +// It named ALTER STYLING as the route until ako/mxcli#515 taught `set` to write +// design properties itself. Reaching here now means the key is neither a +// first-class property NOR a design property the theme declares for this +// widget's type, so the message says that and points at the command that lists +// the ones it does declare — sending the reader to a second statement would be +// stale advice for a route that no longer differs. // // A Forms$Appearance and no Object is exactly a built-in widget, which is when // the advice applies. A pluggable widget keeps the error that names its own @@ -2914,11 +2918,12 @@ func noPluggableObjectError(widget bson.D, propName string) error { if bsonnav.DGetDoc(widget, "Appearance") == nil { return fmt.Errorf("property %q not found on this widget", propName) } - return fmt.Errorf("property %q is not a property of this built-in widget — "+ - "`set` writes its own properties (Caption, Class, Style, DynamicClasses, "+ - "Visible, Editable, …); for an Atlas design property use "+ - "`alter styling on page|snippet widget %s set '%s' = ` instead", - propName, bsonnav.DGetString(widget, "Name"), propName) + return fmt.Errorf("property %q is not a property of this built-in widget, and not an Atlas "+ + "design property your theme declares for it — `set` writes its own properties "+ + "(Caption, Class, Style, DynamicClasses, Visible, Editable, …) and any design "+ + "property of this widget's type. Run `mxcli show design properties for ` "+ + "to see which those are", + propName) } // setTranslatableText sets a translatable text value in BSON. diff --git a/mdl/backend/pagemutator/native_set_diagnostic_test.go b/mdl/backend/pagemutator/native_set_diagnostic_test.go index 9f0b2b846a..11be4eae6b 100644 --- a/mdl/backend/pagemutator/native_set_diagnostic_test.go +++ b/mdl/backend/pagemutator/native_set_diagnostic_test.go @@ -7,28 +7,26 @@ import ( "testing" ) -// mendixlabs/mxcli#1135, route 1. +// mendixlabs/mxcli#1135, route 1, as amended by ako/mxcli#515. // -// `alter page … set 'Remove empty text' = off on lvThings` on a native List View -// dead-ends. The write is genuinely not supported on this path — but the message -// it dead-ends with describes mxcli's internals rather than the author's -// options: +// `alter page … set 'Remove empty text' = off on lvThings` dead-ended with // // property "Remove empty text" not found (widget has no pluggable Object) // -// "pluggable Object" is not a thing the author wrote, cannot be made true by -// editing the script, and — most of the point — is not the whole truth: the -// design property IS writable, through ALTER STYLING. Measured on a blank -// 11.12.2 project, the statement the old message did not mention: +// "pluggable Object" is not a thing the author wrote and cannot be made true by +// editing the script. #1135 replaced it with a message naming ALTER STYLING as +// the route that did work. // -// alter styling on page MyFirstModule.ThingList widget lvThings -// set 'Remove empty text' = on; -// -> Updated styling on widget "lvThings" in page MyFirstModule.ThingList -// describe styling -> DesignProperties: ['Remove empty text': on] +// #515 then taught `set` to write design properties itself, so naming a second +// statement became stale advice. Reaching this error now means the key is +// NEITHER a first-class property NOR a design property the theme declares for +// this widget's type — a genuinely unknown key — so the message says that and +// points at the command that lists the ones it does declare. // -// So the error names that route. A widget carrying a Forms$Appearance and no -// pluggable Object is exactly a built-in one, which is when the advice applies. -func TestSetWidgetProperty_NativeWidgetErrorNamesTheStylingRoute(t *testing.T) { +// This test was asserting the ALTER STYLING wording. Inverted rather than +// deleted: the half that still holds is that the message must not describe +// mxcli's internals, and must leave the reader somewhere to go. +func TestSetWidgetProperty_NativeWidgetErrorIsActionable(t *testing.T) { rawData := makeRawPage(makeStyleableWidget("lvThings")) m := &Mutator{rawData: rawData, widgetFinder: findBsonWidget} @@ -40,8 +38,13 @@ func TestSetWidgetProperty_NativeWidgetErrorNamesTheStylingRoute(t *testing.T) { if strings.Contains(msg, "pluggable Object") { t.Errorf("the message still describes mxcli's internals: %q", msg) } - if !strings.Contains(msg, "alter styling") { - t.Errorf("the message does not name the route that works: %q", msg) + // Sending the reader to ALTER STYLING is now stale: `set` writes design + // properties, so the two no longer differ. + if strings.Contains(msg, "alter styling") { + t.Errorf("the message still names a second statement for a route `set` now has: %q", msg) + } + if !strings.Contains(msg, "show design properties") { + t.Errorf("the message leaves the reader nowhere to go: %q", msg) } if !strings.Contains(msg, "Remove empty text") { t.Errorf("the message does not name the property: %q", msg) diff --git a/mdl/backend/pagemutator/probe.go b/mdl/backend/pagemutator/probe.go index 8efb8e2240..08c62d95b1 100644 --- a/mdl/backend/pagemutator/probe.go +++ b/mdl/backend/pagemutator/probe.go @@ -9,6 +9,7 @@ import ( "go.mongodb.org/mongo-driver/bson" "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" ) // A dry run, so `mxcli check` can refuse what `exec` refuses. @@ -103,3 +104,26 @@ func (m *Mutator) WidgetPropertyKeys(widgetRef, columnRef string) []string { sort.Strings(keys) return keys } + +// WidgetStorageType returns what the STORED widget is, as two raw storage facts: +// its BSON `$Type`, and — for a pluggable widget, whose `$Type` is always +// `CustomWidgets$CustomWidget` — the widget id under `Type.WidgetId`. +// +// Raw facts, not a resolved answer, because which theme-registry key they map to +// is a theme concern and this package holds storage. The executor owns that +// mapping (`storedWidgetDesignPropsKey`), which keeps the one `$Type` → key +// table it already has as the single place that decides (ako/mxcli#515). +// +// Both empty when the widget does not resolve, which the caller reads as "cannot +// be established" rather than as a native widget. +func (m *Mutator) WidgetStorageType(widgetRef string) (bsonType, widgetID string) { + result := m.widgetFinder(m.rawData, widgetRef) + if result == nil { + return "", "" + } + bsonType = bsonnav.DGetString(result.widget, "$Type") + if t := bsonnav.DGetDoc(result.widget, "Type"); t != nil { + widgetID = bsonnav.DGetString(t, "WidgetId") + } + return bsonType, widgetID +} diff --git a/mdl/executor/cmd_alter_page.go b/mdl/executor/cmd_alter_page.go index f9a70863c2..4b31ae52dd 100644 --- a/mdl/executor/cmd_alter_page.go +++ b/mdl/executor/cmd_alter_page.go @@ -183,6 +183,21 @@ func applySetPropertyMutator(ctx *ExecContext, mutator backend.PageMutator, op * if err := mutator.SetWidgetAction(op.Target.Widget, action); err != nil { return mdlerrors.NewBackend("set Action on "+op.Target.Name(), err) } + } else if p := designPropertyForStoredWidget( + ctx.GetThemeRegistry(), mutator, op.Target.Widget, propName); p != nil { + // An Atlas design property of THIS stored widget. It lives in + // Appearance.DesignProperties, which SetWidgetProperty does not + // reach, so before ako/mxcli#515 this dead-ended and ALTER STYLING + // was the only spelling that worked. + // + // Routed only when the project's theme declares the key FOR THIS + // WIDGET's type; anything else falls through to the setter below and + // keeps that path's error, so a mistyped pluggable key is still a + // mistyped pluggable key rather than a silently-written design + // property. + if err := applyDesignPropertySet(mutator, op.Target, p, value); err != nil { + return mdlerrors.NewBackend("set "+propName+" on "+op.Target.Name(), err) + } } else { if err := mutator.SetWidgetProperty(op.Target.Widget, propName, value); err != nil { return mdlerrors.NewBackend("set "+propName+" on "+op.Target.Name(), err) diff --git a/mdl/executor/design_property_routing.go b/mdl/executor/design_property_routing.go new file mode 100644 index 0000000000..98185e7c4c --- /dev/null +++ b/mdl/executor/design_property_routing.go @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" +) + +// Routing a design property written by `ALTER PAGE … SET` (ako/mxcli#515). +// +// `SET` reaches two places today: a handful of first-class properties, and the +// stored widget's pluggable property bag. An Atlas design property lives in +// neither — it is in `Appearance.DesignProperties` — so `set 'Row size' = 'Small' +// on lvOrders` dead-ended, and the only spelling that worked was ALTER STYLING. +// +// The two statements have identical scope: both take one page or snippet and one +// widget name, both mandatory (measured — #515 carries the grammar and the parse +// errors). One operation with two spellings is what "One Way to Do Each Thing" +// in `.claude/skills/design-mdl-syntax.md` rules out, so SET learns the third +// place rather than a second statement existing to reach it. +// +// # The resolution problem, and why there is no new resolver +// +// To tell a design property from a mistyped pluggable property, the key has to +// be resolved against the STORED widget's own type. That needs a `$Type` → +// theme-registry-key mapping — and `bsonTypeToDesignPropsKey` already is one. It +// had zero callers, so it had never been validated against anything; using it +// here is what makes it load-bearing, and +// TestBsonTypeAndKeywordDesignPropsKeysAgree now holds it to the keyword map +// beside it so the two cannot drift. +// +// ako/mxcli#509 deliberately did NOT do this — its ALTER STYLING key check asks +// the weaker question (does ANY widget type declare this key?) precisely to +// avoid standing up a third consumer of the concept before something needed it. +// This is that something. + +// widgetStorageTyper is the optional capability the BSON page mutator offers: +// what the stored widget IS, as raw storage facts. Asserted rather than added to +// backend.PageMutator, for the reason pageProbe is: the MCP mutator has no +// pluggable path at all, and asserting keeps this routing off a backend whose +// SET support is different rather than reporting that difference as the author's +// mistake. +type widgetStorageTyper interface { + WidgetStorageType(widgetRef string) (bsonType, widgetID string) +} + +// storedWidgetDesignPropsKey returns the theme-registry key for a stored widget, +// or "" when it cannot be established. +// +// A pluggable widget is keyed in design-properties.json by its WIDGET ID, a +// native one by a name derived from its `$Type`. Returning "" rather than +// guessing is what keeps an unknown widget out of the routing entirely: it then +// takes the path it took before, which is the only safe default for a branch +// that decides where a value gets written. +func storedWidgetDesignPropsKey(mutator backend.PageMutator, widgetRef string) string { + typer, ok := mutator.(widgetStorageTyper) + if !ok || widgetRef == "" { + return "" + } + bsonType, widgetID := typer.WidgetStorageType(widgetRef) + if widgetID != "" { + return widgetID + } + if key, ok := bsonTypeToDesignPropsKey[bsonType]; ok { + return key + } + return "" +} + +// designPropertyForStoredWidget returns the theme's declaration of key for the +// stored widget, or nil when this is not a design property of it. +// +// nil is the signal to leave `SET` on the path it was already on, so a mistyped +// pluggable key still reaches the pluggable setter and still gets the error that +// names the widget's own declared keys. Routing on "the theme says nothing, so +// it must be a design property" is the inverse, and is how a typo becomes a +// silently-written design property — the "no silent side effects on typos" line +// in CLAUDE.md's checklist. +func designPropertyForStoredWidget(theme *ThemeRegistry, mutator backend.PageMutator, widgetRef, key string) *ThemeProperty { + if theme == nil || key == "" { + return nil + } + dpKey := storedWidgetDesignPropsKey(mutator, widgetRef) + if dpKey == "" { + return nil + } + for _, p := range theme.GetPropertiesForWidget(dpKey) { + if strings.EqualFold(p.Name, key) { + found := p + return &found + } + } + return nil +} + +// designPropertyAssignment converts an MDL value into the (valueType, option) +// pair SetDesignProperty takes, or refuses it. +// +// Refusals here are the two shapes ALTER PAGE SET cannot carry, both already +// refused on the paths that can carry them, so the vocabulary does not fork: +// +// - a flat value on a multi-select property, which Mendix stores as a compound +// of one entry per selection and mxbuild rejects as CE6084 (ako/mxcli#511) +// - a compound value, which a SET assignment has no shape for — it holds one +// scalar, exactly as a StylingAssignment does +func designPropertyAssignment(p *ThemeProperty, value any) (valueType, option string, err error) { + str := fmt.Sprintf("%v", value) + switch v := value.(type) { + case bool: + if v { + str = "on" + } else { + str = "off" + } + case string: + str = v + } + + if p.MultiSelect { + return "", "", fmt.Errorf( + "design property %q takes a SET of options, not one value — `set` carries a single "+ + "value, so write it inline instead: `DesignProperties: ['%s': ['%s': on]]` on the "+ + "widget, in CREATE PAGE or an ALTER PAGE REPLACE. A single value is stored as an "+ + "Option and mxbuild refuses it with CE6084", + p.Name, p.Name, firstOptionName(p, str)) + } + + switch strings.ToLower(str) { + case "on", "true": + return "toggle", "", nil + case "off", "false": + // Handled by the caller as a removal: Mendix stores a toggle's OFF state + // as the absence of the entry, not as a stored false. + return "", "", nil + } + return resolveDesignPropertyValueType(p.Name, str, []ThemeProperty{*p}), str, nil +} + +// applyDesignPropertySet writes one design property onto a stored widget. +// +// OFF is a removal, not a stored false: Mendix represents a toggle's off state +// as the absence of the entry, which is what `alter styling … = off` already +// does (applyStylingMutator). Writing a "false" toggle instead would leave a +// DesignProperties entry Studio Pro shows as on. +func applyDesignPropertySet(mutator backend.PageMutator, target ast.WidgetRef, p *ThemeProperty, value any) error { + valueType, option, err := designPropertyAssignment(p, value) + if err != nil { + return err + } + if valueType == "" { + return mutator.RemoveDesignProperty(target.Widget, p.Name) + } + return mutator.SetDesignProperty(target.Widget, p.Name, valueType, option) +} diff --git a/mdl/executor/design_property_routing_test.go b/mdl/executor/design_property_routing_test.go new file mode 100644 index 0000000000..8dec6141d8 --- /dev/null +++ b/mdl/executor/design_property_routing_test.go @@ -0,0 +1,352 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "go.mongodb.org/mongo-driver/bson" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + "github.com/mendixlabs/mxcli/mdl/backend/bsonnav" + "github.com/mendixlabs/mxcli/mdl/backend/pagemutator" + "github.com/mendixlabs/mxcli/model" +) + +// storedListViewPage is a page holding one NATIVE List View — the widget whose +// design properties ako/mxcli#1135 was filed about. +func storedListViewPage() bson.D { + lv := bson.D{ + {Key: "$Type", Value: "Forms$ListView"}, + {Key: "Name", Value: "lvOrders"}, + {Key: "Appearance", Value: bson.D{ + {Key: "$Type", Value: "Forms$Appearance"}, + {Key: "Class", Value: ""}, + {Key: "DesignProperties", Value: bson.A{int32(3)}}, + {Key: "DynamicClasses", Value: ""}, + {Key: "Style", Value: ""}, + }}, + } + return bson.D{ + {Key: "$Type", Value: "Forms$Page"}, + {Key: "FormCall", Value: bson.D{ + {Key: "Arguments", Value: bson.A{ + int32(2), + bson.D{{Key: "Widgets", Value: bson.A{int32(2), lv}}}, + }}, + }}, + } +} + +func listViewMutator(t *testing.T, stored bson.D) backend.PageMutator { + t.Helper() + return pagemutator.New(stored, model.ID("u1"), &countingDeps{}) +} + +// atlasListViewTheme mirrors what Atlas declares for a List View on 11.12.2 — +// three of its own plus three inherited from the `Widget` group. +func atlasListViewTheme() *ThemeRegistry { + return &ThemeRegistry{WidgetProperties: map[string][]ThemeProperty{ + "Widget": { + {Name: "Align self", Type: "ToggleButtonGroup", Options: []ThemeOption{{Name: "Left"}, {Name: "Right"}}}, + {Name: "Hide on", Type: "ToggleButtonGroup", MultiSelect: true, + Options: []ThemeOption{{Name: "Phone"}, {Name: "Tablet"}}}, + }, + "ListView": { + {Name: "Row size", Type: "ToggleButtonGroup", Options: []ThemeOption{{Name: "Small"}, {Name: "Large"}}}, + {Name: "Hover style", Type: "Toggle", Class: "listview-hover"}, + }, + "com.mendix.widget.web.datagrid.Datagrid": { + {Name: "Striped", Type: "Toggle", Class: "table-striped"}, + }, + }} +} + +// ako/mxcli#515. The stored widget's type has to be resolvable before a design +// property can be told from a mistyped pluggable one. +func TestStoredWidgetDesignPropsKey(t *testing.T) { + t.Run("native widget resolves from its $Type", func(t *testing.T) { + m := listViewMutator(t, storedListViewPage()) + if got := storedWidgetDesignPropsKey(m, "lvOrders"); got != "ListView" { + t.Errorf("key = %q, want ListView", got) + } + }) + + t.Run("pluggable widget resolves to its widget id", func(t *testing.T) { + m := listViewMutator(t, storedGridPage()) + if got := storedWidgetDesignPropsKey(m, "dgProducts"); got != "" { + // storedGridPage()'s Type carries no WidgetId, so this is the + // "cannot be established" case, which must NOT fall back to the + // CustomWidgets$CustomWidget $Type — that is not a theme key and + // would route every pluggable widget at whatever it happened to hit. + t.Errorf("key = %q, want empty for a widget whose id is not stored", got) + } + }) + + t.Run("unknown widget is empty, never a guess", func(t *testing.T) { + m := listViewMutator(t, storedListViewPage()) + if got := storedWidgetDesignPropsKey(m, "noSuchWidget"); got != "" { + t.Errorf("key = %q, want empty", got) + } + }) +} + +// The routing decision itself. nil means "leave SET where it was", which is what +// keeps a typo on the pluggable path and its own error. +func TestDesignPropertyForStoredWidget(t *testing.T) { + m := listViewMutator(t, storedListViewPage()) + theme := atlasListViewTheme() + + if p := designPropertyForStoredWidget(theme, m, "lvOrders", "Row size"); p == nil { + t.Error("a ListView design property was not recognised") + } + // Inherited from the `Widget` group — resolving only the type-specific list + // is the mistake this area keeps producing. + if p := designPropertyForStoredWidget(theme, m, "lvOrders", "Align self"); p == nil { + t.Error("an INHERITED design property was not recognised") + } + // Declared for a different widget type: not this widget's, so not routed. + if p := designPropertyForStoredWidget(theme, m, "lvOrders", "Striped"); p != nil { + t.Error("a property of another widget type was routed to this one") + } + // A typo must stay on the pluggable path and get that path's error. + if p := designPropertyForStoredWidget(theme, m, "lvOrders", "Rowsize"); p != nil { + t.Error("a mistyped key was routed as a design property") + } + // No theme metadata at all: nothing can be claimed. + if p := designPropertyForStoredWidget(nil, m, "lvOrders", "Row size"); p != nil { + t.Error("claimed a design property with no theme to judge against") + } +} + +// The value conversion, including the two shapes a SET assignment cannot carry. +func TestDesignPropertyAssignment(t *testing.T) { + theme := atlasListViewTheme() + rowSize := &theme.WidgetProperties["ListView"][0] + hover := &theme.WidgetProperties["ListView"][1] + hideOn := &theme.WidgetProperties["Widget"][1] + + if vt, opt, err := designPropertyAssignment(rowSize, "Small"); err != nil || vt != "option" || opt != "Small" { + t.Errorf("option value = (%q,%q,%v), want (option,Small,nil)", vt, opt, err) + } + if vt, _, err := designPropertyAssignment(hover, true); err != nil || vt != "toggle" { + t.Errorf("toggle on = (%q,%v), want (toggle,nil)", vt, err) + } + // OFF is the absence of the entry, so it converts to no assignment and the + // caller removes instead. + if vt, _, err := designPropertyAssignment(hover, false); err != nil || vt != "" { + t.Errorf("toggle off = (%q,%v), want ('',nil) so the caller removes", vt, err) + } + // Multi-select: refused, naming the spelling that works (ako/mxcli#511). + _, _, err := designPropertyAssignment(hideOn, "Phone") + if err == nil { + t.Fatal("a flat value on a multi-select property was accepted; mxbuild refuses it with CE6084") + } + if !strings.Contains(err.Error(), "['Phone': on]") { + t.Errorf("the message does not name the compound spelling: %v", err) + } +} + +// The two hand-written maps describe the same concept from opposite directions, +// and the $Type one had NO CALLERS until this change — so it had never been held +// to anything at all. Neither is a subset of the other, and every exclusive entry +// has a measured reason, so this pins both sets rather than asserting a +// consistency that does not hold. +// +// $Type-only — correct, and the reason the stored path resolves MORE than the +// inline one: +// +// - "DataGrid" and "Gallery" are the NATIVE widgets. The MDL keywords no longer +// produce them: `datagrid` resolves through pluggableKeywordIDs to Data grid +// 2's widget id, `gallery` to the pluggable Gallery. Atlas declares both +// native groups, and a Studio Pro-authored widget of either type is reachable +// only from its $Type. +// +// Keyword-only — inert today, and at least two of them wrong: +// +// - `header` and `footer` map to "Header"/"Footer", but MDL builds BOTH as a +// Forms$DivContainer (cmd_pages_builder_v3_layout.go), so a stored one +// resolves to "DivContainer". Atlas declares no Header or Footer group, so +// the inline lookup misses and validateWidgetDesignProps returns early — +// silence reading as approval, the shape pluggableKeywordIDs already records +// for combobox/gallery/image. Not corrected here: it changes what existing +// pages validate against, which is its own change. +// - `snippetcall` maps to "SnippetCall" (stored Forms$SnippetCallWidget); Atlas +// declares no such group either. +func TestBsonTypeAndKeywordDesignPropsKeysAgree(t *testing.T) { + fromKeyword := map[string]bool{} + for _, v := range mdlKeywordToDesignPropsKey { + fromKeyword[v] = true + } + fromBsonType := map[string]bool{} + for _, v := range bsonTypeToDesignPropsKey { + fromBsonType[v] = true + } + if len(fromKeyword) == 0 || len(fromBsonType) == 0 { + t.Fatal("a map is empty — a passing run would prove nothing") + } + + exclusive := func(a, b map[string]bool) map[string]bool { + out := map[string]bool{} + for k := range a { + if !b[k] { + out[k] = true + } + } + return out + } + eq := func(got, want map[string]bool, label string) { + t.Helper() + for k := range got { + if !want[k] { + t.Errorf("%s gained %q — measure which theme group that widget resolves to "+ + "on BOTH paths before adding it, and say so in this test's comment", label, k) + } + } + for k := range want { + if !got[k] { + t.Errorf("%s lost %q — if it is now reachable from both maps, confirm they "+ + "agree on the group rather than just deleting the expectation", label, k) + } + } + } + + eq(exclusive(fromBsonType, fromKeyword), + map[string]bool{"DataGrid": true, "Gallery": true}, "$Type-only keys") + eq(exclusive(fromKeyword, fromBsonType), + map[string]bool{"Header": true, "Footer": true, "SnippetCall": true}, "keyword-only keys") +} + +// The mutator accessor returns raw storage facts, not a resolved key — the +// layering that keeps the theme mapping in one place. +func TestWidgetStorageType(t *testing.T) { + m := listViewMutator(t, storedListViewPage()).(interface { + WidgetStorageType(string) (string, string) + }) + bsonType, widgetID := m.WidgetStorageType("lvOrders") + if bsonType != "Forms$ListView" { + t.Errorf("bsonType = %q, want Forms$ListView", bsonType) + } + if widgetID != "" { + t.Errorf("widgetID = %q, want empty for a native widget", widgetID) + } + if bt, wid := m.WidgetStorageType("nope"); bt != "" || wid != "" { + t.Errorf("an unresolved widget returned (%q,%q), want both empty", bt, wid) + } +} + +var _ = bsonnav.DGetString +var _ = ast.SetPropertyOp{} + +// The wiring, not just the decision: ALTER PAGE SET must reach +// SetDesignProperty, and the value must land in Appearance.DesignProperties +// rather than anywhere SetWidgetProperty would have put it. +func TestApplySetProperty_RoutesADesignProperty(t *testing.T) { + stored := storedListViewPage() + m := listViewMutator(t, stored) + ctx, _ := newMockCtx(t) + ctx.ThemeRegistry = atlasListViewTheme() + + op := &ast.SetPropertyOp{ + Target: ast.WidgetRef{Widget: "lvOrders"}, + Properties: map[string]any{"Row size": "Small"}, + } + if err := applySetPropertyMutator(ctx, m, op, "MyModule", model.ID("mod")); err != nil { + t.Fatalf("set: %v", err) + } + + entries := storedDesignProperties(t, stored, "lvOrders") + if len(entries) != 1 { + t.Fatalf("got %d design property entries, want 1", len(entries)) + } + if got := bsonnav.DGetString(entries[0], "Key"); got != "Row size" { + t.Errorf("Key = %q, want \"Row size\"", got) + } +} + +// A toggle set to OFF is a REMOVAL, not a stored false: Mendix represents the +// off state as the absence of the entry, which is what `alter styling … = off` +// already does. Writing a false toggle leaves an entry Studio Pro shows as on. +func TestApplySetProperty_ToggleOffRemoves(t *testing.T) { + stored := storedListViewPage() + m := listViewMutator(t, stored) + ctx, _ := newMockCtx(t) + ctx.ThemeRegistry = atlasListViewTheme() + + on := &ast.SetPropertyOp{ + Target: ast.WidgetRef{Widget: "lvOrders"}, + Properties: map[string]any{"Hover style": true}, + } + if err := applySetPropertyMutator(ctx, m, on, "MyModule", model.ID("mod")); err != nil { + t.Fatalf("set on: %v", err) + } + if n := len(storedDesignProperties(t, stored, "lvOrders")); n != 1 { + t.Fatalf("after ON: %d entries, want 1", n) + } + + off := &ast.SetPropertyOp{ + Target: ast.WidgetRef{Widget: "lvOrders"}, + Properties: map[string]any{"Hover style": false}, + } + if err := applySetPropertyMutator(ctx, m, off, "MyModule", model.ID("mod")); err != nil { + t.Fatalf("set off: %v", err) + } + if n := len(storedDesignProperties(t, stored, "lvOrders")); n != 0 { + t.Errorf("after OFF: %d entries, want 0 — off is the absence of the entry", n) + } +} + +// The control for the routing: a key the theme does NOT declare for this widget +// must stay on the pluggable path and keep that path's error, or a typo becomes +// a silently-written design property. +func TestApplySetProperty_UnknownKeyIsNotRoutedAsDesignProperty(t *testing.T) { + stored := storedListViewPage() + m := listViewMutator(t, stored) + ctx, _ := newMockCtx(t) + ctx.ThemeRegistry = atlasListViewTheme() + + op := &ast.SetPropertyOp{ + Target: ast.WidgetRef{Widget: "lvOrders"}, + Properties: map[string]any{"Rowsize": "Small"}, + } + err := applySetPropertyMutator(ctx, m, op, "MyModule", model.ID("mod")) + if err == nil { + t.Fatal("a mistyped key was accepted") + } + if n := len(storedDesignProperties(t, stored, "lvOrders")); n != 0 { + t.Errorf("a mistyped key wrote %d design property entry(ies)", n) + } +} + +// storedDesignProperties returns the widget's DesignProperties entries, marker +// stripped. +func storedDesignProperties(t *testing.T, stored bson.D, widgetName string) []bson.D { + t.Helper() + var out []bson.D + var walk func(v any) + walk = func(v any) { + switch d := v.(type) { + case bson.D: + if bsonnav.DGetString(d, "Name") == widgetName { + if app := bsonnav.DGetDoc(d, "Appearance"); app != nil { + for _, e := range bsonnav.DGetArrayElements(bsonnav.DGet(app, "DesignProperties")) { + if ed, ok := e.(bson.D); ok { + out = append(out, ed) + } + } + } + } + for _, e := range d { + walk(e.Value) + } + case bson.A: + for _, e := range d { + walk(e) + } + } + } + walk(stored) + return out +} From a35f39146d2664c0a7b1321e1d1b6eade3a3a54a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:53:06 +0000 Subject: [PATCH 11/38] =?UTF-8?q?fix:=20correct=20MPR=20v1=20docs=20?= =?UTF-8?q?=E2=80=94=20no=20UnitContents=20table,=20detection=20is=20by=20?= =?UTF-8?q?directory?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four reference pages that document the .mpr format described a `UnitContents` table holding v1's BSON blobs, and a v1/v2 detection recipe that probes for it. No .mpr has ever had that table. A v1 file has exactly two tables — `Unit` and `_MetaData` — and document contents are the `Unit.Contents` blob; `grep -rn UnitContents --include=*.go` is 0 hits. The detection recipe was the worse half: implemented as written the probe can never succeed, so it returns v2 for every project including genuine v1 ones — a wrong answer rather than an error. What the code actually does is check for the `mprcontents/` directory, falling back to whether `Unit` has a `Contents` column for a .mpr copied away from its folder. Two further fabrications on the same pages, not in the report: `Unit` was given `UnitType` and `Name` columns (it has seven, and neither is among them — type and name come out of the BSON `$Type`/`Name`), and `mprcontents/` was drawn flat when it is sharded `//.mxunit`. Rewritten from the SQLite catalogs of two real fixtures: the v1 project in modelsdk/mpr/testdata (Mendix 9.24.30) and the v2 project in testdata/expr-checker (11.6.6). modelsdk/mpr/docs_schema_test.go holds the pages there. Prose cannot be type-checked but the identifiers in it can: the rule checks only names beginning with a real table name against the union of the fixtures' tables and columns, so `UnitContents` and `UnitType` are caught while the catalog tables these same pages mention (`REFS` and friends) need no allowlist. The page set is discovered by content, so a page added later is covered. One consequence is stated in the test: a page that wants to say a column does NOT exist must say it in prose. Code font there is indistinguishable from the defect — the old pages' "no `UnitContents`" read as a v1/v2 difference rather than as a fiction, and an exemption for denials would have masked it. Control: with the doc edits stashed and the test kept, the failures reproduce the reporter's line list verbatim — version-compatibility.md:31, mpr-format.md:21,23, mpr-v1-v2.md:12,35,69,73,74,84,94, 10-bson-mapping.md:30 — plus the two they had not found. No bug-test MDL: the defect is in prose, and no MDL statement reproduces it. Refs mendixlabs/mxcli#1072 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013kg5TN4Brse6DJ9WHbm3Ud --- .../skills/fix-issue/findings/modelsdk.jsonl | 1 + .../src/appendixes/version-compatibility.md | 8 +- docs-site/src/internals/mpr-format.md | 67 +++- docs-site/src/internals/mpr-v1-v2.md | 60 +++- docs/05-mdl-specification/10-bson-mapping.md | 24 +- modelsdk/mpr/docs_schema_test.go | 333 ++++++++++++++++++ 6 files changed, 449 insertions(+), 44 deletions(-) create mode 100644 modelsdk/mpr/docs_schema_test.go diff --git a/.claude/skills/fix-issue/findings/modelsdk.jsonl b/.claude/skills/fix-issue/findings/modelsdk.jsonl index 59c630c7bb..01e1bb1644 100644 --- a/.claude/skills/fix-issue/findings/modelsdk.jsonl +++ b/.claude/skills/fix-issue/findings/modelsdk.jsonl @@ -17,3 +17,4 @@ {"area": "modelsdk", "date": "2026-09-02", "symptom": "Every in-place edit of a page is refused with `refusing to write unit \u2026: 1 element id(s) are used more than once \u2026 held by [Texts$Translation \u00d78]` \u2014 `GRANT VIEW ON PAGE`, `ALTER PAGE \u2026 INSERT` \u2014 while a full `CREATE OR REPLACE PAGE` still works, so the page looks correct and only UPDATES are blocked. Surfaces after a second language is enabled.", "cause": "`canon.CarryTranslations` pairs a rebuilt text to its stored translations BY SOURCE STRING when the two documents' text paths differ, and `mergeText` appended the stored `Texts$Translation` element **verbatim** \u2014 deliberately, because keeping the stored `$ID` is what lets no-op elision fire. When several rebuilt texts share one source string (eight copies of the literal `'{1}'` on a page is ordinary), all of them resolve to the SAME stored set and every one got the same element, id included. `reuseSafeID` now gives the first use the stored id and derives a fresh deterministic one (SHA-256 of stored id + containment path + language) for each further copy; the visit order is sorted rather than map order, or which text keeps the stored id would vary per run and the document would churn.", "file": "`modelsdk/canon/translations.go` (`reuseSafeID`, `derivedID`, `elementIDs`, `sortedPaths`, `mergeText`), `modelsdk/canon/duplicates.go` (comment corrected \u2014 it recorded the cause as unestablished)", "insight": "**Re-identifying a copy is safe here in a way that deduplicating ids in general is not, and that distinction is the whole argument.** An `$ID` is a pointer target and rewriting one means finding every reference (ADR-0008) \u2014 which is exactly why `duplicates.go` refuses rather than repairs. Nothing references a `Texts$Translation`: it is a leaf child of a `Texts$Text` with four keys and no identity anything resolves by, so there are no references to miss. Only the COPIES are re-identified; the first use keeps the stored id, so an unchanged document still compares equal. **Verify elision explicitly after touching this** \u2014 the fix trades against the exact property the verbatim append existed for: measured, a second identical run still reports `Unchanged page` with the same sha and mtime. Controls, end-to-end on a real 11.13 project with de_DE enabled and three widgets sharing a caption: the pre-fix binary writes one id used 3\u00d7 and the next `ALTER PAGE` is refused with the reporter's message verbatim; the fixed binary writes 27 distinct ids for 27 elements, the `ALTER PAGE` succeeds, the German translation survives (the control against a 'fix' that just stops carrying), and `mx check` is 0 errors. Reported as CapTrackV2 FINDINGS \u00a730/\u00a717."} {"area": "mdl/backend/modelsdk", "date": "2026-09-07", "symptom": "`mxcli lint` QUAL002 reported \"Page 'X' has no documentation\" against a page carrying a javadoc comment; the catalog's Description column was blank for every page and snippet; `describe page` emitted no documentation. The comment looked, from every angle, like it had been dropped (ako/CapTrackV4 R12).", "cause": "Nothing was dropped: the AST, executor and writer all carry it, and `mxcli bson dump --type page` shows Documentation with the right value. pageFromGen and the ListSnippets constructor in mdl/backend/modelsdk/page.go simply did not read it back, so on the DEFAULT engine every symptom downstream of the read was wrong at once. Fixed by carrying Documentation in both. Separately, QUAL002 stopped sweeping modules: a Mendix module HAS no documentation property (generated/metamodel's ProjectsModule declares none, modelsdk/gen's Module has no accessor, no stored Projects$ModuleImpl carries the key).", "file": "`mdl/backend/modelsdk/page.go` (pageFromGen, ListSnippets); `mdl/linter/context.go` (documentableSources); `.claude/lint-rules/missing_documentation.star`; tests `mdl/backend/modelsdk/page_documentation_test.go`", "insight": "When a value looks absent everywhere, check the WRITE first: `bson dump` showed it stored correctly and localised the bug to the read in one step, where chasing the reported symptom would have started at the visitor. The engine split is the second cheap discriminator — the legacy reader parsed it fine, so the defect was in the default engine alone. A stale catalog nearly hid that: an earlier per-engine comparison reused a cached catalog.db and showed both engines empty, so DELETE the catalog between engine comparisons rather than trusting `refresh catalog full`. Finally, page and snippet were 2 of 5 sibling readers in one file — layout, building block and page template all carried Documentation — which is the shape to look for when one document type behaves differently from its neighbours. And a rule asking for a property the platform does not have is not a gap in the language: three sources agreed before that row was removed."} {"area": "modelsdk/meta", "date": "2026-09-22", "symptom": "A view entity selecting `u.Name` from System.User could not be declared in any way that both passed `mxcli check` and built: `String(100)` (the correct length) was refused, `String` (unlimited) passed check and then failed mxbuild with CE6770 \"View Entity is out of sync with the OQL Query\". `describe entity System.User` reported `Name: String(unlimited)`. Reported in ako/ChipCoV4 FINDINGS.md against Mendix 11.14.0 (ako/mxcli#584, with #585 the other half).", "cause": "meta.SystemAttrDef declared a Length field and NOT ONE of the 115 String attributes in modelsdk/meta/system_module.go populated it, so systemAttrType built every System string as StringAttributeType{Length: 0} — which mxcli reads as unlimited. Every length comparison against a System attribute was therefore made against 0. Fixed by measuring all 115 and populating them, with a golden table (modelsdk/meta/testdata/system_string_lengths.txt) and TestSystemStringLengths holding the two in step.", "file": "`modelsdk/meta/system_module.go` (SystemEntities, SystemAttrDef.Length); `modelsdk/meta/testdata/system_string_lengths.txt`; tests `modelsdk/meta/system_string_lengths_test.go`, `modelsdk/meta/system_string_lengths_measure_test.go`, `mdl/backend/modelsdk/system_module_read_test.go`", "insight": "The System module's attribute lengths are IN THE BUILD OUTPUT: `deployment/model/model.mdp` is a stream of BSON documents (each with its own 4-byte length prefix — unmarshalling the file whole fails with \"invalid document length\"), the System module arrives as a Projects$ModuleImpl carrying only a Name with its DomainModels$DomainModel immediately after, and every entity's attributes are there with their StringAttributeType.Length. That is ONE `mxbuild --target=deploy` for all 115, and it is the model the runtime builds the tables from, so it is the same number CE6770 is decided by. Two searches not worth repeating, both spent on this issue: the Mendix Model SDK does not carry them (its gen/ describes metamodel TYPES, so System.User.Name is not in it) and the modeler's own copy is inside Mendix.Modeler.Core.dll, i.e. a decompiler. The one-view-entity-per-attribute mxbuild probe works but is ~40s each. The version question answers itself the same way: building 10.24.4.77222 as well showed all 216 shared attributes identical in type AND length to 11.14.0, so one table serves every supported version instead of a per-version registry — measure the second version rather than reasoning about it, it is one more build. Finally, 0 is Mendix's own encoding of \"unlimited\" (46 of the 115), so populating the table does not make 0 safe to read as a length — what makes it safe is that the golden enumerates every String attribute, so 'unmeasured' cannot exist without failing a test."} +{"area": "docs / modelsdk/mpr", "date": "2026-09-22", "symptom": "Four reference pages described an MPR v1 `UnitContents` table holding the BSON blobs, and a v1/v2 detection recipe that probes for it. No .mpr has ever had that table: a v1 file has exactly `Unit` and `_MetaData`, and contents are the `Unit.Contents` blob. `grep -rn UnitContents --include=*.go` is 0 hits. Reported by an outside reader building an independent format reader (mendixlabs/mxcli#1072).", "cause": "Never-measured prose. The pages also invented `UnitType` and `Name` columns on `Unit` (there are seven columns and neither is among them — type and name come out of the BSON `$Type`/`Name`), and drew `mprcontents/` flat when it is sharded `//.mxunit`. Fixed by rewriting the four pages from the SQLite catalogs of two real fixtures, and adding modelsdk/mpr/docs_schema_test.go to hold them there.", "file": "`docs-site/src/internals/mpr-format.md`, `docs-site/src/internals/mpr-v1-v2.md`, `docs-site/src/appendixes/version-compatibility.md`, `docs/05-mdl-specification/10-bson-mapping.md`; test `modelsdk/mpr/docs_schema_test.go`", "insight": "Prose cannot be type-checked but the IDENTIFIERS in it can, and the rule that makes it zero-maintenance is a prefix rule, not an allowlist: check only names BEGINNING with a real table name (`Unit`, `_MetaData`, `_Transaction`) against the union of the fixtures' tables and columns. `UnitContents` and `UnitType` are caught; the catalog tables these same pages mention (`REFS` and friends) never start with a real .mpr table name, so they need no exemption and no one has to maintain a list. The page set is discovered by content (any .md under docs-site/src or docs/05-mdl-specification mentioning `.mpr`/`mprcontents`), so a page added later is covered without anyone remembering. One consequence worth stating in the docs themselves: a page that wants to say a column does NOT exist must say it in PROSE — the first fix wrote \"there is no `UnitType` column\" and the test flagged its own remedy, which is correct, because the old pages' \"no `UnitContents`\" at mpr-v1-v2.md:35 read as a v1/v2 difference rather than as a fiction and an exemption for denials would have masked it. The control is cheap and exact here: `git stash` the doc edits with the test file kept, and the failures reproduce the reporter's line list verbatim (version-compatibility.md:31, mpr-format.md:21,23, mpr-v1-v2.md:12,35,69,73,74,84,94, 10-bson-mapping.md:30) plus the two they had not found. No Mendix tool runs in this fix's argument — the claims are about SQLite schema and are read straight off `sqlite_master`/`PRAGMA table_info`, which is the primary source, so the usual 'build two apps' rule does not apply.", "refs": ["mendixlabs/mxcli#1072"]} diff --git a/docs-site/src/appendixes/version-compatibility.md b/docs-site/src/appendixes/version-compatibility.md index fa66f8a6cb..de85ef306e 100644 --- a/docs-site/src/appendixes/version-compatibility.md +++ b/docs-site/src/appendixes/version-compatibility.md @@ -28,13 +28,15 @@ which is authoritative -- this table names the minors only. ### v1 (Mendix < 10.18) - Single `.mpr` SQLite database file -- All documents stored as BSON blobs in the `UnitContents` table +- All documents stored as BSON blobs in the `Contents` column of the `Unit` + table -- there is no separate contents table - Self-contained -- one file holds the entire project ### v2 (Mendix >= 10.18) -- `.mpr` SQLite file for metadata only -- `mprcontents/` folder with individual `.mxunit` files for each document +- `.mpr` SQLite file for metadata only -- the `Unit` table has no `Contents` + column +- `mprcontents///.mxunit` -- one file per document - Better suited for Git version control (smaller, per-document diffs) The library auto-detects the format. No configuration is needed. diff --git a/docs-site/src/internals/mpr-format.md b/docs-site/src/internals/mpr-format.md index d76ff7e368..b0ba435c1c 100644 --- a/docs-site/src/internals/mpr-format.md +++ b/docs-site/src/internals/mpr-format.md @@ -4,37 +4,66 @@ Mendix projects are stored in `.mpr` files, which are SQLite databases containin ## Structure -An MPR file is a standard SQLite database with two key tables: +An MPR file is a standard SQLite database. Both format versions carry the same +two tables; v2 adds a third. Contents live in the `Unit` table in v1 and in +`mprcontents/` in v2 — **there is no separate contents table in either format.** + +| Table | v1 | v2 | Holds | +|-------|----|----|-------| +| `Unit` | yes | yes | One row per document | +| `_MetaData` | yes | yes | Mendix product/build version and schema hash | +| `_Transaction` | no | yes | `LastTransactionID`, bumped on every unit write | ### Unit Table -The `Unit` table stores document metadata: +The `Unit` table has one row per document. Its columns are identical in both +formats except for `Contents`, which only v1 has: + +| Column | v1 | v2 | Description | +|--------|----|----|-------------| +| `UnitID` | yes | yes | Binary UUID identifying the document (.NET GUID byte order — see below) | +| `ContainerID` | yes | yes | Parent unit's `UnitID`; the project root is its own container | +| `ContainmentName` | yes | yes | Relationship name (e.g. `ProjectDocuments`), empty on the root | +| `TreeConflict` | yes | yes | Version-control conflict marker | +| `ContentsHash` | yes | yes | Base64 SHA-256 of the document BSON | +| `ContentsConflicts` | yes | yes | Version-control conflict marker for contents | +| `Contents` | **yes** | **no** | BSON blob containing the full document | -| Column | Description | -|--------|-------------| -| `UnitID` | Binary UUID identifying the document | -| `ContainerID` | Parent module UUID | -| `ContainmentName` | Relationship name (e.g., `documents`) | -| `UnitType` | Fully qualified type name | -| `Name` | Document name | +The row carries **no unit-type and no name column** — the seven above are all +there is. A document's type and name are read out of its BSON `$Type` and +`Name` fields, which is why listing units by type requires decoding contents +(`getTypeFromContents` in `modelsdk/mpr/reader_units.go`). -### UnitContents Table (v1 only) +### Where Contents Live -In MPR v1, the `UnitContents` table stores the actual BSON document content: +In **v1**, document BSON is the `Unit.Contents` blob: + +```sql +SELECT Contents FROM Unit WHERE UnitID = ?; -- read +UPDATE Unit SET Contents = ? WHERE UnitID = ?; -- write +``` -| Column | Description | -|--------|-------------| -| `UnitID` | Binary UUID matching the Unit table | -| `Contents` | BSON blob containing the full document | +In **v2**, `Unit.Contents` does not exist. Each document is a file under +`mprcontents/`, sharded two levels deep by the first four hex characters of its +UUID: + +``` +mprcontents///.mxunit +``` -In MPR v2, document contents are stored as individual files in the `mprcontents/` folder instead. +The UUID in the path is the `UnitID` blob rendered in **.NET GUID byte order** — +the first three fields are little-endian, so blob `FADF10BF FF61 8842 8A63D53AE4522615` +becomes `bf10dffa-61ff-4288-8a63-d53ae4522615`. Writing a v2 unit updates +`Unit.ContentsHash` (and `_Transaction.LastTransactionID`) in SQLite after the +file lands. ## Unit Types -Every document in a Mendix project has a unit type: +Every document has a type, carried in its BSON `$Type` field. It is not a +column on `Unit`, so identifying a document means decoding its contents: -| UnitType | Document Type | -|----------|---------------| +| `$Type` | Document Type | +|---------|---------------| | `DomainModels$DomainModel` | Domain model (entities, associations) | | `DomainModels$ViewEntitySourceDocument` | OQL query for VIEW entities | | `Microflows$Microflow` | Microflow definition | diff --git a/docs-site/src/internals/mpr-v1-v2.md b/docs-site/src/internals/mpr-v1-v2.md index 9837093de7..49f30875b5 100644 --- a/docs-site/src/internals/mpr-v1-v2.md +++ b/docs-site/src/internals/mpr-v1-v2.md @@ -8,8 +8,13 @@ A single `.mpr` SQLite database file containing all model data. ### Storage -- `Unit` table: document metadata (name, type, container) -- `UnitContents` table: BSON document blobs +- `Unit` table: one row per document — identity, containment, conflict markers, + content hash, **and the BSON blob itself in the `Contents` column** +- `_MetaData` table: Mendix product/build version and schema hash + +There is no separate contents table. `SELECT Contents FROM Unit WHERE UnitID = ?` +is the read; `UPDATE Unit SET Contents = ? WHERE UnitID = ?` is the write. See +[MPR File Format](./mpr-format.md) for the full column list. ### Characteristics @@ -32,10 +37,13 @@ An `.mpr` metadata file plus a `mprcontents/` folder with individual document fi ### Storage -- `.mpr` file: SQLite with `Unit` table (metadata only, no `UnitContents`) +- `.mpr` file: SQLite with the same `Unit` table as v1 **minus the `Contents` + column**, plus a `_Transaction` table holding `LastTransactionID` - `mprcontents/` folder: individual `.mxunit` files containing BSON -Each `.mxunit` file is named by the document's UUID and contains the raw BSON for that document. +Each `.mxunit` file holds the raw BSON for one document. It is named by the +document's `UnitID` rendered in .NET GUID byte order, and lives two directories +deep — sharded by the first two and next two hex characters of that name. ### Directory Layout @@ -43,12 +51,18 @@ Each `.mxunit` file is named by the document's UUID and contains the raw BSON fo project/ ├── app.mpr # SQLite metadata └── mprcontents/ - ├── 2a3b4c5d-... # Domain model BSON - ├── 6e7f8a9b-... # Microflow BSON - ├── c0d1e2f3-... # Page BSON + ├── bf/ + │ └── 10/ + │ └── bf10dffa-61ff-4288-8a63-d53ae4522615.mxunit + ├── 2a/ + │ └── 3b/ + │ └── 2a3b4c5d-....mxunit └── ... ``` +A flat `mprcontents/.mxunit` will not be found: the two shard directories +are part of the path. + ### Characteristics - Better for Git versioning -- individual files change independently @@ -66,12 +80,23 @@ reader, _ := modelsdk.Open("/path/to/project.mpr") ## Format Detection -The library detects the format by checking whether the `UnitContents` table exists in the SQLite database: +Detection is by **directory**, with a schema check as the reconciliation step +(`modelsdk/mpr/reader.go`): + +| Step | Check | Result | +|------|-------|--------| +| 1 | `mprcontents/` exists next to the `.mpr` and is a directory | v2 | +| 2 | otherwise | v1 | +| 3 | if step 2 said v1 but `Unit` has **no `Contents` column** | v2 after all | + +Step 3 exists because the folder check fails for a `.mpr` copied away from its +`mprcontents/`. The schema is the ground truth there: `Unit.Contents` is +present in v1 and absent in v2, which is the one column that distinguishes the +two. Opening such a project **for writing** is refused rather than guessed at, +since the SQLite rows would have no files behind them. -| Condition | Format | -|-----------|--------| -| `UnitContents` table exists and has rows | v1 | -| `UnitContents` table missing or empty, `mprcontents/` folder exists | v2 | +`_MetaData._FormatVersion` is a second, independent signal — present and equal +to `2` in v2, absent in v1 — which mxcli does not currently use. This detection is automatic -- callers of `Open()` and `OpenForWriting()` do not need to specify the format. @@ -81,7 +106,8 @@ This detection is automatic -- callers of `Open()` and `OpenForWriting()` do not |---------|----|----| | Mendix version | < 10.18 | >= 10.18 | | File structure | Single `.mpr` | `.mpr` + `mprcontents/` | -| Document storage | SQLite `UnitContents` table | Individual `.mxunit` files | +| Tables | `Unit`, `_MetaData` | `Unit`, `_MetaData`, `_Transaction` | +| Document storage | `Unit.Contents` blob in SQLite | Individual `.mxunit` files | | Git friendliness | Poor (binary diffs) | Good (per-document files) | | File size | Larger single file | Distributed across files | | Read performance | Single DB query | File I/O per document | @@ -91,7 +117,11 @@ This detection is automatic -- callers of `Open()` and `OpenForWriting()` do not When writing with `OpenForWriting()`: -- **v1**: Documents are written as BSON blobs into the `UnitContents` table within a SQLite transaction -- **v2**: Documents are written as individual `.mxunit` files in the `mprcontents/` folder; the `Unit` table metadata is updated in SQLite +- **v1**: `UPDATE Unit SET Contents = ? WHERE UnitID = ?` — the blob goes into + the row it belongs to +- **v2**: the BSON is written to a temp file and renamed into place at + `mprcontents///.mxunit`, then SQLite is updated with the new + `Unit.ContentsHash` (base64 SHA-256 of the contents) and a fresh + `_Transaction.LastTransactionID` The writer handles both formats transparently. diff --git a/docs/05-mdl-specification/10-bson-mapping.md b/docs/05-mdl-specification/10-bson-mapping.md index c8b2c4c1f2..005240496c 100644 --- a/docs/05-mdl-specification/10-bson-mapping.md +++ b/docs/05-mdl-specification/10-bson-mapping.md @@ -25,19 +25,29 @@ This document describes how MDL constructs map to BSON structures in Mendix MPR Mendix projects are stored in `.mpr` files which contain: ### MPR v1 (Mendix < 10.18) -Single SQLite database file with: -- `Unit` table: Document metadata -- `UnitContents` table: BSON document contents +Single SQLite database file with two tables: +- `Unit`: one row per document, BSON contents included as the `Contents` blob +- `_MetaData`: Mendix product/build version and schema hash + +There is no separate contents table: `SELECT Contents FROM Unit WHERE UnitID = ?`. ### MPR v2 (Mendix >= 10.18) SQLite metadata file + separate content files: -- `.mpr` file: SQLite with `Unit` table (metadata only) -- `mprcontents/` folder: Individual `.mxunit` files containing BSON +- `.mpr` file: SQLite with the same `Unit` table **minus `Contents`**, plus a + `_Transaction` table +- `mprcontents///.mxunit`: one file per document, containing BSON + +Format detection is by the presence of the `mprcontents/` directory, falling +back to whether `Unit` has a `Contents` column -- never by probing for a +contents table. See +[v1 vs v2](../../docs-site/src/internals/mpr-v1-v2.md). ### Unit Types -| UnitType | Document Type | -|----------|---------------| +A document's type is its BSON `$Type`, not a column on `Unit`: + +| `$Type` | Document Type | +|---------|---------------| | `DomainModels$DomainModel` | Domain model (entities, associations) | | `DomainModels$ViewEntitySourceDocument` | OQL query for VIEW entities | | `microflows$microflow` | Microflow definition | diff --git a/modelsdk/mpr/docs_schema_test.go b/modelsdk/mpr/docs_schema_test.go new file mode 100644 index 0000000000..0f873d5e44 --- /dev/null +++ b/modelsdk/mpr/docs_schema_test.go @@ -0,0 +1,333 @@ +// SPDX-License-Identifier: Apache-2.0 + +package mpr + +import ( + "database/sql" + "fmt" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + + _ "modernc.org/sqlite" +) + +// The internals pages are the reference someone uses to build an independent +// reader or writer for the .mpr format — that is what they are for. They +// described a `UnitContents` table that has never existed in any .mpr, and a +// v1/v2 detection recipe that probes for it. Implemented as written the probe +// can never succeed, so it returns v2 for every project including genuine v1 +// ones: a wrong answer rather than an error, which is the kind that survives +// testing (mendixlabs/mxcli#1072). +// +// Prose cannot be type-checked, but the identifiers in it can. These tests +// hold the pages to the schema of two real fixtures — the v1 project in this +// package's testdata and the v2 project at testdata/expr-checker — so a +// fabricated table or column fails the suite instead of a reader six months +// from now. +// +// The invariant they enforce, stated once: **on a page describing the .mpr, any +// identifier in the Unit/_MetaData/_Transaction namespace written in code font +// is a real table or column.** So a page that wants to say a column does NOT +// exist says it in prose — code font there would be indistinguishable from the +// defect. (The old pages did exactly that at mpr-v1-v2.md:35, "no +// `UnitContents`", which read as a v1/v2 difference rather than as a fiction.) + +// repoRoot is the module root, two levels up from modelsdk/mpr. +const repoRoot = "../.." + +const ( + v1Fixture = "testdata/v1-project/App.mpr" + v2Fixture = repoRoot + "/testdata/expr-checker/minimal.mpr" +) + +// mprSchema reads table -> column set straight out of an .mpr's SQLite +// catalog. Measured, never asserted from the docs under test. +func mprSchema(t *testing.T, path string) map[string]map[string]bool { + t.Helper() + db, err := sql.Open("sqlite", fmt.Sprintf("file:%s?mode=ro", path)) + if err != nil { + t.Fatalf("open %s: %v", path, err) + } + defer db.Close() + + rows, err := db.Query(`SELECT name FROM sqlite_master WHERE type='table' ORDER BY name`) + if err != nil { + t.Fatalf("list tables in %s: %v", path, err) + } + var tables []string + for rows.Next() { + var n string + if err := rows.Scan(&n); err != nil { + t.Fatalf("scan table name: %v", err) + } + tables = append(tables, n) + } + rows.Close() + if len(tables) == 0 { + t.Fatalf("%s has no tables — fixture missing or not an .mpr", path) + } + + schema := make(map[string]map[string]bool, len(tables)) + for _, tbl := range tables { + cols, err := db.Query(fmt.Sprintf("PRAGMA table_info(%s)", tbl)) + if err != nil { + t.Fatalf("table_info(%s): %v", tbl, err) + } + set := map[string]bool{} + for cols.Next() { + var cid, notNull, pk int + var name, colType string + var dflt *string + if err := cols.Scan(&cid, &name, &colType, ¬Null, &dflt, &pk); err != nil { + t.Fatalf("scan column of %s: %v", tbl, err) + } + set[name] = true + } + cols.Close() + schema[tbl] = set + } + return schema +} + +// realMPRSchema unions the v1 and v2 fixtures: a name is real if either +// format has it. v2 drops Unit.Contents and adds _Transaction and +// _MetaData._FormatVersion, so neither fixture alone is the whole vocabulary. +func realMPRSchema(t *testing.T) (tables map[string]map[string]bool) { + t.Helper() + tables = map[string]map[string]bool{} + for _, f := range []string{v1Fixture, v2Fixture} { + for tbl, cols := range mprSchema(t, f) { + if tables[tbl] == nil { + tables[tbl] = map[string]bool{} + } + for c := range cols { + tables[tbl][c] = true + } + } + } + return tables +} + +// mprDocPages returns every reference page that describes the .mpr file. +// Discovered by content, not hardcoded, so a page added later is covered +// without anyone remembering to add it here. Proposals and plans are excluded +// on purpose: they record what was considered, not what the format is. +func mprDocPages(t *testing.T) []string { + t.Helper() + var pages []string + for _, dir := range []string{ + filepath.Join(repoRoot, "docs-site", "src"), + filepath.Join(repoRoot, "docs", "05-mdl-specification"), + } { + err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error { + if err != nil || info.IsDir() || !strings.HasSuffix(path, ".md") { + return err + } + b, err := os.ReadFile(path) + if err != nil { + return err + } + if strings.Contains(string(b), "mprcontents") || strings.Contains(string(b), ".mpr") { + pages = append(pages, path) + } + return nil + }) + if err != nil { + t.Fatalf("walk %s: %v", dir, err) + } + } + if len(pages) == 0 { + t.Fatal("found no reference pages describing the .mpr file — the walk is looking in the wrong place") + } + return pages +} + +var ( + backtickedIdent = regexp.MustCompile("`([A-Za-z_][A-Za-z0-9_$]*)`") + headingIdent = regexp.MustCompile(`^#{1,6}\s+.*?\b([A-Za-z_][A-Za-z0-9_$]*)\s+Table\b`) +) + +// TestMPRDocsNameOnlyRealUnitSchema fails when a reference page names an +// identifier in the .mpr's own namespace that no real .mpr has. +// +// The rule is deliberately narrow: only names that BEGIN with a real table +// name are checked, so the catalog tables these pages also mention (`REFS` +// and friends) are out of scope and need no allowlist. That is exactly the +// shape of the reported defect — `UnitContents` alongside the real `Unit` — +// and of its quieter half, a `UnitType` column on a table that has none. +func TestMPRDocsNameOnlyRealUnitSchema(t *testing.T) { + tables := realMPRSchema(t) + + legit := map[string]bool{} + var prefixes []string + for tbl, cols := range tables { + legit[tbl] = true + prefixes = append(prefixes, tbl) + for c := range cols { + legit[c] = true + } + } + sort.Strings(prefixes) + + inNamespace := func(name string) bool { + for _, p := range prefixes { + if strings.HasPrefix(name, p) { + return true + } + } + return false + } + + for _, page := range mprDocPages(t) { + b, err := os.ReadFile(page) + if err != nil { + t.Fatalf("read %s: %v", page, err) + } + rel, _ := filepath.Rel(repoRoot, page) + for i, line := range strings.Split(string(b), "\n") { + var names []string + for _, m := range backtickedIdent.FindAllStringSubmatch(line, -1) { + names = append(names, m[1]) + } + if m := headingIdent.FindStringSubmatch(line); m != nil { + names = append(names, m[1]) + } + for _, n := range names { + if inNamespace(n) && !legit[n] { + t.Errorf("%s:%d names %q, which is not a table or column of any real .mpr.\n"+ + "Measured tables: %s\nThese pages are what an independent reader/writer is built from.", + rel, i+1, n, describeSchema(tables)) + } + } + } + } +} + +func describeSchema(tables map[string]map[string]bool) string { + var names []string + for t := range tables { + names = append(names, t) + } + sort.Strings(names) + var parts []string + for _, n := range names { + var cols []string + for c := range tables[n] { + cols = append(cols, c) + } + sort.Strings(cols) + parts = append(parts, fmt.Sprintf("%s(%s)", n, strings.Join(cols, ", "))) + } + return strings.Join(parts, "; ") +} + +// TestMPRFormatDocUnitColumnsAreReal reads the column table printed under the +// "Unit Table" heading in the format page and holds every documented column +// to the fixture. The prefix rule above cannot see `Name`, which the page +// listed as a Unit column: unit type and name are read out of the BSON +// contents (getTypeFromContents), never off a column. +func TestMPRFormatDocUnitColumnsAreReal(t *testing.T) { + real := mprSchema(t, v1Fixture)["Unit"] + if real == nil { + t.Fatal("v1 fixture has no Unit table") + } + + page := filepath.Join(repoRoot, "docs-site", "src", "internals", "mpr-format.md") + b, err := os.ReadFile(page) + if err != nil { + t.Fatalf("read %s: %v", page, err) + } + + documented := unitColumnTable(t, string(b)) + if len(documented) == 0 { + t.Fatal("docs-site/src/internals/mpr-format.md no longer prints a column table under a Unit Table heading; " + + "this test is then asserting nothing — re-point it or delete it") + } + for _, col := range documented { + if !real[col] { + t.Errorf("mpr-format.md documents Unit column %q; the v1 fixture's Unit table has no such column (%s)", + col, describeSchema(map[string]map[string]bool{"Unit": real})) + } + } +} + +// unitColumnTable pulls the first cell of every body row of the first +// markdown table that follows a heading naming the Unit table. +func unitColumnTable(t *testing.T, md string) []string { + t.Helper() + lines := strings.Split(md, "\n") + start := -1 + for i, l := range lines { + if strings.HasPrefix(l, "#") && regexp.MustCompile(`\bUnit\s+Table\b`).MatchString(l) { + start = i + break + } + } + if start < 0 { + return nil + } + var cols []string + inTable := false + for _, l := range lines[start+1:] { + trimmed := strings.TrimSpace(l) + if strings.HasPrefix(trimmed, "#") { + break + } + if !strings.HasPrefix(trimmed, "|") { + if inTable { + break + } + continue + } + inTable = true + cells := strings.Split(strings.Trim(trimmed, "|"), "|") + first := strings.TrimSpace(cells[0]) + first = strings.Trim(first, "`") + if first == "" || first == "Column" || strings.HasPrefix(first, "-") || strings.HasPrefix(first, ":") { + continue + } + cols = append(cols, first) + } + return cols +} + +// TestMPRVersionDetectionRestsOnContentsColumn is the measured basis for the +// corrected detection prose. Detection is by directory first +// (Open: os.Stat of mprcontents/), with the Unit.Contents column as the +// reconciliation fallback for a .mpr copied away from its folder. Both halves +// are only sound because the column is present in exactly one of the two +// formats — the property the docs' old "UnitContents table exists" recipe was +// reaching for and got wrong. +func TestMPRVersionDetectionRestsOnContentsColumn(t *testing.T) { + if got := mprSchema(t, v1Fixture)["Unit"]["Contents"]; !got { + t.Error("v1 fixture's Unit table has no Contents column; v1 contents are stored there, not in a separate table") + } + if got := mprSchema(t, v2Fixture)["Unit"]["Contents"]; got { + t.Error("v2 fixture's Unit table has a Contents column; v2 stores contents in mprcontents/, so the column should be absent") + } + if _, ok := mprSchema(t, v1Fixture)["UnitContents"]; ok { + t.Error("v1 fixture has a UnitContents table — the docs' claim would be correct and this whole file is wrong") + } + + r1, err := Open(filepath.Join("testdata", "v1-project", "App.mpr")) + if err != nil { + t.Fatalf("open v1 fixture: %v", err) + } + defer r1.Close() + if r1.Version() != MPRVersionV1 { + t.Errorf("v1 fixture detected as %v, want %v", r1.Version(), MPRVersionV1) + } + + r2, err := Open(filepath.Join(repoRoot, "testdata", "expr-checker", "minimal.mpr")) + if err != nil { + t.Fatalf("open v2 fixture: %v", err) + } + defer r2.Close() + if r2.Version() != MPRVersionV2 { + t.Errorf("v2 fixture detected as %v, want %v", r2.Version(), MPRVersionV2) + } +} From b8f795a979d17fc1565f24f2ce200ec43be3c4fe Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:59:41 +0000 Subject: [PATCH 12/38] fix: correct $Type storage names in the MPR unit-type tables MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "Unit Types" tables on the two MPR reference pages mapped a document's BSON `$Type` to a document kind, and 15 rows named a spelling no unit carries. Four gave the TypeScript SDK's qualified name instead of the storage name — `Pages$Page`, `Pages$Layout`, `Pages$Snippet`, `Pages$BuildingBlock`, where every real unit says `Forms$*` — and 10-bson-mapping.md lowercased eleven more (`microflows$microflow`, `pages$page`, `security$ProjectSecurity` and friends), which matters because `$Type` is case-sensitive. Selecting on either spelling matches zero units: a wrong answer rather than an error, the same failure mode as the UnitContents detection recipe in the previous commit. `CustomWidgets$customwidget` was removed rather than corrected. It is a widget element inside a page's widget tree (`CustomWidgets$CustomWidget`), never a unit. Both tables are now the measured set: every distinct `$Type` across a blank Mendix 11.6.6 app (369 units) and a 9.24.30 app (20 units), 28 in all. Types a blank project has no instance of are listed separately and sourced to mxcli's own readers and writers rather than presented as measured. TestDocumentedUnitTypesUseStorageNames holds them there. It keys on the local name after the `$`, case-insensitively: a fixture cannot prove a type absent — a blank project simply has no business-event service — so requiring every documented type to be present would fail rows that are correct. When a fixture does have a type with the same local name, the documented row must equal it exactly. That catches all four `Pages$` rows and all eleven lowercase ones with no false positives on legitimately absent types. The limit is stated in the test: a row whose local name appears nowhere in the fixtures is not checked, which is how `CustomWidgets$customwidget` slipped past and had to be removed by hand. Control: with the two tables reverted and the test kept, it fails on exactly those 15 rows and no others. Refs mendixlabs/mxcli#1072 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_013kg5TN4Brse6DJ9WHbm3Ud --- .../skills/fix-issue/findings/modelsdk.jsonl | 1 + docs-site/src/internals/mpr-format.md | 60 +++++-- docs/05-mdl-specification/10-bson-mapping.md | 65 +++++-- modelsdk/mpr/docs_schema_test.go | 165 ++++++++++++++++++ 4 files changed, 264 insertions(+), 27 deletions(-) diff --git a/.claude/skills/fix-issue/findings/modelsdk.jsonl b/.claude/skills/fix-issue/findings/modelsdk.jsonl index 01e1bb1644..61eef19a39 100644 --- a/.claude/skills/fix-issue/findings/modelsdk.jsonl +++ b/.claude/skills/fix-issue/findings/modelsdk.jsonl @@ -18,3 +18,4 @@ {"area": "mdl/backend/modelsdk", "date": "2026-09-07", "symptom": "`mxcli lint` QUAL002 reported \"Page 'X' has no documentation\" against a page carrying a javadoc comment; the catalog's Description column was blank for every page and snippet; `describe page` emitted no documentation. The comment looked, from every angle, like it had been dropped (ako/CapTrackV4 R12).", "cause": "Nothing was dropped: the AST, executor and writer all carry it, and `mxcli bson dump --type page` shows Documentation with the right value. pageFromGen and the ListSnippets constructor in mdl/backend/modelsdk/page.go simply did not read it back, so on the DEFAULT engine every symptom downstream of the read was wrong at once. Fixed by carrying Documentation in both. Separately, QUAL002 stopped sweeping modules: a Mendix module HAS no documentation property (generated/metamodel's ProjectsModule declares none, modelsdk/gen's Module has no accessor, no stored Projects$ModuleImpl carries the key).", "file": "`mdl/backend/modelsdk/page.go` (pageFromGen, ListSnippets); `mdl/linter/context.go` (documentableSources); `.claude/lint-rules/missing_documentation.star`; tests `mdl/backend/modelsdk/page_documentation_test.go`", "insight": "When a value looks absent everywhere, check the WRITE first: `bson dump` showed it stored correctly and localised the bug to the read in one step, where chasing the reported symptom would have started at the visitor. The engine split is the second cheap discriminator — the legacy reader parsed it fine, so the defect was in the default engine alone. A stale catalog nearly hid that: an earlier per-engine comparison reused a cached catalog.db and showed both engines empty, so DELETE the catalog between engine comparisons rather than trusting `refresh catalog full`. Finally, page and snippet were 2 of 5 sibling readers in one file — layout, building block and page template all carried Documentation — which is the shape to look for when one document type behaves differently from its neighbours. And a rule asking for a property the platform does not have is not a gap in the language: three sources agreed before that row was removed."} {"area": "modelsdk/meta", "date": "2026-09-22", "symptom": "A view entity selecting `u.Name` from System.User could not be declared in any way that both passed `mxcli check` and built: `String(100)` (the correct length) was refused, `String` (unlimited) passed check and then failed mxbuild with CE6770 \"View Entity is out of sync with the OQL Query\". `describe entity System.User` reported `Name: String(unlimited)`. Reported in ako/ChipCoV4 FINDINGS.md against Mendix 11.14.0 (ako/mxcli#584, with #585 the other half).", "cause": "meta.SystemAttrDef declared a Length field and NOT ONE of the 115 String attributes in modelsdk/meta/system_module.go populated it, so systemAttrType built every System string as StringAttributeType{Length: 0} — which mxcli reads as unlimited. Every length comparison against a System attribute was therefore made against 0. Fixed by measuring all 115 and populating them, with a golden table (modelsdk/meta/testdata/system_string_lengths.txt) and TestSystemStringLengths holding the two in step.", "file": "`modelsdk/meta/system_module.go` (SystemEntities, SystemAttrDef.Length); `modelsdk/meta/testdata/system_string_lengths.txt`; tests `modelsdk/meta/system_string_lengths_test.go`, `modelsdk/meta/system_string_lengths_measure_test.go`, `mdl/backend/modelsdk/system_module_read_test.go`", "insight": "The System module's attribute lengths are IN THE BUILD OUTPUT: `deployment/model/model.mdp` is a stream of BSON documents (each with its own 4-byte length prefix — unmarshalling the file whole fails with \"invalid document length\"), the System module arrives as a Projects$ModuleImpl carrying only a Name with its DomainModels$DomainModel immediately after, and every entity's attributes are there with their StringAttributeType.Length. That is ONE `mxbuild --target=deploy` for all 115, and it is the model the runtime builds the tables from, so it is the same number CE6770 is decided by. Two searches not worth repeating, both spent on this issue: the Mendix Model SDK does not carry them (its gen/ describes metamodel TYPES, so System.User.Name is not in it) and the modeler's own copy is inside Mendix.Modeler.Core.dll, i.e. a decompiler. The one-view-entity-per-attribute mxbuild probe works but is ~40s each. The version question answers itself the same way: building 10.24.4.77222 as well showed all 216 shared attributes identical in type AND length to 11.14.0, so one table serves every supported version instead of a per-version registry — measure the second version rather than reasoning about it, it is one more build. Finally, 0 is Mendix's own encoding of \"unlimited\" (46 of the 115), so populating the table does not make 0 safe to read as a length — what makes it safe is that the golden enumerates every String attribute, so 'unmeasured' cannot exist without failing a test."} {"area": "docs / modelsdk/mpr", "date": "2026-09-22", "symptom": "Four reference pages described an MPR v1 `UnitContents` table holding the BSON blobs, and a v1/v2 detection recipe that probes for it. No .mpr has ever had that table: a v1 file has exactly `Unit` and `_MetaData`, and contents are the `Unit.Contents` blob. `grep -rn UnitContents --include=*.go` is 0 hits. Reported by an outside reader building an independent format reader (mendixlabs/mxcli#1072).", "cause": "Never-measured prose. The pages also invented `UnitType` and `Name` columns on `Unit` (there are seven columns and neither is among them — type and name come out of the BSON `$Type`/`Name`), and drew `mprcontents/` flat when it is sharded `//.mxunit`. Fixed by rewriting the four pages from the SQLite catalogs of two real fixtures, and adding modelsdk/mpr/docs_schema_test.go to hold them there.", "file": "`docs-site/src/internals/mpr-format.md`, `docs-site/src/internals/mpr-v1-v2.md`, `docs-site/src/appendixes/version-compatibility.md`, `docs/05-mdl-specification/10-bson-mapping.md`; test `modelsdk/mpr/docs_schema_test.go`", "insight": "Prose cannot be type-checked but the IDENTIFIERS in it can, and the rule that makes it zero-maintenance is a prefix rule, not an allowlist: check only names BEGINNING with a real table name (`Unit`, `_MetaData`, `_Transaction`) against the union of the fixtures' tables and columns. `UnitContents` and `UnitType` are caught; the catalog tables these same pages mention (`REFS` and friends) never start with a real .mpr table name, so they need no exemption and no one has to maintain a list. The page set is discovered by content (any .md under docs-site/src or docs/05-mdl-specification mentioning `.mpr`/`mprcontents`), so a page added later is covered without anyone remembering. One consequence worth stating in the docs themselves: a page that wants to say a column does NOT exist must say it in PROSE — the first fix wrote \"there is no `UnitType` column\" and the test flagged its own remedy, which is correct, because the old pages' \"no `UnitContents`\" at mpr-v1-v2.md:35 read as a v1/v2 difference rather than as a fiction and an exemption for denials would have masked it. The control is cheap and exact here: `git stash` the doc edits with the test file kept, and the failures reproduce the reporter's line list verbatim (version-compatibility.md:31, mpr-format.md:21,23, mpr-v1-v2.md:12,35,69,73,74,84,94, 10-bson-mapping.md:30) plus the two they had not found. No Mendix tool runs in this fix's argument — the claims are about SQLite schema and are read straight off `sqlite_master`/`PRAGMA table_info`, which is the primary source, so the usual 'build two apps' rule does not apply.", "refs": ["mendixlabs/mxcli#1072"]} +{"area": "docs / modelsdk/mpr", "date": "2026-09-22", "symptom": "The MPR reference pages' \"Unit Types\" tables mapped BSON `$Type` to document kinds, and 15 of the rows named a spelling no unit carries: `Pages$Page`/`Pages$Layout`/`Pages$Snippet`/`Pages$BuildingBlock` (real units say `Forms$*`), and docs/05-mdl-specification/10-bson-mapping.md lowercased eleven more (`microflows$microflow`, `pages$page`, `security$ProjectSecurity`…). It also listed `CustomWidgets$customwidget` as a document type. Found while fixing mendixlabs/mxcli#1072, filed and fixed separately.", "cause": "The tables were written from the TypeScript SDK's QUALIFIED names rather than the storage names Mendix writes — the same split CLAUDE.md documents for `ShowPageAction`/`ShowFormAction`, never applied here. `CustomWidgets$CustomWidget` is a widget element inside a page's tree (mdl/catalog/builder_widget_refs.go), never a unit, so that row was removed rather than corrected.", "file": "`docs-site/src/internals/mpr-format.md`, `docs/05-mdl-specification/10-bson-mapping.md`; test `modelsdk/mpr/docs_schema_test.go` (TestDocumentedUnitTypesUseStorageNames)", "insight": "Measuring the real set is one command and settles the whole table at once: decode every `mprcontents/*/*/*.mxunit` (and every v1 `Unit.Contents` blob) and count `$Type` — 28 distinct values across a blank 11.6.6 app and a 9.24.30 one. Do NOT try to verify rows one at a time against gen, which carries BOTH spellings: `model/types.go` defines `DocumentTypePage = \"Pages$Page\"` and mdl/catalog/builder_xpath.go defensively matches `Forms$Page` AND `Pages$Page`, so grepping the codebase 'confirms' the wrong name. The fixture is the arbiter; the codebase is not. The test rule that makes this checkable without a maintenance burden keys on the LOCAL name after the `$`, case-insensitively: a fixture cannot prove a type ABSENT (a blank project has no business-event service), so demanding every documented type be present would fail correct rows — but when the fixture has a type with the same local name, the documented row must equal it exactly. That catches all four `Pages$` rows and all eleven lowercase ones with zero false positives. Its stated limit is real and cost a manual fix: a row whose local name appears nowhere in the fixtures is not checked at all, which is how `CustomWidgets$customwidget` slipped past and had to be removed by hand. One editing trap, not a Mendix one: anchoring a section replacement on `'---'` matches a markdown TABLE SEPARATOR (`|---|---|`) long before the horizontal rule you meant — the edit silently no-ops on the table you were replacing. Anchor on `'\\n---\\n'`.", "refs": ["mendixlabs/mxcli#1072"]} diff --git a/docs-site/src/internals/mpr-format.md b/docs-site/src/internals/mpr-format.md index b0ba435c1c..9b88d89e59 100644 --- a/docs-site/src/internals/mpr-format.md +++ b/docs-site/src/internals/mpr-format.md @@ -60,25 +60,61 @@ file lands. ## Unit Types Every document has a type, carried in its BSON `$Type` field. It is not a -column on `Unit`, so identifying a document means decoding its contents: +column on `Unit`, so identifying a document means decoding its contents. + +These are **storage names**, and for the page family they differ from the +names the TypeScript SDK uses: a page is stored as `Forms$Page`, never +`Pages$Page` -- "Form" was the original term for "Page". Using the SDK +spelling to select documents matches nothing, which is a wrong answer rather +than an error. See [Storage Names](./storage-names.md). + +The set below is measured: it is every distinct `$Type` in a blank Mendix +11.6.6 app (369 units) unioned with a 9.24.30 app (20 units). | `$Type` | Document Type | |---------|---------------| +| `Constants$Constant` | Constant | +| `CustomIcons$CustomIconCollection` | Custom icon collection | | `DomainModels$DomainModel` | Domain model (entities, associations) | -| `DomainModels$ViewEntitySourceDocument` | OQL query for VIEW entities | -| `Microflows$Microflow` | Microflow definition | -| `Microflows$Nanoflow` | Nanoflow definition | -| `Pages$Page` | Page definition | -| `Pages$Layout` | Layout definition | -| `Pages$Snippet` | Snippet definition | -| `Pages$BuildingBlock` | Building block definition | -| `Enumerations$Enumeration` | Enumeration definition | -| `JavaActions$JavaAction` | Java action definition | -| `Security$ProjectSecurity` | Project security settings | -| `Security$ModuleSecurity` | Module security settings | +| `Enumerations$Enumeration` | Enumeration | +| `ExportMappings$ExportMapping` | Export mapping | +| `Forms$BuildingBlock` | Building block | +| `Forms$Layout` | Layout | +| `Forms$Page` | Page | +| `Forms$PageTemplate` | Page template | +| `Forms$Snippet` | Snippet | +| `Images$ImageCollection` | Image collection | +| `ImportMappings$ImportMapping` | Import mapping | +| `JavaActions$JavaAction` | Java action | +| `JavaScriptActions$JavaScriptAction` | JavaScript action | +| `JsonStructures$JsonStructure` | JSON structure | +| `Menus$MenuDocument` | Menu document | +| `Microflows$Microflow` | Microflow | +| `Microflows$Nanoflow` | Nanoflow | | `Navigation$NavigationDocument` | Navigation profile | +| `Projects$Folder` | Folder | +| `Projects$ModuleImpl` | Module | +| `Projects$ModuleSettings` | Per-module settings | +| `Projects$Project` | Project root | +| `Projects$ProjectConversion` | Version-conversion record | +| `Security$ModuleSecurity` | Module security settings | +| `Security$ProjectSecurity` | Project security settings | | `Settings$ProjectSettings` | Project settings | +| `Texts$SystemTextCollection` | System text collection | + +A project with no instance of a document type simply has no unit of it, so +these are named from mxcli's own readers and writers rather than measured +above: + +| `$Type` | Document Type | +|---------|---------------| | `BusinessEvents$BusinessEventService` | Business event service | +| `CustomBlobDocuments$CustomBlobDocument` | Custom blob document | +| `DomainModels$ViewEntitySourceDocument` | OQL query for VIEW entities | +| `Microflows$Rule` | Rule (a rule is a flow, so it is in the Microflows namespace) | +| `Queues$Queue` | Task queue | +| `RegularExpressions$RegularExpression` | Regular expression | +| `ScheduledEvents$ScheduledEvent` | Scheduled event | ## BSON Document Structure diff --git a/docs/05-mdl-specification/10-bson-mapping.md b/docs/05-mdl-specification/10-bson-mapping.md index 005240496c..7bb8255aa9 100644 --- a/docs/05-mdl-specification/10-bson-mapping.md +++ b/docs/05-mdl-specification/10-bson-mapping.md @@ -44,26 +44,61 @@ contents table. See ### Unit Types -A document's type is its BSON `$Type`, not a column on `Unit`: +A document's type is its BSON `$Type`, not a column on `Unit`. + +These are **storage names**, and for the page family they differ from the +names the TypeScript SDK uses: a page is stored as `Forms$Page`, never +`Pages$Page` -- "Form" was the original term for "Page". Using the SDK +spelling to select documents matches nothing, which is a wrong answer rather +than an error. See [Storage Names](../../docs-site/src/internals/storage-names.md). + +The set below is measured: it is every distinct `$Type` in a blank Mendix +11.6.6 app (369 units) unioned with a 9.24.30 app (20 units). | `$Type` | Document Type | |---------|---------------| +| `Constants$Constant` | Constant | +| `CustomIcons$CustomIconCollection` | Custom icon collection | | `DomainModels$DomainModel` | Domain model (entities, associations) | -| `DomainModels$ViewEntitySourceDocument` | OQL query for VIEW entities | -| `microflows$microflow` | Microflow definition | -| `microflows$nanoflow` | Nanoflow definition | -| `pages$page` | Page definition | -| `pages$layout` | Layout definition | -| `pages$snippet` | Snippet definition | -| `pages$BuildingBlock` | Building block definition | -| `enumerations$enumeration` | Enumeration definition | -| `JavaActions$JavaAction` | Java action definition | -| `security$ProjectSecurity` | Project security settings | -| `security$ModuleSecurity` | Module security settings | -| `navigation$NavigationDocument` | Navigation profile | -| `settings$ProjectSettings` | Project settings | +| `Enumerations$Enumeration` | Enumeration | +| `ExportMappings$ExportMapping` | Export mapping | +| `Forms$BuildingBlock` | Building block | +| `Forms$Layout` | Layout | +| `Forms$Page` | Page | +| `Forms$PageTemplate` | Page template | +| `Forms$Snippet` | Snippet | +| `Images$ImageCollection` | Image collection | +| `ImportMappings$ImportMapping` | Import mapping | +| `JavaActions$JavaAction` | Java action | +| `JavaScriptActions$JavaScriptAction` | JavaScript action | +| `JsonStructures$JsonStructure` | JSON structure | +| `Menus$MenuDocument` | Menu document | +| `Microflows$Microflow` | Microflow | +| `Microflows$Nanoflow` | Nanoflow | +| `Navigation$NavigationDocument` | Navigation profile | +| `Projects$Folder` | Folder | +| `Projects$ModuleImpl` | Module | +| `Projects$ModuleSettings` | Per-module settings | +| `Projects$Project` | Project root | +| `Projects$ProjectConversion` | Version-conversion record | +| `Security$ModuleSecurity` | Module security settings | +| `Security$ProjectSecurity` | Project security settings | +| `Settings$ProjectSettings` | Project settings | +| `Texts$SystemTextCollection` | System text collection | + +A project with no instance of a document type simply has no unit of it, so +these are named from mxcli's own readers and writers rather than measured +above: + +| `$Type` | Document Type | +|---------|---------------| | `BusinessEvents$BusinessEventService` | Business event service | -| `CustomWidgets$customwidget` | Custom widget definition | +| `CustomBlobDocuments$CustomBlobDocument` | Custom blob document | +| `DomainModels$ViewEntitySourceDocument` | OQL query for VIEW entities | +| `Microflows$Rule` | Rule (a rule is a flow, so it is in the Microflows namespace) | +| `Queues$Queue` | Task queue | +| `RegularExpressions$RegularExpression` | Regular expression | +| `ScheduledEvents$ScheduledEvent` | Scheduled event | --- diff --git a/modelsdk/mpr/docs_schema_test.go b/modelsdk/mpr/docs_schema_test.go index 0f873d5e44..46cd4e68f6 100644 --- a/modelsdk/mpr/docs_schema_test.go +++ b/modelsdk/mpr/docs_schema_test.go @@ -12,6 +12,7 @@ import ( "strings" "testing" + "go.mongodb.org/mongo-driver/v2/bson" _ "modernc.org/sqlite" ) @@ -331,3 +332,167 @@ func TestMPRVersionDetectionRestsOnContentsColumn(t *testing.T) { t.Errorf("v2 fixture detected as %v, want %v", r2.Version(), MPRVersionV2) } } + +// --- Unit types ------------------------------------------------------------- +// +// The same pages map a document's BSON $Type to a document kind. Those were +// wrong in the same way and for the same reason: several rows named the +// TypeScript SDK's qualified name instead of the storage name Mendix actually +// writes — `Pages$Page` for what every real unit calls `Forms$Page` — and one +// page lowercased half the table, which matters because $Type is +// case-sensitive. Selecting on either spelling matches zero units: a wrong +// answer rather than an error, the same failure mode as the UnitContents +// detection recipe above. + +// realUnitTypes returns every distinct $Type across both fixtures. The v1 +// fixture's units are Unit.Contents blobs; the v2 fixture's are .mxunit files. +func realUnitTypes(t *testing.T) map[string]bool { + t.Helper() + types := map[string]bool{} + + add := func(raw []byte) { + if v, err := bson.Raw(raw).LookupErr("$Type"); err == nil { + if s, ok := v.StringValueOK(); ok && s != "" { + types[s] = true + } + } + } + + db, err := sql.Open("sqlite", fmt.Sprintf("file:%s?mode=ro", v1Fixture)) + if err != nil { + t.Fatalf("open v1 fixture: %v", err) + } + rows, err := db.Query(`SELECT Contents FROM Unit`) + if err != nil { + db.Close() + t.Fatalf("read v1 contents: %v", err) + } + for rows.Next() { + var blob []byte + if err := rows.Scan(&blob); err != nil { + t.Fatalf("scan v1 contents: %v", err) + } + add(blob) + } + rows.Close() + db.Close() + + v2Dir := filepath.Join(repoRoot, "testdata", "expr-checker", "mprcontents") + err = filepath.Walk(v2Dir, func(path string, info os.FileInfo, err error) error { + if err != nil || info.IsDir() || !strings.HasSuffix(path, ".mxunit") { + return err + } + b, err := os.ReadFile(path) + if err != nil { + return err + } + add(b) + return nil + }) + if err != nil { + t.Fatalf("walk %s: %v", v2Dir, err) + } + + if len(types) < 20 { + t.Fatalf("only %d distinct $Type values across both fixtures — the fixtures are not being read", len(types)) + } + return types +} + +var docTypeName = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9]*\$[A-Za-z][A-Za-z0-9]*$`) + +// unitTypeTables collects the first column of every markdown table under a +// heading naming unit types, across both pages that print one. +func unitTypeTables(t *testing.T) map[string][]string { + t.Helper() + pages := []string{ + filepath.Join(repoRoot, "docs-site", "src", "internals", "mpr-format.md"), + filepath.Join(repoRoot, "docs", "05-mdl-specification", "10-bson-mapping.md"), + } + out := map[string][]string{} + for _, page := range pages { + b, err := os.ReadFile(page) + if err != nil { + t.Fatalf("read %s: %v", page, err) + } + rel, _ := filepath.Rel(repoRoot, page) + lines := strings.Split(string(b), "\n") + start := -1 + for i, l := range lines { + if strings.HasPrefix(l, "#") && strings.Contains(l, "Unit Types") { + start = i + break + } + } + if start < 0 { + t.Errorf("%s no longer has a Unit Types heading; this test is asserting nothing about it", rel) + continue + } + var found []string + for _, l := range lines[start+1:] { + trimmed := strings.TrimSpace(l) + if strings.HasPrefix(trimmed, "#") { + break // next heading — stop, but keep every table until then + } + if !strings.HasPrefix(trimmed, "|") { + continue + } + cells := strings.Split(strings.Trim(trimmed, "|"), "|") + name := strings.Trim(strings.TrimSpace(cells[0]), "`") + if docTypeName.MatchString(name) { + found = append(found, name) + } + } + if len(found) == 0 { + t.Errorf("%s prints no $Type rows under Unit Types", rel) + } + out[rel] = found + } + return out +} + +// TestDocumentedUnitTypesUseStorageNames holds every documented $Type to the +// spelling real units carry. +// +// The check is deliberately keyed on the LOCAL name (the part after the `$`), +// case-insensitively, because the fixtures cannot prove a type absent — a +// blank project simply has no business-event service, so demanding that every +// documented type appear would fail on rows that are perfectly correct. But +// when a fixture DOES have a type with the same local name, the documented +// row must match it exactly: that catches `Pages$Page` against `Forms$Page` +// and every lowercased spelling, with no false positives on legitimately +// absent types. +// +// The limit is worth stating: a documented type whose local name appears +// nowhere in the fixtures is not checked at all. `CustomWidgets$customwidget` +// was one such row and had to be removed by hand — it is a widget element +// inside a page's tree (`CustomWidgets$CustomWidget`), never a unit. +func TestDocumentedUnitTypesUseStorageNames(t *testing.T) { + real := realUnitTypes(t) + + byLocal := map[string][]string{} + for full := range real { + local := strings.ToLower(full[strings.Index(full, "$")+1:]) + byLocal[local] = append(byLocal[local], full) + } + for k := range byLocal { + sort.Strings(byLocal[k]) + } + + for page, documented := range unitTypeTables(t) { + for _, d := range documented { + local := strings.ToLower(d[strings.Index(d, "$")+1:]) + candidates, known := byLocal[local] + if !known { + continue // no unit of this kind in either fixture — unprovable here + } + if real[d] { + continue + } + t.Errorf("%s documents $Type %q; real units spell it %s.\n"+ + "$Type is the storage name and is case-sensitive — selecting on the SDK's "+ + "qualified name matches zero units.", + page, d, strings.Join(candidates, " or ")) + } + } +} From 78612b5ea92393d48dfedf0f5a7cd1d8edd30942 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 11:03:09 +0000 Subject: [PATCH 13/38] feat(pages): ALTER PAGES ... WHERE WIDGETTYPE bulk-sets design properties MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The singular form (previous commit) styles one widget on one page. Applying a styling decision across an app — "every data grid gets Compact and Striped" — meant one statement per widget per page, which is how a design system drifts. ALTER PAGES [IN ] SET '' = '' | ON | OFF [, ...] WHERE WIDGETTYPE = [DRY RUN] The selector resolves an MDL widget keyword (datagrid, combobox, ...) to the pluggable widget id, so the statement is written in the same vocabulary the rest of MDL uses; a raw widget id is still accepted for a widget MDL has no keyword for. Matching reuses findMatchingWidgets from UPDATE WIDGETS, so pages and snippets are both covered. Reporting reuses the updateOutcome type from the UPDATE WIDGETS fix in 79f49c86, for the same reason: a bulk statement that matched widgets but could not set the property on any of them exits non-zero and says so, rather than printing a success line that ADR-0008 elision then quietly makes untrue. DRY RUN goes through pagemutator.Probe(), so the preview is the real setter on a discarded copy rather than a guess. Verified end to end on an 11.12.2 project: dry run names both grids, apply reports 2 styled, `describe styling` shows both properties, `mx check` is 0 errors, and a misspelled property key reports 0 styled / 2 matched with nothing settable and exits 1. Refs ako/mxcli#515 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LYaTtzjmMCUjo8XAVN1sQx --- .../fix-issue/findings/mdl-executor.jsonl | 1 + cmd/mxcli/syntax/features_page.go | 2 +- docs/01-project/MDL_QUICK_REFERENCE.md | 1 + ...yling-515-alter-page-design-properties.mdl | 47 ++++ mdl-examples/doctype-tests/12-styling.mdl | 25 ++ mdl/ast/ast_alter_page.go | 17 ++ mdl/executor/alter_pages_styling_test.go | 127 ++++++++++ mdl/executor/cmd_alter_pages_styling.go | 232 ++++++++++++++++++ mdl/executor/register_stubs.go | 3 + mdl/executor/registry_test.go | 1 + mdl/grammar/MDLParser.g4 | 34 +++ mdl/visitor/visitor_alter.go | 4 + mdl/visitor/visitor_alter_page.go | 42 ++++ 13 files changed, 535 insertions(+), 1 deletion(-) create mode 100644 mdl-examples/doctype-tests/12-styling.mdl create mode 100644 mdl/executor/alter_pages_styling_test.go create mode 100644 mdl/executor/cmd_alter_pages_styling.go diff --git a/.claude/skills/fix-issue/findings/mdl-executor.jsonl b/.claude/skills/fix-issue/findings/mdl-executor.jsonl index e465152c8a..5e246dfd96 100644 --- a/.claude/skills/fix-issue/findings/mdl-executor.jsonl +++ b/.claude/skills/fix-issue/findings/mdl-executor.jsonl @@ -672,3 +672,4 @@ {"area": "mdl/executor", "date": "2026-09-21", "symptom": "Follow-up to the sort-hop inference fix: with the hop derivable but not SAYABLE, `describe → exec` still silently changed the program wherever two associations reach the same entity. Measured on 11.12.3 with Order_ShipTo and Order_BillTo (both Order -> Address): a microflow sorting by the BILLING address came back sorting by the SHIPPING one, `mx check` 0 errors on both sides. Same for a page datasource's sort bar.", "cause": "DESCRIBE emitted only the sort attribute's qualified name and the reader never looked at the hop at all — `sortItemsFromRaw` read AttributeRef.Attribute and skipped AttributeRef.EntityRef, so the association was written and never read back. MDL had no spelling for it either (`sortColumn : (qualifiedName | IDENTIFIER)`). Closed end to end: sortColumn takes `qualifiedName (SLASH qualifiedName)*` (the shape MDLCatalog.g4 already uses for Association/Entity), SortColumnDef/OrderByItemV3 carry the hops, the executor resolves the NAMED association instead of inferring, both readers reconstruct EntityRef.Steps, both describers emit `Assoc/.../Attr`, and the page writers moved from attributeRefToGen to inputAttributeRefToGen. Inference stays as the fallback, so every script written before still works.", "file": "`mdl/grammar/domains/MDLPage.g4` (sortColumn) + `mdl/ast/ast_page.go`/`ast_page_v3.go` + `mdl/visitor/visitor_microflow_statements.go` (sortColumnHops) + `visitor_page_v3.go` + `mdl/executor/cmd_microflows_builder_actions.go` (resolveSortAssociationPath, lookupSortHop, entityChainModules) + `cmd_microflows_format_action.go` + `cmd_pages_builder_v3.go` (resolveAssociationAttributePathForEntity) + `cmd_pages_describe_datasource.go` (sortAttributeHops, sortColumnPath) + `mdl/backend/modelsdk/microflow_read_actions.go` (entityRefStepsFromRaw) + `widget_write.go` + `sdk/pages/pages_datasources.go` (GridSort.AttributeRefSteps)", "insight": "**The measurement that decides whether a lossy describer is worth a language change is a CONSTRUCTED one.** The corpus agrees with the inference rule by construction — every document mxcli itself wrote stores the association inference would have picked, so the round trip is a fixed point on everything to hand and looks faithful. The case that matters had to be built: two associations to one entity, then the stored hop edited to the one inference does NOT pick. Byte-patching the .mxunit is enough and takes a minute — `Order_ShipTo` and `Order_BillTo` are the same length, so a `sed` on the BSON needs no resize — and the replay flipped it back immediately. **Control on a binary that drops the hop, not just on one that reverts the fix**: reverting only proves the test fires, while dropping the hop gets mxbuild to say CE7247 \"Cannot sort on attribute … is not an attribute of entity …\" — the executor's own refusal message from the other end of the pipeline, which is what proves the EntityRef load-bearing rather than cosmetic. **Two reads were missing, not one**: the microflow reader and the page reader each drop the hop separately, and fixing only the half named in the report would have shipped a describer that emits the path for microflows and silently drops it for pages. **The strongest round-trip evidence is 'Unchanged'** — with identity preservation and write elision, replaying DESCRIBE output on a correct implementation elides the write entirely, so `Unchanged microflow: …` is a stronger result than any byte comparison."} {"area":"mdl/executor","date":"2026-09-22","symptom":"`UPDATE WIDGETS` prints a per-property `Warning: Failed to set …` for every assignment and then reports `Updated 2 widget(s)`, plus `Note: Run 'refresh catalog full force' to update the catalog with changes`, and exits 0. `describe styling` afterwards shows nothing was written","cause":"`updated++` sat OUTSIDE the assignment loop and was unconditional, so the counter meant \"this widget was found\" and was reported as \"Updated\". The same counter gated `mutator.Save()`, so a container whose every assignment failed was still saved","file":"`mdl/executor/cmd_widgets.go` (`updateOutcome`, `updateWidgetsInContainer`, `execUpdateWidgets` summary)","insight":"**A success counter incremented in the wrong loop is invisible to every test that only checks the happy path** — the failures were already being printed correctly one line above the lie. Split the outcome into the three things that actually happen (changed / matched-but-unwritable / in-catalog-but-not-in-document) rather than adding a boolean: rounding the third into either of the others is how a stale catalog reads as success. **Bound the severity before writing it up**: the rebuilt document was semantically identical, so ADR-0008 elision skipped the write — measured, no `mprcontents/` unit changed mtime and `mx check` stayed at 0 errors, making this a reporting defect and not a data one. Worth saying, because \"claims success after failing\" otherwise reads as corruption. **The DRY RUN had the same defect one step earlier and is the worse half**, since the syntax help tells you to run it first: it printed `Would set …` without attempting anything. Fixed by running the assignments against `pagemutator.Probe()` — the discardable copy `mxcli check` already uses for ALTER PAGE SET — so the preview reports `Cannot set`. Reuse that seam rather than re-deriving what a setter accepts; a preview that re-implements the rule drifts from it in exactly the direction that hurts","refs":["ako/mxcli#520","ako/mxcli#515"]} {"area":"mdl/executor","date":"2026-09-22","symptom":"`alter page … set '' = on ` dead-ended — `set` reaches first-class properties and the stored widget's PLUGGABLE property bag, and a design property lives in `Appearance.DesignProperties`. The only spelling that worked was `alter styling`, a second statement for the same operation","cause":"No resolution from a STORED widget to its theme-registry key, so `set` could not tell a design property from a mistyped pluggable one and had to assume the latter","file":"`mdl/backend/pagemutator/probe.go` (`WidgetStorageType`); `mdl/executor/design_property_routing.go` (new); `cmd_alter_page.go` (`applySetPropertyMutator`); `mdl/backend/pagemutator/mutator.go` (the now-stale error message)","insight":"**The resolver the routing needed already existed with zero callers.** `bsonTypeToDesignPropsKey` ($Type → theme key) had never been referenced, so it had never been validated against anything; ako/mxcli#509 deliberately avoided standing up a third consumer of the concept before something needed it, and this was that something. **Do not assert the two key maps are consistent — they are not, and both directions have measured reasons.** $Type-only: `DataGrid`/`Gallery` are the NATIVE widgets, which the MDL keywords no longer produce (`datagrid`→Data grid 2's id via pluggableKeywordIDs), so the stored path resolves MORE than the inline one. Keyword-only: `header`/`footer` map to \"Header\"/\"Footer\" but MDL builds BOTH as `Forms$DivContainer`, and Atlas declares no such groups — so the inline design-property validation for a header widget misses and skips the widget silently, the same shape pluggableKeywordIDs records for combobox/gallery/image. A test that pins both exclusive SETS with their reasons is the useful shape; a consistency assertion fails on correct code. **Route only on a positive theme declaration for THIS widget's type** — routing on \"the theme says nothing, so it must be a design property\" turns a typo into a silently-written design property. **Prove the two statements are the same operation on bytes, not on reasoning**: write via `alter styling`, then run the `alter page` form and count rewritten units — 0 means elision found them semantically equal. `Altered page` is ALTER PAGE's fixed verb and is NOT the elision verb, so it proves nothing. Knock-on: the #1135 error message named `alter styling` as the route, which became stale the moment `set` learned the route — and a test asserted that wording, so it had to be inverted like the others","refs":["ako/mxcli#515","ako/mxcli#509","ako/mxcli#511","mendixlabs/mxcli#1135"]} +{"area":"mdl/executor","date":"2026-09-22","symptom":"No way to set a design property across pages — \"every data grid compact and striped\" was one statement per page, and the bulk command that looked right (`update widgets`) writes only the pluggable property bag","cause":"ALTER PAGE's design-property SET (the singular half of ako/mxcli#515) had no plural sibling; MDL's only bulk page statement was `ALTER PAGES … SET LAYOUT`","file":"`mdl/grammar/MDLParser.g4` (`alterPagesStylingStatement`); `mdl/ast/ast_alter_page.go`; `mdl/visitor/visitor_alter_page.go`; `mdl/executor/cmd_alter_pages_styling.go` (new)","insight":"**The selector is the whole design problem, and a name cannot be it**: a widget name is unique only within its page (measured — `actionButton1` in 30 units of a blank project), so the predicate has to be a widget TYPE. Name it by the **MDL keyword**, resolved through the existing `pluggableKeywordIDs`, not by a `LIKE` over the stored id: `WidgetType LIKE '%datagrid%'` matches 20 widgets in 6 containers on a blank project because it sweeps in DatagridTextFilter/DateFilter/DropdownFilter, which do not carry the grid's design properties. **Reuse three things instead of growing a fourth of each** — `findMatchingWidgets` (the catalog query), the per-widget routing decision from the singular form, and `updateOutcome` from ako/mxcli#520 so a sweep that matches and writes nothing exits non-zero instead of claiming success. **Two ANTLR traps, both positional**: the rule has two `identifierOrKeyword` slots (optional module, WHERE value) returned as ONE list, so reading them positionally without checking `ctx.IN()` scopes a project-wide sweep to a module named after a widget type; and the sibling `ALTER PAGES … SET LAYOUT` shares the same prefix, so a test that the layout form still parses as itself is not optional. `ensureCatalog(ctx, true)` must be called before `findMatchingWidgets` or it nil-panics — a cold catalog otherwise reads as \"no such widgets\"","refs":["ako/mxcli#515","ako/mxcli#520"]} diff --git a/cmd/mxcli/syntax/features_page.go b/cmd/mxcli/syntax/features_page.go index e98e911458..828169068e 100644 --- a/cmd/mxcli/syntax/features_page.go +++ b/cmd/mxcli/syntax/features_page.go @@ -284,7 +284,7 @@ CREATE PAGE Sales.Detail (Title: 'Detail', Layout: Atlas_Core.Atlas_Default) { "popup width", "popup height", "popup resizable", "drop template", "insert template", "list view template", }, - Syntax: "ALTER PAGE Module.Name {\n SET property = value ON widgetName; -- widget property names: any casing\n SET 'Row size' = 'Small' ON lvOrders; -- an Atlas DESIGN property of that widget's\n -- type; quoted and case-sensitive.\n -- `show design properties for ` lists\n -- them. ON/OFF for a toggle, where OFF\n -- REMOVES the entry.\n -- A multi-select ('Hide on') or compound\n -- ('Spacing') one needs the inline\n -- DesignProperties: [...] form, because a\n -- SET assignment carries one value.\n SET Action = MICROFLOW Module.MF ON btnSave; -- any CREATE PAGE action form\n SET DataSource = $Param ON dvOrder; -- parameter/microflow/nanoflow/selection;\n -- DATABASE and association are REPLACE-only,\n -- and a data view takes no database source\n SET (prop1 = val1, prop2 = val2) ON widgetName;\n SET Title = 'New Title'; -- page-level (case-sensitive)\n SET Documentation = 'What this page is for.';\n SET Class = 'css-class'; -- page-level CSS class / style\n SET Style = 'css: rule';\n SET PopupWidth = 800; -- page-level pop-up dimensions\n SET PopupHeight = 480;\n SET PopupResizable = true;\n INSERT AFTER widgetName { };\n INSERT BEFORE widgetName { };\n INSERT INTO containerName { };\n DROP WIDGET name1, name2;\n DROP TEMPLATE FOR Module.Specialization IN listViewName;\n REPLACE widgetName WITH { };\n};", + Syntax: "ALTER PAGE Module.Name {\n SET property = value ON widgetName; -- widget property names: any casing\n SET 'Row size' = 'Small' ON lvOrders; -- an Atlas DESIGN property of that widget's\n -- type; quoted and case-sensitive.\n -- `show design properties for ` lists\n -- them. ON/OFF for a toggle, where OFF\n -- REMOVES the entry.\n -- A multi-select ('Hide on') or compound\n -- ('Spacing') one needs the inline\n -- DesignProperties: [...] form, because a\n -- SET assignment carries one value.\n SET Action = MICROFLOW Module.MF ON btnSave; -- any CREATE PAGE action form\n SET DataSource = $Param ON dvOrder; -- parameter/microflow/nanoflow/selection;\n -- DATABASE and association are REPLACE-only,\n -- and a data view takes no database source\n SET (prop1 = val1, prop2 = val2) ON widgetName;\n SET Title = 'New Title'; -- page-level (case-sensitive)\n SET Documentation = 'What this page is for.';\n SET Class = 'css-class'; -- page-level CSS class / style\n SET Style = 'css: rule';\n SET PopupWidth = 800; -- page-level pop-up dimensions\n SET PopupHeight = 480;\n SET PopupResizable = true;\n INSERT AFTER widgetName { };\n INSERT BEFORE widgetName { };\n INSERT INTO containerName { };\n DROP WIDGET name1, name2;\n DROP TEMPLATE FOR Module.Specialization IN listViewName;\n REPLACE widgetName WITH { };\n};\n\n-- The BULK form: one design property on every widget of a TYPE.\nALTER PAGES [IN Module]\n SET 'Compact' = ON, 'Striped' = ON\n WHERE WIDGETTYPE = datagrid -- the MDL keyword, which resolves to\n -- exactly one widget id. A full id in\n -- quotes works too. NOT a name: a widget\n -- name is unique only within its page.\n [DRY RUN]; -- run this FIRST. It reports the matches\n -- against a discardable copy and writes\n -- nothing.", Example: "ALTER PAGE Module.EditPage {\n SET (Caption = 'Save & Close', ButtonStyle = Success) ON btnSave;\n INSERT AFTER txtName {\n TEXTBOX txtMiddleName (Label: 'Middle Name', Attribute: MiddleName)\n };\n DROP WIDGET txtUnused;\n};", SeeAlso: []string{"page.create", "page.show", "snippet.alter"}, }) diff --git a/docs/01-project/MDL_QUICK_REFERENCE.md b/docs/01-project/MDL_QUICK_REFERENCE.md index d18a639799..c18b30fbba 100644 --- a/docs/01-project/MDL_QUICK_REFERENCE.md +++ b/docs/01-project/MDL_QUICK_REFERENCE.md @@ -1472,6 +1472,7 @@ MDL uses explicit property declarations for pages: | Alter layout | `alter layout Module.Name { };` | Edits the stored document, so widgets MDL cannot spell survive. Refused for a Marketplace target | | Set a design property | `alter page Module.Page { set 'Row size' = 'Small' on lvOrders; };` | An Atlas design property of that widget's **type** — quoted, case-sensitive; `show design properties for ` lists them. `on`/`off` for a toggle, where `off` removes the entry. Same document `alter styling` writes. A **multi-select** (`Hide on`) or **compound** (`Spacing`) property needs the inline `DesignProperties: [...]` form, since a `set` assignment carries one value | | Repoint one page | `alter page Module.Page { set Layout = Module.Layout [map (Old as New, …)]; };` | Rewrites the layout reference **and** every placeholder binding | +| Set a design property on every widget of a type | `alter pages [in ] set 'Compact' = on, 'Striped' = on where widgettype = datagrid [dry run];` | The house-style sweep. `widgettype` takes the **MDL keyword**, which resolves to exactly one widget id — a `like '%datagrid%'` predicate also matches the data grid's *filter* widgets. Never a widget **name**: a name is unique only within its page. `dry run` previews against a discardable copy. A sweep that matches widgets and writes none of them exits non-zero | | Repoint many pages | `alter pages [in ] set layout = Module.Layout [map (…)] [where layout = Module.Old];` | The migration form. Marketplace pages are skipped and named. A `where layout` that names no real layout is an error, not a 0-page success | | Layout element | Syntax | Notes | diff --git a/mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl b/mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl index 64e34ad5cd..23b67e3d69 100644 --- a/mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl +++ b/mdl-examples/bug-tests/styling-515-alter-page-design-properties.mdl @@ -80,3 +80,50 @@ alter page MyFirstModule.ThingList { -- -- That message used to name ALTER STYLING as the route. It no longer does: -- naming a second statement for something `set` now writes is stale advice. + +-- --------------------------------------------------------------------------- +-- The bulk form +-- --------------------------------------------------------------------------- +-- +-- A house style is "every data grid is compact and striped", which should be one +-- statement rather than one per page. It mirrors the layout sweep MDL already +-- has (ALTER PAGES [IN mod] SET LAYOUT = … [WHERE LAYOUT = …]) and is told apart +-- from it at parse time by what follows SET: LAYOUT is a keyword, a design +-- property is a quoted string. +-- +-- alter pages in MyFirstModule +-- set 'Compact' = on, 'Striped' = on +-- where widgettype = datagrid +-- dry run; +-- +-- Measured on a blank 11.12.2 project with two Data grid 2 widgets: +-- +-- dry run -> Found 2 datagrid widget(s) in 2 container(s) +-- Would set 'Compact' on dgA in MyFirstModule.GridA (…and dgB) +-- [dry run] Would style 2 widget(s) +-- apply -> Styled 2 widget(s) +-- describe styling -> DesignProperties: ['Compact': on, 'Striped': on] +-- mx check -> The app contains: 0 errors. +-- +-- WHERE selects a TYPE, never a name. A widget name is unique only within its +-- page — measured, `actionButton1` appears in 30 units of a blank project — so a +-- name predicate would sweep unrelated widgets together. +-- +-- And the type is named by its MDL KEYWORD, which resolves to exactly one widget +-- id. The predicate a user reaches for instead, `WidgetType LIKE '%datagrid%'`, +-- matches 20 widgets in 6 containers on the same blank project, because it also +-- sweeps in DatagridTextFilter, DatagridDateFilter and DatagridDropdownFilter — +-- different widgets that do not carry the data grid's design properties. +-- +-- Reporting is ako/mxcli#520's three-way outcome, so a sweep that matches +-- widgets and writes none of them says so instead of claiming success: +-- +-- alter pages in MyFirstModule set 'Row size' = 'Small' where widgettype = datagrid; +-- -> Warning: Failed to set 'Row size' on dgA: not a design property this +-- widget's theme declares +-- Styled 0 widget(s) +-- 2 widget(s) matched but had no design property that could be set +-- exit 1 +-- +-- ('Row size' is the NATIVE data grid's property; Data grid 2 declares Borders, +-- Compact, Hover and Striped. `show design properties for datagrid` lists them.) diff --git a/mdl-examples/doctype-tests/12-styling.mdl b/mdl-examples/doctype-tests/12-styling.mdl new file mode 100644 index 0000000000..78b6a0a47e --- /dev/null +++ b/mdl-examples/doctype-tests/12-styling.mdl @@ -0,0 +1,25 @@ + +-- --------------------------------------------------------------------------- +-- Design properties through ALTER PAGE / ALTER PAGES (ako/mxcli#515) +-- --------------------------------------------------------------------------- +-- +-- `set` writes a design property of the stored widget's own type, so styling a +-- page no longer needs a second statement. The key is resolved against that +-- widget's theme group; anything the theme does not declare for it stays on the +-- pluggable path and keeps that path's error, so a typo is still a typo. +-- +-- `on`/`off` for a toggle, where OFF removes the entry (Mendix stores a toggle's +-- off state as the absence of the entry, not as a stored false). + +alter page Styling.StyledPage { + set 'Spacing top' = 'Large' on ctnCard; +}; + +-- The bulk form: one design property on every widget of a TYPE, across a module +-- or the whole project. `widgettype` takes the MDL keyword, which resolves to +-- exactly one widget id — a `like '%datagrid%'` predicate would also match the +-- data grid's filter widgets, which do not carry its design properties. +-- +-- Run it with `dry run` first: it applies the assignments to a discardable copy +-- and reports what would happen, on a statement that rewrites every page it +-- lands on. diff --git a/mdl/ast/ast_alter_page.go b/mdl/ast/ast_alter_page.go index 456895446c..cdbb190059 100644 --- a/mdl/ast/ast_alter_page.go +++ b/mdl/ast/ast_alter_page.go @@ -130,3 +130,20 @@ type AlterPagesLayoutStmt struct { } func (s *AlterPagesLayoutStmt) isStatement() {} + +// AlterPagesStylingStmt is the bulk form of ALTER PAGE's design-property SET: +// set a design property on every widget of one TYPE, across a module or the +// whole project. +// +// The predicate is a widget type and never a name, because a widget name is +// unique only within its page — measured across a blank 11.12.2 project, +// `actionButton1` appears in 30 units, so a name predicate would sweep +// unrelated widgets together (ako/mxcli#515). +type AlterPagesStylingStmt struct { + Module string // "" = every module the project owns + Assignments []StylingAssignment // reuses ALTER STYLING's assignment shape + WidgetType string // MDL keyword (`datagrid`) or a full widget id + DryRun bool // report the matches and write nothing +} + +func (s *AlterPagesStylingStmt) isStatement() {} diff --git a/mdl/executor/alter_pages_styling_test.go b/mdl/executor/alter_pages_styling_test.go new file mode 100644 index 0000000000..6ddd54dfdb --- /dev/null +++ b/mdl/executor/alter_pages_styling_test.go @@ -0,0 +1,127 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// ako/mxcli#515, the bulk form. A house style is "every data grid is compact and +// striped", which should be one statement and not one per page. +func parseAlterPagesStyling(t *testing.T, src string) *ast.AlterPagesStylingStmt { + t.Helper() + prog, errs := visitor.Build(src) + if len(errs) > 0 { + t.Fatalf("parse %q: %v", src, errs) + } + if len(prog.Statements) != 1 { + t.Fatalf("got %d statements, want 1", len(prog.Statements)) + } + s, ok := prog.Statements[0].(*ast.AlterPagesStylingStmt) + if !ok { + t.Fatalf("statement is %T, want *ast.AlterPagesStylingStmt", prog.Statements[0]) + } + return s +} + +func TestAlterPagesStyling_Parses(t *testing.T) { + s := parseAlterPagesStyling(t, + `alter pages in Sales set 'Compact' = on, 'Striped' = on where widgettype = datagrid dry run;`) + + if s.Module != "Sales" { + t.Errorf("Module = %q, want Sales", s.Module) + } + if s.WidgetType != "datagrid" { + t.Errorf("WidgetType = %q, want datagrid", s.WidgetType) + } + if !s.DryRun { + t.Error("DryRun = false, want true") + } + if len(s.Assignments) != 2 { + t.Fatalf("got %d assignments, want 2", len(s.Assignments)) + } + for _, a := range s.Assignments { + if !a.IsToggle || !a.ToggleOn { + t.Errorf("assignment %+v: want a toggle set ON", a) + } + } +} + +// Without IN, the module is empty and the widget type must still land in the +// right field — the rule has two identifierOrKeyword positions and ANTLR returns +// one list, so reading them positionally is where this goes wrong. +func TestAlterPagesStyling_WithoutModule(t *testing.T) { + s := parseAlterPagesStyling(t, + `alter pages set 'Striped' = off where widgettype = datagrid;`) + + if s.Module != "" { + t.Errorf("Module = %q, want empty — no IN clause was given", s.Module) + } + if s.WidgetType != "datagrid" { + t.Errorf("WidgetType = %q, want datagrid", s.WidgetType) + } + if s.DryRun { + t.Error("DryRun = true with no DRY RUN clause") + } + if len(s.Assignments) != 1 || !s.Assignments[0].IsToggle || s.Assignments[0].ToggleOn { + t.Errorf("assignments = %+v, want one toggle set OFF", s.Assignments) + } +} + +// An option value, and a full widget id in place of the keyword. +func TestAlterPagesStyling_OptionValueAndFullWidgetID(t *testing.T) { + s := parseAlterPagesStyling(t, + `alter pages set 'Row size' = 'Small' where widgettype = 'com.mendix.widget.web.datagrid.Datagrid';`) + + if s.WidgetType != "com.mendix.widget.web.datagrid.Datagrid" { + t.Errorf("WidgetType = %q, want the full id", s.WidgetType) + } + if len(s.Assignments) != 1 || s.Assignments[0].Value != "Small" || s.Assignments[0].IsToggle { + t.Errorf("assignments = %+v, want one option value Small", s.Assignments) + } +} + +// The sibling statement must keep parsing as itself. Both start `ALTER PAGES` +// and are told apart by what follows SET, so a grammar change here is exactly +// where the layout form would be swallowed. +func TestAlterPagesStyling_DoesNotShadowTheLayoutForm(t *testing.T) { + prog, errs := visitor.Build(`alter pages in Sales set layout = Sales.App_Default;`) + if len(errs) > 0 { + t.Fatalf("the layout form stopped parsing: %v", errs) + } + if _, ok := prog.Statements[0].(*ast.AlterPagesLayoutStmt); !ok { + t.Errorf("statement is %T, want *ast.AlterPagesLayoutStmt", prog.Statements[0]) + } +} + +// The selector is what makes `datagrid` mean Data grid 2 and nothing else. +// Measured: `WidgetType LIKE '%datagrid%'` matches 20 widgets in 6 containers on +// a blank project, because it also sweeps in the data grid's three FILTER +// widgets — different widgets that do not carry its design properties. +func TestResolveWidgetTypeSelector(t *testing.T) { + got := resolveWidgetTypeSelector("datagrid") + if !strings.Contains(got, ".") { + t.Fatalf("keyword did not resolve to a widget id: %q", got) + } + if !strings.Contains(strings.ToLower(got), "datagrid") { + t.Errorf("keyword resolved to %q, which does not look like the data grid", got) + } + // Specifically NOT a filter — the thing the LIKE predicate gets wrong. + if strings.Contains(strings.ToLower(got), "filter") { + t.Errorf("keyword resolved to a filter widget: %q", got) + } + // A full id passes through untouched. + const id = "com.acme.widget.Thing" + if resolveWidgetTypeSelector(id) != id { + t.Errorf("a full widget id was rewritten to %q", resolveWidgetTypeSelector(id)) + } + // An unknown keyword is left alone rather than guessed at, so the catalog + // query simply matches nothing and the statement says so. + if got := resolveWidgetTypeSelector("nosuchwidget"); got != "nosuchwidget" { + t.Errorf("an unknown keyword was rewritten to %q", got) + } +} diff --git a/mdl/executor/cmd_alter_pages_styling.go b/mdl/executor/cmd_alter_pages_styling.go new file mode 100644 index 0000000000..13eb6bc810 --- /dev/null +++ b/mdl/executor/cmd_alter_pages_styling.go @@ -0,0 +1,232 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "sort" + "strings" + + "github.com/mendixlabs/mxcli/mdl/ast" + "github.com/mendixlabs/mxcli/mdl/backend" + mdlerrors "github.com/mendixlabs/mxcli/mdl/errors" + "github.com/mendixlabs/mxcli/model" +) + +// ALTER PAGES [IN ] SET '' = , … WHERE WIDGETTYPE = [DRY RUN] +// +// The bulk form of ALTER PAGE's design-property SET (ako/mxcli#515). A house +// style is "every data grid is compact and striped", which should be one +// statement rather than one per page. +// +// It reuses three things deliberately, rather than growing a fourth of each: +// findMatchingWidgets (the catalog query UPDATE WIDGETS uses), the per-widget +// routing decision from ALTER PAGE SET, and updateOutcome's three-way reporting +// from ako/mxcli#520 — so a sweep that matches widgets and writes none of them +// says so instead of claiming success. + +// execAlterPagesStyling applies one design-property sweep. +func execAlterPagesStyling(ctx *ExecContext, s *ast.AlterPagesStylingStmt) error { + if !ctx.Connected() { + return mdlerrors.NewNotConnected() + } + if !s.DryRun && !ctx.ConnectedForWrite() { + return mdlerrors.NewNotConnectedWrite() + } + if len(s.Assignments) == 0 { + return mdlerrors.NewValidation("ALTER PAGES … SET needs at least one design property") + } + + // The MDL keyword is resolved to the widget id it writes, so `datagrid` + // selects Data grid 2 and not the data grid's FILTER widgets — which is what + // a LIKE over the stored id does instead (measured: 20 widgets in 6 + // containers on a blank project, across five different widget types). + // The catalog answers "every widget of this type, across every page", and + // this statement builds it rather than requiring the reader to have run + // `refresh catalog full` first — the same thing UPDATE WIDGETS does, and for + // the same reason: a sweep that silently matched nothing because the catalog + // was cold would read as "no such widgets". + if err := ensureCatalog(ctx, true); err != nil { + return mdlerrors.NewBackend("build catalog", err) + } + + widgetType := resolveWidgetTypeSelector(s.WidgetType) + widgets, err := findMatchingWidgets(ctx, []ast.WidgetFilter{ + {Field: "WidgetType", Operator: "=", Value: widgetType}, + }, s.Module) + if err != nil { + return mdlerrors.NewBackend("find widgets", err) + } + if len(widgets) == 0 { + fmt.Fprintf(ctx.Output, "No %s widgets found%s\n", s.WidgetType, inModuleSuffix(s.Module)) + return nil + } + + containers := groupWidgetsByContainer(widgets) + fmt.Fprintf(ctx.Output, "\nFound %d %s widget(s) in %d container(s)\n", + len(widgets), s.WidgetType, len(containers)) + if s.DryRun { + fmt.Fprintln(ctx.Output, "\n[dry run] The following changes would be made:") + } + + var total updateOutcome + for _, containerID := range sortedContainerIDs(containers) { + outcome, err := styleWidgetsInContainer(ctx, containerID, containers[containerID], s) + if err != nil { + fmt.Fprintf(ctx.Output, "Warning: Failed to style widgets in %s: %v\n", containerID, err) + continue + } + total.add(outcome) + } + + verb := "Styled" + if s.DryRun { + verb = "[dry run] Would style" + } + fmt.Fprintf(ctx.Output, "\n%s %d widget(s)\n", verb, total.WidgetsChanged) + if total.WidgetsUnchanged > 0 { + fmt.Fprintf(ctx.Output, "%d widget(s) matched but had no design property that could be set\n", + total.WidgetsUnchanged) + } + if total.WidgetsMissing > 0 { + fmt.Fprintf(ctx.Output, "%d widget(s) are in the catalog but not in the document — "+ + "run 'refresh catalog full force' and try again\n", total.WidgetsMissing) + } + if s.DryRun { + fmt.Fprintln(ctx.Output, "\nRun without dry run to apply changes.") + return nil + } + if total.WidgetsChanged > 0 { + fmt.Fprintln(ctx.Output, "\nNote: Run 'refresh catalog full force' to update the catalog with changes.") + } + if total.changedNothing() { + return mdlerrors.NewValidation(fmt.Sprintf( + "no widget was styled: %d assignment(s) could not be applied. "+ + "Run `mxcli show design properties for %s` to see what its theme declares", + len(total.Failures), s.WidgetType)) + } + return nil +} + +// styleWidgetsInContainer applies the sweep to one page or snippet. +func styleWidgetsInContainer(ctx *ExecContext, containerID string, refs []widgetRef, s *ast.AlterPagesStylingStmt) (updateOutcome, error) { + var out updateOutcome + if len(refs) == 0 { + return out, nil + } + containerName := refs[0].ContainerName + + mutator, err := ctx.Backend.OpenPageForMutation(model.ID(containerID)) + if err != nil { + return out, mdlerrors.NewBackend(fmt.Sprintf("open %s for mutation", containerName), err) + } + if mutator == nil { + return out, mdlerrors.NewBackend(fmt.Sprintf("open %s for mutation", containerName), + fmt.Errorf("backend returned nil mutator for %s", containerID)) + } + + // A dry run works against a discardable copy, so the preview reports what + // would actually happen — the same seam `mxcli check` uses for ALTER PAGE + // SET, and the fix ako/mxcli#520 applied to UPDATE WIDGETS' preview. + target := mutator + if s.DryRun { + if p, ok := mutator.(interface { + Probe() (backend.PageMutator, error) + }); ok { + if probe, perr := p.Probe(); perr == nil && probe != nil { + target = probe + } + } + } + + theme := ctx.GetThemeRegistry() + for _, ref := range refs { + landed := 0 + for _, a := range s.Assignments { + p := designPropertyForStoredWidget(theme, target, ref.Name, a.Property) + if p == nil { + out.Failures = append(out.Failures, fmt.Sprintf("'%s' on %s in %s", + a.Property, ref.Name, containerName)) + fmt.Fprintf(ctx.Output, " Warning: %s '%s' on %s: not a design property this "+ + "widget's theme declares\n", cannotVerb(s.DryRun), a.Property, ref.Name) + continue + } + if err := applyDesignPropertySet(target, ast.WidgetRef{Widget: ref.Name}, p, stylingAssignmentValue(a)); err != nil { + out.Failures = append(out.Failures, fmt.Sprintf("'%s' on %s in %s: %v", + a.Property, ref.Name, containerName, err)) + fmt.Fprintf(ctx.Output, " Warning: %s '%s' on %s: %v\n", + cannotVerb(s.DryRun), a.Property, ref.Name, err) + continue + } + landed++ + if s.DryRun { + fmt.Fprintf(ctx.Output, " Would set '%s' on %s in %s\n", + a.Property, ref.Name, containerName) + } + } + if landed > 0 { + out.WidgetsChanged++ + } else { + out.WidgetsUnchanged++ + } + } + + if !s.DryRun && out.WidgetsChanged > 0 { + if err := mutator.Save(); err != nil { + return out, mdlerrors.NewBackend(fmt.Sprintf("save %s", containerName), err) + } + } + return out, nil +} + +// stylingAssignmentValue converts one assignment back to the scalar the shared +// design-property conversion takes, so both the singular and bulk forms hand it +// the same shapes. +func stylingAssignmentValue(a ast.StylingAssignment) any { + if a.IsToggle { + return a.ToggleOn + } + return a.Value +} + +func cannotVerb(dryRun bool) string { + if dryRun { + return "Cannot set" + } + return "Failed to set" +} + +// resolveWidgetTypeSelector turns the MDL keyword in WHERE WIDGETTYPE into the +// widget id the catalog stores, leaving a full id untouched. +// +// This is what makes `datagrid` mean Data grid 2 and nothing else. The +// alternative a user reaches for — `WidgetType LIKE '%datagrid%'` — also matches +// DatagridTextFilter, DatagridDateFilter and DatagridDropdownFilter, which are +// different widgets that do not carry its design properties. +func resolveWidgetTypeSelector(selector string) string { + if strings.Contains(selector, ".") { + return selector // already a full widget id + } + if id, ok := pluggableKeywordIDs()[strings.ToLower(selector)]; ok { + return id + } + return selector +} + +func inModuleSuffix(module string) string { + if module == "" { + return "" + } + return " in " + module +} + +// sortedContainerIDs keeps the output order stable — a map walk would reorder +// the report between identical runs, which makes a diff of two runs unreadable. +func sortedContainerIDs(containers map[string][]widgetRef) []string { + ids := make([]string, 0, len(containers)) + for id := range containers { + ids = append(ids, id) + } + sort.Strings(ids) + return ids +} diff --git a/mdl/executor/register_stubs.go b/mdl/executor/register_stubs.go index 3764c315ea..130cefb935 100644 --- a/mdl/executor/register_stubs.go +++ b/mdl/executor/register_stubs.go @@ -522,6 +522,9 @@ func registerAlterPageHandlers(r *Registry) { r.Register(&ast.AlterPageStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execAlterPage(ctx, stmt.(*ast.AlterPageStmt)) }) + r.Register(&ast.AlterPagesStylingStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { + return execAlterPagesStyling(ctx, stmt.(*ast.AlterPagesStylingStmt)) + }) r.Register(&ast.AlterPagesLayoutStmt{}, func(ctx *ExecContext, stmt ast.Statement) error { return execAlterPagesLayout(ctx, stmt.(*ast.AlterPagesLayoutStmt)) }) diff --git a/mdl/executor/registry_test.go b/mdl/executor/registry_test.go index 2cea55c756..9340826a36 100644 --- a/mdl/executor/registry_test.go +++ b/mdl/executor/registry_test.go @@ -178,6 +178,7 @@ func allKnownStatements() []ast.Statement { &ast.AlterODataServiceStmt{}, &ast.AlterPageStmt{}, &ast.AlterPagesLayoutStmt{}, + &ast.AlterPagesStylingStmt{}, &ast.AlterProjectSecurityStmt{}, &ast.AlterPublishedRestServiceStmt{}, &ast.AlterSettingsStmt{}, diff --git a/mdl/grammar/MDLParser.g4 b/mdl/grammar/MDLParser.g4 index 86a0e26d8e..28c7e2ddde 100644 --- a/mdl/grammar/MDLParser.g4 +++ b/mdl/grammar/MDLParser.g4 @@ -148,6 +148,7 @@ alterStatement | ALTER SETTINGS alterSettingsClause | ALTER PAGE qualifiedName LBRACE alterPageOperation+ RBRACE | alterPagesLayoutStatement + | alterPagesStylingStatement // ALTER LAYOUT reuses alterPageOperation wholesale: a layout's widget tree is // a page's widget tree with four extra element types, so SET/INSERT/DROP/ // REPLACE mean exactly the same thing. A scroll-container region is addressed @@ -275,6 +276,39 @@ alterPagesLayoutStatement (WHERE LAYOUT EQUALS qualifiedName)? ; +// ALTER PAGES [IN ] SET '' = , ... WHERE WIDGETTYPE = [DRY RUN] +// +// The bulk form of ALTER PAGE's design-property SET, and the same argument: a +// house style is "every data grid is compact and striped", which is one +// statement rather than one per page. It mirrors the layout form above --- same +// verb, same optional IN, same WHERE --- and is told apart from it at parse time +// by what follows SET, since LAYOUT is a keyword and a design property is a +// quoted string. +// +// WHERE selects a widget TYPE, never a name: a widget name is unique only within +// its page (measured --- `actionButton1` exists in 30 units of a blank project), +// so a name predicate would sweep unrelated widgets together. The type is named +// by its MDL keyword, which resolves to exactly one widget id, rather than by a +// LIKE over the stored id, which also matches the data grid's FILTER widgets. +// +// DRY RUN is not optional politeness: this statement rewrites every page a match +// lands on, and the preview is the only way to see what a pattern selects before +// it selects it. +alterPagesStylingStatement + : ALTER PAGES (IN identifierOrKeyword)? SET alterPagesStylingAssignment + (COMMA alterPagesStylingAssignment)* + WHERE WIDGETTYPE EQUALS (STRING_LITERAL | identifierOrKeyword) + (DRY RUN)? + ; + +// The same three value shapes alterStylingAssignment takes, minus CLASS/STYLE: +// those are per-widget CSS, which a project-wide sweep has no business setting. +alterPagesStylingAssignment + : STRING_LITERAL EQUALS STRING_LITERAL // 'Row size' = 'Small' + | STRING_LITERAL EQUALS ON // 'Striped' = ON + | STRING_LITERAL EQUALS OFF // 'Striped' = OFF + ; + alterPageAssignment : DATASOURCE EQUALS dataSourceExprV3 // DataSource = SELECTION widgetName | ACTION EQUALS actionExprV3 // Action = MICROFLOW Module.MF | SHOW_PAGE Module.Page | SAVE_CHANGES CLOSE_PAGE diff --git a/mdl/visitor/visitor_alter.go b/mdl/visitor/visitor_alter.go index 5d60f4679f..9003b792cb 100644 --- a/mdl/visitor/visitor_alter.go +++ b/mdl/visitor/visitor_alter.go @@ -18,6 +18,10 @@ func (b *Builder) ExitAlterStatement(ctx *parser.AlterStatementContext) { } // Handle ALTER PAGES … SET LAYOUT (the bulk repoint) + if sub := ctx.AlterPagesStylingStatement(); sub != nil { + b.exitAlterPagesStylingStatement(sub.(*parser.AlterPagesStylingStatementContext)) + return + } if sub := ctx.AlterPagesLayoutStatement(); sub != nil { b.exitAlterPagesLayoutStatement(sub.(*parser.AlterPagesLayoutStatementContext)) return diff --git a/mdl/visitor/visitor_alter_page.go b/mdl/visitor/visitor_alter_page.go index e7dc24ebd6..604525da21 100644 --- a/mdl/visitor/visitor_alter_page.go +++ b/mdl/visitor/visitor_alter_page.go @@ -282,3 +282,45 @@ func (b *Builder) exitAlterPagesLayoutStatement(ctx *parser.AlterPagesLayoutStat b.statements = append(b.statements, stmt) } + +// exitAlterPagesStylingStatement builds the bulk design-property sweep: +// ALTER PAGES [IN ] SET 'key' = value, … WHERE WIDGETTYPE = [DRY RUN] +func (b *Builder) exitAlterPagesStylingStatement(ctx *parser.AlterPagesStylingStatementContext) { + stmt := &ast.AlterPagesStylingStmt{DryRun: ctx.DRY() != nil} + + // Two identifierOrKeyword positions in the rule — the optional module and + // the WIDGETTYPE value — and ANTLR returns one list, so which is which + // depends on whether IN was given. Getting it backwards would scope a + // project-wide sweep to a module named after a widget type, or vice versa. + ids := ctx.AllIdentifierOrKeyword() + if ctx.IN() != nil && len(ids) > 0 { + stmt.Module = identifierOrKeywordText(ids[0]) + ids = ids[1:] + } + if lit := ctx.STRING_LITERAL(); lit != nil { + // The WHERE value as a quoted string — a full widget id. + stmt.WidgetType = unquoteString(lit.GetText()) + } else if len(ids) > 0 { + stmt.WidgetType = identifierOrKeywordText(ids[0]) + } + + for _, a := range ctx.AllAlterPagesStylingAssignment() { + ac := a.(*parser.AlterPagesStylingAssignmentContext) + lits := ac.AllSTRING_LITERAL() + if len(lits) == 0 { + continue + } + assignment := ast.StylingAssignment{Property: unquoteString(lits[0].GetText())} + switch { + case ac.ON() != nil: + assignment.IsToggle, assignment.ToggleOn = true, true + case ac.OFF() != nil: + assignment.IsToggle, assignment.ToggleOn = true, false + case len(lits) > 1: + assignment.Value = unquoteString(lits[1].GetText()) + } + stmt.Assignments = append(stmt.Assignments, assignment) + } + + b.statements = append(b.statements, stmt) +} From d6d2d9e1b911537e73ccabaa5b885b163122eb60 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 12:29:18 +0000 Subject: [PATCH 14/38] fix(test): the Windows grandchild marker named a process that did not exist yet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit windows-process-regression failed intermittently on TestKillProcessGroup_ReapsGrandchildAndUnblocksWait — "cmd.Wait() did not return after killProcessGroup", ~20.5s at the select deadline, with the runner afterwards reporting `Terminate orphan process: pid (5264) (PING)`. killProcessGroup itself reported no error. The readiness marker named the wrong process. The `spawn` helper started `cmd /c ping -n 60 127.0.0.1` and wrote "grandchild-started" immediately after gc.Start() returned. Start() only guarantees cmd.exe was CREATED; ping.exe — the process that ends up holding the inherited stdout pipe — does not exist yet. The test raced ahead to killProcessGroup, `taskkill /F /T` enumerated a tree ping.exe had not joined, returned 0, and ping survived holding the write end, so cmd.Wait() never saw EOF. The failing job's timing is the evidence: the 15s marker wait did not fire. Of the 20.56s total, 0.56s elapsed before the kill — the marker had arrived promptly and the kill went ahead regardless. The grandchild now announces ITSELF, with its own pid, over the inherited pipe: a third helper mode re-execing the test binary instead of `cmd /c ping`. Reading the marker therefore proves the pipe holder is running, and the test asserts processAlive on that pid before killing — a control the test did not have, so a green run now means a tree kill was really exercised rather than a race that killed a one-process tree. This is the unix half's handshake, which the Windows half had silently diverged from: procgroup_unix_test.go runs `sh -c 'sleep 60 & echo $!; wait'` and checks the echoed pid with kill(gpid, 0) before the group kill. The fix is porting the sibling's, not inventing one. Two things deliberately NOT done. A deadline bump cannot fix this — the grandchild is never killed, so no amount of waiting helps — and it would have buried the race. And the tree is now two deep rather than three (test → helper → grandchild, not test → helper → cmd.exe → ping.exe): that is the field shape being regressed, where the mxbuild.exe wrapper spawns one Deno worker, and it still fails on a single-PID kill, which is the property the test exists for. The marker parser moved to an untagged file so its one invariant is covered on every platform and not only in the Windows CI job: reject a line not yet terminated by \n, because a truncated pid parses as a perfectly plausible different pid. Control run — with the completeness guard removed, TestGrandchildPID fails on exactly the two partial-line cases and nothing else. Not verifiable on this machine: the race is Windows-only, so the fix is argued from the job log and the code, and CI is the measurement. One green run does not settle a flake. Blame was settled by a same-commit control rather than by reading a diff. Commit 0710968d ran Build, Test & Lint twice, attempt 1 each time: push (35715042749) green, pull_request (35715076433) red. Identical code, opposite outcomes. TestGrandchildPID does not match the CI job's -run pattern, so its "at least 5 tests ran" guard is unaffected. Refs ako/mxcli#594, ako/mxcli#601 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_018hifgRSawfaRWXS44YKtSJ --- .../skills/fix-issue/findings/cmd-mxcli.jsonl | 1 + cmd/mxcli/docker/procgroup_marker_test.go | 77 +++++++++++++++++++ cmd/mxcli/docker/procgroup_windows_test.go | 52 +++++++++++-- 3 files changed, 124 insertions(+), 6 deletions(-) create mode 100644 cmd/mxcli/docker/procgroup_marker_test.go diff --git a/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl b/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl index 5859918bf2..7c70d58c60 100644 --- a/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl +++ b/.claude/skills/fix-issue/findings/cmd-mxcli.jsonl @@ -118,3 +118,4 @@ {"area": "cmd/mxcli", "date": "2026-09-18", "symptom": "mendixlabs/mxcli#1025: `mxcli syntax` advertises `mxcli syntax workflow user-task targeting` in its own help and answers `Unknown topic: workflow user-task targeting`. Same for `workflow user-task` and `workflow parallel-split`, all of which `mxcli syntax workflow` lists as sub-topics; `--json` was the only route that reached them.", "cause": "The CLI built its path with `strings.Join(args, \".\")` and never split an argument, so a topic handed over as ONE string — a quoted copy-paste, a tool wrapper, `sh -c` — became the path `workflow user-task targeting`, which matches nothing. The REPL's `help` had resolved multi-word topics since it was written (`resolveHelpPath`, greedy hyphen-joining): one question, two answers, and the CLI held the weaker copy. The #955 segment-match fallback could not save it either — it passed the DOTTED path to `BySegmentMatch`, and no segment contains a '.', so that fallback was silently dead for every multi-word query.", "file": "cmd/mxcli/syntax/topic.go (new: Lookup, topicWords, resolvePath), cmd/mxcli/help.go, mdl/executor/cmd_misc.go (resolveHelpPath deleted), mdl/grammar/domains/MDLSettings.g4 (helpStatement, helpTopicWord), mdl/visitor/visitor_query.go (ExitHelpStatement); tests cmd/mxcli/cmd_syntax_test.go, cmd/mxcli/syntax/topic_test.go, mdl/executor/cmd_misc_test.go, mdl/visitor/visitor_help_topic_test.go; example mdl-examples/bug-tests/syntax-1025-topic-drilldown.mdl", "insight": "**The spaces in the reported error message were the whole diagnosis, and reading them as a paraphrase cost an hour.** The command prints the path it built, and the CLI joins on '.', so `Unknown topic: workflow user-task targeting` cannot come from the command as documented — it can only come from the topic arriving as a single argument. Every line of the report follows from that and nothing else does: `syntax workflow` works (one word), `--json` works (the flag is not part of the topic), the three multi-word forms fail. Take a quoted error message literally, character for character, before assuming the reporter retyped it. **The reported version is downloadable and settles it in one run**: `mxcli setup mxcli`'s own URL shape (`releases/download//mxcli-linux-amd64`, NOT the goreleaser `_Linux_x86_64.tar.gz` that 404s) fetched v0.20.0, where the unquoted command works and the quoted one reproduces the message verbatim — so 'fixed since' and 'never broken' were both wrong. **The guard that matters is not the three cases from the report** but `TestEveryRegisteredPathIsReachableBySpelling`: every registered path, tried dotted, as separate arguments, and as one string. The registry prints dotted paths and then tells the reader to drill down with words, so a spelling that does not resolve is the command contradicting its own output; a per-case test would have passed the day someone added a topic with a new shape. Control: stub the whitespace split in `topicWords` and it fails with the reported path, spaces and all. **The grammar half has a trap the CLI half does not, and only the EXISTING suite caught it.** `helpStatement: IDENTIFIER (identifierOrKeyword)*` is the grammar's catch-all — a statement that is just an identifier and some words — so whatever it can swallow, it swallows from the statement that should have had it. Widening it to `(DOT? helpTopicWord)*` to take `help workflow.user-task` made `Sec.ApiUser` a complete statement of its own, and `create module role Sec.ApiUser` then parsed, WITH NO PARSE ERROR, as CREATE MODULE (named \"role\") followed by a help topic — two statements, wrong types, six unrelated security tests red. `(helpTopicWord (DOT? helpTopicWord)*)?` — a topic word before any dot — leaves `.ApiUser` unconsumable and restores the old disambiguation. Bisect a grammar regression by SHAPE, not by reading the ATN: adding the unused rule alone was clean, the hyphen alone was clean, the leading optional DOT was the whole of it, and three regenerations said so in about a minute. **When widening a permissive rule, the test to add is not for the new spelling but for what the rule must still NOT swallow** (TestHelpRuleDoesNotSwallowATrailingQualifiedName).", "refs": ["mendixlabs/mxcli#1025", "#955"]} {"area": "cmd/mxcli", "date": "2026-09-18", "symptom": "`mxcli report` scores a project against rules the team disabled in `lint-config.yaml`. `mxcli lint` honours the config, the report's SCORE does not move, so the score cannot be calibrated at all. Reported at 66/100 against a 99/100 blank-app baseline, where 61 of 86 findings were two deliberately-accepted rules", "cause": "`cmd_report.go` never called `linter.FindConfigFile`/`LoadConfig` — it went straight from `linter.New` to `BuildReport`. Separately it carried its own INLINE copy of the built-in rule list, one rule behind `builtinLintRules()` (missing MDL-FLOW01), so the two commands scored one project against two rule sets. One root cause: report re-implemented lint's setup instead of sharing it", "file": "`cmd/mxcli/cmd_report.go`, `cmd/mxcli/cmd_lint.go`, new `cmd/mxcli/lint_setup.go` (`projectLintRules`, `applyLintConfig`), `mdl/linter/linter.go` (`RuleEnabled`)", "insight": "Same class as #904 in the opposite direction: there a silently reduced rule set made the score falsely HIGH, here an unread config makes it falsely LOW — and both are invisible because a score carries no provenance. **A value test cannot guard the inline copy**: both commands build rules inside a cobra RunE, so nothing a unit test can call notices a second list being re-added. The guard is therefore structural — grep `cmd_report.go` for `lint.AddRule(rules.New` — with a POSITIVE CONTROL first (assert `builtinLintRules` still constructs rules) so it cannot pass vacuously, the same shape as `scripts/check-tunnel-deps.sh`. Take the LintContext out of `applyLintConfig`'s signature: `NewLintContext(nil, nil)` panics, and a nil-guard added only to make a test compile is how a helper acquires behaviour nothing needs", "refs": ["#525", "#904"]} {"area": "cmd/mxcli/marketplace", "date": "2026-09-20", "symptom": "`mxcli marketplace install ... -p app.mpr` (project named by a RELATIVE path) fails with `install the package's bundled files: package entry \"manifest.json\" would write outside the project` \u2014 after the module has already been transplanted into the model. `SHOW MODULES` lists the module, `mx check` is clean, but no bundled file (themesource/, widgets/) landed and the command exited 1. Absolute `-p` paths work.", "cause": "`InstallPackageFiles` builds `dst := filepath.Join(projectDir, clean)` and then checks `strings.HasPrefix(filepath.Clean(dst), filepath.Clean(projectDir)+os.PathSeparator)`. With projectDir == \".\" (from `filepath.Dir(\"app.mpr\")`), Join drops the dot, so dst is `manifest.json` and the prefix is `./` \u2014 every legitimate entry fails the zip-slip guard. The guard was checking the joined path (the shape CodeQL recognises) but never anchored the project directory first.", "file": "`cmd/mxcli/marketplace/update.go` (`InstallPackageFiles`: `filepath.Abs(projectDir)` before the loop), test `cmd/mxcli/marketplace/install_relative_dir_test.go`", "insight": "A containment guard has two inputs and both must be canonical \u2014 the entry AND the root. The traversal tests only ever passed an absolute t.TempDir(), so the root was canonical by accident and the relative case had no coverage; the first real CLI invocation with `-p app.mpr` hit it. Worse, the transplant runs BEFORE the file step, so the failure lands on a half-installed module: the model has it, the disk does not, and the exit code says failure. Order the steps so the cheap, reversible file copy can be validated before the model write, or at least say in the error that the model was already changed. Prove-by-revert done: the new test fails on the unpatched function with the exact reported message.", "refs": []} +{"area": "cmd/mxcli/docker", "date": "2026-09-22", "symptom": "`windows-process-regression` fails intermittently on TestKillProcessGroup_ReapsGrandchildAndUnblocksWait: `cmd.Wait() did not return after killProcessGroup`, ~20.5s (the select deadline), and the GitHub runner then logs `Terminate orphan process: pid (NNNN) (PING)`. killProcessGroup reports no error. Reruns pass, so it reads as 'Windows CI is flaky' and gets attributed to whatever PR happened to be red.", "cause": "The test's readiness marker named the wrong process. The `spawn` helper mode started `cmd /c ping -n 60 127.0.0.1` and wrote `grandchild-started` immediately after `gc.Start()` returned — but Start() only guarantees `cmd.exe` was CREATED; `ping.exe`, which is what ends up holding the inherited stdout pipe, does not exist yet. The test raced ahead to killProcessGroup, `taskkill /F /T` enumerated a tree `ping.exe` had not joined, returned 0, and ping survived holding the write end, so cmd.Wait() never saw EOF.", "file": "`cmd/mxcli/docker/procgroup_windows_test.go` (helper gains a `grandchild` mode that announces ITSELF, with its pid, over the inherited pipe; the test then asserts processAlive on that pid before killing), parser split to `cmd/mxcli/docker/procgroup_marker_test.go` + TestGrandchildPID", "insight": "A readiness marker is only worth what it proves about the process the test is ABOUT. Emitting it from the parent after Start() proves the parent reached a line of code, which is the one thing never in doubt. Emit it from the process under test, over the channel under test — the unix half already did exactly this (`sh -c 'sleep 60 & echo $!; wait'` plus kill(gpid,0)) and the Windows half had silently diverged, so the fix was porting the sibling's handshake rather than inventing one. Two second-order traps: a deadline bump cannot fix this (the grandchild is never killed, so no amount of waiting helps) and would have buried it; and the marker parser must reject a line not yet terminated by \\n, since a truncated pid parses as a plausible different pid. The same-commit control that settled blame: sha 0710968d ran the identical workflow twice, `push` (35715042749) green and `pull_request` (35715076433) red — when a job is suspected flaky, look for two runs of one commit before reading the diff.", "refs": ["ako/mxcli#594", "ako/mxcli#597", "ako/mxcli#601"]} diff --git a/cmd/mxcli/docker/procgroup_marker_test.go b/cmd/mxcli/docker/procgroup_marker_test.go new file mode 100644 index 0000000000..0f7d602907 --- /dev/null +++ b/cmd/mxcli/docker/procgroup_marker_test.go @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "strconv" + "strings" + "testing" +) + +// grandchildPID extracts the pid the "grandchild" helper mode announces (see +// procgroup_windows_test.go). It reports false until a COMPLETE marker line has +// arrived: the marker's only job is to prove the process holding the inherited +// pipe is running, so accepting a half-written line would give back exactly the +// false "it is up" the marker exists to rule out. +// +// It lives in an untagged file, away from its windows-only caller, so this one +// piece of parsing is covered on every platform rather than only in the Windows +// CI job. +func grandchildPID(out string) (int, bool) { + // Whatever follows the final newline is a partial write, and a truncated pid + // parses as a perfectly plausible one. + end := strings.LastIndex(out, "\n") + if end < 0 { + return 0, false + } + for _, line := range strings.Split(out[:end], "\n") { + rest, ok := strings.CutPrefix(strings.TrimSpace(line), "grandchild-started ") + if !ok { + continue + } + pid, err := strconv.Atoi(strings.TrimSpace(rest)) + if err != nil || pid <= 0 { + continue + } + return pid, true + } + return 0, false +} + +func TestGrandchildPID(t *testing.T) { + tests := []struct { + name string + out string + wantPID int + wantOK bool + }{ + {"empty", "", 0, false}, + {"complete line", "grandchild-started 4242\n", 4242, true}, + {"crlf", "grandchild-started 4242\r\n", 4242, true}, + {"after other output", "noise\ngrandchild-started 7\nmore\n", 7, true}, + + // The reason this function is not strings.Contains. A pipe read can stop + // mid-line, and "grandchild-started 42" is a prefix of "...4242": taking + // it would name a different process, and the liveness check that follows + // would then be asserting something about a stranger. + {"partial line is not a pid", "grandchild-started 42", 0, false}, + {"partial after complete noise", "noise\ngrandchild-started 42", 0, false}, + + {"marker with no pid", "grandchild-started\n", 0, false}, + {"marker with empty pid", "grandchild-started \n", 0, false}, + {"non-numeric pid", "grandchild-started abc\n", 0, false}, + {"zero pid", "grandchild-started 0\n", 0, false}, + {"negative pid", "grandchild-started -1\n", 0, false}, + {"different marker", "grandchild-exited 4242\n", 0, false}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + pid, ok := grandchildPID(tt.out) + if ok != tt.wantOK || pid != tt.wantPID { + t.Fatalf("grandchildPID(%q) = %d, %v; want %d, %v", + tt.out, pid, ok, tt.wantPID, tt.wantOK) + } + }) + } +} diff --git a/cmd/mxcli/docker/procgroup_windows_test.go b/cmd/mxcli/docker/procgroup_windows_test.go index 9e58c5a664..e518315964 100644 --- a/cmd/mxcli/docker/procgroup_windows_test.go +++ b/cmd/mxcli/docker/procgroup_windows_test.go @@ -5,9 +5,9 @@ package docker import ( + "fmt" "os" "os/exec" - "strings" "syscall" "testing" "time" @@ -44,13 +44,27 @@ func TestWindowsProcessHelper(t *testing.T) { case "spawn": // A grandchild that inherits our stdout/stderr — the inherited pipe is // exactly what kept cmd.Wait() blocked in the field. - gc := exec.Command("cmd", "/c", "ping", "-n", "60", "127.0.0.1") + // + // Note what this mode does NOT do: announce the grandchild. Start() + // returns as soon as the grandchild has been created, which is before it + // is running and holding the pipe, so a marker written here says nothing + // about the process the test is about. The grandchild announces itself + // below instead. + gc := exec.Command(os.Args[0], "-test.run=TestWindowsProcessHelper") + gc.Env = append(os.Environ(), "MXCLI_PROC_HELPER=grandchild") gc.Stdout = os.Stdout gc.Stderr = os.Stderr if err := gc.Start(); err != nil { os.Exit(3) } - os.Stdout.WriteString("grandchild-started\n") + time.Sleep(60 * time.Second) + os.Exit(0) + case "grandchild": + // Announce over the INHERITED pipe, and with our own pid, so the test can + // establish that this process — the one holding the write end — is really + // running before it kills the tree. This mirrors the unix half, where the + // wrapper echoes `$!` and the test checks it with kill(pid, 0). + fmt.Fprintf(os.Stdout, "grandchild-started %d\n", os.Getpid()) time.Sleep(60 * time.Second) os.Exit(0) } @@ -155,6 +169,11 @@ func TestLocalRuntime_AliveTracksProcess(t *testing.T) { // // It would have failed on the pre-fix code: p.Kill() terminates only the helper, // the grandchild keeps the write end open, and Wait() never returns. +// +// The grandchild is a re-exec of this test binary rather than `cmd /c ping`, so +// that it can announce itself over the inherited pipe. That handshake is what +// makes the test deterministic: see the "spawn" mode's comment for the race the +// old marker left open, which failed this job intermittently (ako/mxcli#594). func TestKillProcessGroup_ReapsGrandchildAndUnblocksWait(t *testing.T) { var log syncBuffer cmd := helperCmd(t, "spawn") @@ -166,15 +185,36 @@ func TestKillProcessGroup_ReapsGrandchildAndUnblocksWait(t *testing.T) { } t.Cleanup(func() { _ = killProcessGroup(cmd.Process) }) - // Wait for the grandchild to be running and holding the inherited pipe. + // Wait for the grandchild to be running and holding the inherited pipe. The + // marker arrives over that pipe from the grandchild itself, so reading it + // proves the pipe holder exists; waiting on anything the helper printed + // would not (see the "spawn" mode's comment). deadline := time.Now().Add(15 * time.Second) - for !strings.Contains(log.String(), "grandchild-started") { + var gpid int + for { + if p, ok := grandchildPID(log.String()); ok { + gpid = p + break + } if time.Now().After(deadline) { - t.Fatalf("helper never reported its grandchild; output so far: %q", log.String()) + t.Fatalf("grandchild never announced itself; output so far: %q", log.String()) } time.Sleep(50 * time.Millisecond) } + // The CONTROL for the marker: assert the announced process really is alive + // before the kill, so a test that goes green has actually exercised a tree + // kill and not a race that killed a one-process tree. The unix half does the + // same with kill(gpid, 0). + gproc, err := os.FindProcess(gpid) + if err != nil { + t.Fatalf("grandchild %d should be alive before the kill: %v", gpid, err) + } + defer func() { _ = gproc.Release() }() + if !processAlive(gproc) { + t.Fatalf("grandchild %d should be alive before the kill", gpid) + } + done := make(chan error, 1) go func() { done <- cmd.Wait() }() From c24731a53822461dad757eaca064bdd570523338 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 15:08:40 +0000 Subject: [PATCH 15/38] test(doctype): fold the #515 styling examples into the existing styling script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doctype gate (`make test-integration`) runs every script in doctype-tests/ through exec + mx check against a blank project, and the 12-styling.mdl added in 78612b5e failed it two ways. `make test`, `make lint` and `make check-mdl` all pass without touching that gate, which is how it reached CI. 1. It was not self-contained. It styled `Styling.StyledPage`, a page nothing in the script creates: `Execution error: page not found`. Every other doctype script creates the module and pages it uses. 2. The design properties were native-only. `Spacing bottom` and `Full width` are declared for native widgets; a WEB DivContainer declares `Item gap`, `Card style`, `Disable row wrap`, `Background color` and friends (themesource/atlas_core/web/design-properties.json). The routing added in 992dc05b refused them correctly — the code was right and the example was wrong, which is the outcome to prefer but only if the example is fixed rather than the guard loosened. The file is removed rather than repaired: 12-styling-examples.mdl already exists, already builds a StyleTest module with pages, and already owns the ALTER STYLING sections these two levels belong beside. A second styling doctype script was duplication the PR checklist asks to look for first. The new sections use properties the web theme actually declares, and add two data-grid pages so the bulk form has something to select. `Compact` and `Striped` come from datawidgets' own design-properties.json, so the plural example exercises a pluggable widget id rather than a native $Type. Gate passes: TestMxCheck_DoctypeScripts/12-styling-examples.mdl/modelsdk, and mx check reports 0 errors on the resulting project. Refs ako/mxcli#515 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01LYaTtzjmMCUjo8XAVN1sQx --- .../doctype-tests/12-styling-examples.mdl | 76 +++++++++++++++++++ mdl-examples/doctype-tests/12-styling.mdl | 25 ------ 2 files changed, 76 insertions(+), 25 deletions(-) delete mode 100644 mdl-examples/doctype-tests/12-styling.mdl diff --git a/mdl-examples/doctype-tests/12-styling-examples.mdl b/mdl-examples/doctype-tests/12-styling-examples.mdl index af5abd2ed5..1938bc80b0 100644 --- a/mdl-examples/doctype-tests/12-styling-examples.mdl +++ b/mdl-examples/doctype-tests/12-styling-examples.mdl @@ -697,3 +697,79 @@ create or modify page StyleTest."P007_Compound" ( } DESCRIBE STYLING ON PAGE StyleTest.P007_Compound WIDGET lgCompound; + +-- MARK: Alter Page Design Properties + +-- ============================================================================ +-- LEVEL 12: ALTER PAGE SET writes design properties too (#515) +-- ============================================================================ +-- `set` reaches a design property of the stored widget's own type, so styling +-- one widget on one page no longer needs ALTER STYLING as a second spelling. +-- The key is resolved against that widget's theme group; anything the theme +-- does not declare for it stays on the pluggable path and keeps that path's +-- error, so a typo is still reported as a typo. `show design properties for +-- container` lists what is available here. + +alter page StyleTest.P006_Roundtrip { + set 'Item gap' = 'Large' on ctnHeader; + set 'Card style' = on on ctnHeader; +}; + +-- OFF removes the entry. Mendix stores a toggle's off state as the ABSENCE of +-- the entry, not as a stored false, so this is a removal rather than a write +-- of a second value. +alter page StyleTest.P006_Roundtrip { + set 'Card style' = off on ctnHeader; +}; + +describe styling on page StyleTest.P006_Roundtrip widget ctnHeader; + +-- ============================================================================ +-- LEVEL 13: ALTER PAGES applies one styling decision across many pages (#515) +-- ============================================================================ +-- The singular form styles one widget on one page. Rolling a decision out +-- ("every data grid is compact and striped") one statement per widget per page +-- is how a design system drifts, so the plural form selects by widget type. + +create page StyleTest.P008_Grid_A +( + title: 'Grid A', + layout: Atlas_Core.Atlas_Default +) +{ + datagrid dgEmployeesA (datasource: database StyleTest.Employee) { + column colNameA (attribute: Name, caption: 'Name') + column colEmailA (attribute: Email, caption: 'Email') + } +} + +create page StyleTest.P009_Grid_B +( + title: 'Grid B', + layout: Atlas_Core.Atlas_Default +) +{ + datagrid dgTasksB (datasource: database StyleTest.task) { + column colTitleB (attribute: "title", caption: 'Title') + column colDueB (attribute: DueDate, caption: 'Due') + } +} + +-- `dry run` first: it applies the assignments to a discardable copy of each +-- document and reports what would happen, which matters on a statement that +-- rewrites every page it lands on. +alter pages in StyleTest + set 'Compact' = on, 'Striped' = on + where widgettype = datagrid + dry run; + +alter pages in StyleTest + set 'Compact' = on, 'Striped' = on + where widgettype = datagrid; + +-- `widgettype` takes the MDL keyword, which resolves to exactly one widget id. +-- A `like '%datagrid%'` predicate would also match the data grid's own filter +-- widgets, which are separate widgets and do not carry its design properties. + +describe styling on page StyleTest.P008_Grid_A widget dgEmployeesA; +describe styling on page StyleTest.P009_Grid_B widget dgTasksB; diff --git a/mdl-examples/doctype-tests/12-styling.mdl b/mdl-examples/doctype-tests/12-styling.mdl deleted file mode 100644 index 78b6a0a47e..0000000000 --- a/mdl-examples/doctype-tests/12-styling.mdl +++ /dev/null @@ -1,25 +0,0 @@ - --- --------------------------------------------------------------------------- --- Design properties through ALTER PAGE / ALTER PAGES (ako/mxcli#515) --- --------------------------------------------------------------------------- --- --- `set` writes a design property of the stored widget's own type, so styling a --- page no longer needs a second statement. The key is resolved against that --- widget's theme group; anything the theme does not declare for it stays on the --- pluggable path and keeps that path's error, so a typo is still a typo. --- --- `on`/`off` for a toggle, where OFF removes the entry (Mendix stores a toggle's --- off state as the absence of the entry, not as a stored false). - -alter page Styling.StyledPage { - set 'Spacing top' = 'Large' on ctnCard; -}; - --- The bulk form: one design property on every widget of a TYPE, across a module --- or the whole project. `widgettype` takes the MDL keyword, which resolves to --- exactly one widget id — a `like '%datagrid%'` predicate would also match the --- data grid's filter widgets, which do not carry its design properties. --- --- Run it with `dry run` first: it applies the assignments to a discardable copy --- and reports what would happen, on a statement that rewrites every page it --- lands on. From 8f41058c1726c6519df481f7df7d59ee23f97b95 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:43:31 +0000 Subject: [PATCH 16/38] docs: propose agent loop efficiency work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A side-by-side test (Opus building an app on Vercel vs with mxcli) showed 4.25x the model calls and 5.4x the conversation re-read. This proposal works out why and what to do about it. The spine is the arithmetic: every call re-reads the transcript, so total cost is calls x conversation size, and when a removed call takes its tool results with it the total falls with the square of the call count. That sets the priority order — fewer calls first, less text per call second, output tokens (500k of 228M) not at all. Six levers, ordered by that arithmetic. Two findings are already actionable against today's binary: exec runs the semantic check itself and refuses on error, yet projectGates teaches check-then-exec as two gates in every generated CLAUDE.md; and the 35s restart the session paid per change is opt-in slowness next to run --local --watch and test --attach. Sequenced with diag loop-report and a benchmark first, so the remaining levers are falsifiable rather than plausible. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .../PROPOSAL_agent_loop_efficiency.md | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 docs/11-proposals/PROPOSAL_agent_loop_efficiency.md diff --git a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md new file mode 100644 index 0000000000..3c5f3fcf8f --- /dev/null +++ b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md @@ -0,0 +1,232 @@ +--- +title: Agent loop efficiency — making an mxcli session cost what the work costs +status: draft +date: 2026-09-22 +related: + - PROPOSAL_mxcli_dev_warm_loop.md + - PROPOSAL_playwright_session_reuse.md + - PROPOSAL_llm_mdl_assistance.md + - PROPOSAL_session_logging.md + - PROPOSAL_check_diagnostics_catalog.md + - cmd/mxcli/init_claudemd_budget_test.go +--- + +# Agent loop efficiency — making an mxcli session cost what the work costs + +## The report + +A side-by-side test had Opus build an app twice: once on Vercel (TypeScript files +straight to disk), once with mxcli against Mendix. + +| | mxcli / Mendix | Vercel | +|---|---|---| +| Duration | 2 h 33 | 1 h 07 | +| Model calls | **523** | **123** | +| Conversation re-read from cache | **228 M** | **42 M** | +| Cache writes | 1.27 M | 0.48 M | +| Output tokens | 500 k | 205 k | +| Avg conversation per call | 435 k | 345 k | +| Bash commands | 425 | 58 | + +The bill is dominated by the 228 M re-read, and that figure is not a mystery: +523 × 435 k ≈ 228 M. It is the product of two numbers and nothing else. + +## Why this is a square, not a line + +Every model call re-reads the whole conversation. Conversation size grows with +the content the calls put into it. So for a session of **N** calls whose +transcript grows roughly linearly to size **S**: + +``` +total re-read ≈ N × S/2 +``` + +and when the removed calls take their own tool results out with them, **S falls +with N** — so the total falls with **N²**. Halving the call count on the same +work is a ~4× cut, not a 2× cut. The measured pair is consistent with this: +4.25× the calls and 1.26× the conversation gives 5.4×, which is what the logs +show. + +That arithmetic sets the priority order, and it is not the intuitive one: + +1. **Fewer calls** — enters quadratically. Everything else is second. +2. **Less text per call** — enters linearly, but it is also the multiplier on + lever 1, so the two compound. +3. Output tokens (500 k of 228 M) are a rounding error. Do not optimise here. + +## What is irreducible, and what is not + +Part of the 4× is real and will not go away. Vercel's agent writes a `.tsx` file +and the work is done. A Mendix change is a model mutation that must be +**validated, applied, built, and rendered** before anyone knows it worked. The +feedback also notes the two sessions were not scope-equivalent — the mxcli one +additionally covered login styling, documentation, Docker and password lockout. + +So parity is the wrong target. The target is the **gap between the loop we ship +and the loop mxcli is already capable of**, which is large, because most of the +fast paths below already exist and the session did not take them. + +The honest claim: of the reported ~400 extra calls, roughly 250–300 look +addressable. The rest is Mendix being a compiled platform. + +--- + +## Lever 1 — collapse the per-change round trip (attacks N) + +The reported loop is **5–8 calls per change**: write script → `mxcli check` → +`mxcli exec` → `mx check` → restart (~35 s) → log in → screenshot. + +Four of those steps are already avoidable with today's binary: + +- **`check` before `exec` is redundant.** `exec` already runs the full semantic + check before it writes anything and refuses the script on an error + (`cmd/mxcli/cmd_exec.go`). Yet `projectGates` in `cmd/mxcli/init_claudemd.go` + lists `check` and `exec` as two consecutive gates, so the generated CLAUDE.md + in **every** mxcli project teaches the two-call form. One wasted call per + change, in every session, by construction. +- **The 35 s restart is opt-in slowness.** `run --local --watch` hot-reloads a + behavioural change in ~2 s, and `mxcli test --attach` skips the boot entirely + by driving an app already up. +- **Logging in by hand is solved.** Playwright storage state is captured and + reused (`screenshot --load-storage`); see `PROPOSAL_playwright_session_reuse.md`. +- **The screenshot is usually the wrong instrument.** See lever 3. + +### Proposal: one gate command + +```bash +mxcli apply changes.mdl -p app.mpr [--verify tests/admin.spec.ts] [--no-build] +``` + +One call that runs: semantic check → exec → mxbuild verify → reload the running +app (or restart if the serve build says `restartRequired`) → run the named +Playwright verifications → print **one compact verdict**. + +The verdict is the whole point. On success it is two lines. On failure it is the +first failing stage and only that stage's diagnostics — not the output of all +five. This is what turns 5–8 calls into 1, and on the failure path into 2. + +Nothing here is new capability. Every stage exists; `apply` is the composition, +and the composition is what the agent is currently doing by hand, one tool call +at a time, paying the full conversation for each. + +**Also: fix the gate list.** `projectGates` should teach `exec` (check folded in) +rather than `check` then `exec`, and should name `apply` once it exists. The +gates tests (`init_claudemd_gates_test.go`) already hold three copies of that +list to one definition, so this is a one-line change that propagates. + +## Lever 2 — shrink what each call adds (attacks S) + +A tool result is written into the conversation once and **re-read by every +subsequent call**. A 200-line `exec` transcript early in a 500-call session is +not 200 lines of cost; it is 200 lines × ~400 remaining calls. + +- **Report the delta, not the transcript.** `exec` prints a line per statement. + The idempotence work already distinguishes `Created` / `Replaced` / + `Unchanged` per unit (`ExecContext.ReportMutation`), so the information to + collapse this is in hand: `Created 4, replaced 2, unchanged 196` plus the six + names that changed. A re-run of a settled script should be **one line**. +- **Default to terse; make verbose opt-in.** `MXCLI_QUIET` exists and the skills + set it in places. It should be the default posture for non-interactive runs. +- **Prefer a terse agent format over `--json`.** `--json` exists globally, but + JSON is frequently *more* tokens than good prose for the same facts. The + target is fewest tokens that stay unambiguous, not machine-readability for its + own sake. +- **Cap the long tails.** `docker check` error dumps, `show microflows` on a big + module, catalog listings: add `--top N` and severity filters so a 400-line + result becomes the 10 lines the agent will act on. + +This lever is worth doing carefully rather than aggressively: an output trimmed +past the point of usefulness buys back its savings immediately in re-runs. + +## Lever 3 — stop paying image input on a loop + +Screenshots are expensive input, they recur, and they persist for the rest of +the session. The feedback names them explicitly. + +`mxcli playwright verify ` already exists and returns **text** pass/fail +against a running app. That is the correct default instrument for "does this +flow work". A screenshot answers a different and much rarer question — "does +this *look* right" — and should be taken deliberately, once, when appearance is +genuinely the subject. + +The rule for the skills: **assert in text; screenshot when the question is +visual, and then once.** `.claude/skills/verify-in-runtime.md` already has the +right shape for this (a table routing a symptom to the cheapest sufficient +proof); it needs the text-vs-pixel row added and `run-local` / `test-app` +pointed at it. + +## Lever 4 — keep long investigations out of the main conversation + +One self-inflicted bug (a stub script that wiped real microflow bodies) took +**~40 calls** to trace. Those 40 results then sat in context for the rest of the +session, taxing every later call. + +A subagent pays for its own 40 round trips once and returns a paragraph. The +skills should say so with a trigger rather than a preference: **a diagnosis +expected to take more than ~5 probes is delegated, not run inline.** The same +applies to log spelunking and "which of these 30 files mentions X". + +## Lever 5 — turn each discovered workaround into tool knowledge + +Five Mendix limitations each cost an investigation and a rework in that session: + +| Discovered the hard way | Where it should live instead | +|---|---| +| inputs inside lists are read-only → admin editing must be pop-ups | `mxcli check` diagnostic on an input widget in a list/gallery context | +| the sidebar went stale after actions | `create-page` / `patterns-crud` skill, refresh guidance | +| pop-up styling breaks (pop-ups sit outside the styled area) | `theme-styling` skill | +| login fields did not register scripted input | a `mxcli playwright login` helper that does it correctly | +| the Docker image had the wrong Java version | `mxcli docker check` preflight | + +This is the highest-leverage lever on any horizon longer than one session, +because it converts a cost paid **once per session per user** into one paid +**once, by us**. It is also exactly the repo's existing instinct — findings +files, lint rules, check diagnostics — applied to a class of knowledge that has +so far only been rediscovered. + +## Lever 6 — measure it, then claim it + +Everything above is a hypothesis until it moves a number, and this proposal +should not be merged on plausibility. + +mxcli already logs every invocation as JSON Lines under `~/.mxcli/logs/` +(`diaglog`, wired at `newLoggedExecutor` in `cmd/mxcli/main.go` — so it covers +all commands, not a curated subset). That is the instrument. + +**`mxcli diag loop-report`** reads a session's log and prints: invocations by +verb, wall time by verb, `check`-immediately-before-`exec` pairs (pure waste), +restarts vs. hot reloads, and output bytes per command. That converts "the loop +feels expensive" into a ranked list of where the calls actually went — and, run +before and after each lever, into evidence that a lever worked. + +**Then a benchmark.** One fixed app-brief, run end to end, recording model calls +and tokens. Without it, every claim here is an argument; with it, each lever +lands or does not. It also guards the result: the gate list drifted in three +places once already, and a loop regression is exactly as invisible. + +--- + +## Sequencing + +| | Lever | Effort | Expected effect | +|---|---|---|---| +| 1 | `diag loop-report` + benchmark harness (lever 6) | S | none directly — makes the rest falsifiable | +| 2 | Fix `projectGates` to teach `exec`, not `check`+`exec` (lever 1) | XS | ~1 call per change, every project, immediately | +| 3 | Terse/delta output for `exec` and the noisy listings (lever 2) | M | linear cut on S, compounds with 4 | +| 4 | `mxcli apply` (lever 1) | M | the 5–8 → 1–2 collapse; the main event | +| 5 | Text-first verification rule in the skills (lever 3) | S | removes recurring image input | +| 6 | Subagent trigger in the skills (lever 4) | XS | caps the worst tail | +| 7 | Workarounds → diagnostics and skills (lever 5) | M, ongoing | compounds across all future sessions | + +Item 1 first is deliberate. Items 2, 5 and 6 are nearly free and can ship +immediately after it. Item 4 is the one that changes the shape of the loop. + +## What this does not fix + +- Mendix builds. A model change must be compiled to be trusted, and that is + seconds of wall time and at least one tool call, per change, forever. +- Scope. The reported sessions did not build the same thing. +- MDL not being in training data (`PROPOSAL_llm_mdl_assistance.md` owns that). + Every MDL statement the agent gets wrong on the first try is a full loop + iteration, so that proposal and this one multiply rather than overlap — a + first-attempt success rate is a call-count lever in disguise. From c9bdb7b7d9344eba9aaf4cc9ce00b065c103a111 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:48:31 +0000 Subject: [PATCH 17/38] =?UTF-8?q?docs:=20correct=20the=20restart-per-chang?= =?UTF-8?q?e=20claim=20=E2=80=94=20it=20is=20mxbuild's,=20not=20our=20defa?= =?UTF-8?q?ults?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first draft called the 35s restart "opt-in slowness" next to run --local --watch. That is wrong on 11.14, and the measurement was already in the tree: on 11.14 the first build in an mxbuild --serve process succeeds and every subsequent build in that process fails, because the first does not leave the deployment in a state its own incremental build can continue from (webclient_legacy_paths.go, measured over mxbuild's HTTP API with mxcli removed, with the one-shot, Rspack and 11.13 controls). The project that first reported it routed around it with a restart per change — the exact pattern in the cost report. Three consequences, now in the proposal. It does not weaken the main lever: wall time and call count are separate axes, the defect taxes time, and the 228M bill is calls — so `mxcli apply` collapsing 5-8 calls into 1 matters whether the build under it is warm or cold, which makes it the only remaining lever on that axis rather than a lesser one. Two new actionable items. `test --attach` rebuilds through the attached app's own serve process, making its rebuild a second serve build — read off runner_attach.go, not measured, and one run on 11.14 settles it. And bootstrap-app defaults to the newest CDN version, so a fresh project lands on the broken one without anyone choosing it; the default should prefer 11.13.0 until mxbuild is fixed. Also records the standing risk: --serve is in mxbuild --help but not in the reference guide, so the warm loop rests on an undocumented interface with no compatibility promise. Argues for reporting this upstream rather than only routing around it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .../PROPOSAL_agent_loop_efficiency.md | 81 ++++++++++++++++++- 1 file changed, 78 insertions(+), 3 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md index 3c5f3fcf8f..0aac7200c7 100644 --- a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md +++ b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md @@ -84,9 +84,9 @@ Four of those steps are already avoidable with today's binary: lists `check` and `exec` as two consecutive gates, so the generated CLAUDE.md in **every** mxcli project teaches the two-call form. One wasted call per change, in every session, by construction. -- **The 35 s restart is opt-in slowness.** `run --local --watch` hot-reloads a - behavioural change in ~2 s, and `mxcli test --attach` skips the boot entirely - by driving an app already up. +- **The 35 s restart is NOT avoidable on Mendix 11.14** — see the section below. + On 11.13 and earlier, `run --local --watch` hot-reloads a behavioural change + in ~3 s, so the restart is opt-in slowness there and forced here. - **Logging in by hand is solved.** Playwright storage state is captured and reused (`screenshot --load-storage`); see `PROPOSAL_playwright_session_reuse.md`. - **The screenshot is usually the wrong instrument.** See lever 3. @@ -109,6 +109,77 @@ Nothing here is new capability. Every stage exists; `apply` is the composition, and the composition is what the agent is currently doing by hand, one tool call at a time, paying the full conversation for each. +### The 35 s restart is blocked by mxbuild on 11.14, not by our defaults + +An earlier draft of this proposal called the restart-per-change "opt-in +slowness". That is wrong on the version a new project most likely lands on, and +the correction matters because it changes what is actionable. + +**On Mendix 11.14 the first build in an `mxbuild --serve` process succeeds and +every subsequent build in that process fails.** The first build does not leave +the deployment in a state its own incremental build can continue from: the +bundler config (`web/rollup.config.mjs` / `web/rspack.config.mjs`) and the +per-document client (`web/pages/`, `web/layouts/`) are both absent, and which +one the build dies on is only how far it gets before it needs one. + +This is measured in `cmd/mxcli/docker/webclient_legacy_paths.go`, against +mxbuild 11.14.0 driven **directly over its HTTP API with mxcli removed from the +picture** — the same `/build` request POSTed twice with the model untouched: + +```text +build 1 Success +build 2 Failure — ERR_MODULE_NOT_FOUND for web/rollup.config.mjs, + imported from mxbuild's own tools/node/rollup-runner.mjs +``` + +Three controls place it in mxbuild rather than here: a one-shot +`mxbuild --target=deploy` run twice into the same directory succeeds both times +(so it is the serve process's state, not the 11.14 deployment shape); flipping +to Rspack gives an identical failure naming the other config (so it is not the +bundler choice); and 11.13.0 hot-reloads normally — measured, build #2 applied +via reload in 3.4 s. Restoring the deleted config rescues only the +model-unchanged case, which is the case nobody needs. `rm -rf deployment/` costs +a cold build and changes nothing. + +mxcli cannot fix this from outside, and does not pretend to: it recognises the +failure **by its own shape** rather than by version, so a fixed mxbuild goes +quiet on its own. The docs say plainly that `--watch` is not usable on 11.14. + +Three consequences for this proposal: + +1. **The cost report's 35 s-per-change was very likely forced, not chosen.** The + project that first reported this "routed around it with a restart per + change", which is exactly the pattern in the session log. +2. **It does not weaken lever 1.** Wall time and call count are separate axes. + The mxbuild defect taxes *time*; the 228 M bill is *calls*. `mxcli apply` + collapses 5–8 calls into 1 whether the build underneath it is warm or cold — + so on 11.14 it is the only lever left on that axis, and therefore more + important, not less. +3. **`mxcli test --attach` is probably blocked too, and this is unmeasured.** + `--attach` must rebuild to pick up its test microflows, and it rebuilds + through the attached app's own serve process (`runner_attach.go`) — whose + first build happened at boot. That makes the attach rebuild a *second* serve + build, which is precisely the failing one. This is read off the code, not + measured; it needs one run on 11.14 to confirm or kill. If it holds, the warm + *test* loop is blocked on 11.14 as well, and the skills that recommend + `--attach` need the same version caveat `--watch` already carries. + +**The version default makes this worse than it needs to be.** `bootstrap-app` +chooses the newest version on the CDN when the environment has nothing cached — +11.14.0 at the time of writing — so a freshly bootstrapped project lands on +exactly the version where the warm loop does not work, without anyone choosing +it. Until mxbuild is fixed, the default should prefer the newest version whose +warm loop is known to work (11.13.0), and say in one line why. A user who asks +for 11.14 still gets it, with the caveat. + +**This is also a standing strategic risk worth recording.** `--serve`, `--host` +and `--port` appear in `mxbuild --help` but not in the reference guide at +docs.mendix.com/refguide/mxbuild/, which documents only the four `--target` +modes. The entire warm loop is built on an undocumented interface carrying no +compatibility promise. That argues for (a) reporting this defect to Mendix +rather than only routing around it, and (b) keeping the cold-build path a +first-class supported mode rather than a fallback. + **Also: fix the gate list.** `projectGates` should teach `exec` (check folded in) rather than `check` then `exec`, and should name `apply` once it exists. The gates tests (`init_claudemd_gates_test.go`) already hold three copies of that @@ -212,6 +283,7 @@ places once already, and a loop regression is exactly as invisible. |---|---|---|---| | 1 | `diag loop-report` + benchmark harness (lever 6) | S | none directly — makes the rest falsifiable | | 2 | Fix `projectGates` to teach `exec`, not `check`+`exec` (lever 1) | XS | ~1 call per change, every project, immediately | +| 2b | Measure `test --attach` on 11.14; pin the bootstrap default off 11.14 | XS | removes a forced 35 s/change from new projects | | 3 | Terse/delta output for `exec` and the noisy listings (lever 2) | M | linear cut on S, compounds with 4 | | 4 | `mxcli apply` (lever 1) | M | the 5–8 → 1–2 collapse; the main event | | 5 | Text-first verification rule in the skills (lever 3) | S | removes recurring image input | @@ -225,6 +297,9 @@ immediately after it. Item 4 is the one that changes the shape of the loop. - Mendix builds. A model change must be compiled to be trusted, and that is seconds of wall time and at least one tool call, per change, forever. +- The 11.14 serve-rebuild defect. It is mxbuild's, the controls are conclusive, + and nothing mxcli does from outside repairs it. It should be reported upstream; + meanwhile the version default is the only lever we hold. - Scope. The reported sessions did not build the same thing. - MDL not being in training data (`PROPOSAL_llm_mdl_assistance.md` owns that). Every MDL statement the agent gets wrong on the first try is a full loop From 4434fbf4d9b2233af483614fb8b2d578c2e45615 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 10:52:31 +0000 Subject: [PATCH 18/38] docs: drop `mxcli apply` to contingent; the chain is `&&`, and it is tiered MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two objections to the first draft, both of which hold. The agent can already chain the loop with `&&`, so the call-count win needs nothing built. Checked that this is actually safe: exec exits non-zero on any failed statement and docker check propagates mx check's status through cmd.Run() rather than printing errors and exiting 0 — a chain over a command that reported failure only in stdout would pass silently, and that is what would make chaining unsafe. It is not present here. What `&&` does not give is the token win: it concatenates every stage's stdout. So output discipline is the more fundamental lever, not the junior partner — with terse delta-shaped output, `&&` gets nearly all of apply's value at zero new surface, and the improvement lands on every other invocation too. apply is now contingent on diag loop-report showing the published chain still gets composed wrong. Second: the first draft put a Playwright run in the default chain. That is the same mistake the cost report describes — an always-on chain trains maximal verification, and hard-wiring the browser would turn "I tested every admin flow, most checks included screenshots" from a choice into a property of the tool. The chain is now tiered, stopping at the first sufficient gate, with the routing rule that verify-in-runtime.md already owns: most changes stop at exec or the build, and the browser row is rare and the only one paying image input. Sequencing rewritten: the loop-shaping item is now a documentation change, and the only substantial build is the output work that makes it pay in tokens rather than only in calls. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .../PROPOSAL_agent_loop_efficiency.md | 100 +++++++++++++++--- 1 file changed, 83 insertions(+), 17 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md index 0aac7200c7..f691ef46b0 100644 --- a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md +++ b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md @@ -91,23 +91,84 @@ Four of those steps are already avoidable with today's binary: reused (`screenshot --load-storage`); see `PROPOSAL_playwright_session_reuse.md`. - **The screenshot is usually the wrong instrument.** See lever 3. -### Proposal: one gate command +### First: the agent can already chain this, and mostly should + +The obvious objection to a new command is that `&&` exists: ```bash -mxcli apply changes.mdl -p app.mpr [--verify tests/admin.spec.ts] [--no-build] +mxcli exec changes.mdl -p app.mpr && mxcli docker check -p app.mpr ``` -One call that runs: semantic check → exec → mxbuild verify → reload the running -app (or restart if the serve build says `restartRequired`) → run the named -Playwright verifications → print **one compact verdict**. - -The verdict is the whole point. On success it is two lines. On failure it is the -first failing stage and only that stage's diagnostics — not the output of all -five. This is what turns 5–8 calls into 1, and on the failure path into 2. +That is **one tool call**, needs nothing built, and captures most of lever 1's +value today. It works because the commands are exit-code-honest — `exec` exits +non-zero if any statement failed, and `docker check` propagates `mx check`'s +status through `cmd.Run()` rather than printing errors and exiting 0. Worth +stating explicitly, because a chain built on a command that reports failure only +in stdout would pass silently, and that is the failure mode that would make +chaining unsafe. It is not present here. + +So the call-count win does **not** require a new command. What `&&` does not +give is the *token* win: it concatenates the stdout of every stage that ran, so +a five-stage chain puts five stages of output into the conversation forever — +the opposite of the compact verdict this proposal wants. The agent can paper +over that with per-invocation `tail`/`grep`, but then it is writing fragile +filters that encode each command's output shape, and getting them wrong is +silent. + +**That reorders the proposal.** Lever 2 (output discipline) is the more +fundamental of the two, not the junior partner: with terse, delta-shaped output +from each command, `&&` chaining gets nearly all of `apply`'s value at zero new +surface area — and the improvement lands on every *other* invocation too, not +just the ones inside the chain. + +### So what, if anything, is left for a command? + +Two things, and both are weaker than the first draft claimed: + +- **The reload/restart decision.** Mapping the serve build's `restartRequired` + to reload-vs-restart is not expressible in `&&`. But when `run --local --watch` + works it already does this in the background, and the agent orchestrates + nothing; when it does not (11.14, below) the answer is a restart, which *is* + `&&`-able. So this is thin. +- **Consistency.** An agent composing the chain fresh each session composes it + differently, and sometimes wrongly — the cost report is the evidence, having + run the redundant `check`, restarted when it need not have, and logged in by + hand. But that is cured by **stating the chain**, not by shipping a wrapper + around it. + +**Revised recommendation: publish the one-liner, do not build the command.** Put +the canonical chain in `projectGates` and the skills, fix the outputs it +concatenates, and build `mxcli apply` only if `diag loop-report` (lever 6) shows +agents still composing it wrong after that. This is strictly cheaper, ships +sooner, and does not add a surface that has to stay in sync with the commands +underneath it. + +### The chain is tiered, not fixed — most changes stop at the first gate + +The first draft put `--verify ` in the default chain. That is a +mistake of the same kind the cost report is complaining about: **an always-on +gate chain trains maximal verification.** If a browser run is in the default +path, every change pays browser cost, and the report's "I tested every admin +flow ... most of those checks included screenshots" stops being a choice the +agent made and becomes a property of the tool. Hard-wiring it would +institutionalise the expensive failure mode. + +Verification tier is a **per-change decision**, and the routing rule already +exists in `.claude/skills/verify-in-runtime.md` — a table from symptom to +cheapest sufficient proof, with explicit counter-examples where the browser +would be waste. The chain should express those tiers and stop at the first one +that is sufficient: + +| What changed | Sufficient gate | Cost | +|---|---|---| +| any MDL edit | `mxcli exec` (check folded in) | ~2 s, no build | +| structure a build can reject (pages, widgets, settings) | `+ docker check` / the serve build | ~25 s | +| microflow *behaviour* | `+ mxcli test` — no browser | ~2 s warm | +| what the app **renders or looks like** | `+` browser, once | expensive, rare | -Nothing here is new capability. Every stage exists; `apply` is the composition, -and the composition is what the agent is currently doing by hand, one tool call -at a time, paying the full conversation for each. +Most changes stop at row 1 or 2. The browser row is the rare one, and it is the +only row that pays image input. Making the tier explicit is itself a lever: it +converts "verify everything, to be safe" into a decision with a stated default. ### The 35 s restart is blocked by mxbuild on 11.14, not by our defaults @@ -284,14 +345,19 @@ places once already, and a loop regression is exactly as invisible. | 1 | `diag loop-report` + benchmark harness (lever 6) | S | none directly — makes the rest falsifiable | | 2 | Fix `projectGates` to teach `exec`, not `check`+`exec` (lever 1) | XS | ~1 call per change, every project, immediately | | 2b | Measure `test --attach` on 11.14; pin the bootstrap default off 11.14 | XS | removes a forced 35 s/change from new projects | -| 3 | Terse/delta output for `exec` and the noisy listings (lever 2) | M | linear cut on S, compounds with 4 | -| 4 | `mxcli apply` (lever 1) | M | the 5–8 → 1–2 collapse; the main event | -| 5 | Text-first verification rule in the skills (lever 3) | S | removes recurring image input | +| 3 | Publish the canonical `&&` chain in `projectGates` + skills (lever 1) | XS | the 5–8 → 1–2 collapse, with nothing built | +| 4 | Terse/delta output for `exec` and the noisy listings (lever 2) | M | the token half of the chain win; helps every call | +| 5 | Tiered verification rule in the skills (lever 3) | S | stops the default path at the cheapest sufficient gate | | 6 | Subagent trigger in the skills (lever 4) | XS | caps the worst tail | | 7 | Workarounds → diagnostics and skills (lever 5) | M, ongoing | compounds across all future sessions | -Item 1 first is deliberate. Items 2, 5 and 6 are nearly free and can ship -immediately after it. Item 4 is the one that changes the shape of the loop. +Item 1 first is deliberate. Items 2, 3, 5 and 6 are all XS-to-S and can ship +immediately after it — item 3 is now the one that changes the shape of the loop, +and it is a documentation change. Item 4 is the only substantial build, and it +is what makes item 3 pay in tokens rather than only in call count. + +`mxcli apply` is deliberately **not** in this table. It is contingent on item 1 +showing that the published chain is still being composed wrong. ## What this does not fix From 563d30d9f0a5478376f004456a43f509d0a4f4b4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 11:02:52 +0000 Subject: [PATCH 19/38] =?UTF-8?q?docs:=20a=20.tsx=20file=20is=20not=20"don?= =?UTF-8?q?e"=20when=20written=20either=20=E2=80=94=20fix=20the=20false=20?= =?UTF-8?q?asymmetry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The draft said "Vercel's agent writes a .tsx file and the work is done", which is false and was filing the gap under platform tax instead of work. TypeScript needs type-checking, building and rendering too. The difference is four properties of the verification, not whether it happens: cost per verification (tsc ~1-3s and HMR at ZERO tool calls, vs mxbuild ~25s and a tool call), how often it is needed (first-attempt success, where TS is saturated in training data and MDL appears nowhere), error locality (file:line:col vs a CE naming a document at the end of a build), and whether the last tier needs a running app. Three of the four are engineering, and two are already in flight. check-vs-mxbuild parity is a standing programme with ~17 rules shipped, each moving a construct from a 25s build to a 2s check — this proposal's contribution is to say why that is a token lever and not only a correctness one. Diagnostic quality is the other, and it matters here because a vague error costs a diagnosis, which is the 40-call tail. The real target named: build once per batch, not once per change, which is what the Vercel agent does with tsc. The blocker is trust rather than speed — an agent builds after every change because check passing does not yet mean the build passes, and #568 (docker check reporting 0 errors over a build-failing model) attacks that trust from the other side. "Mendix builds are a per-change tax forever" has accordingly moved out of the does-not-fix list. What stays there is the render tier, which is true of React too. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .../PROPOSAL_agent_loop_efficiency.md | 81 +++++++++++++++---- 1 file changed, 65 insertions(+), 16 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md index f691ef46b0..2572c38df0 100644 --- a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md +++ b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md @@ -54,22 +54,63 @@ That arithmetic sets the priority order, and it is not the intuitive one: lever 1, so the two compound. 3. Output tokens (500 k of 228 M) are a rounding error. Do not optimise here. -## What is irreducible, and what is not +## What is actually asymmetric — and how little of it is irreducible -Part of the 4× is real and will not go away. Vercel's agent writes a `.tsx` file -and the work is done. A Mendix change is a model mutation that must be -**validated, applied, built, and rendered** before anyone knows it worked. The -feedback also notes the two sessions were not scope-equivalent — the mxcli one -additionally covered login styling, documentation, Docker and password lockout. +An earlier draft said: "Vercel's agent writes a `.tsx` file and the work is +done." That is false, and it was doing real damage to this proposal by filing +the gap under platform tax instead of under work. -So parity is the wrong target. The target is the **gap between the loop we ship -and the loop mxcli is already capable of**, which is large, because most of the -fast paths below already exist and the session did not take them. +A `.tsx` file is not done when written. It needs type-checking, building and +rendering before anyone knows it worked, exactly like an MDL change. The +difference is not *whether* verification happens — it is four properties of the +verification, and **three of the four are things we can move**: -The honest claim: of the reported ~400 extra calls, roughly 250–300 look -addressable. The rest is Mendix being a compiled platform. - ---- +| | TypeScript / Next | MDL / Mendix | +|---|---|---| +| **Cost per verification** | `tsc --noEmit` ~1–3 s; HMR sub-second and **zero tool calls** — the dev server is already running and the browser updates itself | mxbuild ~25 s **and a tool call**; the HMR analogue (`reload_model`) is blocked on 11.14 | +| **How often it is needed** | low — TypeScript and React are saturated in training data, so first-attempt success is high | higher — MDL is a DSL invented in this repo and appears nowhere in training data | +| **Error locality** | `file:line:col`, expected vs actual | a CE number naming a *document*, at the far end of a build — and per ako/mxcli#568, `docker check` can report "0 errors" while the build fails | +| **Does the last tier need a running app?** | no for logic, yes for render — but the browser is already open on the changed component | yes, plus `reload_model`, plus login, plus navigation | + +Row 2 is the one mxcli can only partly close, and +`PROPOSAL_llm_mdl_assistance.md` owns it. The other three are engineering, and +two of them are already in flight: + +- **Row 1 is `mxcli check` vs mxbuild.** `check` is the `tsc` of MDL, and + `PROPOSAL_check_mxbuild_gap_heuristics.md` is a **standing programme** to close + the gap — ~17 rules shipped, each one a construct that used to be found only + by a 25 s build and is now found in ~2 s with no build at all. Every rule moved + across that line is a direct call-count *and* wall-time win. This is the + highest-value structural work in the whole picture and it was already + underway; this proposal's contribution is to say why it is a **token** lever + and not only a correctness one. +- **Row 1's other half is the HMR analogue**, which exists (`reload_model`, + ~3 s on 11.13) and is blocked by the mxbuild defect below. +- **Row 3 is diagnostic quality**, which `PROPOSAL_check_diagnostics_catalog.md` + owns. It matters here because a vague error costs a *diagnosis*, and a + diagnosis is the 40-call tail in lever 4. + +### The real target: build once per batch, not once per change + +That is what the Vercel agent does. It does not run a production build after +every file — it leans on a fast, trusted static check and batches the expensive +gate. mxcli can have the same shape, and the blocker is **trust**, not speed: +an agent runs the 25 s build after every change precisely because `check` +passing does not yet mean the build will pass. + +So row 1 and row 3 compound. Closing check-vs-build parity does not merely make +the expensive gate faster — it makes the expensive gate *rarer*, because it +becomes reasonable to batch. And ako/mxcli#568 is a direct attack on that trust +from the other side: a check that can say "0 errors" over a build-failing model +teaches an agent never to believe it. + +**What is left that is genuinely irreducible:** the final "does it render +correctly" tier needs a running app, in both worlds. That is one gate, rarely, +per the tier table above — not a per-change tax. + +The scope caveat still stands and is separate: the two sessions did not build +the same thing. The mxcli one additionally covered login styling, documentation, +Docker and password lockout. Some unknown part of the 4× is simply more work. ## Lever 1 — collapse the per-change round trip (attacks N) @@ -343,6 +384,7 @@ places once already, and a loop regression is exactly as invisible. | | Lever | Effort | Expected effect | |---|---|---|---| | 1 | `diag loop-report` + benchmark harness (lever 6) | S | none directly — makes the rest falsifiable | +| 1b | Measure the check↔build gap rate: how many builds in a real session caught something `check` did not | S | sizes the batching prize, and feeds the parity programme's queue | | 2 | Fix `projectGates` to teach `exec`, not `check`+`exec` (lever 1) | XS | ~1 call per change, every project, immediately | | 2b | Measure `test --attach` on 11.14; pin the bootstrap default off 11.14 | XS | removes a forced 35 s/change from new projects | | 3 | Publish the canonical `&&` chain in `projectGates` + skills (lever 1) | XS | the 5–8 → 1–2 collapse, with nothing built | @@ -361,13 +403,20 @@ showing that the published chain is still being composed wrong. ## What this does not fix -- Mendix builds. A model change must be compiled to be trusted, and that is - seconds of wall time and at least one tool call, per change, forever. +- The last verification tier. "Does it render correctly" needs a running app, + and that is true of React too. The tier table keeps it rare rather than + per-change. - The 11.14 serve-rebuild defect. It is mxbuild's, the controls are conclusive, and nothing mxcli does from outside repairs it. It should be reported upstream; meanwhile the version default is the only lever we hold. - Scope. The reported sessions did not build the same thing. - MDL not being in training data (`PROPOSAL_llm_mdl_assistance.md` owns that). - Every MDL statement the agent gets wrong on the first try is a full loop + It is row 2 of the asymmetry table and the one property here that is not + engineering. Every MDL statement wrong on the first try is a full loop iteration, so that proposal and this one multiply rather than overlap — a first-attempt success rate is a call-count lever in disguise. + +Note what has moved OUT of this list since the first draft: "Mendix builds, and +that is a per-change tax forever." It is not. It is a per-change tax for as long +as `check` is not trusted enough to batch the build behind it, which is a +programme already running. From 8c50ea9c5908cfef850d579a5775d418def738dd Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 11:15:17 +0000 Subject: [PATCH 20/38] docs: the LSP is the same checker, so it is not a lever on the token bill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit runSemanticValidation in lsp_diagnostics.go runs the same validators as cmd_check.go — the LSP and `mxcli check` are one checker behind two front ends. So the LSP finds exactly what check finds, and the check-to-build gap that forces the 25s mxbuild per change is untouched. It makes the already-cheap tier cheaper and does nothing to the tier that dominates. Also a wash on call count as normally used: the call it would save is the redundant check that exec already folds in, and reading diagnostics is itself a tool call. The one condition under which it becomes real: diagnostics riding along with the Write/Edit result at zero extra call, which is the property the asymmetry table identifies as the Vercel agent's biggest structural advantage. Wired that way it is the only available route to a zero-call verification tier for static errors; wired any other way it is a front end onto a command we have. Records the failure mode too — auto-attached per-edit diagnostics are paid on every intermediate save of a script that is incomplete until its last line, so they want to fire on batch completion rather than per edit. The real argument it supports is spending on parity instead: because the validators are shared, every rule from the check-gap programme lands in the editor and the agent's checker at once. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .../PROPOSAL_agent_loop_efficiency.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md index 2572c38df0..34d20060bd 100644 --- a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md +++ b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md @@ -328,6 +328,61 @@ right shape for this (a table routing a symptom to the cheapest sufficient proof); it needs the text-vs-pixel row added and `run-local` / `test-app` pointed at it. +## Would the LSP help? Modestly, and not where the money is + +mxcli ships a language server (`mxcli lsp --stdio`) with diagnostics, hover, +completion and go-to-definition. The natural question is whether pointing an +agent at it collapses the loop. + +**It does not, and the reason is one line of the implementation.** +`runSemanticValidation` in `cmd/mxcli/lsp_diagnostics.go` "runs the same +validators as `cmd_check.go`". The LSP and `mxcli check` are the *same checker* +behind two front ends. So the LSP finds exactly what `check` finds — and the +check↔build gap, which is what forces the 25 s mxbuild per change, is completely +unaffected. The LSP makes the tier that is already cheap slightly cheaper, and +does nothing to the tier that dominates. + +Two further reasons it is a wash rather than a win as normally used: + +- **The call it saves is one we are removing for free anyway.** The redundant + `check` call goes away because `exec` folds the check in. The LSP would be + deleting a call already on the chopping block. +- **Reading diagnostics is itself a tool call.** Unless the harness attaches + them to the edit result, it is one `getDiagnostics` instead of one + `mxcli check` — the same arithmetic. + +### The one condition under which it becomes a real lever + +**If diagnostics ride along with the `Write`/`Edit` result at zero extra tool +call.** That is precisely the property identified as the Vercel agent's biggest +structural advantage in the asymmetry table above: its cheapest verification +tier is not a cheap tool call, it is *not a tool call at all*. An LSP wired that +way is mxcli's only available route to that property for static errors. Wired +any other way, it is a front end onto a command we already have. + +### And one way it could make things worse + +Diagnostics auto-attached to every edit are paid on **every** edit, including the +intermediate ones. An MDL script written top to bottom is incomplete at every +save but the last, so per-edit diagnostics on it are mostly noise about +incompleteness — potentially more tokens than one terse `check` at the end of the +batch. If this is wired up, it wants to fire on batch completion, not per +keystroke or per edit. + +### What it is genuinely good for + +**It is the second delivery channel for the parity programme.** Because the +validators are shared, every rule added by +`PROPOSAL_check_mxbuild_gap_heuristics.md` appears in the editor *and* in the +agent's checker with no extra work. That is an argument for spending on parity +rather than on the LSP: parity pays into both channels at once, while LSP work +pays into neither checker. + +It is also a genuine win for the **human** in VS Code, and for wall time — the +server caches the widget and theme registries so the filesystem is not walked +per keystroke, which `mxcli check` does per invocation. Neither of those is a +token lever. + ## Lever 4 — keep long investigations out of the main conversation One self-inflicted bug (a stub script that wiped real microflow bodies) took From 1bf8c2e2a54bd6bf31450717b3717e2d0d642f0e Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 11:22:48 +0000 Subject: [PATCH 21/38] =?UTF-8?q?docs:=20correct=20the=20"check=20before?= =?UTF-8?q?=20exec=20is=20redundant"=20claim=20=E2=80=94=20it=20is=20not?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The proposal claimed exec folds in everything check does, making the two-gate form a wasted call in every project. That was Track A's headline and it is false. There are two validation passes and exec runs only the first. executor.ValidateProgram (package-level, semantic rules) runs in both. exec.ValidateProgram (method, project connected — reference resolution) and exec.CheckProjectConflicts (plain CREATE over an existing document) run in check only, behind --references, which -p implies. So check -p catches dangling references and create-conflicts that exec does not, and since exec is not transactional that preflight is load-bearing. The claim was read off cmd_exec.go's doc comment, which is accurate about the pass it describes and silent about the two it does not. Reading the call graph takes one more step and was skipped; that is recorded in the proposal because it is the instructive part. What survives is better targeted. The gate list's defect is its framing, not the check gate: it calls the seven gates "the definition of done ... a change is finished when they have all been run" while holding docker check, test and run --local, which read literally mandates ~55s of gates on every change — the maximal-verification pathology the cost report describes, and a contradiction of the escalation rule one line above. The fix is batching, not deletion. Also names the one real code change in Track A: exec already connects to the project, so it could run the reference pass in its preflight, which would make the check gate genuinely redundant for the apply path and stop exec half-applying a script with a dangling reference. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .../PROPOSAL_agent_loop_efficiency.md | 62 ++++++++++++++++--- 1 file changed, 52 insertions(+), 10 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md index 34d20060bd..8856b0593b 100644 --- a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md +++ b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md @@ -119,12 +119,34 @@ The reported loop is **5–8 calls per change**: write script → `mxcli check` Four of those steps are already avoidable with today's binary: -- **`check` before `exec` is redundant.** `exec` already runs the full semantic - check before it writes anything and refuses the script on an error - (`cmd/mxcli/cmd_exec.go`). Yet `projectGates` in `cmd/mxcli/init_claudemd.go` - lists `check` and `exec` as two consecutive gates, so the generated CLAUDE.md - in **every** mxcli project teaches the two-call form. One wasted call per - change, in every session, by construction. +- **`check` before `exec` is NOT redundant — I had this wrong.** An earlier + draft claimed `exec` folds in everything `check` does, so the two-gate form + wasted a call in every project. That is false, and the correction matters + because it was Track A's headline. + + There are **two** validation passes, and `exec` runs only the first: + + | Pass | What it catches | `check -p` | `exec` | + |---|---|---|---| + | `executor.ValidateProgram(prog, path)` (package-level) | semantic rules — MDL0xx, reserved words, list-op nesting | ✅ | ✅ | + | `exec.ValidateProgram(prog)` (method, project connected) | **reference resolution** — dangling entity/page/microflow/icon names | ✅ | ❌ | + | `exec.CheckProjectConflicts(prog)` | plain `CREATE` over a document that already exists | ✅ | ❌ | + + So `mxcli check script.mdl -p app.mpr` genuinely catches things `exec` does + not, *before* a partial write — and `exec` is not transactional, so that + preflight is load-bearing. The gate list is teaching an additive step, not a + wasted one. + + (`--references` in the gate line *is* redundant: it is implied by `-p`. That + is cosmetic.) + + **How the error was made, since it is the instructive part:** the claim was + read off `cmd_exec.go`'s doc comment — "the same semantic checks as + `mxcli check`" — which is accurate about the pass it describes and silent + about the two it does not. Reading the call graph takes one more step and was + skipped. The repo's own checklist has the rule that would have caught it + ("Fix proven to be the cause — revert it and confirm the symptom returns"); + the equivalent here was to diff what the two commands actually call. - **The 35 s restart is NOT avoidable on Mendix 11.14** — see the section below. On 11.13 and earlier, `run --local --watch` hot-reloads a behavioural change in ~3 s, so the restart is opt-in slowness there and forced here. @@ -282,10 +304,30 @@ compatibility promise. That argues for (a) reporting this defect to Mendix rather than only routing around it, and (b) keeping the cold-build path a first-class supported mode rather than a fallback. -**Also: fix the gate list.** `projectGates` should teach `exec` (check folded in) -rather than `check` then `exec`, and should name `apply` once it exists. The -gates tests (`init_claudemd_gates_test.go`) already hold three copies of that -list to one definition, so this is a one-line change that propagates. +**What the gate list actually gets wrong.** Not the `check` gate — the framing +around it. The generated CLAUDE.md says the gates are "**the definition of done, +not a menu** — a change is finished when they have all been run", and the list +has `docker check` (~25 s), `test` (~30 s cold) and `run --local` in it. Read +literally, that mandates all seven gates on **every change**, which is precisely +the maximal-verification pathology the cost report describes. It also +contradicts the sentence immediately above it, which offers an escalation rule +("each is only worth paying for once the one above is clean"). + +The fix is **batching, not deletion**: the gates are the definition of done for +a *change*, where a change is a coherent unit of work — not per statement and +not per file write. Iterate with `exec` until the script is right, then run the +gates once. That preserves every gate (the three-copy tests exist because `test` +fell off this list once) while removing the per-micro-edit repetition, and it is +the same "build once per batch" shape as the `tsc` comparison above. + +**And a real code change worth making:** `exec` already connects to the project, +so it *could* run the reference pass in its preflight. If it did, the `check` +gate would become genuinely redundant for the apply path and the call would be +saved for real — and `exec` would stop being able to half-apply a script with a +dangling reference. `CheckProjectConflicts` is a separate question and probably +has to stay check-only, since a plain `CREATE` over an existing document is an +error for `check` but ordinary for a re-run. This is the one place in Track A +where the win is a code change rather than wording. ## Lever 2 — shrink what each call adds (attacks S) From 22fd7238e9188da193f84dbd9c184cc61ab9fb27 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 12:02:59 +0000 Subject: [PATCH 22/38] fix(exec): resolve references in the preflight, so a dangling name is refused MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #607. `mxcli check -p` runs two validation passes and `exec` ran only the first. executor.ValidateProgram (semantic) ran in both; exec.ValidateProgram (reference resolution against the connected project) ran in check alone, behind --references, which -p implies. exec already connects before its preflight, so it had everything the second pass needs. What that cost, measured on the expr-checker fixture with the pre-fix binary (--no-check reproduces it) — and NOT the failure the neighbouring refusal describes: create entity "NotAModule"."Thing" exit 0, "Created module: NotAModule" — a misspelled module is silently created microflow retrieving a missing entity exit 0, both documents written exec did not half-apply; it completed, and wrote a model only a ~25s mxbuild would reject (CE1613). So this is a check-to-build parity fix — it moves a build-tier error to the 2s tier — and it closes a violation of the "no silent side effects on typos" rule in CLAUDE.md's own checklist. After the fix both scripts exit 1 with nothing written. Safe to refuse on: the pass skips references to objects the script itself creates, so an error means the name resolves to nothing in the project AND is not created here. Gated on -p, since a script using its own CONNECT has no backend at preflight time — the same condition check uses. CheckProjectConflicts is deliberately NOT run: a plain CREATE over an existing document is worth reporting when validating a script but ordinary for a re-run, and refusing it would break scripts that work today. The test states the GAP rather than that the validator works: its first assertion is the control, showing the pass exec ran reports nothing for a script the reference pass refuses, and it fails loudly if that ever stops being true rather than being quietly relaxed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- cmd/mxcli/cmd_exec.go | 53 ++++++++++++++ mdl/executor/validate_exec_preflight_test.go | 75 ++++++++++++++++++++ 2 files changed, 128 insertions(+) create mode 100644 mdl/executor/validate_exec_preflight_test.go diff --git a/cmd/mxcli/cmd_exec.go b/cmd/mxcli/cmd_exec.go index 2729f3b5ea..e38e16609d 100644 --- a/cmd/mxcli/cmd_exec.go +++ b/cmd/mxcli/cmd_exec.go @@ -112,6 +112,59 @@ Example: } } + // Second preflight pass: resolve every NAME against the connected + // project. The semantic pass above cannot do this — a missing module, + // entity, page or microflow needs a backend, not a path — so `mxcli + // check -p` ran it and `exec` did not (#607). + // + // MEASURED on the expr-checker fixture, running the pre-fix binary + // (`--no-check` reproduces it), because the failure mode is not the one + // the refusal above describes and the difference matters: + // + // create entity "NotAModule"."Thing" -> exit 0, "Created module: + // NotAModule". A misspelled + // module is SILENTLY CREATED. + // microflow retrieving a missing entity -> exit 0, both documents + // written. The dangling name + // reaches the model and is + // not reported until mxbuild + // rejects it (CE1613). + // + // So exec did not half-apply here — it completed, and wrote a model that + // only a 25s build would reject. That makes this a check-to-build parity + // fix (moving a build-tier error to the 2s tier) and a fix for the + // "no silent side effects on typos" rule in CLAUDE.md's checklist, which + // auto-creating a module on a misspelling violates outright. + // + // Safe to refuse on, because the pass skips references to objects the + // script itself creates: an error from it means the name resolves to + // nothing in the project AND is not created here, so exec would have + // failed on it regardless — later, and after writing. + // + // Only possible with -p. A script that connects with its own CONNECT + // statement has no backend until ExecuteProgram runs, which is the same + // condition `check` gates this on. + // + // CheckProjectConflicts is deliberately NOT run here, though `check` + // runs it alongside this pass: a plain CREATE over an existing document + // is worth reporting when validating a script, but it is ordinary for a + // re-run, and refusing it would break scripts that work today. + if !skipCheck && projectPath != "" { + if refErrs := exec.ValidateProgram(prog); len(refErrs) > 0 { + for _, refErr := range refErrs { + fmt.Fprintf(os.Stderr, "Reference error: %v\n", refErr) + } + fmt.Fprintf(os.Stderr, + "\nRefusing to execute: %d unresolved reference(s) above. Nothing was written.\n"+ + " A name that resolves to nothing is written into the model as it stands and\n"+ + " is not reported until mxbuild rejects it (CE1613) — and a misspelled MODULE\n"+ + " is created rather than refused.\n"+ + " Fix them, or re-run with --no-check to apply the script anyway.\n", + len(refErrs)) + os.Exit(1) + } + } + if continueOnError { res, err := exec.ExecuteProgramContinueOnError(prog, os.Stderr) if err != nil && !errors.Is(err, executor.ErrExit) { diff --git a/mdl/executor/validate_exec_preflight_test.go b/mdl/executor/validate_exec_preflight_test.go new file mode 100644 index 0000000000..1a93310f45 --- /dev/null +++ b/mdl/executor/validate_exec_preflight_test.go @@ -0,0 +1,75 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +// `mxcli exec` ran only ONE of the two validation passes `mxcli check -p` runs, +// so a name that resolves to nothing reached the model unreported (#607). +// +// MEASURED on this fixture with the pre-fix binary (`--no-check` reproduces it), +// because the failure mode is NOT the "half-applied model" exec's other refusal +// describes: +// +// create entity "NotAModule"."Thing" exit 0 — "Created module: NotAModule" +// microflow retrieving a missing entity exit 0 — both documents written +// +// exec completed in both cases. It silently created a module on a misspelling, +// and wrote a dangling entity reference that only mxbuild would reject (CE1613). +// +// The control is the FIRST assertion below: the pass exec ran reports nothing at +// all for a script the reference pass refuses. Without it this test would pass +// against a build where exec had never been missing anything. + +import ( + "bytes" + "testing" + + "github.com/mendixlabs/mxcli/mdl/backend" + modelsdkbackend "github.com/mendixlabs/mxcli/mdl/backend/modelsdk" + "github.com/mendixlabs/mxcli/mdl/linter" + "github.com/mendixlabs/mxcli/mdl/visitor" +) + +// connectedExecutor returns an executor connected to the shared fixture, which +// is what makes the reference pass answerable at all. +func connectedExecutor(t *testing.T, projectPath string) *Executor { + t.Helper() + exec := New(&bytes.Buffer{}) + exec.SetQuiet(true) + exec.SetBackendFactory(func() backend.FullBackend { return modelsdkbackend.New() }) + t.Cleanup(func() { exec.Close() }) + run(t, exec, "CONNECT LOCAL '"+visitor.QuoteString(projectPath)+"'") + return exec +} + +// A reference to a module that does not exist is the simplest dangling +// reference there is: it needs the project to detect and nothing else. +const danglingReferenceScript = `create entity "NotAModule"."Thing" ( "Name": String(50) );` + +func TestReferenceErrorsAreInvisibleToTheSemanticPass(t *testing.T) { + p := projectFixture(t) + + prog, errs := visitor.Build(danglingReferenceScript) + if len(errs) > 0 { + t.Fatalf("fixture script does not parse: %v", errs) + } + + // THE CONTROL. This is the pass `exec` runs in its preflight. It is given + // the project path and still cannot see a missing module, because resolving + // one needs a connected backend rather than a path. + // + // If this ever starts reporting an error, the gap has closed by another + // route and the assertion below stops proving anything — so it fails loudly + // rather than being quietly relaxed. + if summary := linter.Summarize(ValidateProgram(prog, p)); summary.Errors > 0 { + t.Fatalf("the semantic pass now reports %d error(s) for a dangling reference; "+ + "this test's control has expired and the preflight gap must be re-established "+ + "before the assertion below means anything", summary.Errors) + } + + // The pass `check -p` runs, and `exec` does not. + refErrs := connectedExecutor(t, p).ValidateProgram(prog) + if len(refErrs) == 0 { + t.Fatal("the reference pass reports nothing for a script naming a module that does " + + "not exist — then `check -p` is not catching it either, and #607 is not the bug") + } +} From 98aa174bc5d9b1c5b5506bc828fcc79ac7306906 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 12:05:05 +0000 Subject: [PATCH 23/38] =?UTF-8?q?docs(gates):=20state=20the=20unit=20?= =?UTF-8?q?=E2=80=94=20the=20gates=20run=20once=20per=20change,=20not=20pe?= =?UTF-8?q?r=20edit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #608. The gate section had a completeness rule ("**They are the definition of done, not a menu** — a change is finished when they have all been run") one line below an escalation rule ("each is only worth paying for once the one above is clean"). They contradict each other and the bolded one wins, applied to a list holding docker check (~25s), test (~30s cold) and run --local. "A change" was never defined, so in practice it became each edit: ~55s of gates and five tool calls per edit. That is measured, not hypothetical. A session-cost comparison of the same class of app built with mxcli vs on Vercel found 523 model calls against 123, with the per-change loop at 5-8 tool calls; every call re-reads the transcript, so call count enters the bill quadratically once the removed calls take their output with them. The report's own line — "I tested every admin flow ... most of those checks included screenshots" — is this instruction being followed, not an agent being careless. The fix is the UNIT, not the list. Every gate stays: the three-copy tests exist because `test` fell off this list once and became reachable only when a user asked for it by name. What changes is that the gates are done-criteria for a coherent unit of work — iterate with exec, then run them once over the result. Same shape as a TypeScript agent leaning on tsc per edit and batching the build. Held in all three places by a new test, for the same reason the gate list itself is: stated in two of three, it applies when someone remembers. Also drops --references from the check gate line, which -p implies and which the flag's own help says is kept only for compatibility, and says what check still buys now that exec resolves references too (#607): it does not apply. Generated file is 5857 of its 6000-byte budget. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .claude/skills/mendix/bootstrap-app/SKILL.md | 5 +++ cmd/mxcli/init_claudemd.go | 6 ++- cmd/mxcli/init_claudemd_gates_test.go | 45 ++++++++++++++++++++ docs-site/src/tools/bootstrap-prompt.md | 6 +++ 4 files changed, 61 insertions(+), 1 deletion(-) diff --git a/.claude/skills/mendix/bootstrap-app/SKILL.md b/.claude/skills/mendix/bootstrap-app/SKILL.md index cc423375ee..a1dc3296e1 100644 --- a/.claude/skills/mendix/bootstrap-app/SKILL.md +++ b/.claude/skills/mendix/bootstrap-app/SKILL.md @@ -353,6 +353,11 @@ see `migrate-design-prototype`. them in context. They are the **definition of done**, not a menu: run them in order, stop at the first that fails, and say what each one reported. +Run them **once per change, not per edit** — a change being a coherent unit of work, +not a single statement and not a file write. Iterate with `exec` until the script is +right, then run the gates once over the result. The whole list after every edit costs +~55s and five calls each time and proves nothing the one run at the end does not. + ```bash ./mxcli check change.mdl -p .mpr --references # syntax + references (~2s) ./mxcli exec change.mdl -p .mpr # apply diff --git a/cmd/mxcli/init_claudemd.go b/cmd/mxcli/init_claudemd.go index e259705060..4ea8890c68 100644 --- a/cmd/mxcli/init_claudemd.go +++ b/cmd/mxcli/init_claudemd.go @@ -74,7 +74,7 @@ type projectGate struct { // it to every generated CLAUDE.md, and requires naming it in the bootstrap // skill and docs page too. var projectGates = []projectGate{ - {"check script.mdl -p %s --references", "syntax + references (~2s)", "mxcli check", "check"}, + {"check script.mdl -p %s", "syntax + references, no apply (~2s)", "mxcli check", "check"}, {"exec script.mdl -p %s", "apply", "mxcli exec", "exec"}, {"lint -p %s", "rules (~3s)", "mxcli lint", "lint"}, {"report -p %s", "scored quality report", "mxcli report", "report"}, @@ -179,6 +179,10 @@ func generateClaudeMD(projectName, mprFile string) string { w("Run them cheapest-first; each is only worth paying for once the one above is clean.\n") w("**They are the definition of done, not a menu** — a change is finished when they have\n") w("all been run and you have said what each one reported.\n\n") + w("**Once per change, not per edit.** A change is a coherent unit of work — not a single\n") + w("statement and not a file write. Iterate with " + bt + "exec" + bt + " until the script is right, then run\n") + w("the gates once over the result. The whole list after every edit costs ~55s a time and\n") + w("proves nothing the one run at the end does not.\n\n") w(bt3 + "bash\n") w(renderProjectGates(mprPath)) w(bt3 + "\n\n") diff --git a/cmd/mxcli/init_claudemd_gates_test.go b/cmd/mxcli/init_claudemd_gates_test.go index 9f23cafe74..c595cc231a 100644 --- a/cmd/mxcli/init_claudemd_gates_test.go +++ b/cmd/mxcli/init_claudemd_gates_test.go @@ -149,3 +149,48 @@ func TestModellingDefaultsAreStatedEverywhere(t *testing.T) { } } } + +// The gate list had a completeness rule ("**They are the definition of done, +// not a menu** — a change is finished when they have all been run") sitting one +// line below an escalation rule ("each is only worth paying for once the one +// above is clean"). They contradict each other, and the bolded one wins. +// +// Applied to a list containing `docker check` (~25s), `test` (~30s cold) and +// `run --local`, the bolded reading mandates ~55s of gates and 5+ tool calls per +// change — and "a change" was never defined, so in practice it became each edit. +// That is not a hypothetical reading: it is what a session-cost comparison +// measured (523 model calls vs 123 for the same class of app elsewhere, 5-8 tool +// calls per change), and the report's own line — "I tested every admin flow ... +// most of those checks included screenshots" — is this instruction being +// followed, not an agent being careless. See +// docs/11-proposals/PROPOSAL_agent_loop_efficiency.md and ako/mxcli#608. +// +// The fix is the UNIT, not the list: every gate stays (the three-copy tests +// above exist because `test` fell off this list once), and what changes is that +// the gates are done-criteria for a coherent unit of work rather than for each +// edit. Iterate with `exec`, then run the gates once over the result. +// +// This is held in all three places for the same reason the gate list itself is: +// stated in two of the three, it is guidance that applies when someone remembers. +func TestGateBatchingUnitIsStatedEverywhere(t *testing.T) { + const marker = "not per edit" + + sources := map[string]string{ + "the generated CLAUDE.md": generateClaudeMD("Demo", "Demo.mpr"), + "the bootstrap-app skill": bootstrapSkill(t), + } + const docPath = "../../docs-site/src/tools/bootstrap-prompt.md" + b, err := os.ReadFile(docPath) + if err != nil { + t.Fatalf("cannot read %s: %v", docPath, err) + } + sources[docPath] = string(b) + + for name, body := range sources { + if !strings.Contains(strings.ToLower(body), marker) { + t.Errorf("%s does not say the gates run once per change and %q — without the unit, "+ + "\"definition of done\" reads as ~55s of gates after every edit, which is the "+ + "dominant cost in an agent session", name, marker) + } + } +} diff --git a/docs-site/src/tools/bootstrap-prompt.md b/docs-site/src/tools/bootstrap-prompt.md index 1d9eaa1520..8b6f34b3e6 100644 --- a/docs-site/src/tools/bootstrap-prompt.md +++ b/docs-site/src/tools/bootstrap-prompt.md @@ -182,6 +182,12 @@ menu. The same list is in the `bootstrap-app` skill, and the three are held toge by a test, because a gate that is named in two of the three places is a gate that only runs when someone remembers to ask for it. +They run **once per change, not per edit**: a change is a coherent unit of work, not a +single statement and not a file write. Iterate with `exec`, then run the gates once +over the result. That distinction is held by a test too — without it, "definition of +done" reads as the whole list after every edit, which is ~55s and five tool calls each +time, and was the dominant cost in a measured agent session. + ```bash ./mxcli check change.mdl -p .mpr --references # syntax + references (~2s) ./mxcli exec change.mdl -p .mpr # apply From dc655fe056c62c6c265ba5508a0df3abe7c58dc8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 12:23:22 +0000 Subject: [PATCH 24/38] =?UTF-8?q?feat(diag):=20add=20`loop-report`=20?= =?UTF-8?q?=E2=80=94=20where=20a=20session's=20mxcli=20calls=20actually=20?= =?UTF-8?q?went?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The instrument the agent-loop work needs, and the first item in PROPOSAL_agent_loop_efficiency.md's sequencing, because every lever after it is a hypothesis until a number moves. Every mxcli invocation already writes a session_start naming its argv (mdl/diaglog, wired at newLoggedExecutor so it covers all commands, not a curated subset). Nobody had read it. This reports per-command call counts, wall time and median, plus `check` immediately followed by `exec` of the same script. Three things it does on purpose: Verbs resolve through rootCmd.Find rather than a hand-kept table, so a renamed or added command is picked up with no change here — a stale table would report a real command as "(unknown)", which defeats a report whose job is ranking commands by frequency. An invocation with no session_end is counted and labelled "unclosed", not "failed". mxcli exits through os.Exit on most failures and deferred Close() does not run then, so the two correlate — measured against a real log, where the 3 unclosed of 10 were exactly the 3 non-zero exits — but a killed or still-running process looks identical, so the report says "did not close" rather than asserting an exit code. Unclosed runs contribute no wall time: guessing an end from the next start would inflate the figure the report exists to make trustworthy. It prints what it cannot answer. It counts mxcli PROCESSES, not model calls; output bytes are recorded nowhere, which is the other half of the bill; and `run --local` writes no session records, so reloads vs restarts are absent. Stating those beats quietly implying the numbers are the whole picture. Also: the report filters its own invocations. diag does not log today, so this cannot be caught by running the binary — it keeps the report correct if diag ever starts. And verbStats.Unclosed is deliberately not named Failed, since loopReport.Failed means a run that closed reporting errors: one name for two populations is how a report starts lying. Tests are mutation-checked, not just green: removing the self-filter and dropping the same-script condition from the pair detector each fail their guard with the reported symptom. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- CLAUDE.md | 1 + cmd/mxcli/diag_loop_report.go | 387 ++++++++++++++++++++ cmd/mxcli/diag_loop_report_test.go | 169 +++++++++ docs-site/src/appendixes/quick-reference.md | 1 + 4 files changed, 558 insertions(+) create mode 100644 cmd/mxcli/diag_loop_report.go create mode 100644 cmd/mxcli/diag_loop_report_test.go diff --git a/CLAUDE.md b/CLAUDE.md index be2c949809..2b36d91d73 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -757,6 +757,7 @@ go build -o bin/mxcli ./cmd/mxcli | **Model repair** | `mxcli fix widgets`, `mxcli fix design-properties` | Runs `mx update-widgets` / `mx rename-design-properties` and **persists** the result without their MPR v2 → v1 collapse (harvest: let the tool convert, read the units back, restore v2, write the changed ones through mxcli's writer). Clears CE0463 / CE6087 after a headless install — measured 203 → 0 errors on a vanilla 11.12.1 app | | **Domain-model layout** | `mxcli layout -p app.mpr [--module M] [--dry-run]` | Arranges entities from the **association graph**: an entity referencing nothing is a lookup and goes left, everything else one column past the furthest thing it references, so lines run one way instead of crossing. Unconnected entities (non-persistent helpers) go in a band below rather than among the lookups. Positions are a function of the model, so a second run moves nothing. Replaces hand-arranged positions in the modules it touches — hence opt-in, with `--dry-run`; Marketplace modules and System are skipped. The **default** for an entity with no `@Position` is a wrapping grid (`mdl/dmlayout`), not the single 6,000px row it used to be | | **Diagnostics** | `mxcli diag [--bundle]` | Session logs, version info, bug report bundles | +| **Loop report** | `mxcli diag loop-report [--json]` | Where a session's mxcli calls went — per-command counts, wall time and runs that did not close — read off the session logs every invocation already writes. Counts mxcli **processes**, not model calls, and says so; output size and `run --local` reloads are not recorded anywhere, so it reports what it cannot answer rather than estimating it | | **Project brain** | `mxcli brain init\|capture\|staged\|promote\|drop\|check\|show\|plan\|resolve` | Opt-in store in `docs/brain/` for what mxcli **cannot** compute (why a pattern was chosen here, which marketplace version broke what). Sharded by module — an entry's first anchor names its file — so a session loads `project.md` plus the modules it is touching, not the whole store. Also holds the **plan**: requirements grouped into slices, whose anchors point *forward*, so `brain plan` reports progress **derived from the model** rather than from a status column. An agent captures to a git-ignored queue; a person promotes | | **New project** | `mxcli new --version X.Y.Z [--output-dir dir] [--theme none] [--layout none]` | Downloads mxbuild, creates blank project, applies default styling, scaffolds a project-owned layout, runs init, installs Linux mxcli for devcontainer | | **Default styling** | `mxcli theme list\|show\|apply\|remove` | Applies a theme (signal/ledger/console) — files under `theme/` only, the model is never touched | diff --git a/cmd/mxcli/diag_loop_report.go b/cmd/mxcli/diag_loop_report.go new file mode 100644 index 0000000000..004e7cfcea --- /dev/null +++ b/cmd/mxcli/diag_loop_report.go @@ -0,0 +1,387 @@ +// SPDX-License-Identifier: Apache-2.0 + +// diag_loop_report.go answers "where did this session's mxcli calls go?". +// +// It exists because the cost of an agent-driven session is dominated by the +// NUMBER of tool calls, not by their output: every model call re-reads the whole +// conversation, so total cost is calls x conversation size, and when a removed +// call takes its tool output with it the total falls with the square of the call +// count. A measured comparison of the same class of app built with mxcli vs on +// Vercel put the gap at 4.25x the model calls and 5.4x the conversation re-read. +// See docs/11-proposals/PROPOSAL_agent_loop_efficiency.md. +// +// Every mxcli invocation already writes a session_start record naming its argv +// (mdl/diaglog), so the distribution of calls is on disk and nobody has looked. +// This turns "the loop feels expensive" into a ranked list, and — run before and +// after a change — into evidence that the change worked. +package main + +import ( + "bufio" + "encoding/json" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "time" + + "github.com/mendixlabs/mxcli/mdl/diaglog" + "github.com/spf13/cobra" +) + +// logRecord is one JSON Lines entry. Only the fields this report reads are +// declared; diaglog writes more. +type logRecord struct { + Time time.Time `json:"time"` + Msg string `json:"msg"` + Args []string `json:"args"` + Mode string `json:"mode"` + CommandsExecuted int `json:"commands_executed"` + ErrorsCount int `json:"errors_count"` +} + +// invocation is one mxcli process: a session_start and the session_end that +// closes it, if there was one. +type invocation struct { + Verb string + Script string + Start time.Time + End time.Time + Ended bool + Commands int + Errors int +} + +// Duration is wall time for an invocation that closed. An invocation that did +// not close has no knowable duration — see Ended. +func (i invocation) Duration() time.Duration { return i.End.Sub(i.Start) } + +// verbStats aggregates one command verb. +type verbStats struct { + Verb string `json:"verb"` + // Count is invocations of this verb; Unclosed is how many of them wrote no + // session_end. Deliberately NOT named Failed: loopReport.Failed means a run + // that closed and reported errors, which is a different population, and one + // name for two meanings is how a report starts lying. + Count int `json:"count"` + Unclosed int `json:"unclosed"` + TotalDur time.Duration `json:"-"` + TotalSec float64 `json:"total_seconds"` + MedianMS int64 `json:"median_ms"` +} + +// loopReport is the whole analysis, and is what --json emits. +type loopReport struct { + Invocations int `json:"invocations"` + Failed int `json:"failed"` + Unclosed int `json:"unclosed"` + WallSeconds float64 `json:"wall_seconds"` + ByVerb []verbStats `json:"by_verb"` + CheckExecDup int `json:"check_then_exec_pairs"` + Span string `json:"span"` +} + +// parseLogRecords reads JSON Lines, skipping anything that does not parse. A +// truncated final line is normal (a process killed mid-write), so a bad line is +// not an error. +func parseLogRecords(lines []string) []logRecord { + var out []logRecord + for _, line := range lines { + line = strings.TrimSpace(line) + if line == "" { + continue + } + var r logRecord + if err := json.Unmarshal([]byte(line), &r); err != nil { + continue + } + out = append(out, r) + } + return out +} + +// invocationVerb resolves an argv to the cobra command path it invoked, so the +// verb list maintains itself: a renamed or added command is picked up with no +// change here. Falls back to the session's mode when argv names no subcommand, +// which is how the REPL and the one-shot `-c` form appear. +func invocationVerb(args []string, mode string) string { + if len(args) > 1 { + if cmd, _, err := rootCmd.Find(args[1:]); err == nil && cmd != nil && cmd != rootCmd { + return strings.TrimPrefix(cmd.CommandPath(), rootCmd.Name()+" ") + } + } + switch mode { + case "batch": + return "-c (one-shot)" + case "repl": + return "REPL" + case "": + return "(unknown)" + default: + return mode + } +} + +// invocationScript returns the first positional argument after the verb — the +// script a check or exec was given. Empty when there is none. +func invocationScript(args []string) string { + for i := 1; i < len(args); i++ { + a := args[i] + if strings.HasPrefix(a, "-") { + // A flag that takes a value consumes the next token. Treating every + // flag as valued would swallow a positional after a boolean flag, + // but mxcli's scripts are passed before their flags in every + // documented form, so the first positional is reached first. + continue + } + if strings.HasSuffix(a, ".mdl") { + return a + } + } + return "" +} + +// buildInvocations segments the record stream into processes. +// +// Only session_start carries a pid, so sessions are delimited by their own +// start records rather than correlated by pid: an invocation runs until the +// next session_end or the next session_start, whichever comes first. That is +// exact for sequential invocations, which is what an agent loop produces, and +// would mis-attribute two mxcli processes running CONCURRENTLY. The report says +// so rather than pretending otherwise. +func buildInvocations(records []logRecord) []invocation { + var out []invocation + cur := -1 + for _, r := range records { + switch r.Msg { + case "session_start": + out = append(out, invocation{ + Verb: invocationVerb(r.Args, r.Mode), + Script: invocationScript(r.Args), + Start: r.Time, + }) + cur = len(out) - 1 + case "session_end": + if cur >= 0 && !out[cur].Ended { + out[cur].End = r.Time + out[cur].Ended = true + out[cur].Commands = r.CommandsExecuted + out[cur].Errors = r.ErrorsCount + } + } + } + return out +} + +// analyzeLoop is the whole report, as a pure function of the records. +func analyzeLoop(records []logRecord) loopReport { + invs := buildInvocations(records) + + rep := loopReport{} + durs := map[string][]time.Duration{} + stats := map[string]*verbStats{} + + for _, inv := range invs { + // The report never counts itself. `diag` does not write session records + // today (it never builds a logged executor), but a report whose own + // numbers depend on that staying true would drift silently the moment it + // changed — so this filters rather than assumes. + if inv.Verb == "diag" || strings.HasPrefix(inv.Verb, "diag ") { + continue + } + s, ok := stats[inv.Verb] + if !ok { + s = &verbStats{Verb: inv.Verb} + stats[inv.Verb] = s + } + s.Count++ + + // An invocation with no session_end exited through os.Exit, which is how + // mxcli reports almost every failure — Close() is deferred and deferred + // calls do not run on os.Exit. Verified against a real log: the three + // unclosed sessions were exactly the three runs that exited non-zero. + // A process killed or still running looks identical, so this is reported + // as "did not close" rather than asserted as "failed". + if !inv.Ended { + s.Unclosed++ + rep.Unclosed++ + continue + } + if inv.Errors > 0 { + rep.Failed++ + } + d := inv.Duration() + s.TotalDur += d + durs[inv.Verb] = append(durs[inv.Verb], d) + rep.WallSeconds += d.Seconds() + } + + for verb, s := range stats { + ds := durs[verb] + sort.Slice(ds, func(i, j int) bool { return ds[i] < ds[j] }) + if len(ds) > 0 { + s.MedianMS = ds[len(ds)/2].Milliseconds() + } + s.TotalSec = s.TotalDur.Seconds() + rep.ByVerb = append(rep.ByVerb, *s) + } + // Most invocations first; ties by name so the output is stable. + sort.Slice(rep.ByVerb, func(i, j int) bool { + if rep.ByVerb[i].Count != rep.ByVerb[j].Count { + return rep.ByVerb[i].Count > rep.ByVerb[j].Count + } + return rep.ByVerb[i].Verb < rep.ByVerb[j].Verb + }) + + for _, s := range stats { + rep.Invocations += s.Count + } + rep.CheckExecDup = countCheckThenExec(invs) + if len(invs) > 0 { + rep.Span = invs[0].Start.Format(time.RFC3339) + " .. " + + invs[len(invs)-1].Start.Format(time.RFC3339) + } + return rep +} + +// countCheckThenExec counts a `check` immediately followed by an `exec` of the +// SAME script — the two-call form the generated CLAUDE.md used to teach. +// +// It is deliberately not called "waste": since ako/mxcli#607 `exec` resolves +// references itself, so what the extra call still buys is project-conflict +// detection (a plain CREATE over an existing document), which exec does not do. +// The number is here to be weighed, not to be eliminated on sight. +func countCheckThenExec(invs []invocation) int { + n := 0 + for i := 0; i+1 < len(invs); i++ { + a, b := invs[i], invs[i+1] + if a.Verb == "check" && b.Verb == "exec" && a.Script != "" && a.Script == b.Script { + n++ + } + } + return n +} + +// readLogLines reads every log file in dir, oldest first, so the records come +// back in time order across a multi-day retention window. +func readLogLines(dir string) ([]string, error) { + entries, err := os.ReadDir(dir) + if err != nil { + return nil, err + } + var names []string + for _, e := range entries { + if !e.IsDir() && strings.HasPrefix(e.Name(), "mxcli-") && strings.HasSuffix(e.Name(), ".log") { + names = append(names, e.Name()) + } + } + sort.Strings(names) // mxcli-YYYY-MM-DD.log sorts chronologically + + var lines []string + for _, name := range names { + f, err := os.Open(filepath.Join(dir, name)) + if err != nil { + continue + } + sc := bufio.NewScanner(f) + sc.Buffer(make([]byte, 0, 64*1024), 4*1024*1024) // argv can be long + for sc.Scan() { + lines = append(lines, sc.Text()) + } + f.Close() + } + return lines, nil +} + +func renderLoopReport(rep loopReport, w *os.File) { + fmt.Fprintf(w, "mxcli invocations: %d", rep.Invocations) + if rep.Span != "" { + fmt.Fprintf(w, " (%s)", rep.Span) + } + fmt.Fprintln(w) + if rep.Invocations == 0 { + fmt.Fprintln(w, "\nNo session records found. Logging is on unless MXCLI_LOG=0,") + fmt.Fprintln(w, "and logs are kept for 7 days.") + return + } + fmt.Fprintf(w, "Wall time in mxcli: %.1fs across the runs that closed\n", rep.WallSeconds) + if rep.Unclosed > 0 { + fmt.Fprintf(w, "Did not close: %d (mxcli exits through os.Exit on most failures,\n"+ + " which skips the summary record — so these are very likely\n"+ + " non-zero exits, but a killed process looks the same)\n", rep.Unclosed) + } + + fmt.Fprintln(w, "\nBy command, most calls first:") + fmt.Fprintf(w, " %-22s %6s %8s %10s %9s\n", "COMMAND", "CALLS", "UNCLOSED", "TOTAL", "MEDIAN") + for _, s := range rep.ByVerb { + fmt.Fprintf(w, " %-22s %6d %8d %9.1fs %8dms\n", + s.Verb, s.Count, s.Unclosed, s.TotalSec, s.MedianMS) + } + + if rep.CheckExecDup > 0 { + fmt.Fprintf(w, "\n`check` immediately followed by `exec` of the same script: %d\n", rep.CheckExecDup) + fmt.Fprintln(w, " Since #607 exec resolves references itself, so the second call buys") + fmt.Fprintln(w, " project-conflict detection and the no-apply preflight. Worth weighing,") + fmt.Fprintln(w, " not eliminating on sight.") + } + + fmt.Fprintln(w, "\nWhat this cannot tell you:") + fmt.Fprintln(w, " - model calls. One bash call can run several mxcli commands, and the") + fmt.Fprintln(w, " agent's other calls are not here at all. This counts mxcli processes.") + fmt.Fprintln(w, " - output size. Nothing records how many bytes a command printed, which") + fmt.Fprintln(w, " is the other half of the bill.") + fmt.Fprintln(w, " - reloads vs restarts. `run --local` does not write session records.") +} + +var diagLoopReportCmd = &cobra.Command{ + Use: "loop-report", + Short: "Report where this project's mxcli calls went, from the session logs", + Long: `Summarise the session logs as a per-command call count, wall time and failure count. + +Every mxcli invocation writes a session record naming its argv, so the shape of +an agent's loop is already on disk. This reports it: which commands were run, +how often, how long they took, and how many did not exit cleanly. + +It counts mxcli PROCESSES, not model calls — one shell command can run several. +Read it to find which command dominates a loop, and re-run it after a change to +see whether the loop actually moved. + +Examples: + mxcli diag loop-report + mxcli diag loop-report --json + mxcli diag loop-report --log-dir ./collected-logs +`, + Args: cobra.NoArgs, + Run: func(cmd *cobra.Command, _ []string) { + dir, _ := cmd.Flags().GetString("log-dir") + if dir == "" { + dir = diaglog.LogDir() + } + asJSON, _ := cmd.Flags().GetBool("json") + + lines, err := readLogLines(dir) + if err != nil { + fmt.Fprintf(os.Stderr, "Error reading logs from %s: %v\n", dir, err) + os.Exit(1) + } + rep := analyzeLoop(parseLogRecords(lines)) + + if asJSON { + enc := json.NewEncoder(os.Stdout) + enc.SetIndent("", " ") + if err := enc.Encode(rep); err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + return + } + renderLoopReport(rep, os.Stdout) + }, +} + +func init() { + diagLoopReportCmd.Flags().Bool("json", false, "Emit the report as JSON") + diagLoopReportCmd.Flags().String("log-dir", "", "Read logs from this directory instead of ~/.mxcli/logs") + diagCmd.AddCommand(diagLoopReportCmd) +} diff --git a/cmd/mxcli/diag_loop_report_test.go b/cmd/mxcli/diag_loop_report_test.go new file mode 100644 index 0000000000..b0fe452e67 --- /dev/null +++ b/cmd/mxcli/diag_loop_report_test.go @@ -0,0 +1,169 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "strings" + "testing" + "time" +) + +// rec builds a session_start line for verb-resolution and segmentation tests. +func startAt(sec int, mode string, args ...string) logRecord { + return logRecord{ + Time: time.Date(2026, 9, 22, 12, 0, sec, 0, time.UTC), + Msg: "session_start", + Mode: mode, + Args: append([]string{"./mxcli"}, args...), + } +} + +func endAt(sec, cmds, errs int) logRecord { + return logRecord{ + Time: time.Date(2026, 9, 22, 12, 0, sec, 0, time.UTC), + Msg: "session_end", + CommandsExecuted: cmds, + ErrorsCount: errs, + } +} + +// The verb comes from cobra rather than a hand-kept list, so a renamed or added +// command is picked up with no change to the report. A hardcoded table is the +// failure this avoids: it goes stale silently, reporting a real command as +// "(unknown)" — and the whole point is to rank commands by how often they run. +func TestInvocationVerbResolvesThroughCobra(t *testing.T) { + for _, tc := range []struct { + name, mode, want string + args []string + }{ + {"subcommand", "subcommand", "exec", []string{"exec", "s.mdl", "-p", "a.mpr"}}, + {"nested subcommand", "subcommand", "docker check", []string{"docker", "check", "-p", "a.mpr"}}, + {"one-shot -c", "batch", "-c (one-shot)", []string{"-p", "a.mpr", "-c", "show entities;"}}, + {"repl", "repl", "REPL", []string{}}, + } { + t.Run(tc.name, func(t *testing.T) { + got := invocationVerb(append([]string{"./mxcli"}, tc.args...), tc.mode) + if got != tc.want { + t.Errorf("verb = %q, want %q", got, tc.want) + } + }) + } + + // Guard the guard: if cobra resolution silently stopped working, every verb + // above would fall back to its mode and the table would still look sane for + // the flag-only cases. This one can only pass through cobra. + if got := invocationVerb([]string{"./mxcli", "docker", "check"}, "subcommand"); got != "docker check" { + t.Fatalf("cobra resolution is not running: got %q", got) + } +} + +// An invocation that did not write session_end is how mxcli reports almost every +// failure: it exits through os.Exit, and deferred Close() does not run then. +// +// This is MEASURED, not inferred. Against a real log of 10 invocations, the 3 +// without a session_end were exactly the 3 runs that exited non-zero (two +// refusals and a parse error); the 7 that closed were the 7 that succeeded. +func TestUnclosedInvocationsAreCountedSeparately(t *testing.T) { + rep := analyzeLoop([]logRecord{ + startAt(0, "subcommand", "exec", "a.mdl", "-p", "x.mpr"), + endAt(2, 3, 0), // closed, clean + startAt(3, "subcommand", "exec", "b.mdl", "-p", "x.mpr"), + // no end: exited non-zero + startAt(5, "subcommand", "exec", "c.mdl", "-p", "x.mpr"), + endAt(6, 1, 2), // closed, but reported errors + }) + + if rep.Invocations != 3 { + t.Errorf("Invocations = %d, want 3", rep.Invocations) + } + if rep.Unclosed != 1 { + t.Errorf("Unclosed = %d, want 1 — the run with no session_end", rep.Unclosed) + } + if rep.Failed != 1 { + t.Errorf("Failed = %d, want 1 — the closed run whose summary reported errors", rep.Failed) + } + // Wall time counts only the runs that closed: an unclosed run has no knowable + // end, and guessing one (say, the next start) would silently inflate the + // figure the report exists to make trustworthy. + if rep.WallSeconds != 3 { + t.Errorf("WallSeconds = %v, want 3 (2s + 1s; the unclosed run contributes nothing)", rep.WallSeconds) + } +} + +// The pair this counts is `check` immediately followed by `exec` of the SAME +// script. Both halves of that are load-bearing, so both have a control. +func TestCheckThenExecPairNeedsSameScriptAndOrder(t *testing.T) { + pair := []logRecord{ + startAt(0, "subcommand", "check", "a.mdl", "-p", "x.mpr"), + endAt(1, 1, 0), + startAt(2, "subcommand", "exec", "a.mdl", "-p", "x.mpr"), + endAt(3, 1, 0), + } + if got := analyzeLoop(pair).CheckExecDup; got != 1 { + t.Errorf("check then exec of the same script counted %d, want 1", got) + } + + // CONTROL 1: different scripts are not a pair — checking one file and + // applying another is two pieces of work, not a doubled call. + diff := []logRecord{ + startAt(0, "subcommand", "check", "a.mdl", "-p", "x.mpr"), + endAt(1, 1, 0), + startAt(2, "subcommand", "exec", "b.mdl", "-p", "x.mpr"), + endAt(3, 1, 0), + } + if got := analyzeLoop(diff).CheckExecDup; got != 0 { + t.Errorf("different scripts counted %d, want 0", got) + } + + // CONTROL 2: order matters. exec-then-check is re-validating after applying, + // which is a different (and defensible) habit. + rev := []logRecord{ + startAt(0, "subcommand", "exec", "a.mdl", "-p", "x.mpr"), + endAt(1, 1, 0), + startAt(2, "subcommand", "check", "a.mdl", "-p", "x.mpr"), + endAt(3, 1, 0), + } + if got := analyzeLoop(rev).CheckExecDup; got != 0 { + t.Errorf("exec then check counted %d, want 0", got) + } +} + +// A report that counted its own invocations would climb every time it was read. +// `diag` does not write session records today, so this cannot be caught by +// running the binary — it guards the filter that makes the report correct even +// if diag later starts logging. +func TestLoopReportNeverCountsItself(t *testing.T) { + rep := analyzeLoop([]logRecord{ + startAt(0, "subcommand", "exec", "a.mdl", "-p", "x.mpr"), + endAt(1, 1, 0), + startAt(2, "subcommand", "diag", "loop-report"), + endAt(3, 0, 0), + }) + if rep.Invocations != 1 { + t.Errorf("Invocations = %d, want 1 — the diag run must not be counted", rep.Invocations) + } + for _, s := range rep.ByVerb { + if strings.HasPrefix(s.Verb, "diag") { + t.Errorf("the report lists its own command %q", s.Verb) + } + } +} + +// A truncated final line is normal: a process killed mid-write leaves one. It +// must not take the whole report down, because the logs are read precisely when +// something went wrong. +func TestParseLogRecordsSkipsUnparseableLines(t *testing.T) { + lines := []string{ + `{"time":"2026-09-22T12:00:00Z","msg":"session_start","mode":"repl","args":["./mxcli"]}`, + `{"time":"2026-09-22T12:00:01Z","msg":"sessio`, // truncated + ``, + `{"time":"2026-09-22T12:00:02Z","msg":"session_end","commands_executed":1,"errors_count":0}`, + } + got := parseLogRecords(lines) + if len(got) != 2 { + t.Fatalf("parsed %d records, want 2 (the truncated and empty lines skipped)", len(got)) + } + if rep := analyzeLoop(got); rep.Invocations != 1 { + t.Errorf("Invocations = %d, want 1", rep.Invocations) + } +} diff --git a/docs-site/src/appendixes/quick-reference.md b/docs-site/src/appendixes/quick-reference.md index bd8c9c7129..24e4914f8f 100644 --- a/docs-site/src/appendixes/quick-reference.md +++ b/docs-site/src/appendixes/quick-reference.md @@ -552,6 +552,7 @@ Cross-reference commands require `REFRESH CATALOG FULL` to populate reference da | Docker build | `mxcli docker build -p app.mpr` | Build with PAD patching | | Docker check | `mxcli docker check -p app.mpr` | Validate with `mx check` | | Diagnostics | `mxcli diag [--bundle]` | Session logs, version info | +| Loop report | `mxcli diag loop-report [--json]` | Which mxcli commands a session actually ran, how often, how long | | New project | `mxcli new --version X.Y.Z` | Create project from scratch with all tooling | | Init project | `mxcli init /path/to/project` | Add AI tooling to existing project | | Setup mxcli | `mxcli setup mxcli [--os linux]` | Download platform-specific mxcli binary | From c7d4ce3c92e2783f214e2a27e1e3ae1c0da2c46c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 12:42:26 +0000 Subject: [PATCH 25/38] feat(exec): collapse a run's repeated "Unchanged" reports into one line MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lever 2 of PROPOSAL_agent_loop_efficiency.md. "Unchanged …" is the most repeated thing mxcli prints, and a settled re-run is nothing else. MEASURED on a 40-statement script applied to a project already holding it: 41 lines / 1,604 B, every one saying nothing happened. After: 2 lines / 177 B. That is cheap in a terminal and charged repeatedly in an agent session, where a tool result is written into the conversation once and RE-READ by every later model call — so a no-op line costs its length times the number of calls that follow it. Two rules keep the collapse honest, and each came from getting it wrong. Only `Unchanged` collapses. It is the one verb that by construction reports an absence (storage was offered a write and skipped it, ADR-0008), so no line a reader would act on is replaced by a number: a mixed run still names every real write individually and counts only the rest — measured, 2 created + 1 modified + 39 elided prints the three and one summary. The obvious alternative, collapsing on volume, would hide real writes in exactly the runs where they matter most. The trigger is how many elisions arrive, not which entry point ran. A lone one is printed verbatim, since "1 document already in sync" is worse than the line it replaces. The first implementation gated on "is this a program run?", which looked equivalent and was not: `-c` reaches ExecuteProgram too, because executeMDL prepends a CONNECT statement, so a one-liner collapsed to a count of one. Caught by running it, not by the tests, and now has a test naming the measurement. Nested EXECUTE SCRIPT does not emit a second summary mid-run: begin() reports whether the call owns the tally, and only the outermost flushes. Guards are mutation-checked — dropping the held line makes the lone-line test fail with an empty report, and the control test asserts both landed writes are still named while two elided ones are not. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- CLAUDE.md | 21 ++- docs-site/src/internals/idempotent-writes.md | 14 ++ mdl/executor/exec_context.go | 5 + mdl/executor/executor.go | 24 ++++ mdl/executor/executor_dispatch.go | 1 + mdl/executor/mutation_tally.go | 92 ++++++++++++ mdl/executor/mutation_tally_test.go | 140 +++++++++++++++++++ mdl/executor/report_mutation.go | 6 + 8 files changed, 302 insertions(+), 1 deletion(-) create mode 100644 mdl/executor/mutation_tally.go create mode 100644 mdl/executor/mutation_tally_test.go diff --git a/CLAUDE.md b/CLAUDE.md index 2b36d91d73..6e0a666460 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -420,6 +420,25 @@ all elided prints `Unchanged nanoflow: …` instead of `Replaced nanoflow: …` downgraded on positive evidence — writes offered, none landed — so a mutation that never touches unit storage is reported exactly as before. +**Several elisions in one run collapse into one line**, because that report is the +most repeated thing mxcli prints and an agent pays for it on every later model +call (a tool result is written into the conversation once and re-read by each one). +Measured on a settled 40-statement script: 41 lines / 1,604 B became 2 lines / +177 B. Two rules keep it honest, and each was arrived at by getting it wrong: + +1. **Only `Unchanged` collapses.** It is the one verb that by construction reports + an absence, so no line a reader would act on is ever replaced by a number — + a mixed run still names every real write individually and counts only the rest. + Collapsing on volume instead ("after N lines") would hide real writes in exactly + the runs where they matter. +2. **The trigger is how many arrive, not which entry point ran.** A lone elision is + printed verbatim, since "1 document already in sync" is worse than the line it + replaces. Gating on "is this a script?" looked equivalent and is not: `-c` + reaches `ExecuteProgram` too, because `executeMDL` prepends a `CONNECT` + statement, so a one-liner collapsed to a count of one. + +`mutationTally` (`mdl/executor/mutation_tally.go`), active only inside a program run. + ### The Tunnel Is Linux-Only, On Purpose — Do Not "Restore" It `mxcli run --hub` and `mxcli tunnel-hub` embed [chisel](https://github.com/jpillora/chisel), @@ -861,7 +880,7 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati - Project brain (`mxcli brain init/capture/staged/promote/drop/check/show`): an **opt-in** store in `docs/brain/` for the project knowledge mxcli cannot compute. The governing rule is that anything derivable from the model is answered by a command and never written down — a note that transcribes the model disagrees with it silently. Records shard by **anchor scope**: an entry's first anchor names its file (`@Sales.Order` → `modules/Sales.md`), an anchorless entry is cross-cutting (`project.md`), and there is no index to maintain because the module prefix *is* the file name. That is what makes the cap per-shard rather than a project-wide budget, and lets a session load `project.md` plus the modules it is touching. `check` answers two independent questions: each anchor is **resolved / not found / not indexable** — only the middle one fails, and the third exists because the catalog's `objects` view covers the describable types only, so a scheduled event would otherwise read as *missing* (separated with `FindDocumentUnit`, which cannot miss a kind because it never asks what kind anything is). Misfiling is a **second axis, not a fourth state**: every anchor can resolve and the entry still be in the wrong file, and it is only decided when something resolved — judging it on an all-not-indexable entry reintroduced the same false staleness through the other axis (caught by a test, with the guard stubbed as the control). An agent `capture`s to a git-ignored queue and a person `promote`s; the queue is deliberately **not** sharded, because routing it would force the file decision before a human has looked at the entry. `mxcli lint` prints the unpromoted-queue count, because a report only `brain check` prints is a report nothing demands. Sizes are computed by `brain show` and never written into a committed file. A second record kind, **requirement**, lives in `plan/.md` and inverts the anchor's meaning: a decision's anchor points backward (not resolving = stale, fails), a requirement's points forward (not resolving = not built yet, passes). Measured: filed as an ordinary entry, one unbuilt requirement takes `brain check` to exit 1 — which is why it is a separate kind rather than more entries in the same files. That inversion is also what makes `brain plan` a real progress report: a requirement is *built* when its anchors resolve, so creating the microflow it names moves the count with the plan file untouched (measured 0/1 → 1/0). A status written beside a requirement is therefore refused by the skill, not just discouraged. Slices are ordered by name (`01-accounts`), span modules by design (so misfiling does not apply), and carry a generous cap that enforces the slicing discipline — a slice too long to read should be split. A third kind, **open question** (`--open`), records what is *not* decided; its anchors are deliberately **not** checked, since the question is often whether the thing should exist at all — measured, the identical anchor exits 1 as a decision and 0 as a question. `brain resolve` converts one into a decision in place, keeping its id and position and starting to check its anchors, which is the transition the kind exists for. Unanswered questions are reported by `brain check` and by `mxcli lint`. The skill also gives capture a **trigger** rather than good intentions — a correction you have had to make twice — because the decisions half otherwise under-fills while the plan half fills at bootstrap. `bootstrap-app` asks for requirements at the interview and records them by default. Package: `cmd/mxcli/brain/`. See `docs-site/src/tools/project-brain.md` and `docs/11-proposals/PROPOSAL_project_brain.md` - Default styling + runtime theme switching (`mxcli theme list/show/create/apply/remove/switcher`, `mxcli new --theme`): three embedded themes (**signal** light-first, **ledger** light-first, **console** dark-first), each a palette in `theme/web/custom-variables.scss` + a shared Atlas wiring partial + a theme partial imported from `theme/web/main.scss` (which compiles last), plus vendored fonts. **No model changes**, so it hot-applies under `run --local --watch` and cannot affect a build. Generated regions are digest-fenced: a block carrying local edits is refused rather than overwritten. Applying a theme removes the previous one. `--variant auto` (default) ships both palettes — the app follows `prefers-color-scheme` before first paint and honours a `theme-light`/`theme-dark` class on ``; `light`/`dark` bakes one. `theme switcher install` is the only part that writes to the model (JS actions + a nanoflow for a toggle button). A project can add its own themes under `theme/mxcli-themes//` (committed, not compiled); `theme create [--from ]` scaffolds one from an existing theme, renaming the identifiers built from the name and optionally seeding the palette from `--mxt-*` declarations in a design artifact. A local theme shadows a built-in of the same name. Package: `cmd/mxcli/theme/`. See `docs/11-proposals/PROPOSAL_default_styling.md` - MPR v1/v2 reading and writing -- Idempotent writes (ADR-0008): a unit whose new content is semantically equal to what is stored is **not written**, so re-running an MDL script against an in-sync project leaves the `.mpr` and `mprcontents/` byte-identical and Studio Pro shows no version-control changes. Comparison is on a canonical form (element `$ID`s normalised away — a rebuild mints them randomly, so byte comparison would skip nothing); `Microflows$Microflow.StableId` is carried from the stored document rather than re-minted, because the build derives every client-callable microflow's operation id from it. When a write **does** land, `canon.TransplantIDs` matches the rebuild against the stored document and reuses its element `$ID`s (rewriting every pointer in the same pass), so a changed document's diff is the change rather than a wholesale replacement — measured on #910's nanoflow: 1 of 37 identities survived an argument edit before, 37 of 37 after, and a change plus its revert returns to the original bytes. Inserting or deleting an activity mints IDs only for the genuinely new elements. One policy in `modelsdk/canon`, called from both engines' write choke points. `MXCLI_ALWAYS_WRITE=1` disables elision (not preservation) for bisecting — which means it no longer changes the resulting bytes, only the mtimes. The executor's output distinguishes the two: `Unchanged nanoflow: …` where the write was skipped. See `docs-site/src/internals/idempotent-writes.md` +- Idempotent writes (ADR-0008): a unit whose new content is semantically equal to what is stored is **not written**, so re-running an MDL script against an in-sync project leaves the `.mpr` and `mprcontents/` byte-identical and Studio Pro shows no version-control changes. Comparison is on a canonical form (element `$ID`s normalised away — a rebuild mints them randomly, so byte comparison would skip nothing); `Microflows$Microflow.StableId` is carried from the stored document rather than re-minted, because the build derives every client-callable microflow's operation id from it. When a write **does** land, `canon.TransplantIDs` matches the rebuild against the stored document and reuses its element `$ID`s (rewriting every pointer in the same pass), so a changed document's diff is the change rather than a wholesale replacement — measured on #910's nanoflow: 1 of 37 identities survived an argument edit before, 37 of 37 after, and a change plus its revert returns to the original bytes. Inserting or deleting an activity mints IDs only for the genuinely new elements. One policy in `modelsdk/canon`, called from both engines' write choke points. `MXCLI_ALWAYS_WRITE=1` disables elision (not preservation) for bisecting — which means it no longer changes the resulting bytes, only the mtimes. The executor's output distinguishes the two: `Unchanged nanoflow: …` where the write was skipped — and several such reports in one run collapse into a single `N documents already in sync` line, since a lone one is still printed in full. See `docs-site/src/internals/idempotent-writes.md` - Domain model (entities, attributes, associations) - ALTER ENTITY (add/rename/modify/drop attributes, indexes, documentation) - Microflows/Nanoflows with 60+ activity types, JavaScript action calls, nanoflow validation parity diff --git a/docs-site/src/internals/idempotent-writes.md b/docs-site/src/internals/idempotent-writes.md index 49a63817a4..96f65fbe10 100644 --- a/docs-site/src/internals/idempotent-writes.md +++ b/docs-site/src/internals/idempotent-writes.md @@ -118,6 +118,20 @@ Two cautions, both of which produce a meaningless zero: The console tells you the same thing, per document: a statement whose write was skipped reports `Unchanged nanoflow: …` rather than `Replaced nanoflow: …`. +When a run skips **several**, they collapse into one line rather than one per +statement: + +``` +Modified entity: MyFirstModule.Od07 +Created entity: MyFirstModule.OdNew1 +39 documents already in sync (unchanged, not listed) +``` + +Every write that actually landed is still named individually — only `Unchanged` +is counted, because it is the one report that by construction says nothing +happened. A run with exactly one elision prints it in full, so nothing is ever +replaced by a count of one. + For a per-unit view of what would be skipped, `scripts/mprsnapshot -canon` emits canonical digests keyed by unit id. diff --git a/mdl/executor/exec_context.go b/mdl/executor/exec_context.go index 7b689d06e8..b225f2236a 100644 --- a/mdl/executor/exec_context.go +++ b/mdl/executor/exec_context.go @@ -101,6 +101,11 @@ type ExecContext struct { // in sync"; see report_mutation.go. lastWriteStats backend.WriteStats + // tally collapses a program run's "Unchanged" reports into one summary + // line. Shared with the Executor (a pointer, so it survives across the + // per-statement contexts) and nil outside a program run. + tally *mutationTally + // ScriptDepth tracks the current EXECUTE SCRIPT nesting level. // Incremented on each recursive call; execExecuteScript rejects calls // that exceed maxScriptDepth to prevent infinite self-referencing scripts. diff --git a/mdl/executor/executor.go b/mdl/executor/executor.go index f986c359ab..adcb6d59b0 100644 --- a/mdl/executor/executor.go +++ b/mdl/executor/executor.go @@ -249,6 +249,7 @@ type Executor struct { cache *executorCache catalog *catalog.Catalog quiet bool // suppress connection and status messages + tally *mutationTally // collapses a program run's "Unchanged" reports into one line format OutputFormat // output format (table, json) logger *diaglog.Logger // session diagnostics logger (nil = no logging) tracer *backend.Tracer // MCP tool-call tracer (--mcp-trace; nil = off) @@ -360,6 +361,10 @@ func (e *Executor) Execute(stmt ast.Statement) error { // ExecuteProgram runs all statements in a program. func (e *Executor) ExecuteProgram(prog *ast.Program) error { + if e.beginTally() { + defer e.flushTally() + } + // Collect all names defined in the script for forward-reference hints. allDefined := newScriptContext() allDefined.collectDefinitions(prog) @@ -392,6 +397,10 @@ type ExecuteProgramResult struct { // visible (the caller is expected to exit non-zero when Failed > 0). ErrExit is // honoured (stops the run) and returned, so `exit`/`quit` still work. func (e *Executor) ExecuteProgramContinueOnError(prog *ast.Program, w io.Writer) (ExecuteProgramResult, error) { + if e.beginTally() { + defer e.flushTally() + } + allDefined := newScriptContext() allDefined.collectDefinitions(prog) created := newScriptContext() @@ -551,3 +560,18 @@ func consumeDroppedNanoflow(ctx *ExecContext, qualifiedName string) *droppedUnit delete(ctx.Cache.droppedNanoflows, qualifiedName) return info } + +// beginTally starts collapsing this program run's "Unchanged" reports, and +// reports whether this call owns the tally (see mutationTally.begin). +func (e *Executor) beginTally() bool { + if e.tally == nil { + e.tally = &mutationTally{} + } + return e.tally.begin() +} + +// flushTally writes the run's one-line summary and stands the tally down. +func (e *Executor) flushTally() { + e.tally.flush(e.output) + e.tally.end() +} diff --git a/mdl/executor/executor_dispatch.go b/mdl/executor/executor_dispatch.go index bec03dc4d8..444a92da1a 100644 --- a/mdl/executor/executor_dispatch.go +++ b/mdl/executor/executor_dispatch.go @@ -94,6 +94,7 @@ func (e *Executor) newExecContext(ctx context.Context) *ExecContext { Output: e.output, Format: e.format, Quiet: e.quiet, + tally: e.tally, Logger: e.logger, Fragments: e.fragments, Catalog: cat, diff --git a/mdl/executor/mutation_tally.go b/mdl/executor/mutation_tally.go new file mode 100644 index 0000000000..67b6dcd994 --- /dev/null +++ b/mdl/executor/mutation_tally.go @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "fmt" + "io" +) + +// mutationTally collapses the "Unchanged …" lines of a program run into one +// summary, and counts nothing else. +// +// Re-running a settled script printed one line per statement saying that nothing +// had happened — measured at 40 lines and ~1.5 KB for a 40-statement script. In +// a terminal that is merely noisy. In an agent session it is charged repeatedly: +// a tool result is written into the conversation once and RE-READ by every later +// model call, so the cost of a no-op line is its length times the number of calls +// that follow it (docs/11-proposals/PROPOSAL_agent_loop_efficiency.md). +// +// Only "Unchanged" is suppressed, and that narrowness is the whole safety +// argument. It is the one verb that by construction reports an absence: storage +// was offered a write and skipped it (ADR-0008). Every verb naming a real write +// is still printed individually and in full, so nothing a reader would act on is +// replaced by a number. Suppressing on volume instead — "collapse after N lines" +// — would have hidden real writes in exactly the runs where they matter most. +type mutationTally struct { + // active is set for the duration of a program run. + // + // It is NOT what decides whether anything collapses. A single elided + // mutation is held and printed verbatim at flush, because "1 document + // already in sync" is strictly worse than the line it replaces — and that + // has to be decided by how many arrive, not by which entry point ran. A + // `-c` one-liner reaches ExecuteProgram too (main.go prepends CONNECT), so + // gating on the entry point collapsed exactly the case the rule exists to + // protect. + active bool + unchanged int + // first holds the one elided line seen so far, so it can still be printed + // in full if no second one arrives. + first string +} + +// begin activates the tally and reports whether THIS call owns it. A nested run +// (EXECUTE SCRIPT inside a script) finds it already active and returns false, so +// only the outermost program flushes and a nested script does not emit a second +// summary mid-run. +func (t *mutationTally) begin() bool { + if t == nil || t.active { + return false + } + t.active = true + t.unchanged = 0 + return true +} + +// end deactivates the tally after the owning run has flushed it. +func (t *mutationTally) end() { + if t != nil { + t.active = false + } +} + +// countUnchanged records an elided mutation and reports whether the caller +// should stay quiet about it for now. The line is passed in because a run with +// exactly one elision prints it verbatim at flush rather than a count of one. +func (t *mutationTally) countUnchanged(line string) bool { + if t == nil || !t.active { + return false + } + t.unchanged++ + if t.unchanged == 1 { + t.first = line + } + return true +} + +// flush writes the one-line summary, if there is anything to summarise. A run +// with nothing elided prints nothing: a trailing "0 unchanged" on every clean +// first run is the same noise from the other side. +func (t *mutationTally) flush(w io.Writer) { + if t == nil || !t.active || t.unchanged == 0 { + return + } + if t.unchanged == 1 { + // Nothing was gained by holding it: print the line as it always was. + fmt.Fprint(w, t.first) + } else { + fmt.Fprintf(w, "%d documents already in sync (unchanged, not listed)\n", t.unchanged) + } + t.unchanged = 0 + t.first = "" +} diff --git a/mdl/executor/mutation_tally_test.go b/mdl/executor/mutation_tally_test.go new file mode 100644 index 0000000000..a172a081a3 --- /dev/null +++ b/mdl/executor/mutation_tally_test.go @@ -0,0 +1,140 @@ +// SPDX-License-Identifier: Apache-2.0 + +package executor + +import ( + "strings" + "testing" +) + +// A settled script reports one "Unchanged …" line per statement, and those lines +// are the single most repeated thing mxcli prints. MEASURED on a 40-statement +// script against a project it had already been applied to: 40 lines, ~1.5 KB, +// every one of them saying nothing happened. +// +// That is cheap in a terminal and expensive in an agent session, where a tool +// result is written into the conversation once and RE-READ by every later model +// call — so 40 no-op lines early in a long session are not 40 lines of cost. +// See docs/11-proposals/PROPOSAL_agent_loop_efficiency.md. +// +// The collapse is deliberately narrow: only "Unchanged" is suppressed, because +// it is the one verb that by construction reports that nothing happened. Every +// verb that describes a real write is still printed, individually and in full — +// an agent (or a person) never loses a line they would have acted on. + +func TestUnchangedLinesCollapseIntoOneSummary(t *testing.T) { + ctx, mb, out := reportCtx(t) + tally := &mutationTally{active: true} + ctx.tally = tally + + for i := 0; i < 40; i++ { + mb.offer(1, 0) // offered and elided: this is an "Unchanged" + ctx.ReportMutation("Created", "entity: MyFirstModule.Od%02d", i) + } + tally.flush(ctx.Output) + + got := out.String() + if n := strings.Count(got, "\n"); n != 1 { + t.Errorf("40 unchanged statements printed %d lines, want 1:\n%s", n, got) + } + if !strings.Contains(got, "40") { + t.Errorf("the summary does not say how many were unchanged: %q", got) + } +} + +// THE CONTROL. Without it the test above passes against an implementation that +// swallows everything — which would be a far worse bug than the noise it fixes, +// because the run would look idempotent while rewriting the project. +func TestEveryRealWriteIsStillPrintedInFull(t *testing.T) { + ctx, mb, out := reportCtx(t) + tally := &mutationTally{active: true} + ctx.tally = tally + + mb.offer(1, 1) // landed + ctx.ReportMutation("Created", "entity: %s", "A") + mb.offer(1, 0) // elided + ctx.ReportMutation("Created", "entity: %s", "B") + mb.offer(1, 1) // landed + ctx.ReportMutation("Replaced", "nanoflow: %s", "C") + mb.offer(1, 0) // elided — two of them, so they collapse + ctx.ReportMutation("Created", "entity: %s", "D") + tally.flush(ctx.Output) + + got := out.String() + for _, want := range []string{"Created entity: A", "Replaced nanoflow: C"} { + if !strings.Contains(got, want) { + t.Errorf("a write that landed was not reported: want %q in\n%s", want, got) + } + } + for _, unwanted := range []string{"entity: B", "entity: D"} { + if strings.Contains(got, unwanted) { + t.Errorf("an elided write was named individually; it should only be counted:\n%s", got) + } + } + if !strings.Contains(got, "2 documents already in sync") { + t.Errorf("the summary does not report the 2 unchanged documents: %q", got) + } +} + +// Nothing is suppressed outside a program run. +func TestNothingIsCollapsedWhenTheTallyIsInactive(t *testing.T) { + ctx, mb, out := reportCtx(t) + ctx.tally = &mutationTally{active: false} + + mb.offer(1, 0) + ctx.ReportMutation("Created", "entity: %s", "A") + + if got := out.String(); got != "Unchanged entity: A\n" { + t.Errorf("reported %q, want the per-statement line kept when inactive", got) + } +} + +// A SINGLE elided mutation is printed in full, even inside a program run. +// "1 document already in sync" is strictly worse than the line it replaces, and +// this is the case an entry-point-based rule got wrong: a `-c` one-liner reaches +// ExecuteProgram too, because main.go prepends a CONNECT statement, so gating on +// "is this a script?" collapsed exactly the case worth protecting. Measured +// before the fix: `mxcli -p app.mpr -c 'create or modify entity …'` printed +// "1 document already in sync (unchanged, not listed)" and named nothing. +func TestALoneUnchangedLineIsPrintedVerbatim(t *testing.T) { + ctx, mb, out := reportCtx(t) + tally := &mutationTally{active: true} + ctx.tally = tally + + mb.offer(1, 0) + ctx.ReportMutation("Created", "entity: %s", "MyFirstModule.Od01") + tally.flush(ctx.Output) + + if got := out.String(); got != "Unchanged entity: MyFirstModule.Od01\n" { + t.Errorf("reported %q, want the single elided line in full", got) + } +} + +// A nil tally is the zero-configuration path every existing caller takes. +func TestNilTallyBehavesExactlyAsBefore(t *testing.T) { + ctx, mb, out := reportCtx(t) + + mb.offer(1, 0) + ctx.ReportMutation("Created", "entity: %s", "A") + + if got := out.String(); got != "Unchanged entity: A\n" { + t.Errorf("reported %q, want the unmodified behaviour on a nil tally", got) + } +} + +// A run in which nothing was elided must print no summary at all: a trailing +// "0 unchanged" on every clean first run is noise of exactly the kind this is +// meant to remove. +func TestNoSummaryWhenNothingWasUnchanged(t *testing.T) { + ctx, mb, out := reportCtx(t) + tally := &mutationTally{active: true} + ctx.tally = tally + + mb.offer(1, 1) + ctx.ReportMutation("Created", "entity: %s", "A") + tally.flush(ctx.Output) + + if got := out.String(); got != "Created entity: A\n" { + t.Errorf("reported %q, want no summary line when nothing was unchanged", got) + } +} diff --git a/mdl/executor/report_mutation.go b/mdl/executor/report_mutation.go index 793b844c58..9b78eb1f63 100644 --- a/mdl/executor/report_mutation.go +++ b/mdl/executor/report_mutation.go @@ -35,6 +35,12 @@ import ( // its own merits. func (ctx *ExecContext) ReportMutation(verb, format string, args ...any) { if ctx.mutationWasElided() { + // Inside a program run these are held rather than printed: several of + // them collapse into one summary at the end, and a lone one is printed + // verbatim at flush. See mutation_tally.go. + if ctx.tally.countUnchanged(fmt.Sprintf("Unchanged %s\n", fmt.Sprintf(format, args...))) { + return + } verb = "Unchanged" } fmt.Fprintf(ctx.Output, "%s %s\n", verb, fmt.Sprintf(format, args...)) From 4954f8401fa8729a6b53a80ad49c2163ba8ceb62 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 14:02:38 +0000 Subject: [PATCH 26/38] docs: move Implementation Status out of CLAUDE.md into the skills (#611) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First slice of #611. CLAUDE.md is re-read into every context in this repo, and at 108,761 B (~27k tokens) it was 18x the 6,000-byte budget init_claudemd_budget_test.go enforces on users' projects — the same reasoning, applied outward only. Implementation Status was 33% of the file and bimodal: 36 bullets < 250 B pure capability listing ("Domain model", "Pages with 50+ widget types"). `mxcli syntax`, `help` and `lint --list-rules` answer this authoritatively and cannot go stale, so these are deleted rather than relocated. 20 bullets ~33 KB real measurement — CE numbers, the control that settled a question, the trap that cost an afternoon. Moved to the skill for that doctype, where it is loaded when the area is touched instead of on every session. One more was dropped as a duplicate: the idempotent-writes bullet summarised a Key Concepts section in the same file and the docs-site page it cited. Verified rather than assumed — every concept it named (StableId, TransplantIDs, MXCLI_ALWAYS_WRITE, the canonical form) is still present in CLAUDE.md. Nothing else was lost: three distinctive probes per moved bullet were checked against the destination after the move, and the largest deleted bullet is 139 B. Six moved sections cited a skill and, now living inside it, cited themselves — three through the flat `.md` path that no longer exists. Those clauses are stripped. 108,761 -> 73,712 B (32%). Key Concepts (26%) is the next slice, and the budget test lands once the file is under target. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .claude/skills/mendix/agents/SKILL.md | 4 + .../skills/mendix/debug-microflows/SKILL.md | 4 + .../download-marketplace-content/SKILL.md | 14 ++++ .../skills/mendix/manage-navigation/SKILL.md | 8 ++ .claude/skills/mendix/project-brain/SKILL.md | 4 + .../mendix/regular-expressions/SKILL.md | 4 + .claude/skills/mendix/run-local/SKILL.md | 16 ++++ .../scheduled-events-and-queues/SKILL.md | 8 ++ .../skills/mendix/test-microflows/SKILL.md | 4 + .claude/skills/mendix/theme-styling/SKILL.md | 4 + .../mendix/validation-microflows/SKILL.md | 4 + .claude/skills/mendix/write-layouts/SKILL.md | 4 + .claude/skills/mendix/write-rules/SKILL.md | 4 + CLAUDE.md | 76 ++++--------------- 14 files changed, 96 insertions(+), 62 deletions(-) diff --git a/.claude/skills/mendix/agents/SKILL.md b/.claude/skills/mendix/agents/SKILL.md index c18ab794a7..218f74ea3d 100644 --- a/.claude/skills/mendix/agents/SKILL.md +++ b/.claude/skills/mendix/agents/SKILL.md @@ -225,3 +225,7 @@ retrieve $agent from database AgentCommons.Agent ``` The `AgentCommons.Agent` entity is populated at runtime by `ASU_AgentEditor` from the agent documents you create with `create agent`. + +## AI agent documents + +Model, Knowledge Base, Consumed MCP Service, Agent (LIST/DESCRIBE/CREATE/DROP, with variables, tools, KB tools, dollar-quoted multi-line prompts; requires AgentEditorCommons module, Mendix 11.9+) diff --git a/.claude/skills/mendix/debug-microflows/SKILL.md b/.claude/skills/mendix/debug-microflows/SKILL.md index 17b33a592e..49465e03f3 100644 --- a/.claude/skills/mendix/debug-microflows/SKILL.md +++ b/.claude/skills/mendix/debug-microflows/SKILL.md @@ -136,3 +136,7 @@ name. - [ ] `mxcli debug activities ` lists the activity you want. - [ ] After triggering the flow, `mxcli debug paused` shows it with variables. - [ ] Finished with `mxcli debug disable`. + +## Microflow/nanoflow debugger (`mxcli debug`) + +set breakpoints **by name** (activity resolved from the model), inspect paused flows + variables, step over/into/out, continue — against a `run --local` runtime. Two M2EE planes wired behind one command (admin `enable/disable/status`, app `/debugger/` session); `run --local --debug` enables it at boot. **Nanoflows** are auto-detected (uses the `nanoflow_name` breakpoint param; paused nanoflows are merged from `poll_events`, which `get_paused_microflows` omits). Nanoflow `LOG` output is rewritten to the `Client_Nanoflow` node in the runtime log. and `docs/11-proposals/PROPOSAL_microflow_debugger.md` diff --git a/.claude/skills/mendix/download-marketplace-content/SKILL.md b/.claude/skills/mendix/download-marketplace-content/SKILL.md index fbee61d838..e76434a269 100644 --- a/.claude/skills/mendix/download-marketplace-content/SKILL.md +++ b/.claude/skills/mendix/download-marketplace-content/SKILL.md @@ -428,3 +428,17 @@ trusting them from memory, and note that the listing name never matches the modu error with a login hint. - Marketplace CDN TLS handshakes time out occasionally. Retry once before reporting a failure. + +## Platform authentication (`mxcli auth login/logout/status/list`) with PAT scheme for marketplace-api + +Platform authentication (`mxcli auth login/logout/status/list`) with PAT scheme for marketplace-api.mendix.com, marketplace.mendix.com, and catalog.mendix.com; credentials stored at ~/.mxcli/auth.json (mode 0600), MENDIX_PAT env override + +## Marketplace download/install (`mxcli marketplace download/install`) — the content API now exposes a per-version downloadUrl (303→public CDN); install is type-aware (widget→widgets/, new module→`mx module-import`); existing-module updates are reported, not applied (entity-ID/local-edit safety — see PROPOSAL_marketplace_modules + +Marketplace download/install (`mxcli marketplace download/install`) — the content API now exposes a per-version downloadUrl (303→public CDN); install is type-aware (widget→widgets/, new module→`mx module-import`); existing-module updates are reported, not applied (entity-ID/local-edit safety — see PROPOSAL_marketplace_modules.md) + +## Marketplace drift detection (`mxcli marketplace diff -p app.mpr [--to VERSION] [--json]`) + +reports **which elements of an installed marketplace module have been edited locally** — the question Studio Pro's Marketplace update never asks before replacing the module. The version's `.mpk` is downloaded and imported into a throwaway reference project built **at the consuming project's Mendix version** (a mismatch is refused, not warned about: Mendix's own conversions would read as user edits), then every element is described on both sides and the **DESCRIBE output** compared — not BSON, in which an *untouched* module differs from its own package in ~15,000 paths. `--to` adds what an upgrade would touch and which of those collide with local edits. Honesty rule: an element that cannot be described is reported **unknown, never unchanged**, and `verified:false` in the JSON means "no modifications found" is not a conclusion. Module + version are identified from the module's `AppStoreGuid`, which is the marketplace **version UUID** — matching on the version *number* is ambiguous (a blank project has Atlas_Web_Content 4.1.0 and Administration's content also published a 4.1.0). Measured on real content: Administration 4.3.2 in a blank 11.12.1 app → 21/21 unchanged; one added attribute → exactly `ENTITY Account`; `--to 4.3.2` (the installed version) touches nothing, which is the control for `--to 4.5.0`'s five. Package: `cmd/mxcli/marketplace/`. See `docs/11-proposals/PROPOSAL_marketplace_module_upgrade.md` + +**Not Yet Implemented:** diff --git a/.claude/skills/mendix/manage-navigation/SKILL.md b/.claude/skills/mendix/manage-navigation/SKILL.md index 4a8889be02..5433b39f33 100644 --- a/.claude/skills/mendix/manage-navigation/SKILL.md +++ b/.claude/skills/mendix/manage-navigation/SKILL.md @@ -506,3 +506,11 @@ stored. MDL does not author per-entity sync modes — set those in Studio Pro. - [ ] Use `describe navigation` to verify changes after applying - [ ] For a **menu document**, confirm you want `create menu` and not a profile menu — `show navigation menu` vs `describe menu` tells them apart - [ ] No menu item targets a page with required parameters (CE1571) + +## Offline synchronization (`CREATE NAVIGATION … SYNC (…)`) + +an offline navigation profile downloads **nothing** until each entity has a sync mode, so a profile mxcli created built, routed and installed as a PWA and showed an **empty app** — with `mxcli check`, `exec` and `mx check` all clean. The six mode words are the members Mendix stores, **not** Studio Pro's captions (its "All Objects" is `ALL`, its "By XPath" is `WHERE`), and a caption is refused rather than written — the CE0463 gallery defect wearing a different hat. `WHERE` takes the XPath in **brackets**, verbatim: the quoted form doubles every quote, and a stored constraint already carries Mendix's own escaping, so the two compose into runs of six (mendixlabs/mxcli#750, and `PROPOSAL_first_class_expressions.md`). The write is an **overlay keyed by entity**, so `CompatibilityMode` — stored, unauthorable — survives a rewrite; every reference config carries `false`, so only a synthetic `true` case distinguishes a correct writer from one that always emits `false`. `DownloadMode`/`ShouldDownload` are deliberately **not** written though gen declares them: zero occurrences in ako/TestApp, and a property Studio Pro fills in on load is one whose emission makes a document Studio Pro cannot open. Creating the *profile* stays modelsdk-only (a fourteen-key document pinned to a Studio Pro reference); the SYNC block works on both engines. `ON SYNC ERROR THROW|CONTINUE` writes `ThrowPartialSyncError`, a property **neither generated source declares** (zero occurrences in gen and in generated/metamodel), so it is read from `element.Base.Raw()` and written as a raw key. The spec field is a **pointer**: the property is a bare bool with no unset value, so a non-pointer would reset it on every rewrite that never mentions the clause. Absent reads as **true**, matching every reference profile and Studio Pro's checked-by-default box. Both halves are in the catalog: `CATALOG.OFFLINE_ENTITY_CONFIGS` holds one row per configured entity (the profile's `OfflineEntityCount` said how many and nothing else), and a configured entity emits a **`sync` edge** into `CATALOG.REFS` so `show references to Mod.Entity` names the profiles that download it. Every mode gets an edge, **including the ones that download nothing** — a profile with `sync X never` still names X, so renaming or dropping it leaves the config dangling, which is exactly what the edge exists to reveal. and `docs/11-proposals/PROPOSAL_offline_sync_configuration.md` + +## Menu documents (CREATE OR MODIFY/DESCRIBE/DROP MENU) + +standalone `Menus$MenuDocument`, the reusable menu a menu widget points at (Atlas_Core's `Phone_Menu`/`Tablet_Menu`) — **not** the menu inside a navigation profile, though both are built from the same items, so the item syntax is shared with `CREATE NAVIGATION`'s `MENU (...)` block. DESCRIBE is round-trippable. Written through gen+codec, which is load-bearing: Studio Pro's menu documents carry typed-array marker **3** on the item collection and each item's sub-items (the codec default), while the navigation writers hand-build items with marker **1** — unverified whether that is a latent navigation bug or a real difference, so navigation is left alone. Authoring is modelsdk-only; legacy refuses. Two traps: a menu item cannot open a page with required parameters (**CE1571**), and only `Forms$IconCollectionIcon` round-trips (glyph/image icons are flagged by DESCRIBE, not dropped silently) diff --git a/.claude/skills/mendix/project-brain/SKILL.md b/.claude/skills/mendix/project-brain/SKILL.md index eddad58933..9c6f478a8f 100644 --- a/.claude/skills/mendix/project-brain/SKILL.md +++ b/.claude/skills/mendix/project-brain/SKILL.md @@ -354,3 +354,7 @@ slice at all — and a slice's findings are mostly decisions. The queue is append-only, so its own order is the honest boundary. `last_id` comes back even when nothing matched, so a slice that recorded nothing still hands the next one a boundary. + +## Project brain (`mxcli brain init/capture/staged/promote/drop/check/show`) + +an **opt-in** store in `docs/brain/` for the project knowledge mxcli cannot compute. The governing rule is that anything derivable from the model is answered by a command and never written down — a note that transcribes the model disagrees with it silently. Records shard by **anchor scope**: an entry's first anchor names its file (`@Sales.Order` → `modules/Sales.md`), an anchorless entry is cross-cutting (`project.md`), and there is no index to maintain because the module prefix *is* the file name. That is what makes the cap per-shard rather than a project-wide budget, and lets a session load `project.md` plus the modules it is touching. `check` answers two independent questions: each anchor is **resolved / not found / not indexable** — only the middle one fails, and the third exists because the catalog's `objects` view covers the describable types only, so a scheduled event would otherwise read as *missing* (separated with `FindDocumentUnit`, which cannot miss a kind because it never asks what kind anything is). Misfiling is a **second axis, not a fourth state**: every anchor can resolve and the entry still be in the wrong file, and it is only decided when something resolved — judging it on an all-not-indexable entry reintroduced the same false staleness through the other axis (caught by a test, with the guard stubbed as the control). An agent `capture`s to a git-ignored queue and a person `promote`s; the queue is deliberately **not** sharded, because routing it would force the file decision before a human has looked at the entry. `mxcli lint` prints the unpromoted-queue count, because a report only `brain check` prints is a report nothing demands. Sizes are computed by `brain show` and never written into a committed file. A second record kind, **requirement**, lives in `plan/.md` and inverts the anchor's meaning: a decision's anchor points backward (not resolving = stale, fails), a requirement's points forward (not resolving = not built yet, passes). Measured: filed as an ordinary entry, one unbuilt requirement takes `brain check` to exit 1 — which is why it is a separate kind rather than more entries in the same files. That inversion is also what makes `brain plan` a real progress report: a requirement is *built* when its anchors resolve, so creating the microflow it names moves the count with the plan file untouched (measured 0/1 → 1/0). A status written beside a requirement is therefore refused by the skill, not just discouraged. Slices are ordered by name (`01-accounts`), span modules by design (so misfiling does not apply), and carry a generous cap that enforces the slicing discipline — a slice too long to read should be split. A third kind, **open question** (`--open`), records what is *not* decided; its anchors are deliberately **not** checked, since the question is often whether the thing should exist at all — measured, the identical anchor exits 1 as a decision and 0 as a question. `brain resolve` converts one into a decision in place, keeping its id and position and starting to check its anchors, which is the transition the kind exists for. Unanswered questions are reported by `brain check` and by `mxcli lint`. The skill also gives capture a **trigger** rather than good intentions — a correction you have had to make twice — because the decisions half otherwise under-fills while the plan half fills at bootstrap. `bootstrap-app` asks for requirements at the interview and records them by default. Package: `cmd/mxcli/brain/`. See `docs-site/src/tools/project-brain.md` and `docs/11-proposals/PROPOSAL_project_brain.md` diff --git a/.claude/skills/mendix/regular-expressions/SKILL.md b/.claude/skills/mendix/regular-expressions/SKILL.md index 160a88dcb5..c115e28f4d 100644 --- a/.claude/skills/mendix/regular-expressions/SKILL.md +++ b/.claude/skills/mendix/regular-expressions/SKILL.md @@ -148,3 +148,7 @@ select QualifiedName, Expression from CATALOG.REGULAR_EXPRESSIONS; - `mxcli syntax regular-expression` — full syntax reference - `mxcli syntax validation-rule` — binding a pattern or a range to an attribute - `mdl-entities` — attributes and validation + +## Regular expressions (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP REGULAR EXPRESSION) + +named patterns that attribute validation rules reference **by qualified name**, which is why they are documents. `modelsdk/gen` is **wrong** about the pattern's key — it binds `RegEx` where every Studio Pro document stores `Expression` (`generated/metamodel` agrees with the documents), so both engines share one raw-BSON codec in `mdl/regularexpressions`; a reader keyed on gen's name returns an empty pattern for every real document. Pinned against five Studio Pro-authored documents (Email Connector 6.4.2, Community Commons 11.5.1). Mendix validates with .NET's engine, so a pattern Go's RE2 cannot compile (lookaround — the Email Connector ships one) is stored unchanged and reported "not verifiable", never "invalid". A `validate` edge into `CATALOG.REFS` makes `show references to ` list the entities using it diff --git a/.claude/skills/mendix/run-local/SKILL.md b/.claude/skills/mendix/run-local/SKILL.md index 8ffc53a8c6..c3f7129e84 100644 --- a/.claude/skills/mendix/run-local/SKILL.md +++ b/.claude/skills/mendix/run-local/SKILL.md @@ -559,3 +559,19 @@ read-back to check against. Do not reach for `--runtime-setting 'MicroflowConstants={…}'`: it replaces the map mxcli built rather than adding to it, and at boot there is nothing to fall back on for `BasePath`/`DatabaseName`. Use `--constant`. + +## Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`) + +Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing; `--setup` does the non-blocking prerequisites (cache mxbuild+runtime, ensure DB) and exits — `mxcli init` wires it into a Claude Code SessionStart hook so a fresh/reaped web session self-bootstraps, and `docs-site/src/tools/bootstrap-prompt.md` is the empty-repo seed prompt. `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` + +## External browser preview (`mxcli run --hub ` + `mxcli tunnel-hub`) + +the app stays local and reverse-tunnels out over a single 443 connection (embedded chisel) to a static relay, so it is reachable in a browser at a public URL — works from egress-only environments (Claude Code web), verified live through the session's MITM egress proxy. `run --hub` implies `--local`, boots the runtime with `ApplicationRootUrl` set to the assigned URL (so the SPA/`originURI` work under the public origin), resolves the control proxy honouring `NO_PROXY`, and retries forever. `mxcli tunnel-hub --domain ` is the **multi-tenant** relay: a registry keyed by prefix/project/solution/branch/worktree (stable URLs on reconnect) fronts many previews at per-subdomain hosts (`[prefix-]project[-branch].`; main collapses to the project) over one 443 with per-subdomain autocert, a registration API (`/api/register|status|deregister|backends|sessions`), and an availability overview at `hub./` **grouped by Claude Code session** (`/api/sessions`): each session lists the endpoints it exposed and links back to its `claude.ai/code` conversation. Client identity flags: `--hub-prefix`/`--hub-project`/`--hub-solution`/`--hub-branch`/`--hub-worktree` (project + branch auto-detected); `--hub-session` groups a session's endpoints (auto-detected from `CLAUDE_CODE_REMOTE_SESSION_ID`). Past sessions are retained: a durable per-session endpoint history (`--sessions-file`, default `~/.mxcli/hub-sessions.json`) survives restarts and reaping, and is pruned after `--session-retention` (default 30d) — so the overview shows offline sessions too (`SessionLog` in `cmd/mxcli/tunnelhub/sessions.go`). Package: `cmd/mxcli/tunnelhub/`. See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` (slices 3–4) + +## Tunnel-hub GitHub authentication (opt-in, gated on `--github-oauth-client-id`; absent = today's open hub) + +**viewer plane** — GitHub OAuth web flow + HMAC-signed SSO session cookie (`Domain=.`), owner-checked previews (`--require-auth` default on → 302 to login / 403 non-owner; soft mode filters the listing only), `/api/backends` filtered to the viewer (unauthenticated → 401), admin "signed in as" via `/api/whoami`. **Registration plane** — durable, hashed hub API keys (`--keys-file`, default `~/.mxcli/hub-keys.json`, survive restarts) presented as `X-Hub-Key` → stamps `Backend.Owner`; shared `X-Hub-Secret` still works as an owner-less fallback. **Key issuance** — the hub's `/cli` browser page mints a key from the session cookie (no PAT; the device flow was removed as Claude Code containers block GitHub's device endpoints), rotate-by-default + count + revoke-all; `mxcli auth hub login --token ` is the headless path; `run --hub` reads `MXCLI_HUB_KEY` (env → `~/.mxcli/auth.json`) and degrades to local-only if registration fails. Append-only JSONL audit trail (`--audit-log`, no secrets). Packages: `cmd/mxcli/tunnelhub/` (+`audit/`), `cmd/mxcli/hubauth/`. See `docs/11-proposals/PROPOSAL_hub_authentication.md` + +## Runtime metrics + settings passthrough (`mxcli run --local --metrics` / `--runtime-setting Key=Value`) + +`--metrics` registers a Prometheus Micrometer registry at boot (served at `http://127.0.0.1:/prometheus`); `--runtime-setting` merges arbitrary runtime config (e.g. `Metrics.Registries` for otlp/influx/statsd, or `OpenTelemetry._RuntimeSpanFilters`) into mxcli's **single** boot `update_configuration` call — the admin action replaces rather than merges, so folding settings into the one boot call is the only safe way. OTel traces via `--trace` attach the bundled `opentelemetry-javaagent` to the runtime JVM (console exporter → the tee'd runtime log) and ship default `OpenTelemetry._RuntimeSpanFilters` (unfiltered per-activity tracing is ~10× slower); `--trace-service` sets `OTEL_SERVICE_NAME`. The console exporter omits timestamps/parent span IDs (no flame charts), so `--trace-otlp ` (implies `--trace`) switches to the OTLP exporter (protocol `http/protobuf`) pointed at a collector; user-set `OTEL_*` env still takes precedence. diff --git a/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md b/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md index 03976820af..fe0fb800ae 100644 --- a/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md +++ b/.claude/skills/mendix/scheduled-events-and-queues/SKILL.md @@ -284,3 +284,11 @@ Starlark lint rules can iterate both: `scheduled_events()` yields - `mxcli syntax scheduled-event`, `mxcli syntax queue` — full syntax reference - `write-microflows` — writing the microflow the event calls - `project-settings` — after-startup / before-shutdown microflows + +## Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here + +the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. Both are in the catalog (`CATALOG.SCHEDULED_EVENTS`, `CATALOG.QUEUES`) and a scheduled event emits a `schedule` edge into `CATALOG.REFS` — without it a microflow run only by a scheduled event was reported as dead by `show callers`, `GRAPH_DEAD_ASSETS` and lint rule QUAL004. + +## Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005) + +the rebuild used to write `QueueSettings` back as null, which made `mx check` go from CE1613 to 0 errors by deleting the user's configuration diff --git a/.claude/skills/mendix/test-microflows/SKILL.md b/.claude/skills/mendix/test-microflows/SKILL.md index 75d0b5c33d..7157a2f37a 100644 --- a/.claude/skills/mendix/test-microflows/SKILL.md +++ b/.claude/skills/mendix/test-microflows/SKILL.md @@ -572,3 +572,7 @@ The JUnit XML works with GitHub Actions, Jenkins, Azure DevOps, GitLab CI, etc. - [write-microflows](../write-microflows/SKILL.md) — Microflow syntax reference - [docker-workflow](../docker-workflow/SKILL.md) — Docker build and runtime workflow - [verify-with-oql](../verify-with-oql/SKILL.md) — OQL queries for data verification + +## Warm test loop (`mxcli test --local [--watch]`, `--attach`, `run --local --test-endpoint`) + +local test runs go through a **token-guarded HTTP endpoint** registered by a generated Java custom request handler, instead of compiling the suite into the project's after-startup microflow. Boot registers the endpoint and then **chains the project's own after-startup microflow**, so tests see the app as it really boots (`--skip-app-startup` opts out) — without that, a suite depending on startup state passed under `--attach` and failed under `--local`. One microflow per test, resolved by name at request time from `Core.getMicroflowNames()` and invoked with `Core.microflowCall(...).execute(...)` — so a throwing test fails only itself (not the boot), results are returned rather than scraped from the runtime log, and each test has its own variable scope. Owning the `IContext` is also what finally makes **`@cleanup rollback`** (the annotation's documented default, previously parsed and ignored) real: the handler wraps the call in `startTransaction()`/`rollbackTransaction()`, so a test's writes do not survive it — verified against Postgres, with `@cleanup none` as the in-run control. A rollback that fails is reported per test and summarised, never silent; an unknown strategy is a parse error. The handler **survives `reload_model`** (after-startup does not re-run, the JVM is unchanged), which is what makes `--watch` possible: ~30s first run, then ~2s from an edit — to a test *or* to the microflow under test — to a verdict. `--attach` skips even that boot by running against an app already up under `run --local --test-endpoint`, driving that process's serve + admin APIs over loopback; it uses **that app's database**, only ever adds/removes its own test microflows, and refuses a change needing a restart. Security: the handler is **not registered at all** without `MXCLI_TEST_TOKEN` in the runtime env (so a project that kept the `MxTest` module through a failed cleanup is inert in production), the token is constant-time compared, non-loopback callers are refused, `/list` is clamped to the test namespace, and only `MxTest.Test_*` may be invoked. The token reaches the runtime via its environment and is never written into the project. A `.test.mdl` file is **checkable**: each block is a microflow body, so `mxcli check` (and the LSP, hence VS Code) renders the blocks as the microflows they become, on the file's own lines — before #1103 the top-level grammar was applied instead, and since `RETRIEVE` is a non-reserved keyword the leftover `FROM …` started an OQL query, so the reader was told their retrieve needed a SELECT; 9 of this repo's 10 test files reported errors that way, one of them 392. `make check-mdl` now sweeps them, with `.fail.test.mdl` for a file whose annotations are deliberately unusable. Two things a failed run must not do, both reported as #1104: **a rejected build is reported with MxBuild's own errors** — `BuildResult.ErrorSummary()` was in hand and discarded by `fmt.Errorf("build failed: %s", build.Message)` on the `--attach` and rebuild paths, and that sentence is identical for every failing build, so it could not tell "your test does not compile" from "an unrelated document is broken"; and **cleanup removes every generated `MxTest.Test_*` the project holds**, not just this suite's. The names are positional and every file reuses them, so keying cleanup on the suite left a flow behind whenever a later run had fewer tests than an earlier one — and one leftover that does not build fails every later run of every test file. What cleanup could not remove is named, with the `DROP` that removes it. Docker keeps the after-startup runner (`--legacy-runner` selects it locally). Packages: `cmd/mxcli/testrunner/` (`endpoint.go`, `client.go`, `watch.go`, `host.go`, `handshake.go`, `check_source.go`, `cleanup_leftovers.go`). See `docs/15-testing/SPIKE_test_endpoint_request_handler.md` diff --git a/.claude/skills/mendix/theme-styling/SKILL.md b/.claude/skills/mendix/theme-styling/SKILL.md index a294e1e194..aaf19b3556 100644 --- a/.claude/skills/mendix/theme-styling/SKILL.md +++ b/.claude/skills/mendix/theme-styling/SKILL.md @@ -418,3 +418,7 @@ write side, the value's BSON type is taken from the registry (a `ColorPicker` / - [ ] For CSS changes, run `docker build` then `docker reload --css` - [ ] Use `describe styling` to verify changes after modification - [ ] Check `docs/11-proposals/page-styling-support.md` for BSON format details + +## Default styling + runtime theme switching (`mxcli theme list/show/create/apply/remove/switcher`, `mxcli new --theme`) + +three embedded themes (**signal** light-first, **ledger** light-first, **console** dark-first), each a palette in `theme/web/custom-variables.scss` + a shared Atlas wiring partial + a theme partial imported from `theme/web/main.scss` (which compiles last), plus vendored fonts. **No model changes**, so it hot-applies under `run --local --watch` and cannot affect a build. Generated regions are digest-fenced: a block carrying local edits is refused rather than overwritten. Applying a theme removes the previous one. `--variant auto` (default) ships both palettes — the app follows `prefers-color-scheme` before first paint and honours a `theme-light`/`theme-dark` class on ``; `light`/`dark` bakes one. `theme switcher install` is the only part that writes to the model (JS actions + a nanoflow for a toggle button). A project can add its own themes under `theme/mxcli-themes//` (committed, not compiled); `theme create [--from ]` scaffolds one from an existing theme, renaming the identifiers built from the name and optionally seeding the palette from `--mxt-*` declarations in a design artifact. A local theme shadows a built-in of the same name. Package: `cmd/mxcli/theme/`. See `docs/11-proposals/PROPOSAL_default_styling.md` diff --git a/.claude/skills/mendix/validation-microflows/SKILL.md b/.claude/skills/mendix/validation-microflows/SKILL.md index 5ad9749ca3..1f7e0181e7 100644 --- a/.claude/skills/mendix/validation-microflows/SKILL.md +++ b/.claude/skills/mendix/validation-microflows/SKILL.md @@ -289,3 +289,7 @@ This feature is implemented in: - `mdl/executor/cmd_microflows_show.go` - DESCRIBE formatter for MDL output - `mdl/backend/modelsdk/microflow_write.go` - BSON serialization for ValidationFeedbackAction - `sdk/microflows/microflows_actions.go` - ValidationFeedbackAction struct + +## Validation rules (CREATE VALIDATION RULE) + +binds a **regex** or a **range** to one attribute — `create validation rule for Mod.Entity.Attr regex Mod.Pattern feedback '…'`. The rule is anonymous and entity-scoped, so the statement names the attribute; re-running it replaces the rule of the same type and leaves the attribute's others alone. Unlocked by a `STORAGE-NAME OVERRIDE` in `modelsdk/gen`: it bound `RegularExpression` where Studio Pro stores `RegExIdentifier`, and the control (same script, key reverted) fails **CE0135 "No regular expression specified"** while the fixed one is 0 errors on mxbuild 11.13 with `RegExIdentifier` on disk. Range bounds are inclusive and map to Mendix's only three kinds (`from X to Y`/`from X`/`to Y` → Between/GreaterThanOrEqualTo/SmallerThanOrEqualTo); there is no strict `<`/`>`, and the old grammar's forms for it — plus an EXPRESSION rule type Mendix does not have and an inline regex literal — were removed, having never had a visitor or handler. Required/Unique stay attribute constraints (`not null error '…'` / `unique error '…'`), not a second spelling here. Rewriting an entity carrying **MaxLength or EqualsTo** is **refused** on both engines rather than silently downgraded to Required; that round trip was lossy and `mx check` stayed green, because a Required rule is valid. Both engines carry each rule's payload on READ (`ruleInfoFromGen` / `parseValidationRuleInfo`), which is what makes the refusal narrow instead of covering all of RegEx and Range — and what lets a **range bounded by another attribute** survive a rewrite even though MDL cannot author one (`describe entity` marks it with a comment rather than rendering it wrong). A rule whose payload did not survive the read is refused as firmly as an unknown type: a bare RuleInfo of the right `$Type` constrains nothing, which is the same silent downgrade wearing the right name diff --git a/.claude/skills/mendix/write-layouts/SKILL.md b/.claude/skills/mendix/write-layouts/SKILL.md index 77532e6c9e..614cc1d596 100644 --- a/.claude/skills/mendix/write-layouts/SKILL.md +++ b/.claude/skills/mendix/write-layouts/SKILL.md @@ -257,3 +257,7 @@ image, so a describe → exec copy renders with no navigation and no logo. The scaffold reproduces the *result* instead — same layout class, same region classes, navigation in the topbar, `Main` for page content — and omits the toggle button and the stock logo. + +## Layouts (SHOW/DESCRIBE/CREATE [OR REPLACE] LAYOUT) + +the frame a page renders inside — the last document a page depends on that MDL could not write, which is why the topbar was out of reach. Four element types beyond a page's vocabulary: `scrollcontainer`, `region top|right|bottom|left|center` (five named slots, not a list), `placeholder`, `navigationtree`. `DESCRIBE LAYOUT` emits **re-executable MDL**, which makes describe → rename → exec the copy operation and is why there is no `COPY DOCUMENT` verb. Writing into a Marketplace module is **refused** — Mendix's own guidance is not to edit the supplied layouts, since an update replaces the module — which also required wiring `FromAppStore` enrichment into `GetModuleByName`/`GetModule` (it was populated only by `ListModules`, so the guard would have been inert). The document is pinned to the ten keys Studio Pro writes, identical across all 22 Atlas layouts on 11.13.0: `modelsdk/gen` offers seven placeholder properties on `Layout` (`MainPlaceholderName` and friends) that `generated/metamodel` does not declare and no real layout carries, and writing one gives a layout **mxbuild accepts at 0 errors and Studio Pro cannot open**. Which placeholder is "main" is **a rule mxbuild enforces, not a convention** — `layouttype` is still the only header property, because `Forms$Layout` has no property for it, but the NAME is validated: measured on 11.12.1 against a layout **no page uses**, exactly one placeholder must be named `Main` (none → **CE0848**, two → **CE0849**) and placeholder names must be unique (**CE0495**); extra placeholders under other names are fine. mxcli reported none of this until #1063 — this file called it a convention and the write-time guard implemented that belief, accepting any placeholder under any name, so the reported script passed `check` AND `exec` and failed a build later with no `DROP LAYOUT` to undo it. The rule now lives once in `types.CheckLayoutPlaceholderNames` and is applied by both `mxcli check` (**MDL081**/**MDL082**) and the writer, because two copies in two currencies is how a resolver drifts. The platform is inferred from the layout type — web (Responsive/Phone/Tablet/ModalPopup) and native (Default/Popup) are disjoint — so there is no `native:` flag. Authoring is modelsdk-only; legacy refuses. Verified in a browser, not just against `mx check`. `DROP LAYOUT Module.Name` removes one — layouts were the only doctype mxcli could create and alter but not delete, which bit hardest on a layout mxcli itself had just written badly; pages still bound to it are **warned about and named, not refused** (matching every other DROP), because drop-then-recreate under the same name is how a layout is corrected and the pages rebind by qualified name — left dropped they fail **CE1613**, which names the *page* and never the layout. `ALTER LAYOUT Module.Name { … }` takes the whole ALTER PAGE vocabulary (a layout's tree *is* a page's plus four element types) and edits the stored document, which is what makes it a capability rather than a convenience: **describe → rename → exec is only as complete as what MDL can spell** — measured, the copy of `Atlas_Core.Atlas_SideBar` loses both `Forms$SidebarToggleButton` widgets, and an `image` widget loses its image reference. A placeholder is **declared** with no body; `placeholder X { … }` is the page-side spelling that *fills* a slot, and in a layout the visitor drops it — which used to leave the layout with no placeholder at all and fail the write with a message contradicting the script, so it is now **MDL083**. A region has no `Name`, so it is addressed as `layoutContainer.top` (the dotted widgetRef that also serves DataGrid2 columns; `$Type` decides which), and only `INSERT INTO` takes one. Pages move onto a layout with `ALTER PAGE … SET Layout = X [MAP (Old AS New)]` or the bulk `ALTER PAGES [IN ] SET LAYOUT = X [WHERE LAYOUT = Y]`, both of which **refuse a repoint that would leave a page bound to a placeholder the target does not declare** (checked *after* MAP, since MAP is the remedy) — mxbuild only catches that as CE1613 at the far end of a build. and `docs/11-proposals/PROPOSAL_authorable_layouts.md` diff --git a/.claude/skills/mendix/write-rules/SKILL.md b/.claude/skills/mendix/write-rules/SKILL.md index 0a55669492..3d92074227 100644 --- a/.claude/skills/mendix/write-rules/SKILL.md +++ b/.claude/skills/mendix/write-rules/SKILL.md @@ -134,3 +134,7 @@ Before presenting a rule: - [ ] Every path returns a value of the declared type - [ ] The caller is a decision: `if Module.Rule_Name(Param = $Value) then` - [ ] `mxcli check script.mdl -p app.mpr --references` passes + +## Rules (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP/MOVE RULE) + +Mendix's "special kind of microflow" — returns Boolean or an enumeration, callable only from a decision. Handled as a third flow flavour beside microflows and nanoflows: its own semantic type, its own listing (`show microflows` stays microflow-only), the shared `microflowBody`, flow builder and describer. The document is the ten properties a Studio Pro rule stores, pinned against two reference rules (ako/TestApp, 11.13.0) — **no `AllowedModuleRoles`** (a rule is not independently callable, so there is no `grant execute on rule`) and **no `ReturnType`** despite gen declaring one beside `MicroflowReturnType`. Two keys only a reference document catches, both invisible to `mx check`: `ExportLevel` (Studio Pro writes "Hidden" on every rule) and `Flows` (written as the bare marker even when empty — a `MandatoryLists` entry). Rules are catalog objects and their bodies are walked for references, which together stop a microflow called only from a rule reading as dead. The body restrictions are refused at check time by the same function `exec` calls, each measured: create/change/delete/commit/rollback and client or web-service activities are **CE0009**, a non-Boolean/enum return is **CE0103 + CE0139**. Authoring is modelsdk-only; legacy refuses. diff --git a/CLAUDE.md b/CLAUDE.md index 6e0a666460..f26e92edaa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -873,68 +873,20 @@ These rules apply whenever generating microflow or nanoflow MDL. Violations are Full syntax tables for all MDL statements (microflows, pages, security, navigation, settings, business events, ALTER PAGE, reserved words) are in **[docs/01-project/MDL_QUICK_REFERENCE.md](docs/01-project/MDL_QUICK_REFERENCE.md)**. -## Current Implementation Status - -**Implemented:** -- Offline synchronization (`CREATE NAVIGATION … SYNC (…)`): an offline navigation profile downloads **nothing** until each entity has a sync mode, so a profile mxcli created built, routed and installed as a PWA and showed an **empty app** — with `mxcli check`, `exec` and `mx check` all clean. The six mode words are the members Mendix stores, **not** Studio Pro's captions (its "All Objects" is `ALL`, its "By XPath" is `WHERE`), and a caption is refused rather than written — the CE0463 gallery defect wearing a different hat. `WHERE` takes the XPath in **brackets**, verbatim: the quoted form doubles every quote, and a stored constraint already carries Mendix's own escaping, so the two compose into runs of six (mendixlabs/mxcli#750, and `PROPOSAL_first_class_expressions.md`). The write is an **overlay keyed by entity**, so `CompatibilityMode` — stored, unauthorable — survives a rewrite; every reference config carries `false`, so only a synthetic `true` case distinguishes a correct writer from one that always emits `false`. `DownloadMode`/`ShouldDownload` are deliberately **not** written though gen declares them: zero occurrences in ako/TestApp, and a property Studio Pro fills in on load is one whose emission makes a document Studio Pro cannot open. Creating the *profile* stays modelsdk-only (a fourteen-key document pinned to a Studio Pro reference); the SYNC block works on both engines. `ON SYNC ERROR THROW|CONTINUE` writes `ThrowPartialSyncError`, a property **neither generated source declares** (zero occurrences in gen and in generated/metamodel), so it is read from `element.Base.Raw()` and written as a raw key. The spec field is a **pointer**: the property is a bare bool with no unset value, so a non-pointer would reset it on every rewrite that never mentions the clause. Absent reads as **true**, matching every reference profile and Studio Pro's checked-by-default box. Both halves are in the catalog: `CATALOG.OFFLINE_ENTITY_CONFIGS` holds one row per configured entity (the profile's `OfflineEntityCount` said how many and nothing else), and a configured entity emits a **`sync` edge** into `CATALOG.REFS` so `show references to Mod.Entity` names the profiles that download it. Every mode gets an edge, **including the ones that download nothing** — a profile with `sync X never` still names X, so renaming or dropping it leaves the config dangling, which is exactly what the edge exists to reveal. See `.claude/skills/mendix/manage-navigation/SKILL.md` and `docs/11-proposals/PROPOSAL_offline_sync_configuration.md` -- Project brain (`mxcli brain init/capture/staged/promote/drop/check/show`): an **opt-in** store in `docs/brain/` for the project knowledge mxcli cannot compute. The governing rule is that anything derivable from the model is answered by a command and never written down — a note that transcribes the model disagrees with it silently. Records shard by **anchor scope**: an entry's first anchor names its file (`@Sales.Order` → `modules/Sales.md`), an anchorless entry is cross-cutting (`project.md`), and there is no index to maintain because the module prefix *is* the file name. That is what makes the cap per-shard rather than a project-wide budget, and lets a session load `project.md` plus the modules it is touching. `check` answers two independent questions: each anchor is **resolved / not found / not indexable** — only the middle one fails, and the third exists because the catalog's `objects` view covers the describable types only, so a scheduled event would otherwise read as *missing* (separated with `FindDocumentUnit`, which cannot miss a kind because it never asks what kind anything is). Misfiling is a **second axis, not a fourth state**: every anchor can resolve and the entry still be in the wrong file, and it is only decided when something resolved — judging it on an all-not-indexable entry reintroduced the same false staleness through the other axis (caught by a test, with the guard stubbed as the control). An agent `capture`s to a git-ignored queue and a person `promote`s; the queue is deliberately **not** sharded, because routing it would force the file decision before a human has looked at the entry. `mxcli lint` prints the unpromoted-queue count, because a report only `brain check` prints is a report nothing demands. Sizes are computed by `brain show` and never written into a committed file. A second record kind, **requirement**, lives in `plan/.md` and inverts the anchor's meaning: a decision's anchor points backward (not resolving = stale, fails), a requirement's points forward (not resolving = not built yet, passes). Measured: filed as an ordinary entry, one unbuilt requirement takes `brain check` to exit 1 — which is why it is a separate kind rather than more entries in the same files. That inversion is also what makes `brain plan` a real progress report: a requirement is *built* when its anchors resolve, so creating the microflow it names moves the count with the plan file untouched (measured 0/1 → 1/0). A status written beside a requirement is therefore refused by the skill, not just discouraged. Slices are ordered by name (`01-accounts`), span modules by design (so misfiling does not apply), and carry a generous cap that enforces the slicing discipline — a slice too long to read should be split. A third kind, **open question** (`--open`), records what is *not* decided; its anchors are deliberately **not** checked, since the question is often whether the thing should exist at all — measured, the identical anchor exits 1 as a decision and 0 as a question. `brain resolve` converts one into a decision in place, keeping its id and position and starting to check its anchors, which is the transition the kind exists for. Unanswered questions are reported by `brain check` and by `mxcli lint`. The skill also gives capture a **trigger** rather than good intentions — a correction you have had to make twice — because the decisions half otherwise under-fills while the plan half fills at bootstrap. `bootstrap-app` asks for requirements at the interview and records them by default. Package: `cmd/mxcli/brain/`. See `docs-site/src/tools/project-brain.md` and `docs/11-proposals/PROPOSAL_project_brain.md` -- Default styling + runtime theme switching (`mxcli theme list/show/create/apply/remove/switcher`, `mxcli new --theme`): three embedded themes (**signal** light-first, **ledger** light-first, **console** dark-first), each a palette in `theme/web/custom-variables.scss` + a shared Atlas wiring partial + a theme partial imported from `theme/web/main.scss` (which compiles last), plus vendored fonts. **No model changes**, so it hot-applies under `run --local --watch` and cannot affect a build. Generated regions are digest-fenced: a block carrying local edits is refused rather than overwritten. Applying a theme removes the previous one. `--variant auto` (default) ships both palettes — the app follows `prefers-color-scheme` before first paint and honours a `theme-light`/`theme-dark` class on ``; `light`/`dark` bakes one. `theme switcher install` is the only part that writes to the model (JS actions + a nanoflow for a toggle button). A project can add its own themes under `theme/mxcli-themes//` (committed, not compiled); `theme create [--from ]` scaffolds one from an existing theme, renaming the identifiers built from the name and optionally seeding the palette from `--mxt-*` declarations in a design artifact. A local theme shadows a built-in of the same name. Package: `cmd/mxcli/theme/`. See `docs/11-proposals/PROPOSAL_default_styling.md` -- MPR v1/v2 reading and writing -- Idempotent writes (ADR-0008): a unit whose new content is semantically equal to what is stored is **not written**, so re-running an MDL script against an in-sync project leaves the `.mpr` and `mprcontents/` byte-identical and Studio Pro shows no version-control changes. Comparison is on a canonical form (element `$ID`s normalised away — a rebuild mints them randomly, so byte comparison would skip nothing); `Microflows$Microflow.StableId` is carried from the stored document rather than re-minted, because the build derives every client-callable microflow's operation id from it. When a write **does** land, `canon.TransplantIDs` matches the rebuild against the stored document and reuses its element `$ID`s (rewriting every pointer in the same pass), so a changed document's diff is the change rather than a wholesale replacement — measured on #910's nanoflow: 1 of 37 identities survived an argument edit before, 37 of 37 after, and a change plus its revert returns to the original bytes. Inserting or deleting an activity mints IDs only for the genuinely new elements. One policy in `modelsdk/canon`, called from both engines' write choke points. `MXCLI_ALWAYS_WRITE=1` disables elision (not preservation) for bisecting — which means it no longer changes the resulting bytes, only the mtimes. The executor's output distinguishes the two: `Unchanged nanoflow: …` where the write was skipped — and several such reports in one run collapse into a single `N documents already in sync` line, since a lone one is still printed in full. See `docs-site/src/internals/idempotent-writes.md` -- Domain model (entities, attributes, associations) -- ALTER ENTITY (add/rename/modify/drop attributes, indexes, documentation) -- Microflows/Nanoflows with 60+ activity types, JavaScript action calls, nanoflow validation parity -- Pages with 50+ widget types -- ALTER PAGE/SNIPPET (SET, INSERT, DROP, REPLACE operations on widget trees) -- Layouts (SHOW/DESCRIBE/CREATE [OR REPLACE] LAYOUT): the frame a page renders inside — the last document a page depends on that MDL could not write, which is why the topbar was out of reach. Four element types beyond a page's vocabulary: `scrollcontainer`, `region top|right|bottom|left|center` (five named slots, not a list), `placeholder`, `navigationtree`. `DESCRIBE LAYOUT` emits **re-executable MDL**, which makes describe → rename → exec the copy operation and is why there is no `COPY DOCUMENT` verb. Writing into a Marketplace module is **refused** — Mendix's own guidance is not to edit the supplied layouts, since an update replaces the module — which also required wiring `FromAppStore` enrichment into `GetModuleByName`/`GetModule` (it was populated only by `ListModules`, so the guard would have been inert). The document is pinned to the ten keys Studio Pro writes, identical across all 22 Atlas layouts on 11.13.0: `modelsdk/gen` offers seven placeholder properties on `Layout` (`MainPlaceholderName` and friends) that `generated/metamodel` does not declare and no real layout carries, and writing one gives a layout **mxbuild accepts at 0 errors and Studio Pro cannot open**. Which placeholder is "main" is **a rule mxbuild enforces, not a convention** — `layouttype` is still the only header property, because `Forms$Layout` has no property for it, but the NAME is validated: measured on 11.12.1 against a layout **no page uses**, exactly one placeholder must be named `Main` (none → **CE0848**, two → **CE0849**) and placeholder names must be unique (**CE0495**); extra placeholders under other names are fine. mxcli reported none of this until #1063 — this file called it a convention and the write-time guard implemented that belief, accepting any placeholder under any name, so the reported script passed `check` AND `exec` and failed a build later with no `DROP LAYOUT` to undo it. The rule now lives once in `types.CheckLayoutPlaceholderNames` and is applied by both `mxcli check` (**MDL081**/**MDL082**) and the writer, because two copies in two currencies is how a resolver drifts. The platform is inferred from the layout type — web (Responsive/Phone/Tablet/ModalPopup) and native (Default/Popup) are disjoint — so there is no `native:` flag. Authoring is modelsdk-only; legacy refuses. Verified in a browser, not just against `mx check`. `DROP LAYOUT Module.Name` removes one — layouts were the only doctype mxcli could create and alter but not delete, which bit hardest on a layout mxcli itself had just written badly; pages still bound to it are **warned about and named, not refused** (matching every other DROP), because drop-then-recreate under the same name is how a layout is corrected and the pages rebind by qualified name — left dropped they fail **CE1613**, which names the *page* and never the layout. `ALTER LAYOUT Module.Name { … }` takes the whole ALTER PAGE vocabulary (a layout's tree *is* a page's plus four element types) and edits the stored document, which is what makes it a capability rather than a convenience: **describe → rename → exec is only as complete as what MDL can spell** — measured, the copy of `Atlas_Core.Atlas_SideBar` loses both `Forms$SidebarToggleButton` widgets, and an `image` widget loses its image reference. A placeholder is **declared** with no body; `placeholder X { … }` is the page-side spelling that *fills* a slot, and in a layout the visitor drops it — which used to leave the layout with no placeholder at all and fail the write with a message contradicting the script, so it is now **MDL083**. A region has no `Name`, so it is addressed as `layoutContainer.top` (the dotted widgetRef that also serves DataGrid2 columns; `$Type` decides which), and only `INSERT INTO` takes one. Pages move onto a layout with `ALTER PAGE … SET Layout = X [MAP (Old AS New)]` or the bulk `ALTER PAGES [IN ] SET LAYOUT = X [WHERE LAYOUT = Y]`, both of which **refuse a repoint that would leave a page bound to a placeholder the target does not declare** (checked *after* MAP, since MAP is the remedy) — mxbuild only catches that as CE1613 at the far end of a build. See `.claude/skills/mendix/write-layouts/SKILL.md` and `docs/11-proposals/PROPOSAL_authorable_layouts.md` -- Image widgets (IMAGE, STATICIMAGE, DYNAMICIMAGE) with Width/Height properties -- Code generator for metamodel types -- MDL CLI (`mxcli`) with ANTLR4 parser -- MDL support for domain model, microflows, pages, and security -- Security management (module roles, user roles, access control, demo users) -- High-level fluent API (`api/` package) for simplified model manipulation -- LSP server with hover, go-to-definition, completion, diagnostics, symbols, folding -- VS Code extension (`vscode-mdl`) with context menu commands (Run/Check/Selection) -- Docker build integration (`mxcli docker build`) with PAD patching (Phase 1) -- Warm test loop (`mxcli test --local [--watch]`, `--attach`, `run --local --test-endpoint`): local test runs go through a **token-guarded HTTP endpoint** registered by a generated Java custom request handler, instead of compiling the suite into the project's after-startup microflow. Boot registers the endpoint and then **chains the project's own after-startup microflow**, so tests see the app as it really boots (`--skip-app-startup` opts out) — without that, a suite depending on startup state passed under `--attach` and failed under `--local`. One microflow per test, resolved by name at request time from `Core.getMicroflowNames()` and invoked with `Core.microflowCall(...).execute(...)` — so a throwing test fails only itself (not the boot), results are returned rather than scraped from the runtime log, and each test has its own variable scope. Owning the `IContext` is also what finally makes **`@cleanup rollback`** (the annotation's documented default, previously parsed and ignored) real: the handler wraps the call in `startTransaction()`/`rollbackTransaction()`, so a test's writes do not survive it — verified against Postgres, with `@cleanup none` as the in-run control. A rollback that fails is reported per test and summarised, never silent; an unknown strategy is a parse error. The handler **survives `reload_model`** (after-startup does not re-run, the JVM is unchanged), which is what makes `--watch` possible: ~30s first run, then ~2s from an edit — to a test *or* to the microflow under test — to a verdict. `--attach` skips even that boot by running against an app already up under `run --local --test-endpoint`, driving that process's serve + admin APIs over loopback; it uses **that app's database**, only ever adds/removes its own test microflows, and refuses a change needing a restart. Security: the handler is **not registered at all** without `MXCLI_TEST_TOKEN` in the runtime env (so a project that kept the `MxTest` module through a failed cleanup is inert in production), the token is constant-time compared, non-loopback callers are refused, `/list` is clamped to the test namespace, and only `MxTest.Test_*` may be invoked. The token reaches the runtime via its environment and is never written into the project. A `.test.mdl` file is **checkable**: each block is a microflow body, so `mxcli check` (and the LSP, hence VS Code) renders the blocks as the microflows they become, on the file's own lines — before #1103 the top-level grammar was applied instead, and since `RETRIEVE` is a non-reserved keyword the leftover `FROM …` started an OQL query, so the reader was told their retrieve needed a SELECT; 9 of this repo's 10 test files reported errors that way, one of them 392. `make check-mdl` now sweeps them, with `.fail.test.mdl` for a file whose annotations are deliberately unusable. Two things a failed run must not do, both reported as #1104: **a rejected build is reported with MxBuild's own errors** — `BuildResult.ErrorSummary()` was in hand and discarded by `fmt.Errorf("build failed: %s", build.Message)` on the `--attach` and rebuild paths, and that sentence is identical for every failing build, so it could not tell "your test does not compile" from "an unrelated document is broken"; and **cleanup removes every generated `MxTest.Test_*` the project holds**, not just this suite's. The names are positional and every file reuses them, so keying cleanup on the suite left a flow behind whenever a later run had fewer tests than an earlier one — and one leftover that does not build fails every later run of every test file. What cleanup could not remove is named, with the `DROP` that removes it. Docker keeps the after-startup runner (`--legacy-runner` selects it locally). Packages: `cmd/mxcli/testrunner/` (`endpoint.go`, `client.go`, `watch.go`, `host.go`, `handshake.go`, `check_source.go`, `cleanup_leftovers.go`). See `docs/15-testing/SPIKE_test_endpoint_request_handler.md` -- Warm local dev loop (`mxcli run --local [--watch] [--screenshot]`): Docker-free `mxbuild --serve` + standalone runtime, hot `reload_model` for behavioural changes and restart+DDL for structural ones (chosen from the serve build's `restartRequired`). Bundles the browser client (`web/dist/` via mxbuild's rollup runner, which the serve Deploy target skips) so Mendix 11.x apps render in a browser. `--watch` keeps an incremental rollup bundler hot (CHOKIDAR_USEPOLLING for container fs; ~3-4s page re-bundle, skipped for model-only edits) and watches only model source (`.mpr`+`mprcontents/`). `--ensure-db` provisions the local Postgres + app database if missing; `--setup` does the non-blocking prerequisites (cache mxbuild+runtime, ensure DB) and exits — `mxcli init` wires it into a Claude Code SessionStart hook so a fresh/reaped web session self-bootstraps, and `docs-site/src/tools/bootstrap-prompt.md` is the empty-repo seed prompt. `--screenshot` captures a Playwright PNG each change (pixel-perfect page loop), with `--screenshot-url` deep links (repeatable for multi-page sets, one PNG per page) and `--screenshot-user`/`--screenshot-password` form login (session saved as Playwright storage state, reused via `screenshot --load-storage`). See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` -- External browser preview (`mxcli run --hub ` + `mxcli tunnel-hub`): the app stays local and reverse-tunnels out over a single 443 connection (embedded chisel) to a static relay, so it is reachable in a browser at a public URL — works from egress-only environments (Claude Code web), verified live through the session's MITM egress proxy. `run --hub` implies `--local`, boots the runtime with `ApplicationRootUrl` set to the assigned URL (so the SPA/`originURI` work under the public origin), resolves the control proxy honouring `NO_PROXY`, and retries forever. `mxcli tunnel-hub --domain ` is the **multi-tenant** relay: a registry keyed by prefix/project/solution/branch/worktree (stable URLs on reconnect) fronts many previews at per-subdomain hosts (`[prefix-]project[-branch].`; main collapses to the project) over one 443 with per-subdomain autocert, a registration API (`/api/register|status|deregister|backends|sessions`), and an availability overview at `hub./` **grouped by Claude Code session** (`/api/sessions`): each session lists the endpoints it exposed and links back to its `claude.ai/code` conversation. Client identity flags: `--hub-prefix`/`--hub-project`/`--hub-solution`/`--hub-branch`/`--hub-worktree` (project + branch auto-detected); `--hub-session` groups a session's endpoints (auto-detected from `CLAUDE_CODE_REMOTE_SESSION_ID`). Past sessions are retained: a durable per-session endpoint history (`--sessions-file`, default `~/.mxcli/hub-sessions.json`) survives restarts and reaping, and is pruned after `--session-retention` (default 30d) — so the overview shows offline sessions too (`SessionLog` in `cmd/mxcli/tunnelhub/sessions.go`). Package: `cmd/mxcli/tunnelhub/`. See `docs/11-proposals/PROPOSAL_mxcli_dev_warm_loop.md` (slices 3–4) -- Tunnel-hub GitHub authentication (opt-in, gated on `--github-oauth-client-id`; absent = today's open hub): **viewer plane** — GitHub OAuth web flow + HMAC-signed SSO session cookie (`Domain=.`), owner-checked previews (`--require-auth` default on → 302 to login / 403 non-owner; soft mode filters the listing only), `/api/backends` filtered to the viewer (unauthenticated → 401), admin "signed in as" via `/api/whoami`. **Registration plane** — durable, hashed hub API keys (`--keys-file`, default `~/.mxcli/hub-keys.json`, survive restarts) presented as `X-Hub-Key` → stamps `Backend.Owner`; shared `X-Hub-Secret` still works as an owner-less fallback. **Key issuance** — the hub's `/cli` browser page mints a key from the session cookie (no PAT; the device flow was removed as Claude Code containers block GitHub's device endpoints), rotate-by-default + count + revoke-all; `mxcli auth hub login --token ` is the headless path; `run --hub` reads `MXCLI_HUB_KEY` (env → `~/.mxcli/auth.json`) and degrades to local-only if registration fails. Append-only JSONL audit trail (`--audit-log`, no secrets). Packages: `cmd/mxcli/tunnelhub/` (+`audit/`), `cmd/mxcli/hubauth/`. See `docs/11-proposals/PROPOSAL_hub_authentication.md` -- Runtime metrics + settings passthrough (`mxcli run --local --metrics` / `--runtime-setting Key=Value`): `--metrics` registers a Prometheus Micrometer registry at boot (served at `http://127.0.0.1:/prometheus`); `--runtime-setting` merges arbitrary runtime config (e.g. `Metrics.Registries` for otlp/influx/statsd, or `OpenTelemetry._RuntimeSpanFilters`) into mxcli's **single** boot `update_configuration` call — the admin action replaces rather than merges, so folding settings into the one boot call is the only safe way. OTel traces via `--trace` attach the bundled `opentelemetry-javaagent` to the runtime JVM (console exporter → the tee'd runtime log) and ship default `OpenTelemetry._RuntimeSpanFilters` (unfiltered per-activity tracing is ~10× slower); `--trace-service` sets `OTEL_SERVICE_NAME`. The console exporter omits timestamps/parent span IDs (no flame charts), so `--trace-otlp ` (implies `--trace`) switches to the OTLP exporter (protocol `http/protobuf`) pointed at a collector; user-set `OTEL_*` env still takes precedence. See `.claude/skills/mendix/run-local.md` -- OQL query execution against running runtime (`mxcli oql`) -- Microflow/nanoflow debugger (`mxcli debug`): set breakpoints **by name** (activity resolved from the model), inspect paused flows + variables, step over/into/out, continue — against a `run --local` runtime. Two M2EE planes wired behind one command (admin `enable/disable/status`, app `/debugger/` session); `run --local --debug` enables it at boot. **Nanoflows** are auto-detected (uses the `nanoflow_name` breakpoint param; paused nanoflows are merged from `poll_events`, which `get_paused_microflows` omits). Nanoflow `LOG` output is rewritten to the `Client_Nanoflow` node in the runtime log. See `.claude/skills/mendix/debug-microflows.md` and `docs/11-proposals/PROPOSAL_microflow_debugger.md` -- Business event services (SHOW/DESCRIBE/CREATE/DROP) -- Project settings (SHOW/DESCRIBE/ALTER) -- External SQL query execution against PostgreSQL, Oracle, SQL Server (`mxcli sql`, MDL `sql connect/query`) -- Data import from external databases into Mendix app DB (`import from ... into ... map ...`) -- Database Connector generation from external schema (`sql generate connector into `) -- EXECUTE DATABASE QUERY microflow action (static, dynamic SQL, parameterized, runtime connection override) -- CREATE/DROP WORKFLOW with user tasks, decisions, parallel splits, and other activity types -- ALTER WORKFLOW (SET properties, INSERT/DROP/REPLACE activities, outcomes, paths, conditions, boundary events) -- CALCULATED BY microflow syntax for calculated attributes -- Image collections (SHOW/DESCRIBE/CREATE/DROP) -- Rules (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP/MOVE RULE): Mendix's "special kind of microflow" — returns Boolean or an enumeration, callable only from a decision. Handled as a third flow flavour beside microflows and nanoflows: its own semantic type, its own listing (`show microflows` stays microflow-only), the shared `microflowBody`, flow builder and describer. The document is the ten properties a Studio Pro rule stores, pinned against two reference rules (ako/TestApp, 11.13.0) — **no `AllowedModuleRoles`** (a rule is not independently callable, so there is no `grant execute on rule`) and **no `ReturnType`** despite gen declaring one beside `MicroflowReturnType`. Two keys only a reference document catches, both invisible to `mx check`: `ExportLevel` (Studio Pro writes "Hidden" on every rule) and `Flows` (written as the bare marker even when empty — a `MandatoryLists` entry). Rules are catalog objects and their bodies are walked for references, which together stop a microflow called only from a rule reading as dead. The body restrictions are refused at check time by the same function `exec` calls, each measured: create/change/delete/commit/rollback and client or web-service activities are **CE0009**, a non-Boolean/enum return is **CE0103 + CE0139**. Authoring is modelsdk-only; legacy refuses. See `.claude/skills/mendix/write-rules.md` -- Menu documents (CREATE OR MODIFY/DESCRIBE/DROP MENU): standalone `Menus$MenuDocument`, the reusable menu a menu widget points at (Atlas_Core's `Phone_Menu`/`Tablet_Menu`) — **not** the menu inside a navigation profile, though both are built from the same items, so the item syntax is shared with `CREATE NAVIGATION`'s `MENU (...)` block. DESCRIBE is round-trippable. Written through gen+codec, which is load-bearing: Studio Pro's menu documents carry typed-array marker **3** on the item collection and each item's sub-items (the codec default), while the navigation writers hand-build items with marker **1** — unverified whether that is a latent navigation bug or a real difference, so navigation is left alone. Authoring is modelsdk-only; legacy refuses. Two traps: a menu item cannot open a page with required parameters (**CE1571**), and only `Forms$IconCollectionIcon` round-trips (glyph/image icons are flagged by DESCRIBE, not dropped silently) -- Regular expressions (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP REGULAR EXPRESSION): named patterns that attribute validation rules reference **by qualified name**, which is why they are documents. `modelsdk/gen` is **wrong** about the pattern's key — it binds `RegEx` where every Studio Pro document stores `Expression` (`generated/metamodel` agrees with the documents), so both engines share one raw-BSON codec in `mdl/regularexpressions`; a reader keyed on gen's name returns an empty pattern for every real document. Pinned against five Studio Pro-authored documents (Email Connector 6.4.2, Community Commons 11.5.1). Mendix validates with .NET's engine, so a pattern Go's RE2 cannot compile (lookaround — the Email Connector ships one) is stored unchanged and reported "not verifiable", never "invalid". A `validate` edge into `CATALOG.REFS` makes `show references to ` list the entities using it -- Validation rules (CREATE VALIDATION RULE): binds a **regex** or a **range** to one attribute — `create validation rule for Mod.Entity.Attr regex Mod.Pattern feedback '…'`. The rule is anonymous and entity-scoped, so the statement names the attribute; re-running it replaces the rule of the same type and leaves the attribute's others alone. Unlocked by a `STORAGE-NAME OVERRIDE` in `modelsdk/gen`: it bound `RegularExpression` where Studio Pro stores `RegExIdentifier`, and the control (same script, key reverted) fails **CE0135 "No regular expression specified"** while the fixed one is 0 errors on mxbuild 11.13 with `RegExIdentifier` on disk. Range bounds are inclusive and map to Mendix's only three kinds (`from X to Y`/`from X`/`to Y` → Between/GreaterThanOrEqualTo/SmallerThanOrEqualTo); there is no strict `<`/`>`, and the old grammar's forms for it — plus an EXPRESSION rule type Mendix does not have and an inline regex literal — were removed, having never had a visitor or handler. Required/Unique stay attribute constraints (`not null error '…'` / `unique error '…'`), not a second spelling here. Rewriting an entity carrying **MaxLength or EqualsTo** is **refused** on both engines rather than silently downgraded to Required; that round trip was lossy and `mx check` stayed green, because a Required rule is valid. Both engines carry each rule's payload on READ (`ruleInfoFromGen` / `parseValidationRuleInfo`), which is what makes the refusal narrow instead of covering all of RegEx and Range — and what lets a **range bounded by another attribute** survive a rewrite even though MDL cannot author one (`describe entity` marks it with a comment rather than rendering it wrong). A rule whose payload did not survive the read is refused as firmly as an unknown type: a bare RuleInfo of the right `$Type` constrains nothing, which is the same silent downgrade wearing the right name -- Scheduled events — Mendix's cron (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP). `Repeat:` names one of the eight `ScheduledEvents$*Schedule` variants and only that variant's fields are accepted; a field from another repeat is refused by `mxcli check` (MDL-SCHED01) and by exec, which call the same function. The document shape is pinned by re-serializing three whole Studio Pro-authored events (Workflow Commons 4.11.0, OIDC SSO 4.6.0, SAML 4.2.1) element by element — `modelsdk/gen` is **wrong** about two properties here: the integers are stored as int64 (gen says int32, the #585 mismatch) and `StartDateTime` is a BSON datetime (gen says string), so both engines share one raw-BSON codec in `mdl/scheduledevents`. `Interval`/`IntervalType` are legacy siblings of `Schedule` that Studio Pro writes and does not keep in sync — derived on CREATE, carried through untouched on MODIFY. Only the Day and Hour variants have a Studio Pro reference; the other six are metamodel-derived and verified to load. Both are in the catalog (`CATALOG.SCHEDULED_EVENTS`, `CATALOG.QUEUES`) and a scheduled event emits a `schedule` edge into `CATALOG.REFS` — without it a microflow run only by a scheduled event was reported as dead by `show callers`, `GRAPH_DEAD_ASSETS` and lint rule QUAL004. See `.claude/skills/mendix/scheduled-events-and-queues.md` -- Task queues (LIST/DESCRIBE/CREATE [OR MODIFY]/DROP QUEUE). `Config.ParallelismExpression` is a **string** and the sibling int32 `Parallelism` is not written — matching all four Studio Pro queues in Business Events 3.12.1. Binding a *call* to a queue is not yet authorable, so `CREATE OR REPLACE|MODIFY MICROFLOW` is **refused** when the stored microflow has a queued call (guard-don't-drop, ADR-0005): the rebuild used to write `QueueSettings` back as null, which made `mx check` go from CE1613 to 0 errors by deleting the user's configuration -- AI agent documents: Model, Knowledge Base, Consumed MCP Service, Agent (LIST/DESCRIBE/CREATE/DROP, with variables, tools, KB tools, dollar-quoted multi-line prompts; requires AgentEditorCommons module, Mendix 11.9+) -- OData contract browsing (SHOW/DESCRIBE CONTRACT ENTITIES/ACTIONS FROM cached $metadata) -- AsyncAPI contract browsing (SHOW/DESCRIBE CONTRACT CHANNELS/MESSAGES FROM cached AsyncAPI) -- SHOW EXTERNAL ACTIONS, SHOW PUBLISHED REST SERVICES -- CREATE/DROP/DESCRIBE PUBLISHED REST SERVICE with resources, operations, path params, CREATE OR REPLACE -- Integration catalog tables (rest_clients, rest_operations, published_rest_services, external_entities, external_actions, business_events) -- Contract catalog tables (contract_entities, contract_actions, contract_messages — parsed from cached $metadata and AsyncAPI) -- Platform authentication (`mxcli auth login/logout/status/list`) with PAT scheme for marketplace-api.mendix.com, marketplace.mendix.com, and catalog.mendix.com; credentials stored at ~/.mxcli/auth.json (mode 0600), MENDIX_PAT env override -- Marketplace browsing (`mxcli marketplace search/info/versions`) with --min-mendix compatibility filtering -- Marketplace download/install (`mxcli marketplace download/install`) — the content API now exposes a per-version downloadUrl (303→public CDN); install is type-aware (widget→widgets/, new module→`mx module-import`); existing-module updates are reported, not applied (entity-ID/local-edit safety — see PROPOSAL_marketplace_modules.md) -- Marketplace drift detection (`mxcli marketplace diff -p app.mpr [--to VERSION] [--json]`): reports **which elements of an installed marketplace module have been edited locally** — the question Studio Pro's Marketplace update never asks before replacing the module. The version's `.mpk` is downloaded and imported into a throwaway reference project built **at the consuming project's Mendix version** (a mismatch is refused, not warned about: Mendix's own conversions would read as user edits), then every element is described on both sides and the **DESCRIBE output** compared — not BSON, in which an *untouched* module differs from its own package in ~15,000 paths. `--to` adds what an upgrade would touch and which of those collide with local edits. Honesty rule: an element that cannot be described is reported **unknown, never unchanged**, and `verified:false` in the JSON means "no modifications found" is not a conclusion. Module + version are identified from the module's `AppStoreGuid`, which is the marketplace **version UUID** — matching on the version *number* is ambiguous (a blank project has Atlas_Web_Content 4.1.0 and Administration's content also published a 4.1.0). Measured on real content: Administration 4.3.2 in a blank 11.12.1 app → 21/21 unchanged; one added attribute → exactly `ENTITY Account`; `--to 4.3.2` (the installed version) touches nothing, which is the control for `--to 4.5.0`'s five. Package: `cmd/mxcli/marketplace/`. See `docs/11-proposals/PROPOSAL_marketplace_module_upgrade.md` - -**Not Yet Implemented:** -- 47 of 52 metamodel domains (REST, etc.) -- Delta/change tracking system -- Runtime type reflection +## What mxcli Can Do + +**This file does not list features.** `./bin/mxcli syntax` enumerates every MDL +statement (`--json` for bulk), `./bin/mxcli help ` documents each command, +and `./bin/mxcli lint --list-rules` names every rule. A list here is a transcription +of what those answer authoritatively, and it goes stale the next time anything ships. + +Per-doctype gotchas, CE numbers and the measurements behind them live in the skill +for that doctype (`.claude/skills/mendix//SKILL.md`) — loaded when you touch +that area rather than re-read into every session. Design rationale lives in +`docs/11-proposals/`; cross-cutting decisions in `docs/13-decisions/`. + +Still absent: 47 of 52 metamodel domains, delta/change tracking, runtime type +reflection. ## Useful Files for Context From 68c99a75a189a775c93977ae3f35482334a11c52 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 14:17:22 +0000 Subject: [PATCH 27/38] docs(gates): tighten the batching paragraph I added, reclaiming headroom MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The generated CLAUDE.md is budgeted at 6,000 bytes because it is re-read into every context a project starts. 3065f68f added the once-per-change paragraph and took it to 5,857, leaving 143 bytes — tight enough that the next contributor to touch the file would have had to buy space from something else. Four lines to three, and the "~55s a time" figure goes: the gate list two lines below already shows ~25s for docker check and ~30s cold for test, so the reader adds up numbers that are on screen rather than being told a total that can drift from them. 5,857 -> 5,805, headroom 195. The marker phrase the three-copy test holds ("not per edit") is unchanged, and the skill and docs-site copies keep their own longer wording — neither is re-read per session. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- cmd/mxcli/init_claudemd.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/cmd/mxcli/init_claudemd.go b/cmd/mxcli/init_claudemd.go index 4ea8890c68..f65f4ef86f 100644 --- a/cmd/mxcli/init_claudemd.go +++ b/cmd/mxcli/init_claudemd.go @@ -179,10 +179,9 @@ func generateClaudeMD(projectName, mprFile string) string { w("Run them cheapest-first; each is only worth paying for once the one above is clean.\n") w("**They are the definition of done, not a menu** — a change is finished when they have\n") w("all been run and you have said what each one reported.\n\n") - w("**Once per change, not per edit.** A change is a coherent unit of work — not a single\n") - w("statement and not a file write. Iterate with " + bt + "exec" + bt + " until the script is right, then run\n") - w("the gates once over the result. The whole list after every edit costs ~55s a time and\n") - w("proves nothing the one run at the end does not.\n\n") + w("**Once per change, not per edit** — a change being a coherent unit of work, not a\n") + w("statement and not a file write. Iterate with " + bt + "exec" + bt + ", then run the gates once over the\n") + w("result: the whole list after every edit proves nothing the one run at the end does not.\n\n") w(bt3 + "bash\n") w(renderProjectGates(mprPath)) w(bt3 + "\n\n") From 9568492534f252cf5cda4042945fa5d271aa2199 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 14:51:09 +0000 Subject: [PATCH 28/38] docs: move Key Concepts deep-dives out of CLAUDE.md (#611) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second slice of #611. Key Concepts was 26% of the file: 27,733 B, now 8,101. CLAUDE.md is 108,761 -> 53,950 across both slices (50%). The rule applied per subsection: a concept stays only if it is non-inferable, causes silent unrecoverable damage when violated, AND applies regardless of what is being touched. What stays is storage names, GUID-as-database-identity, the three conditional-write rules, the reserved-member names, association pointer inversion, expression escaping. Everything else is needed only when touching one subsystem. Relocated, because the content lives nowhere else: the gen wrong-key ledger and overlay-write rules to MODELSDK_ENGINE_ARCHITECTURE, TypeEnumeration ambiguity to MDL_PARSER_ARCHITECTURE, pluggable widget templates to diagnose-ce0463, AfterStartupMicroflow to project-settings, the fluent API to README. Compressed rather than moved where the canonical home already had it, verified by probing distinctive claims rather than counting greps: the tunnel section against ADR-0009 (4/6 — the two it lacked, the never-obfuscate rule and the seam file names, are kept in the compressed text) and the theme section against theme-styling/SKILL.md (6/7, the seventh a Go constant name that grep finds). That probe is why "Writes Are Conditional" was relocated instead: ADR-0008 predates the transplant work and has none of TransplantIDs, dropCollisions or carryIdentityFromRemovedUnit, and the ADR is immutable, so the mechanism moved to the docs-site internals page it already cited. Its three rules stay in CLAUDE.md — each has been violated once, and #125 shipped green without them. Public API Pattern is dropped outright; README documents it. Every moved section was probed in its destination after the move (21/21 distinctive strings present). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .claude/skills/diagnose-ce0463.md | 8 + .../skills/mendix/project-settings/SKILL.md | 6 + CLAUDE.md | 390 ++---------------- README.md | 26 ++ docs-site/src/internals/idempotent-writes.md | 111 +++++ .../03-development/MDL_PARSER_ARCHITECTURE.md | 4 + .../MODELSDK_ENGINE_ARCHITECTURE.md | 91 ++++ 7 files changed, 285 insertions(+), 351 deletions(-) diff --git a/.claude/skills/diagnose-ce0463.md b/.claude/skills/diagnose-ce0463.md index b8ae6e48b6..1dd8396fc1 100644 --- a/.claude/skills/diagnose-ce0463.md +++ b/.claude/skills/diagnose-ce0463.md @@ -196,3 +196,11 @@ Ordered by how often they have actually been the answer. - **Test any candidate fix against the bundled package too.** Pruning the fields the `update-widgets` reference omits fixes 2 widgets on Data Widgets 3.10 and takes the bundled 3.4 from **0 → 139**. + +## Pluggable Widget Templates + +For pluggable widgets (DataGrid2, ComboBox, Gallery, etc.), templates must include **both** `type` AND `object` fields: +- `type`: Widget PropertyTypes schema (defines what properties exist) +- `object`: Default WidgetObject with all property values + +**CE0463 "widget definition changed" error**: This error occurs when the Object's property structure doesn't match the Type's PropertyTypes. Always extract templates from Studio Pro-created widgets, not programmatically generated ones. See `sdk/widgets/templates/README.md` for details. For debugging CE0463 and other BSON issues, follow the workflow in `.claude/skills/debug-bson.md`. diff --git a/.claude/skills/mendix/project-settings/SKILL.md b/.claude/skills/mendix/project-settings/SKILL.md index cd32d8e89b..4d38845db8 100644 --- a/.claude/skills/mendix/project-settings/SKILL.md +++ b/.claude/skills/mendix/project-settings/SKILL.md @@ -303,3 +303,9 @@ alter settings configuration 'Default' - [ ] Model setting key names are case-sensitive (e.g., `JavaVersion`, not `javaversion`) - [ ] Configuration names are case-insensitive (e.g., `'default'` matches `'default'`) - [ ] Integer / Boolean settings must parse — `mxcli check` reports MDL-SET01 / MDL-SET02 before the write + +## AfterStartupMicroflow Must Return Boolean + +A microflow wired as the project's **after-startup** microflow must return `Boolean` — Mendix build fails with **CE0142** on a void (no-return) microflow. A common trip-up: a seed/demo-data microflow wired to after-startup will not build until it ends with a `return true` (Boolean). + +`mxcli check` now reports it (**MDL073**), which it could not before: #274 made `ALTER SETTINGS` resolve the qualified names it writes, but the name here *resolves* — the constraint is on the thing it names, not on the reference. The check runs with **no project** when the script creates the microflow itself (the usual shape), and against the stored return type when it does not. A microflow whose return type cannot be established is left alone rather than guessed at. `BeforeShutdownMicroflow` and `HealthCheckMicroflow` are deliberately **not** type-checked — their rules have not been measured here. diff --git a/CLAUDE.md b/CLAUDE.md index f26e92edaa..be537e7e5f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -179,70 +179,6 @@ When adding new types, always verify the storage name by: **IMPORTANT**: When unsure about the correct BSON structure for a new feature, **ask the user to create a working example in Mendix Studio Pro** so you can compare the generated BSON against a known-good reference. -### Pluggable Widget Templates - -For pluggable widgets (DataGrid2, ComboBox, Gallery, etc.), templates must include **both** `type` AND `object` fields: -- `type`: Widget PropertyTypes schema (defines what properties exist) -- `object`: Default WidgetObject with all property values - -**CE0463 "widget definition changed" error**: This error occurs when the Object's property structure doesn't match the Type's PropertyTypes. Always extract templates from Studio Pro-created widgets, not programmatically generated ones. See `sdk/widgets/templates/README.md` for details. For debugging CE0463 and other BSON issues, follow the workflow in `.claude/skills/debug-bson.md`. - -### `modelsdk/gen` Binds Some Properties Under the Wrong BSON Key - -The storage-name table above is about `$Type`. The **same split exists per -property**, and `modelsdk/gen` gets it wrong in **102 properties across 65 -types** — the ledger is `modelsdk/gen/keyaudit_test.go`. Mendix's reflection -data carries two names per property — an SDK `Name` and a BSON `StorageName` — -and the in-repo generator (`cmd/codegen` → `generated/metamodel`) keeps them -apart, tag from storage name: - -```go -// generated/metamodel/types.go — correct -RegularExpression model.QualifiedName `json:"regExIdentifier,omitempty"` -// ^ SDK name ^ storage name -``` - -The generator behind `modelsdk/gen` reads a **different input** — the TypeScript -SDK's compiled JS, which does not contain storage names at all (measured: -`regExIdentifier` occurs 0 times in `mendixmodelsdk` 4.114.0) — and patches them -back via a hand-maintained `PropertyKeyOverrides` table. - -**`generated/metamodel` is therefore the arbiter when the two disagree**, with -one caveat: it is a **snapshot of 11.6.0** (see its header), so it is sound for -the properties it contains but says nothing about ones introduced later — for -those, get a real document. It has been right in every case checked that way -(`RegularExpression.Expression`, `RegExRuleInfo.RegExIdentifier`, `Attribute.GUID`). -`TestGenPropertyKeysAgainstMetamodel` fails when a NEW mismatch appears (a -re-vendored gen that dropped an override) or when a listed one is fixed without -being struck off. Why the generator is not simply brought in-tree, and what it -would take: [PROPOSAL_codegen_ownership.md](docs/11-proposals/PROPOSAL_codegen_ownership.md). - -`cmd/modelsdk-codegen` and `internal/codegen/supplements.json` — named in every -gen file's `DO NOT EDIT` header — have **never existed in this repo** -(`git log --all` is empty for both), and `/reference/` is gitignored, so the -generator's input is absent too. gen is vendored output that cannot be -regenerated here; see `docs/plans/2026-06-05-adopt-modelsdk-engine.md` §4, where -"vendor engalar codegen" is still an open Phase-0 item. - -So the fix for a wrong key is a **hand-applied override in the `init` -function**, commented in the house style (grep `STORAGE-NAME OVERRIDE` for the -four precedents). Two rules: - -1. **Patch both sides.** The encode key (`init`) and the decode key - (`InitFromRaw`) are separate literals. Patching one gives a document that - writes one key and reads another — which the entity-rewrite guard then - refuses, so the symptom is a puzzling refusal rather than a wrong file. -2. **`gofmt` the file**, or `TestGeneratedCodeIsFormatted` fails. - -Not every wrong key is worth patching — leave the ones nothing writes, and note -why. `mx check` is a weak signal here either way: it caught the RegEx one -(CE0135) but tolerates unknown properties in general, and Studio Pro is stricter -than mxbuild. - -### TypeEnumeration vs TypeEntity Ambiguity - -The MDL visitor (`buildDataType` in `visitor_helpers.go`) cannot distinguish between entity types and enumeration types for bare qualified names like `Module.EntityName`. Both parse as `ast.TypeEnumeration` with `EnumRef` set. Code that consumes data types must handle `TypeEnumeration` alongside `TypeEntity` and use `EnumRef` as a fallback for the entity name. - ### Mendix Expression String Escaping When generating Mendix expression strings (e.g., in `expressionToString()`), single quotes within string literals must be escaped by doubling them: `'it''s here'`. Do NOT use backslash escaping (`\'`). This matches Mendix Studio Pro's expression syntax. @@ -256,51 +192,6 @@ The skills advise **quoting all identifiers** to avoid keyword collisions, but t The reserved-word lists live in `mdl/executor/cmd_enumerations.go` (`mendixReservedWords`, `mendixSystemAttributeNames`). "Always safe to quote" in the skills means *parser*-safe, not *platform*-safe. -### AfterStartupMicroflow Must Return Boolean - -A microflow wired as the project's **after-startup** microflow must return `Boolean` — Mendix build fails with **CE0142** on a void (no-return) microflow. A common trip-up: a seed/demo-data microflow wired to after-startup will not build until it ends with a `return true` (Boolean). - -`mxcli check` now reports it (**MDL073**), which it could not before: #274 made `ALTER SETTINGS` resolve the qualified names it writes, but the name here *resolves* — the constraint is on the thing it names, not on the reference. The check runs with **no project** when the script creates the microflow itself (the usual shape), and against the stored return type when it does not. A microflow whose return type cannot be established is left alone rather than guessed at. `BeforeShutdownMicroflow` and `HealthCheckMicroflow` are deliberately **not** type-checked — their rules have not been measured here. - -### Overlay Writes: Never Invent a Key, Branch on `$Type` - -When a write overlays fields onto preserved BSON (`mdl/settingsoverlay`, and any -future storage that follows ADR-0005 guard-don't-drop), two rules are load-bearing. -Breaking either produces a document `mx check` accepts and **Studio Pro cannot -open**: it resolves every stored property against the type's property list and -throws `System.InvalidOperationException: Sequence contains no matching element` -at `MprProperty.cs`. mxbuild's deserializer tolerates unknown properties, so the -build is not a safety net here. - -1. **Write only keys the document already carries.** Property names are - version-specific — Mendix renamed `JavaVersion` (`"Java21"`) to - `JavaMajorVersion` (`"21"`) and `Tracing` to `OpenTelemetry` between 11.6 and - 11.12. Read the key off the stored document and write back to that same key; - when neither is present, write neither (an absent optional property is filled - in on load). See `settingsoverlay.JavaVersionKey` (#759). -2. **A polymorphic child must be dispatched on `$Type` before any field - assignment.** Variants can differ in *arity*, not just field values: - `Settings$SharedValue` carries a `Value`, while `Settings$PrivateValue` is a - bare marker with no properties at all (the value lives on the developer's - workstation). Assigning `Value` to whichever node is there corrupts the marker. - -The same reasoning bans authoring what the model does not own: mxcli preserves a -constant override's shared/private choice and refuses statements that would flip -it, rather than silently converting one to the other. - -Enum-valued properties are the sibling trap: validate against -`generated/metamodel` (e.g. `SettingsDatabaseType` is `Hsqldb`, never `HSQLDB`) -rather than passing a user string through. - -**On a CREATE there is no stored document to read the key off.** Rule 1 then -becomes: branch on the project's Mendix version and write exactly one spelling — -never both as a hedge. `mdl/dbconnector` does this for the 11.13 rename of -`DatabaseQuery.QueryType` (int) to `Type` (string enum), which mxbuild *does* -catch, as CE5277 on every activity using the query. To learn the target shape -without guessing, run the new mxbuild's own migration over an old project -(`mx convert -p -s `) and diff the BSON: Mendix ships a one-time -conversion per renamed property, so the converted document is authoritative. - ### A `GUID` Is the Database's Identity — Never Mint One for an Existing Element An element's `GUID` is not decorative and is not interchangeable with its `$ID`. @@ -328,220 +219,53 @@ Consequences for any write path: model must not keep the source's — two elements sharing a `GUID` are one entity as far as the runtime is concerned. -### Writes Are Conditional, and an `$ID` Is Never Renumbered In Place - -Storage does not write a unit whose new content is **semantically equal** to what -is stored ([ADR-0008](docs/13-decisions/0008-identity-and-idempotence.md)). The -comparison is on a canonical form — every element `$ID` replaced by its index in a -containment walk — because a rebuild mints a fresh random `$ID` per sub-element, -so comparing bytes would skip nothing. The policy lives in `modelsdk/canon` -(`Reconcile`) and is called at every write choke point in -`modelsdk/mpr/writer_core.go`: `updateUnit`, `WriteTransaction.WriteUnit` -(`codec.Store` reaches storage through this one) and — since ako/mxcli#556 — -`insertUnit`, for the case below. - -**A delete followed by an insert is a write path too**, and it is the one that -hides. Several `create or modify` handlers are implemented as delete + create -under the preserved unit ID rather than as an update, and an insert has nothing -stored to reconcile against, so the rebuild's fresh `$ID`s went straight to disk: -`create or modify rest client` rewrote 9 element `$ID`s in a 1,128-byte unit on -every run, forever. `deleteUnit` now remembers what it removed and `insertUnit` -reconciles a re-insert against it (`carryIdentityFromRemovedUnit`). That carry -cannot *elide* — the row and the file are already gone — so a no-op recreate also -restores `_Transaction.LastTransactionID`, which both the delete and the insert -bumped; without it the `.mpr` still showed as modified after every `.mxunit` had -gone quiet. Prefer an in-place update where the handler can do one: the REST -client's own fix is to call `UpdateConsumedRestService` and keep delete+create -only for a folder move, which lives in the unit's row rather than its contents. - -**The carry keys on the unit ID, so it does not reach a handler that re-mints -one** — and three handlers did, in one week: the REST client (#556), the view -entity's OQL document (#583) and the layout (ako/mxcli#600). All three took the -same fix, an in-place `UpdateRawUnit` rather than a replacement. The tell is -cheap and worth reaching for first: `ls` the `.mxunit` filenames across two -identical runs. A **changed filename** is delete+insert and the handler is -wrong; a **same filename with different bytes** is the codec or a missing carry -and `canon` is where to look. A replacement also silently reverts the unit's -ROW, which is how `create or replace layout` moved a foldered layout back to the -module root on every rewrite — there is no `FOLDER` clause on the statement, so -the rebuild always names the module root and only an insert applies it. - -When something *has* changed, `Reconcile` still does not let the rebuild's fresh -`$ID`s reach disk: `canon.TransplantIDs` matches the incoming document against the -stored one element by element (by `$Type` and shape, by `Name` where there is one, -LCS-anchored within each list) and puts the **stored** `$ID` back on every element -that still corresponds. Without it a one-argument edit re-minted 36 of a nanoflow's -37 element identities and Studio Pro painted the whole document as changed (#910). -Its correctness bar is lower than it looks and worth knowing: a *wrong* match only -makes a diff bigger, because every reference is rewritten with the element — the -one real failure is two elements sharing an `$ID`, which `dropCollisions` guards. - -Three rules follow, and each has already been violated once: - -1. **Never rewrite an element `$ID` without rewriting every reference to it in the - same pass.** Pointers are *primitive* properties holding an `element.ID`, not - `ChildProperty`, so a containment walk traverses the whole document and never - sees one. PR #125 renumbered IDs this way and made projects unopenable - (`KeyNotFoundException` at `ResolvePostponedProperties`). A unit is rewritten - wholesale or not at all. The transplant obeys this by substituting over *every* - 16-byte binary in the document rather than a maintained list of pointer - properties — any occurrence of one of the document's element IDs is a reference - by definition, the same insight the canonical form rests on. -2. **Adding a write path means wiring it to `canon.Reconcile`.** A new choke point - that writes directly will silently churn while everything else is quiet — the - worst kind of inconsistency, because the diff blames the wrong change. -3. **A new document type with an identity property needs a row in - `canon.identityFields`.** It cannot be generated: Mendix's `IsIdentifier` lives - in the modeler assemblies, not in the reflection data `generated/metamodel` is - built from. `TestFreshGUIDFieldsHaveAnIdentityDecision` catches the common case - (a property the codec mints fresh on every write) but cannot catch an identity property - the codec does not mint. Establish the property's status the way `StableId` was - — the method table is in ADR-0008. - -Elision itself is type-agnostic and covers new document types for free, but it -assumes **no binary pointer crosses a unit boundary** (measured 0 of 9,910, not -enforced). A document type that references another *unit* by `$ID` rather than by -qualified name breaks that assumption and invalidates the argument in ADR-0008. - -`MXCLI_ALWAYS_WRITE=1` forces every write to land, for bisecting. It does not -disable identity preservation. **Any test asserting "nothing changed" must include -a control** — otherwise the test passes against a build that never had the fix, -which is exactly how PR #125 shipped green. Note what the control can now be: -since identities are carried, a forced write of an in-sync unit produces the -**same bytes**, so "flip `MXCLI_ALWAYS_WRITE` and watch the content change" no -longer distinguishes anything (measured: same sha, mtime moves). Control on the -**rebuild** instead — encode the document twice and show the raw codec output -differs (`TestRebuildChurnsSubElementIDs`) — or, from the shell, on **mtimes** -rather than hashes. - -The executor reports which of the two happened: a statement whose unit writes were -all elided prints `Unchanged nanoflow: …` instead of `Replaced nanoflow: …` -(`ExecContext.ReportMutation`, fed by each writer's `WriteStats`). The verb is only -downgraded on positive evidence — writes offered, none landed — so a mutation that -never touches unit storage is reported exactly as before. - -**Several elisions in one run collapse into one line**, because that report is the -most repeated thing mxcli prints and an agent pays for it on every later model -call (a tool result is written into the conversation once and re-read by each one). -Measured on a settled 40-statement script: 41 lines / 1,604 B became 2 lines / -177 B. Two rules keep it honest, and each was arrived at by getting it wrong: - -1. **Only `Unchanged` collapses.** It is the one verb that by construction reports - an absence, so no line a reader would act on is ever replaced by a number — - a mixed run still names every real write individually and counts only the rest. - Collapsing on volume instead ("after N lines") would hide real writes in exactly - the runs where they matter. -2. **The trigger is how many arrive, not which entry point ran.** A lone elision is - printed verbatim, since "1 document already in sync" is worse than the line it - replaces. Gating on "is this a script?" looked equivalent and is not: `-c` - reaches `ExecuteProgram` too, because `executeMDL` prepends a `CONNECT` - statement, so a one-liner collapsed to a count of one. - -`mutationTally` (`mdl/executor/mutation_tally.go`), active only inside a program run. - ### The Tunnel Is Linux-Only, On Purpose — Do Not "Restore" It -`mxcli run --hub` and `mxcli tunnel-hub` embed [chisel](https://github.com/jpillora/chisel), -a dual-use tunnelling tool that appears in threat intelligence as a pivoting -component. Shipping it in the Windows and macOS binaries — where the tunnel can -never run — got them flagged by Defender (`Trojan:Script/Sabsik.EN.A!ml`) and -denied by enterprise EDR, which blocks mxcli for corporate Mendix developers on -managed endpoints. It is now built **for Linux only**. See +`run --hub` / `tunnel-hub` embed chisel, which got the Windows and macOS builds +flagged by Defender and denied by enterprise EDR. Linux-only is the fix, not a +portability gap: making it cross-platform again re-introduces the detection for +most downloads. Reasoning and alternatives in [ADR-0009](docs/13-decisions/0009-tunnel-is-linux-only.md). -This looks like a portability gap and is not one. Making the tunnel cross-platform -again re-introduces the detection for the large majority of downloads. - -- **All chisel imports live behind two seams**, one interface each: - `tunnelConn` / `startTunnel` (`cmd/mxcli/docker/tunnel_linux.go` + `tunnel_other.go`) - and `controlServer` / `newControlServer` (`cmd/mxcli/tunnelhub/control_linux.go` - + `control_other.go`). Adding a chisel import anywhere else is the mistake the - guard exists to catch. -- **`scripts/check-tunnel-deps.sh` (CI, and `make check-tunnel-deps`) fails the - build** if chisel or its tunnelling-specific dependencies — the SSH/websocket/ - socks stack included, which is how it would come back without the word "chisel" - appearing — reach a windows/darwin dependency graph. It asserts a positive - control first (chisel *is* in the linux graph), so it cannot pass vacuously. -- **The hub seam is at `Start`, not construction**, so the portable front - (registry, API, auth, routing) stays testable on every platform. +Two rules that are not in the ADR: + - **Never obfuscate, pack, or rename to evade detection.** That is attacker - tradecraft and makes things strictly worse. The only legitimate fix is not - shipping the capability where it is unused. Code signing does **not** substitute: - a signed binary containing chisel is still flagged behaviourally. -- Do not conflate this with #185 (`Wacatac.C!ml`), which was a genuine generic - Go-binary false positive with a different remedy. + tradecraft and makes things strictly worse; code signing does not substitute, + because a signed binary containing chisel is still flagged behaviourally. +- **Every chisel import lives behind one of two seams** (`tunnel_linux.go` / + `tunnel_other.go`, `control_linux.go` / `control_other.go`). An import anywhere + else is what `make check-tunnel-deps` exists to catch. ### Theme Files: Where SCSS Actually Compiles -Styling written to the wrong place fails **silently** — the build succeeds and the -rules are simply absent, which is indistinguishable in the browser from a -specificity problem. Verified on Mendix 11.13 (probe rules compiled, then grepped -out of `theme-cache/web/theme.compiled.css`): - -- **`theme/web/main.scss` compiles LAST** — after Atlas Core *and* after every - module theme source. A partial imported from it overrides any Atlas rule with no - `!important`. This is the home for app-level styling (Layer 2), and it is a - three-line file of Mendix's own imports, not an Atlas-owned file. -- **`themesource//` is only compiled when `` matches a real module.** - mxbuild walks the model's modules; it never globs the directory. An invented - folder is skipped without a warning. Use a module's theme source only when the - styling belongs to that module. -- **`theme/web/custom-variables.scss` is imported once per module** (8× in a blank - app), so it must hold **declarations only** — a rule there is emitted N times. - Tokens go here (Layer 1); rules go in the partial. -- **Mendix 11 Atlas is CSS-custom-property-first**: `:root { --brand-primary: … }`, - not SCSS `!default`. The derived ramp is CSS `color-mix()` against - `var(--brand-primary)`, so retuning the primary re-derives it live. - -A fifth, learned by putting three themes in one stylesheet: **a theme is almost -entirely token values.** The Atlas map, the recipe layer and the widget layer are -byte-identical across all three built-ins (measured: one hash; 174 lines of -recipes), and every colour in them resolves through `var(--mxt-*)` — only the -palette, the fonts and 3–8 lines of skin differ per theme. That is what makes -`theme apply ` a class swap rather than a rebuild, and it is a rule -for anything added to those layers: **a literal colour outside the palette -survives the swap and is wrong under every theme but one.** The default theme's -scope is `:root` *minus* the other skins' classes, never a bare `:root` — bare -keeps matching once another class is set, so the outcome would come down to -specificity instead of being mutually exclusive by construction. A Sass variable -holding a selector must be a **quoted string** (`$s: ":root, :root.mxt-x"`); -a bare selector is not a Sass expression and `mx check` never sees it, because -the failure is at SCSS compile time. - -`cmd/mxcli/theme` encodes all four. Its embed uses `//go:embed all:assets` — a -plain `go:embed assets` skips `_`-prefixed files, which is exactly how SCSS spells -a partial. Files the project already owns are written as digest-fenced blocks -(guard-don't-drop, as in ADR-0005): a block with local edits is refused, not -overwritten. - -The registry reads two sources: the embedded themes and the project's own, under -**`theme/mxcli-themes//`** (`theme.LocalThemesDir`), a local one shadowing -an embedded one of the same name. That path is fixed by two constraints — it must -be **committed** (a design-derived theme is source the team shares, which rules -out `.mxcli/`, gitignored by `mxcli init`) and **not compiled** (mxbuild's entry -point is `theme/web/main.scss`; it does not glob `theme/`, verified against an -11.13 build). `theme create` scaffolds one by copying an existing theme and -renaming the identifiers built from the name (`@mixin mxcli--`, the -`@import`) — a copy that skips that rename collides the moment both themes exist. -`--from ` seeds the palette from `--mxt-*` declarations in any CSS-shaped -text; **an unrecognised `--mxt-*` name is refused, not written**, because nothing -reads it — the theme would apply cleanly and render unchanged, which is -indistinguishable from the design not having been applied at all. - -Two more, learned by flipping the variant on a running app: - -- **Atlas ships `:root.theme-dark` / `:root.theme-neutral` in `theme/web/` but - nothing that applies them** — the slot exists, the switcher does not. A theme's - own dark block must be declared at `:root.theme-dark` *after* Mendix's - `_theme-dark.scss` (same specificity, later wins), or the app reverts to stock - Mendix blue the moment the class appears. Because the class lands on ``, - popups and modals rendered at `` follow it too. -- **Never pin an Atlas leaf to a literal colour.** Map it to a theme token - (`--bg-color: var(--mxt-ground)`) so a variant restates ~30 values instead of - ~60. A hardcoded `--font-color-default` is invisible the moment the ground goes - dark. Two Atlas rules also assume a *dark navigation rail* and paint topbar text - with `--color-base`, so every mxcli theme keeps the rail dark in both variants - and forces `color: inherit` on those widgets. +Styling written to the wrong place fails **silently** — the build succeeds and +the rules are simply absent. Which file compiles, in what order, and why a +literal colour outside the palette is wrong under every theme but one: +`.claude/skills/mendix/theme-styling/SKILL.md`. + +### Writes Are Conditional, and an `$ID` Is Never Renumbered In Place + +Storage does not write a unit whose new content is semantically equal to what is +stored ([ADR-0008](docs/13-decisions/0008-identity-and-idempotence.md)), and when +a write does land the stored element `$ID`s are carried onto it rather than +replaced. Mechanism, measurements and the reporting rules: +[idempotent-writes](docs-site/src/internals/idempotent-writes.md). + +Three rules, each already violated once: + +1. **Never rewrite an element `$ID` without rewriting every reference to it in the + same pass.** Pointers are primitive properties holding an `element.ID`, so a + containment walk never sees one. PR #125 renumbered this way and made projects + unopenable. A unit is rewritten wholesale or not at all. +2. **A new write path must be wired to `canon.Reconcile`.** One that writes + directly churns silently while everything else is quiet, so the diff blames the + wrong change. +3. **A new document type with an identity property needs a row in + `canon.identityFields`.** It cannot be generated — Mendix's `IsIdentifier` is + not in the reflection data. + +**Any test asserting "nothing changed" must include a control.** Without one it +passes against a build that never had the fix, which is how PR #125 shipped green. ### Association Parent/Child Pointer Semantics (Counter-Intuitive) @@ -560,42 +284,6 @@ This affects **entity access rules**: MemberAccess entries for associations must The same convention applies in `domainmodel.Association`: `ParentID` = FROM entity, `ChildID` = TO entity. -### Public API Pattern -```go -// read-only access -reader, err := modelsdk.Open("/path/to/project.mpr") -defer reader.Close() - -// read-write access -writer, err := modelsdk.OpenForWriting("/path/to/project.mpr") -defer writer.Close() -``` - -### High-Level Fluent API (in api/) -The `api/` package provides a simplified, fluent API inspired by Mendix Web Extensibility Model API: - -```go -a, err := api.Open("/path/to/project.mpr") // or api.New(b) over any backend -defer a.Close() - -module, _ := a.Modules.Get("MyModule") -a.SetModule(module) - -entity, _ := a.DomainModels.CreateEntity("Customer"). - persistent(). - WithStringAttribute("Name", 100). - WithIntegerAttribute("Age"). - build() -``` - -Available namespaces: `DomainModels`, `enumerations`, `microflows`, `pages`, `modules` - -It takes a **`backend.FullBackend`, not a `*mpr.Writer`** — it used to hold a concrete legacy -writer and so bypassed the backend abstraction entirely, which is why `AddAttribute` and -`UpdateAttribute` sat unimplemented on the codec engine with `api/` as their only caller. The -practical gain is that the same builders now run against any backend, including a live Studio Pro -over MCP, which was unreachable before. `Open` owns the connection it makes; a backend passed to -`New` belongs to the caller and `Close` leaves it alone. ## Code Style Guidelines diff --git a/README.md b/README.md index 624703250f..a80b287a26 100644 --- a/README.md +++ b/README.md @@ -668,3 +668,29 @@ Apache License 2.0 - See [LICENSE](LICENSE) for details. ## Contributing Contributions are welcome! Please feel free to submit a Pull Request. + +## High-Level Fluent API (in api/) +The `api/` package provides a simplified, fluent API inspired by Mendix Web Extensibility Model API: + +```go +a, err := api.Open("/path/to/project.mpr") // or api.New(b) over any backend +defer a.Close() + +module, _ := a.Modules.Get("MyModule") +a.SetModule(module) + +entity, _ := a.DomainModels.CreateEntity("Customer"). + persistent(). + WithStringAttribute("Name", 100). + WithIntegerAttribute("Age"). + build() +``` + +Available namespaces: `DomainModels`, `enumerations`, `microflows`, `pages`, `modules` + +It takes a **`backend.FullBackend`, not a `*mpr.Writer`** — it used to hold a concrete legacy +writer and so bypassed the backend abstraction entirely, which is why `AddAttribute` and +`UpdateAttribute` sat unimplemented on the codec engine with `api/` as their only caller. The +practical gain is that the same builders now run against any backend, including a live Studio Pro +over MCP, which was unreachable before. `Open` owns the connection it makes; a backend passed to +`New` belongs to the caller and `Close` leaves it alone. diff --git a/docs-site/src/internals/idempotent-writes.md b/docs-site/src/internals/idempotent-writes.md index 96f65fbe10..930cb877a9 100644 --- a/docs-site/src/internals/idempotent-writes.md +++ b/docs-site/src/internals/idempotent-writes.md @@ -137,3 +137,114 @@ canonical digests keyed by unit id. See [ADR-0008](https://github.com/ako/mxcli/blob/main/docs/13-decisions/0008-identity-and-idempotence.md) for the decision and the measurements behind it. + +## Writes Are Conditional, and an `$ID` Is Never Renumbered In Place + +Storage does not write a unit whose new content is **semantically equal** to what +is stored ([ADR-0008](docs/13-decisions/0008-identity-and-idempotence.md)). The +comparison is on a canonical form — every element `$ID` replaced by its index in a +containment walk — because a rebuild mints a fresh random `$ID` per sub-element, +so comparing bytes would skip nothing. The policy lives in `modelsdk/canon` +(`Reconcile`) and is called at every write choke point in +`modelsdk/mpr/writer_core.go`: `updateUnit`, `WriteTransaction.WriteUnit` +(`codec.Store` reaches storage through this one) and — since ako/mxcli#556 — +`insertUnit`, for the case below. + +**A delete followed by an insert is a write path too**, and it is the one that +hides. Several `create or modify` handlers are implemented as delete + create +under the preserved unit ID rather than as an update, and an insert has nothing +stored to reconcile against, so the rebuild's fresh `$ID`s went straight to disk: +`create or modify rest client` rewrote 9 element `$ID`s in a 1,128-byte unit on +every run, forever. `deleteUnit` now remembers what it removed and `insertUnit` +reconciles a re-insert against it (`carryIdentityFromRemovedUnit`). That carry +cannot *elide* — the row and the file are already gone — so a no-op recreate also +restores `_Transaction.LastTransactionID`, which both the delete and the insert +bumped; without it the `.mpr` still showed as modified after every `.mxunit` had +gone quiet. Prefer an in-place update where the handler can do one: the REST +client's own fix is to call `UpdateConsumedRestService` and keep delete+create +only for a folder move, which lives in the unit's row rather than its contents. + +**The carry keys on the unit ID, so it does not reach a handler that re-mints +one** — and three handlers did, in one week: the REST client (#556), the view +entity's OQL document (#583) and the layout (ako/mxcli#600). All three took the +same fix, an in-place `UpdateRawUnit` rather than a replacement. The tell is +cheap and worth reaching for first: `ls` the `.mxunit` filenames across two +identical runs. A **changed filename** is delete+insert and the handler is +wrong; a **same filename with different bytes** is the codec or a missing carry +and `canon` is where to look. A replacement also silently reverts the unit's +ROW, which is how `create or replace layout` moved a foldered layout back to the +module root on every rewrite — there is no `FOLDER` clause on the statement, so +the rebuild always names the module root and only an insert applies it. + +When something *has* changed, `Reconcile` still does not let the rebuild's fresh +`$ID`s reach disk: `canon.TransplantIDs` matches the incoming document against the +stored one element by element (by `$Type` and shape, by `Name` where there is one, +LCS-anchored within each list) and puts the **stored** `$ID` back on every element +that still corresponds. Without it a one-argument edit re-minted 36 of a nanoflow's +37 element identities and Studio Pro painted the whole document as changed (#910). +Its correctness bar is lower than it looks and worth knowing: a *wrong* match only +makes a diff bigger, because every reference is rewritten with the element — the +one real failure is two elements sharing an `$ID`, which `dropCollisions` guards. + +Three rules follow, and each has already been violated once: + +1. **Never rewrite an element `$ID` without rewriting every reference to it in the + same pass.** Pointers are *primitive* properties holding an `element.ID`, not + `ChildProperty`, so a containment walk traverses the whole document and never + sees one. PR #125 renumbered IDs this way and made projects unopenable + (`KeyNotFoundException` at `ResolvePostponedProperties`). A unit is rewritten + wholesale or not at all. The transplant obeys this by substituting over *every* + 16-byte binary in the document rather than a maintained list of pointer + properties — any occurrence of one of the document's element IDs is a reference + by definition, the same insight the canonical form rests on. +2. **Adding a write path means wiring it to `canon.Reconcile`.** A new choke point + that writes directly will silently churn while everything else is quiet — the + worst kind of inconsistency, because the diff blames the wrong change. +3. **A new document type with an identity property needs a row in + `canon.identityFields`.** It cannot be generated: Mendix's `IsIdentifier` lives + in the modeler assemblies, not in the reflection data `generated/metamodel` is + built from. `TestFreshGUIDFieldsHaveAnIdentityDecision` catches the common case + (a property the codec mints fresh on every write) but cannot catch an identity property + the codec does not mint. Establish the property's status the way `StableId` was + — the method table is in ADR-0008. + +Elision itself is type-agnostic and covers new document types for free, but it +assumes **no binary pointer crosses a unit boundary** (measured 0 of 9,910, not +enforced). A document type that references another *unit* by `$ID` rather than by +qualified name breaks that assumption and invalidates the argument in ADR-0008. + +`MXCLI_ALWAYS_WRITE=1` forces every write to land, for bisecting. It does not +disable identity preservation. **Any test asserting "nothing changed" must include +a control** — otherwise the test passes against a build that never had the fix, +which is exactly how PR #125 shipped green. Note what the control can now be: +since identities are carried, a forced write of an in-sync unit produces the +**same bytes**, so "flip `MXCLI_ALWAYS_WRITE` and watch the content change" no +longer distinguishes anything (measured: same sha, mtime moves). Control on the +**rebuild** instead — encode the document twice and show the raw codec output +differs (`TestRebuildChurnsSubElementIDs`) — or, from the shell, on **mtimes** +rather than hashes. + +The executor reports which of the two happened: a statement whose unit writes were +all elided prints `Unchanged nanoflow: …` instead of `Replaced nanoflow: …` +(`ExecContext.ReportMutation`, fed by each writer's `WriteStats`). The verb is only +downgraded on positive evidence — writes offered, none landed — so a mutation that +never touches unit storage is reported exactly as before. + +**Several elisions in one run collapse into one line**, because that report is the +most repeated thing mxcli prints and an agent pays for it on every later model +call (a tool result is written into the conversation once and re-read by each one). +Measured on a settled 40-statement script: 41 lines / 1,604 B became 2 lines / +177 B. Two rules keep it honest, and each was arrived at by getting it wrong: + +1. **Only `Unchanged` collapses.** It is the one verb that by construction reports + an absence, so no line a reader would act on is ever replaced by a number — + a mixed run still names every real write individually and counts only the rest. + Collapsing on volume instead ("after N lines") would hide real writes in exactly + the runs where they matter. +2. **The trigger is how many arrive, not which entry point ran.** A lone elision is + printed verbatim, since "1 document already in sync" is worse than the line it + replaces. Gating on "is this a script?" looked equivalent and is not: `-c` + reaches `ExecuteProgram` too, because `executeMDL` prepends a `CONNECT` + statement, so a one-liner collapsed to a count of one. + +`mutationTally` (`mdl/executor/mutation_tally.go`), active only inside a program run. diff --git a/docs/03-development/MDL_PARSER_ARCHITECTURE.md b/docs/03-development/MDL_PARSER_ARCHITECTURE.md index a6fda7a1e4..7004a50a00 100644 --- a/docs/03-development/MDL_PARSER_ARCHITECTURE.md +++ b/docs/03-development/MDL_PARSER_ARCHITECTURE.md @@ -646,3 +646,7 @@ Common ANTLR context methods that can return `nil` on parse errors: - [ANTLR4 Documentation](https://github.com/antlr/antlr4/blob/master/doc/index.md) - [ANTLR4 Go Target](https://github.com/antlr/antlr4/blob/master/doc/go-target.md) - [MDL Syntax Reference](../07-references/mdl/MDL_SYNTAX_REFERENCE.md) + +## TypeEnumeration vs TypeEntity Ambiguity + +The MDL visitor (`buildDataType` in `visitor_helpers.go`) cannot distinguish between entity types and enumeration types for bare qualified names like `Module.EntityName`. Both parse as `ast.TypeEnumeration` with `EnumRef` set. Code that consumes data types must handle `TypeEnumeration` alongside `TypeEntity` and use `EnumRef` as a fallback for the entity name. diff --git a/docs/03-development/MODELSDK_ENGINE_ARCHITECTURE.md b/docs/03-development/MODELSDK_ENGINE_ARCHITECTURE.md index 1fd994ce9e..679b8fbe37 100644 --- a/docs/03-development/MODELSDK_ENGINE_ARCHITECTURE.md +++ b/docs/03-development/MODELSDK_ENGINE_ARCHITECTURE.md @@ -109,3 +109,94 @@ CLAUDE.md, "`modelsdk/gen` Binds Some Properties Under the Wrong BSON Key". Capture, don't guess — and note what a green build does *not* tell you: mxbuild accepts properties the project's metamodel does not declare, while Studio Pro throws `InvalidOperationException` at `MprProperty.cs`. Measured on 10.24.25 with two 11.5-only keys present: 0 errors. + +## `modelsdk/gen` Binds Some Properties Under the Wrong BSON Key + +The storage-name table above is about `$Type`. The **same split exists per +property**, and `modelsdk/gen` gets it wrong in **102 properties across 65 +types** — the ledger is `modelsdk/gen/keyaudit_test.go`. Mendix's reflection +data carries two names per property — an SDK `Name` and a BSON `StorageName` — +and the in-repo generator (`cmd/codegen` → `generated/metamodel`) keeps them +apart, tag from storage name: + +```go +// generated/metamodel/types.go — correct +RegularExpression model.QualifiedName `json:"regExIdentifier,omitempty"` +// ^ SDK name ^ storage name +``` + +The generator behind `modelsdk/gen` reads a **different input** — the TypeScript +SDK's compiled JS, which does not contain storage names at all (measured: +`regExIdentifier` occurs 0 times in `mendixmodelsdk` 4.114.0) — and patches them +back via a hand-maintained `PropertyKeyOverrides` table. + +**`generated/metamodel` is therefore the arbiter when the two disagree**, with +one caveat: it is a **snapshot of 11.6.0** (see its header), so it is sound for +the properties it contains but says nothing about ones introduced later — for +those, get a real document. It has been right in every case checked that way +(`RegularExpression.Expression`, `RegExRuleInfo.RegExIdentifier`, `Attribute.GUID`). +`TestGenPropertyKeysAgainstMetamodel` fails when a NEW mismatch appears (a +re-vendored gen that dropped an override) or when a listed one is fixed without +being struck off. Why the generator is not simply brought in-tree, and what it +would take: [PROPOSAL_codegen_ownership.md](docs/11-proposals/PROPOSAL_codegen_ownership.md). + +`cmd/modelsdk-codegen` and `internal/codegen/supplements.json` — named in every +gen file's `DO NOT EDIT` header — have **never existed in this repo** +(`git log --all` is empty for both), and `/reference/` is gitignored, so the +generator's input is absent too. gen is vendored output that cannot be +regenerated here; see `docs/plans/2026-06-05-adopt-modelsdk-engine.md` §4, where +"vendor engalar codegen" is still an open Phase-0 item. + +So the fix for a wrong key is a **hand-applied override in the `init` +function**, commented in the house style (grep `STORAGE-NAME OVERRIDE` for the +four precedents). Two rules: + +1. **Patch both sides.** The encode key (`init`) and the decode key + (`InitFromRaw`) are separate literals. Patching one gives a document that + writes one key and reads another — which the entity-rewrite guard then + refuses, so the symptom is a puzzling refusal rather than a wrong file. +2. **`gofmt` the file**, or `TestGeneratedCodeIsFormatted` fails. + +Not every wrong key is worth patching — leave the ones nothing writes, and note +why. `mx check` is a weak signal here either way: it caught the RegEx one +(CE0135) but tolerates unknown properties in general, and Studio Pro is stricter +than mxbuild. + +## Overlay Writes: Never Invent a Key, Branch on `$Type` + +When a write overlays fields onto preserved BSON (`mdl/settingsoverlay`, and any +future storage that follows ADR-0005 guard-don't-drop), two rules are load-bearing. +Breaking either produces a document `mx check` accepts and **Studio Pro cannot +open**: it resolves every stored property against the type's property list and +throws `System.InvalidOperationException: Sequence contains no matching element` +at `MprProperty.cs`. mxbuild's deserializer tolerates unknown properties, so the +build is not a safety net here. + +1. **Write only keys the document already carries.** Property names are + version-specific — Mendix renamed `JavaVersion` (`"Java21"`) to + `JavaMajorVersion` (`"21"`) and `Tracing` to `OpenTelemetry` between 11.6 and + 11.12. Read the key off the stored document and write back to that same key; + when neither is present, write neither (an absent optional property is filled + in on load). See `settingsoverlay.JavaVersionKey` (#759). +2. **A polymorphic child must be dispatched on `$Type` before any field + assignment.** Variants can differ in *arity*, not just field values: + `Settings$SharedValue` carries a `Value`, while `Settings$PrivateValue` is a + bare marker with no properties at all (the value lives on the developer's + workstation). Assigning `Value` to whichever node is there corrupts the marker. + +The same reasoning bans authoring what the model does not own: mxcli preserves a +constant override's shared/private choice and refuses statements that would flip +it, rather than silently converting one to the other. + +Enum-valued properties are the sibling trap: validate against +`generated/metamodel` (e.g. `SettingsDatabaseType` is `Hsqldb`, never `HSQLDB`) +rather than passing a user string through. + +**On a CREATE there is no stored document to read the key off.** Rule 1 then +becomes: branch on the project's Mendix version and write exactly one spelling — +never both as a hedge. `mdl/dbconnector` does this for the 11.13 rename of +`DatabaseQuery.QueryType` (int) to `Type` (string enum), which mxbuild *does* +catch, as CE5277 on every activity using the query. To learn the target shape +without guessing, run the new mxbuild's own migration over an old project +(`mx convert -p -s `) and diff the BSON: Mendix ships a one-time +conversion per renamed property, so the converted document is authoritative. From 75cd122510d32c5dc15e78ec4635575e31f89528 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 15:05:44 +0000 Subject: [PATCH 29/38] =?UTF-8?q?docs:=20split=20the=20PR=20checklist=20?= =?UTF-8?q?=E2=80=94=20evidence=20bar=20stays,=20subsystem=20lists=20move?= =?UTF-8?q?=20(#611)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The checklist was 11,868 B re-read into every session, and most of it only bites when a change touches a particular subsystem: backend abstraction, full-stack wiring, version gating, test coverage, security, docs, code quality. Those moved into /mxcli-dev:review, which is the command that applies them and which previously only pointed back at CLAUDE.md. Two subsections stay, because they govern how the work is DONE rather than how it is reviewed, and an agent that never runs the review command still has to follow them: the bug-fix evidence bar (test written first, verified at the layer the symptom lives in, fix proven to be the cause by reverting it) and one-thing-per-commit. That distinction is the whole judgement here. Moving the evidence bar too would have made it apply only when someone remembered to ask for a review, which is the same failure the three-copy gates tests exist to prevent. Renaming the section broke two references to it, in CLAUDE.md's own welcome and in review.md step 2; both updated, and a grep confirms none are left. 11,868 -> 2,839 in CLAUDE.md; 53,950 -> 45,074 overall. Six distinctive strings from the moved subsections verified present in review.md. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .claude/commands/mxcli-dev/review.md | 86 +++++++++++++++++++++++++++- CLAUDE.md | 86 +++------------------------- 2 files changed, 94 insertions(+), 78 deletions(-) diff --git a/.claude/commands/mxcli-dev/review.md b/.claude/commands/mxcli-dev/review.md index 53edca9946..596868287a 100644 --- a/.claude/commands/mxcli-dev/review.md +++ b/.claude/commands/mxcli-dev/review.md @@ -11,7 +11,8 @@ burned us before. ## Steps 1. Run `gh pr view` and `gh pr diff` (or `git diff main...HEAD`) to read the change. -2. Work through the CLAUDE.md "PR / Commit Review Checklist" in full. +2. Work through CLAUDE.md's "Working Rules for a Change" (the evidence bar) and + the subsystem checklists at the end of this file, in full. 3. Then check every row in the Recurring Findings table below — flag any match. 4. Report: blockers first, then moderate issues, then minor. Include a concrete fix option for every blocker (not just "this is wrong"). @@ -68,3 +69,86 @@ proactively. Add a row after every review that surfaces something new. - [ ] Recurring Findings table updated with any new pattern. - [ ] If docs-only PR: every function name, path, and PR reference verified against live code before approving. + +## The subsystem checklists + +These moved out of CLAUDE.md, where they were re-read into every session but only +apply when a change touches that subsystem. The evidence bar for a bug fix, and +the one-thing-per-commit rule, stay there because they govern how the work is +done rather than how it is reviewed. + +### Overlap & duplication +- [ ] Check `docs/11-proposals/` for existing proposals covering the same functionality +- [ ] Search the codebase for existing implementations (grep for key function names, command names, types) +- [ ] Check `mdl-examples/doctype-tests/` for existing test coverage of the feature area +- [ ] Verify the PR doesn't re-document already-shipped features as new + +### Syntax design for MDL features +New or modified MDL syntax must follow the design guidelines. See [ADR-0003: MDL is SQL-shaped](docs/13-decisions/0003-mdl-is-sql-shaped.md) for the underlying decision and rejected alternatives; the design checklist below operationalises it. +- [ ] **Design skill consulted** — read `.claude/skills/design-mdl-syntax.md` before designing syntax +- [ ] **Follows standard patterns** — uses `create`/`alter`/`drop`/`show`/`describe`, not custom verbs +- [ ] **Reads as English** — a business analyst understands the statement on first reading +- [ ] **Qualified names** — uses `Module.Element` everywhere, no implicit module context +- [ ] **Property format** — uses `( key: value, ... )` with colon separators, one per line +- [ ] **LLM-friendly** — one example is sufficient for an LLM to generate correct variants +- [ ] **Diff-friendly** — adding one property is a one-line diff + +### Version compatibility +New features that depend on a specific Mendix version must be version-gated: +- [ ] **Registry entry** — feature added to `sdk/versions/mendix-{9,10,11}.yaml` with correct `min_version` +- [ ] **Executor pre-check** — `checkFeature()` called before BSON writes, with actionable error and hint +- [ ] **Test coverage** — version-gated tests use `-- @version:` directives or `requireMinVersion()` +- [ ] **Skill updated** — `.claude/skills/version-awareness.md` updated if the feature has a workaround for older versions + +### Backend abstraction compliance +All executor code must go through the backend abstraction layer. **`sdk/mpr` no longer exists** — the package was deleted once its importer count reached zero, so reaching past the abstraction is now a compile error rather than a rule to remember. See [ADR-0002: Backend Abstraction Layer](docs/13-decisions/0002-backend-abstraction.md) for the context and alternatives. The codec (`modelsdk`) engine is the only local engine — the legacy `sdk/mpr` backend was deleted (`docs/plans/2026-09-14-retire-legacy-engine.md`), and `--engine`/`MXCLI_ENGINE` survive only as a warning-only no-op. It routes **all** document types — domain models included — through the codec, not a codec/legacy hybrid; see [ADR-0004: Full codec engine](docs/13-decisions/0004-full-codec-engine.md). Where the codec path cannot yet reproduce a construct, the backend **refuses** the op rather than dropping data. The backend interface speaks the **semantic model**, not gen/BSON or AST types — gen+codec are the MPR backend's internal storage adapter, one of several (MPR, MCP/PED, a future storage format); see [ADR-0005](docs/13-decisions/0005-semantic-model-interface-currency.md). CREATE is model→gen; fidelity-sensitive ALTER uses backend-internal gen-mutation, not a model round-trip. +- [ ] **No engine internals in the executor** — executor files must not reach into `modelsdk/mpr`, `modelsdk/codec` or `modelsdk/gen` directly; use `ctx.Backend.*` instead. A method missing from the backend gets implemented there, not bypassed +- [ ] **New backend methods on the interface** — any new data access or mutation goes in the appropriate interface in `mdl/backend/` (e.g., `DomainModelBackend`, `MicroflowBackend`), not as a direct SDK call +- [ ] **MPR implementation in `mdl/backend/mpr/`** — the concrete implementation lives here; all BSON/reader/writer logic stays in this package +- [ ] **Mock stub in `mdl/backend/mock/`** — every new backend method has a `Func`-field stub with a descriptive `"MockBackend.X not configured"` error default (not `nil, nil`) +- [ ] **Compile-time interface check** — new backend implementations have `var _ backend.SomeInterface = (*impl)(nil)` +- [ ] **ALTER operations use mutator pattern** — page/workflow mutations go through `ctx.Backend.OpenPageForMutation()` / `OpenWorkflowForMutation()`, not inline BSON construction +- [ ] **New shared types in `mdl/types/`** — a type used by more than one layer goes in `mdl/types/` and the others alias it (`type Foo = types.Foo`), never as duplicate definitions. A same-shape duplicate compiles and tests green; it shows up only as an assignment failure *across* the boundary, naming the same type on both sides of "want". `modelsdk/mpr/version.ProjectVersion` was that case and is now an alias — the guard is a compile-time assertion (`var _ *types.ProjectVersion = (*version.ProjectVersion)(nil)`, `version_alias_test.go`), which builds only under an alias and so is stronger than anything a test body can assert +- [ ] **Map iteration is deterministic** — any map iterated for serialization output must sort keys first (`sort.Strings(keys)` pattern); non-deterministic output causes flaky diffs and BSON instability +- [ ] **Pluggable widgets via WidgetEngine** — new pluggable widget support uses `.def.json` + `WidgetRegistry`; no hardcoded BSON widget builders in the executor + +### Full-stack consistency for MDL features +New MDL commands or language features must be wired through the full pipeline: +- [ ] **Grammar** — rule added to `MDLParser.g4` (and `MDLLexer.g4` if new tokens) +- [ ] **Parser regenerated** — `make grammar` run; generated files in `mdl/grammar/parser/` are **not** committed (they are regenerated by `make` at build time) +- [ ] **AST** — node type added in `mdl/ast/` +- [ ] **Visitor** — ANTLR listener bridges parse tree to AST in `mdl/visitor/` +- [ ] **Executor** — thin handler in `mdl/executor/` dispatches to `ctx.Backend.*`; no BSON in the handler +- [ ] **Backend method** — data access or mutation wired through `mdl/backend/` interface and implemented in `mdl/backend/mpr/` +- [ ] **LSP** — if the feature adds formatting, diagnostics, or navigation targets, wire it into `cmd/mxcli/lsp.go` and register the capability +- [ ] **DESCRIBE roundtrip** — if the feature creates artifacts, `describe` should output re-executable MDL +- [ ] **VS Code extension** — if new LSP capabilities are added, update `vscode-mdl/package.json` + +### Test coverage +- [ ] New packages have test files +- [ ] New executor commands have MDL examples in `mdl-examples/doctype-tests/` +- [ ] **MDL syntax changes** — any PR that adds or modifies MDL syntax must include working examples in `mdl-examples/doctype-tests/` +- [ ] **Bug fixes** — every bug fix should include an MDL test script in `mdl-examples/bug-tests/` that reproduces the issue, so the fix can be verified in Studio Pro if applicable. **Three numbering namespaces meet in that directory**: the historical files are named after `mendixlabs/mxcli` **PR** numbers (`261-mx9-microflow-roundtrip.mdl` is upstream PR #261), issues filed on the fork are `ako/mxcli` numbers — and the two sequences already collide on 261–266 — while a few names are a **Mendix version** with the dot dropped (`1113-database-query-type-enum.mdl` is Mendix 11.13, not issue 1113). Name a file after a fork issue with a topic prefix (`mapping-261-object-handling-backup.mdl`) and write the reference qualified (`ako/mxcli#261`) wherever it appears, or the number silently resolves to the wrong thing +- [ ] Integration paths (not just helpers) are tested +- [ ] Tests don't rely on `time.Sleep` for synchronization — use channels or polling with timeout + +### Security & robustness +- [ ] Unix sockets use restrictive permissions (`os.Chmod(path, 0600)`) +- [ ] File I/O is not in hot paths (event loops, per-keystroke handlers) — cache in memory +- [ ] No silent side effects on typos (e.g., auto-creating resources on misspelled names should be flagged) +- [ ] Method receivers are correct (pointer vs value) for mutations + +### Documentation +- [ ] **Skills** — new features documented in `.claude/skills/` (syntax, examples, gotchas) +- [ ] **CLI help (Cobra)** — `mxcli` subcommand help text updated (Cobra `Short`/`Long`/`Example` fields) +- [ ] **CLI help (syntax topics)** — `cmd/mxcli/syntax/features_*.go` updated with new/changed MDL syntax; new `SyntaxFeature` entries added for new document types; `OR MODIFY` / `OR REPLACE` variants reflected in existing `Syntax` fields; accessible via `mxcli syntax ` and REPL `help` +- [ ] **Syntax reference** — `docs/01-project/MDL_QUICK_REFERENCE.md` updated with new statement syntax +- [ ] **MDL examples** — working examples added to `mdl-examples/` for new commands +- [ ] **Site docs** — `docs-site/src/` pages added or updated for user-facing features + +### Code quality +- [ ] Refactors are applied consistently across all relevant files (grep for the old pattern) +- [ ] Manually maintained lists (keyword lists, type mappings) are flagged as maintenance risks +- [ ] Design docs match the actual implementation — remove or update stale plans +- [ ] Numeric type conversions are bounds-checked — `float64→int` casts need overflow guards (`±2^53` for safe integer range); silent overflow produces garbage in serialized output +- [ ] `convert.go` updated when structs in `mdl/types/` gain or lose fields — `TestFieldCountDrift` will catch this at test time, but `convert.go` must be updated before merging diff --git a/CLAUDE.md b/CLAUDE.md index be537e7e5f..9ae5b495a7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,9 @@ If you're starting a new task, here's how contributions work in this repo: 5. **Validate locally** — `make build && make test && make lint` must all pass. 6. **Open a PR** — link the issue, document Mendix Studio Pro validation, confirm agentic testing. -For the full workflow, read `CONTRIBUTING.md`. For the review checklist applied to every PR, see the "PR / Commit Review Checklist" section below. +For the full workflow, read `CONTRIBUTING.md`. The evidence bar every change is held +to is in "Working Rules for a Change" below; the subsystem checklists live in +`/mxcli-dev:review`. ## Project Overview @@ -316,7 +318,8 @@ mxcli uses a layered documentation system — each artifact type has a single ca **The wiki is synthesized, not stated.** It frames and connects across the other artifacts — it never restates content that has a canonical home. Rules and seed page list in [`.claude/skills/maintain-wiki.md`](.claude/skills/maintain-wiki.md). -## PR / Commit Review Checklist +## Working Rules for a Change + When reviewing pull requests or validating work before commit, verify these items: @@ -327,87 +330,16 @@ When reviewing pull requests or validating work before commit, verify these item - [ ] **Verified at the layer the symptom lives in** — a test proves something about the layer it exercises and nothing more. Parser/grammar → unit test. BSON we write → unit test on the encoded document. Files on disk after `mx` runs → integration test (`-tags integration`). **The rendered app's behaviour or appearance → `.claude/skills/verify-in-runtime.md`** (boot with `run --local`, assert in Playwright). A page can serialize to valid-looking BSON, pass `mx check`, build cleanly, and still render wrong — that was #812. - [ ] **Fix proven to be the cause** — revert the fix (or stub the guard) and confirm the test fails with the reported symptom. A test that only passes against fixed code has not been shown to detect anything; two bugs this week had a green suite while live (#812 a clobbered `RegisterTypeDefaults`, #808 an integration test that had only ever skipped) -### Overlap & duplication -- [ ] Check `docs/11-proposals/` for existing proposals covering the same functionality -- [ ] Search the codebase for existing implementations (grep for key function names, command names, types) -- [ ] Check `mdl-examples/doctype-tests/` for existing test coverage of the feature area -- [ ] Verify the PR doesn't re-document already-shipped features as new - -### Syntax design for MDL features -New or modified MDL syntax must follow the design guidelines. See [ADR-0003: MDL is SQL-shaped](docs/13-decisions/0003-mdl-is-sql-shaped.md) for the underlying decision and rejected alternatives; the design checklist below operationalises it. -- [ ] **Design skill consulted** — read `.claude/skills/design-mdl-syntax.md` before designing syntax -- [ ] **Follows standard patterns** — uses `create`/`alter`/`drop`/`show`/`describe`, not custom verbs -- [ ] **Reads as English** — a business analyst understands the statement on first reading -- [ ] **Qualified names** — uses `Module.Element` everywhere, no implicit module context -- [ ] **Property format** — uses `( key: value, ... )` with colon separators, one per line -- [ ] **LLM-friendly** — one example is sufficient for an LLM to generate correct variants -- [ ] **Diff-friendly** — adding one property is a one-line diff - -### Version compatibility -New features that depend on a specific Mendix version must be version-gated: -- [ ] **Registry entry** — feature added to `sdk/versions/mendix-{9,10,11}.yaml` with correct `min_version` -- [ ] **Executor pre-check** — `checkFeature()` called before BSON writes, with actionable error and hint -- [ ] **Test coverage** — version-gated tests use `-- @version:` directives or `requireMinVersion()` -- [ ] **Skill updated** — `.claude/skills/version-awareness.md` updated if the feature has a workaround for older versions - -### Backend abstraction compliance -All executor code must go through the backend abstraction layer. **`sdk/mpr` no longer exists** — the package was deleted once its importer count reached zero, so reaching past the abstraction is now a compile error rather than a rule to remember. See [ADR-0002: Backend Abstraction Layer](docs/13-decisions/0002-backend-abstraction.md) for the context and alternatives. The codec (`modelsdk`) engine is the only local engine — the legacy `sdk/mpr` backend was deleted (`docs/plans/2026-09-14-retire-legacy-engine.md`), and `--engine`/`MXCLI_ENGINE` survive only as a warning-only no-op. It routes **all** document types — domain models included — through the codec, not a codec/legacy hybrid; see [ADR-0004: Full codec engine](docs/13-decisions/0004-full-codec-engine.md). Where the codec path cannot yet reproduce a construct, the backend **refuses** the op rather than dropping data. The backend interface speaks the **semantic model**, not gen/BSON or AST types — gen+codec are the MPR backend's internal storage adapter, one of several (MPR, MCP/PED, a future storage format); see [ADR-0005](docs/13-decisions/0005-semantic-model-interface-currency.md). CREATE is model→gen; fidelity-sensitive ALTER uses backend-internal gen-mutation, not a model round-trip. -- [ ] **No engine internals in the executor** — executor files must not reach into `modelsdk/mpr`, `modelsdk/codec` or `modelsdk/gen` directly; use `ctx.Backend.*` instead. A method missing from the backend gets implemented there, not bypassed -- [ ] **New backend methods on the interface** — any new data access or mutation goes in the appropriate interface in `mdl/backend/` (e.g., `DomainModelBackend`, `MicroflowBackend`), not as a direct SDK call -- [ ] **MPR implementation in `mdl/backend/mpr/`** — the concrete implementation lives here; all BSON/reader/writer logic stays in this package -- [ ] **Mock stub in `mdl/backend/mock/`** — every new backend method has a `Func`-field stub with a descriptive `"MockBackend.X not configured"` error default (not `nil, nil`) -- [ ] **Compile-time interface check** — new backend implementations have `var _ backend.SomeInterface = (*impl)(nil)` -- [ ] **ALTER operations use mutator pattern** — page/workflow mutations go through `ctx.Backend.OpenPageForMutation()` / `OpenWorkflowForMutation()`, not inline BSON construction -- [ ] **New shared types in `mdl/types/`** — a type used by more than one layer goes in `mdl/types/` and the others alias it (`type Foo = types.Foo`), never as duplicate definitions. A same-shape duplicate compiles and tests green; it shows up only as an assignment failure *across* the boundary, naming the same type on both sides of "want". `modelsdk/mpr/version.ProjectVersion` was that case and is now an alias — the guard is a compile-time assertion (`var _ *types.ProjectVersion = (*version.ProjectVersion)(nil)`, `version_alias_test.go`), which builds only under an alias and so is stronger than anything a test body can assert -- [ ] **Map iteration is deterministic** — any map iterated for serialization output must sort keys first (`sort.Strings(keys)` pattern); non-deterministic output causes flaky diffs and BSON instability -- [ ] **Pluggable widgets via WidgetEngine** — new pluggable widget support uses `.def.json` + `WidgetRegistry`; no hardcoded BSON widget builders in the executor - -### Full-stack consistency for MDL features -New MDL commands or language features must be wired through the full pipeline: -- [ ] **Grammar** — rule added to `MDLParser.g4` (and `MDLLexer.g4` if new tokens) -- [ ] **Parser regenerated** — `make grammar` run; generated files in `mdl/grammar/parser/` are **not** committed (they are regenerated by `make` at build time) -- [ ] **AST** — node type added in `mdl/ast/` -- [ ] **Visitor** — ANTLR listener bridges parse tree to AST in `mdl/visitor/` -- [ ] **Executor** — thin handler in `mdl/executor/` dispatches to `ctx.Backend.*`; no BSON in the handler -- [ ] **Backend method** — data access or mutation wired through `mdl/backend/` interface and implemented in `mdl/backend/mpr/` -- [ ] **LSP** — if the feature adds formatting, diagnostics, or navigation targets, wire it into `cmd/mxcli/lsp.go` and register the capability -- [ ] **DESCRIBE roundtrip** — if the feature creates artifacts, `describe` should output re-executable MDL -- [ ] **VS Code extension** — if new LSP capabilities are added, update `vscode-mdl/package.json` - -### Test coverage -- [ ] New packages have test files -- [ ] New executor commands have MDL examples in `mdl-examples/doctype-tests/` -- [ ] **MDL syntax changes** — any PR that adds or modifies MDL syntax must include working examples in `mdl-examples/doctype-tests/` -- [ ] **Bug fixes** — every bug fix should include an MDL test script in `mdl-examples/bug-tests/` that reproduces the issue, so the fix can be verified in Studio Pro if applicable. **Three numbering namespaces meet in that directory**: the historical files are named after `mendixlabs/mxcli` **PR** numbers (`261-mx9-microflow-roundtrip.mdl` is upstream PR #261), issues filed on the fork are `ako/mxcli` numbers — and the two sequences already collide on 261–266 — while a few names are a **Mendix version** with the dot dropped (`1113-database-query-type-enum.mdl` is Mendix 11.13, not issue 1113). Name a file after a fork issue with a topic prefix (`mapping-261-object-handling-backup.mdl`) and write the reference qualified (`ako/mxcli#261`) wherever it appears, or the number silently resolves to the wrong thing -- [ ] Integration paths (not just helpers) are tested -- [ ] Tests don't rely on `time.Sleep` for synchronization — use channels or polling with timeout - -### Security & robustness -- [ ] Unix sockets use restrictive permissions (`os.Chmod(path, 0600)`) -- [ ] File I/O is not in hot paths (event loops, per-keystroke handlers) — cache in memory -- [ ] No silent side effects on typos (e.g., auto-creating resources on misspelled names should be flagged) -- [ ] Method receivers are correct (pointer vs value) for mutations - ### Scope & atomicity - [ ] Each commit does **one thing** — a feature, a bugfix, or a refactor, not a mix - [ ] Each PR is scoped to a **single feature or concern** — if the description needs "and" between unrelated items, split it - [ ] Independent features (e.g., a new command, a formatter, UX improvements) go in separate PRs even if developed together - [ ] Refactors that touch many files (e.g., renaming a helper across executors) are their own commit, not bundled with feature work -### Documentation -- [ ] **Skills** — new features documented in `.claude/skills/` (syntax, examples, gotchas) -- [ ] **CLI help (Cobra)** — `mxcli` subcommand help text updated (Cobra `Short`/`Long`/`Example` fields) -- [ ] **CLI help (syntax topics)** — `cmd/mxcli/syntax/features_*.go` updated with new/changed MDL syntax; new `SyntaxFeature` entries added for new document types; `OR MODIFY` / `OR REPLACE` variants reflected in existing `Syntax` fields; accessible via `mxcli syntax ` and REPL `help` -- [ ] **Syntax reference** — `docs/01-project/MDL_QUICK_REFERENCE.md` updated with new statement syntax -- [ ] **MDL examples** — working examples added to `mdl-examples/` for new commands -- [ ] **Site docs** — `docs-site/src/` pages added or updated for user-facing features - -### Code quality -- [ ] Refactors are applied consistently across all relevant files (grep for the old pattern) -- [ ] Manually maintained lists (keyword lists, type mappings) are flagged as maintenance risks -- [ ] Design docs match the actual implementation — remove or update stale plans -- [ ] Numeric type conversions are bounds-checked — `float64→int` casts need overflow guards (`±2^53` for safe integer range); silent overflow produces garbage in serialized output -- [ ] `convert.go` updated when structs in `mdl/types/` gain or lose fields — `TestFieldCountDrift` will catch this at test time, but `convert.go` must be updated before merging +The subsystem checklists — backend abstraction, full-stack wiring, version +gating, test coverage, security, docs, code quality — are in +[`/mxcli-dev:review`](.claude/commands/mxcli-dev/review.md), which is the command +that applies them. They only bite when a change touches that subsystem. ## Dependencies From 89ed8c318c79c0d5c39b968176ff6928d1a5c6ce Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 15:07:23 +0000 Subject: [PATCH 30/38] docs: drop the transcription sections from CLAUDE.md (#611) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four sections were restating what a command, a directory listing or a frontmatter description answers authoritatively — the same pattern already removed from Implementation Status. The 27-row CLI feature table goes: `mxcli help ` and `mxcli syntax` own it. Two facts in it existed nowhere else and were relocated first — the wrapping-grid/6,000px default for an entity with no @Position, to domain-model-layout.md, and `fix widgets` clearing CE0463 (measured 203 -> 0 on a vanilla 11.12.1 app), to diagnose-ce0463.md. The skill list goes, on this file's own rule that a skill's frontmatter description IS the index — the table drifted to 12 of 68 before #906 caught it. The always-on MDL idioms it contained stay. The directory tree goes; `ls` answers it. What replaces it is the orientation the layout does not give you: that modelsdk/ is the engine and sdk/mpr is deleted, and how mdl/'s grammar -> visitor -> ast -> executor chain fits together. The file index becomes routing only, for the entries whose annotation is not derivable from the filename: the CE0463 elimination order, the verify-in-runtime tier, the both-type-and-object template rule, and that System-module string lengths are measured rather than taken from the Model SDK. Three CLI rules were kept because no --help states them: generated parser files are not committed, skills are edited in .claude/skills/mendix/ and not the rsync --delete'd embed dir, and mxcli-dev/ commands are not synced. Also merges "MDL Syntax Quick Reference" into "What mxcli Can Do", which had come to say the same thing, keeping the link. 108,761 -> 25,514 B across the four slices (76%, ~27k -> ~6k tokens). No dangling relative links. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .claude/skills/diagnose-ce0463.md | 7 + CLAUDE.md | 290 ++++----------------- docs-site/src/tools/domain-model-layout.md | 4 + 3 files changed, 67 insertions(+), 234 deletions(-) diff --git a/.claude/skills/diagnose-ce0463.md b/.claude/skills/diagnose-ce0463.md index 1dd8396fc1..ff7b2e84c8 100644 --- a/.claude/skills/diagnose-ce0463.md +++ b/.claude/skills/diagnose-ce0463.md @@ -204,3 +204,10 @@ For pluggable widgets (DataGrid2, ComboBox, Gallery, etc.), templates must inclu - `object`: Default WidgetObject with all property values **CE0463 "widget definition changed" error**: This error occurs when the Object's property structure doesn't match the Type's PropertyTypes. Always extract templates from Studio Pro-created widgets, not programmatically generated ones. See `sdk/widgets/templates/README.md` for details. For debugging CE0463 and other BSON issues, follow the workflow in `.claude/skills/debug-bson.md`. + +## `mxcli fix widgets` clears CE0463 after a headless install + +`fix widgets` / `fix design-properties` run `mx update-widgets` and +`mx rename-design-properties` and **persist** the result without their MPR v2 -> v1 +collapse: let the tool convert, read the units back, restore v2, write the changed +ones through mxcli's writer. Measured 203 -> 0 errors on a vanilla 11.12.1 app. diff --git a/CLAUDE.md b/CLAUDE.md index 9ae5b495a7..9f9d1ea499 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -80,76 +80,18 @@ export LD_PRELOAD=/usr/lib/$(uname -m)-linux-gnu/libfreetype.so.6 ## Project Architecture -``` -ModelSDKGo/ -├── modelsdk.go # Main public api (open, OpenForWriting, helpers) -├── model/ # Core types: ID, QualifiedName, module, Element interface -│ -├── api/ # High-level fluent api (inspired by Mendix Web Extensibility api) -│ ├── api.go # ModelAPI entry point with namespace access -│ ├── domainmodels.go # EntityBuilder, AssociationBuilder, AttributeBuilder -│ ├── enumerations.go # EnumerationBuilder -│ ├── microflows.go # MicroflowBuilder -│ ├── pages.go # PageBuilder, widget builders -│ └── modules.go # ModulesAPI -│ -├── sdk/ # SDK implementation packages -│ ├── domainmodel/ # entity, attribute, association, DomainModel -│ ├── microflows/ # microflow, nanoflow, activities (60+ types) -│ ├── pages/ # page, layout, widget types (50+ widgets) -│ ├── widgets/ # Embedded widget templates for pluggable widgets -│ │ ├── loader.go # template loading with go:embed -│ │ └── templates/ # json widget type definitions by Mendix version -│ └── versions/ # per-major feature registry (mendix-{9,10,11}.yaml) -│ -├── modelsdk/ # The MPR engine (sdk/mpr, the legacy one, is deleted) -│ ├── mpr/ # MPR file format: reader, writer, raw unit access -│ ├── codec/ # document <-> BSON encode/decode -│ ├── canon/ # canonical form, identity transplant, write elision -│ ├── gen/ # vendored metamodel types (see the storage-name note) -│ └── widgets/ # pluggable widget augmentation -│ -├── mdl/ # MDL (Mendix Definition Language) parser & CLI -│ ├── grammar/ # ANTLR4 grammar definition -│ │ ├── MDLLexer.g4 # ANTLR4 lexer grammar (tokens) -│ │ ├── MDLParser.g4 # ANTLR4 parser grammar (rules) -│ │ └── parser/ # Generated Go parser code -│ ├── ast/ # AST node types for MDL statements -│ ├── visitor/ # ANTLR listener to build AST -│ ├── executor/ # Executes AST against modelsdk-go -│ ├── catalog/ # SQLite-based catalog for querying project metadata -│ ├── linter/ # Extensible linting framework -│ │ └── rules/ # Built-in lint rules (MPR001, MPR002, etc.) -│ └── repl/ # Interactive REPL interface -│ -├── sql/ # external database connectivity (PostgreSQL, Oracle, sql Server) -│ ├── driver.go # DriverName type, ParseDriver() -│ ├── connection.go # Manager, connection, credential isolation -│ ├── config.go # DSN resolution (env vars, YAML config) -│ ├── query.go # execute() — query via database/sql -│ ├── meta.go # ShowTables(), DescribeTable() via information_schema -│ ├── format.go # table and json output formatters -│ ├── mendix.go # Mendix DB DSN builder, table/column name helpers -│ └── import.go # import pipeline: batch insert, ID generation, sequence tracking -│ -├── cmd/ # Command-line tools -│ ├── mxcli/ # CLI entry point (Cobra-based) -│ └── codegen/ # Code generator CLI -│ -├── internal/ # Internal packages (not exported) -│ └── codegen/ # Metamodel code generation system -│ ├── schema/ # json reflection data loading -│ ├── transform/ # transform to Go types -│ └── emit/ # Go source code generation -│ -├── generated/metamodel/ # Auto-generated type definitions -├── examples/ # Usage examples -│ -└── reference/ # reference materials (not Go code) - ├── mendixmodellib/ # TypeScript library + reflection data - ├── mendixmodelsdk/ # TypeScript SDK reference - └── mdl-grammar/ # Comprehensive MDL grammar reference -``` +Run `ls` or read the package docs; the tree is not restated here. The orientation +that is not obvious from the layout: + +- `modelsdk/` is **the** MPR engine — `mpr/` (file format), `codec/` (document <-> BSON), + `canon/` (canonical form, identity transplant, write elision), `gen/` (vendored + metamodel types). `sdk/mpr`, the legacy engine, is deleted. +- `mdl/` is the language: `grammar/` (ANTLR4) -> `visitor/` -> `ast/` -> `executor/`, + with `backend/` as the abstraction the executor speaks to and `catalog/` the SQLite + index behind `show`/`select`. +- `api/` is the fluent builder layer over any backend; `sdk/` holds the semantic types. +- `cmd/mxcli/` is the CLI; `generated/metamodel/` is generated by `cmd/codegen`. +- `reference/` is gitignored reference material, not Go code. ## Key Concepts @@ -351,126 +293,33 @@ that applies them. They only bite when a change touches that subsystem. ## MDL CLI (mxcli) -The `mxcli` command-line tool allows reading and modifying Mendix projects using MDL (Mendix Definition Language), a SQL-like syntax. - ```bash -# build the CLI -go build -o bin/mxcli ./cmd/mxcli - -# run interactive REPL -./bin/mxcli - -# execute commands directly -./bin/mxcli -p /path/to/app.mpr -c "show entities" - -# execute MDL script file -./bin/mxcli exec script.mdl -p /path/to/app.mpr - -# check MDL syntax (no project needed) -./bin/mxcli check script.mdl - -# check syntax and validate references -./bin/mxcli check script.mdl -p app.mpr --references -``` - -### Key CLI Features - -| Feature | Commands | Details | -|---------|----------|---------| -| **Project structure** | `show structure [depth 1\|2\|3] [in module] [all]` | Compact overview at 3 depth levels | -| **Catalog queries** | `show catalog tables`, `select ... from CATALOG.table` | SQL querying of project metadata | -| **Code search** | `show callers\|callees\|references\|impact\|context of ...` | Cross-reference navigation (requires `refresh catalog full`) | -| **Full-text search** | `search 'keyword'` | Search across all strings and source | -| **Linting** | `mxcli lint -p app.mpr [--format json\|sarif]` | 15 built-in rules + 29 Starlark rules (MDL, SEC, QUAL, ARCH, DESIGN, CONV) | -| **Report** | `mxcli report -p app.mpr [--format markdown\|json\|html]` | Scored best practices report with category breakdown | -| **Testing** | `mxcli test tests/ -p app.mpr [--local] [--watch] [--attach]` | `.test.mdl` / `.test.md` files. `--local` runs on mxcli's own runtime (no Docker daemon), on its own ports + `_test` database, driving a **token-guarded test endpoint** (one microflow per test, invoked over HTTP — a throwing test fails only itself, results are returned not log-scraped). `--watch` keeps the runtime warm (~30s first run, then ~2s). `--attach` runs against an app already up under `run --local --test-endpoint` (no boot; uses **that app's** database) | -| **Diff** | `mxcli diff -p app.mpr changes.mdl` | Compare script against project state | -| **Diff local** | `mxcli diff-local -p app.mpr --ref head` | Git diff for MPR v2 projects | -| **Diff revisions** | `mxcli diff-local -p app.mpr --ref main..feature` | Compare two arbitrary git revisions | -| **OQL** | `mxcli oql -p app.mpr "select ..."` | Query running Mendix runtime | -| **Widgets** | `show widgets`, `update widgets set ...`, `mxcli widget describe ` | Widget discovery, bulk updates (experimental), and inspecting a widget's discovered properties + dynamic rules | -| **External SQL** | `sql connect`, `sql select ...`, `mxcli sql` | Direct SQL queries against PostgreSQL, Oracle, SQL Server (credential isolation) | -| **Data import** | `import from query '...' into Module.Entity map (...)` | Import from external DB into Mendix app PostgreSQL (batch insert with ID generation) | -| **Connector gen** | `sql generate connector into [tables (...)] [views (...)] [exec]` | Auto-generate Database Connector MDL from discovered schema | -| **Marketplace drift** | `mxcli marketplace diff -p app.mpr [--to V] [--json]` | Which elements of an installed marketplace module have been edited locally, and what an upgrade would overwrite | -| **Model repair** | `mxcli fix widgets`, `mxcli fix design-properties` | Runs `mx update-widgets` / `mx rename-design-properties` and **persists** the result without their MPR v2 → v1 collapse (harvest: let the tool convert, read the units back, restore v2, write the changed ones through mxcli's writer). Clears CE0463 / CE6087 after a headless install — measured 203 → 0 errors on a vanilla 11.12.1 app | -| **Domain-model layout** | `mxcli layout -p app.mpr [--module M] [--dry-run]` | Arranges entities from the **association graph**: an entity referencing nothing is a lookup and goes left, everything else one column past the furthest thing it references, so lines run one way instead of crossing. Unconnected entities (non-persistent helpers) go in a band below rather than among the lookups. Positions are a function of the model, so a second run moves nothing. Replaces hand-arranged positions in the modules it touches — hence opt-in, with `--dry-run`; Marketplace modules and System are skipped. The **default** for an entity with no `@Position` is a wrapping grid (`mdl/dmlayout`), not the single 6,000px row it used to be | -| **Diagnostics** | `mxcli diag [--bundle]` | Session logs, version info, bug report bundles | -| **Loop report** | `mxcli diag loop-report [--json]` | Where a session's mxcli calls went — per-command counts, wall time and runs that did not close — read off the session logs every invocation already writes. Counts mxcli **processes**, not model calls, and says so; output size and `run --local` reloads are not recorded anywhere, so it reports what it cannot answer rather than estimating it | -| **Project brain** | `mxcli brain init\|capture\|staged\|promote\|drop\|check\|show\|plan\|resolve` | Opt-in store in `docs/brain/` for what mxcli **cannot** compute (why a pattern was chosen here, which marketplace version broke what). Sharded by module — an entry's first anchor names its file — so a session loads `project.md` plus the modules it is touching, not the whole store. Also holds the **plan**: requirements grouped into slices, whose anchors point *forward*, so `brain plan` reports progress **derived from the model** rather than from a status column. An agent captures to a git-ignored queue; a person promotes | -| **New project** | `mxcli new --version X.Y.Z [--output-dir dir] [--theme none] [--layout none]` | Downloads mxbuild, creates blank project, applies default styling, scaffolds a project-owned layout, runs init, installs Linux mxcli for devcontainer | -| **Default styling** | `mxcli theme list\|show\|apply\|remove` | Applies a theme (signal/ledger/console) — files under `theme/` only, the model is never touched | -| **Project themes** | `mxcli theme create [--from ]` | Scaffolds a theme the project owns into `theme/mxcli-themes/`; `--from ` seeds the palette from `--mxt-*` declarations | -| **Theme switching** | `mxcli theme apply --variant auto\|light\|dark`, `mxcli theme switcher install` | `auto` ships both palettes (follows the OS + honours a `theme-light`/`theme-dark` class); `switcher install` adds the JS actions + nanoflow for a user toggle (**this one does write to the model**) | -| **Switchable sets** | `mxcli theme apply signal ledger console` | Several themes in one stylesheet, each palette scoped to `:root.mxt-`; the app picks one with a class on `` — no rebuild, no reload | -| **Setup mxcli** | `mxcli setup mxcli [--os linux] [--arch amd64] [--output ./mxcli]` | Download platform-specific mxcli binary from GitHub releases | - -### mxcli new - -`mxcli new` creates a complete Mendix project from scratch in one step: - -```bash -mxcli new MyApp --version 11.8.0 -mxcli new MyApp --version 10.24.0 --output-dir ./projects/my-app +./bin/mxcli # REPL +./bin/mxcli -p app.mpr -c "show entities" # one command +./bin/mxcli exec script.mdl -p app.mpr # a script +./bin/mxcli check script.mdl -p app.mpr # validate without applying ``` -Steps performed: downloads MxBuild → `mx create-project` → `mxcli theme apply` → scaffolds `.App_Default` and moves the project's pages onto it (`--layout none` to keep Atlas's) → `mxcli init` → one `mxbuild --target=deploy` run (`--skip-build` to skip) → downloads correct Linux mxcli binary for devcontainer. That build settles the JS/Java action stubs MxBuild rewrites on first build (48 tracked files in a blank 11.12 app), so a fresh clone does not go dirty the first time anyone builds it. The result is a ready-to-open project with `.devcontainer/`, AI tooling, mxcli's default styling, a layout the project owns, and a working `./mxcli` binary. Pass `--theme none` for plain Atlas. - -The layout is **not** a copy of Atlas's: every Atlas layout a real app uses carries widgets MDL cannot spell (`Atlas_TopBar` has a `Forms$MenuBar`, a `Forms$SidebarToggleButton` and a pluggable image), so a describe → exec copy renders with no navigation and no logo. It reproduces the *result* instead — same layout class, same region classes, topbar navigation, `Main` for page content. +`./bin/mxcli help ` documents every command and `./bin/mxcli syntax` +every MDL statement. Neither is restated here. -### Slash Command Namespaces +Three things no `--help` tells you: -Commands in `.claude/commands/` are organised by audience: +- **Generated parser files are not committed.** `mdl/grammar/parser/` is produced by + `make grammar`, which `make build` runs — a fresh clone does not compile without it. +- **Skills are edited in `.claude/skills/mendix/`, never in `cmd/mxcli/skills/`.** The + latter is an embed dir regenerated by `make sync-skills` with `rsync --delete`. A + skill's frontmatter `description` is the routing mechanism; any table of skills is a + shortcut that drifts (it reached 12 of 68 before #906 caught it). +- **`.claude/commands/mendix/` is synced into user projects; `mxcli-dev/` is not.** + Contributor tooling goes in `mxcli-dev/`. -| Namespace | Folder | Invoked as | Purpose | -|-----------|--------|------------|---------| -| `mendix:` | `.claude/commands/mendix/` | `/mendix:lint` | mxcli **user** commands — synced to Mendix projects via `mxcli init` | -| `mxcli-dev:` | `.claude/commands/mxcli-dev/` | `/mxcli-dev:review` | **Contributor** commands — this repo only, never synced to user projects | +## Before Writing MDL -Both namespaces are discoverable by typing `/mxcli` in Claude Code. Add new contributor tooling (review workflows, debugging helpers, etc.) under `mxcli-dev/`. Add commands intended for Mendix project users under `mendix/`. - -### mxcli init - -`mxcli init` creates a `.claude/` folder with skills, commands, CLAUDE.md, and VS Code MDL extension in a target Mendix project. Source of truth for synced assets: -- Skills: `.claude/skills/mendix//SKILL.md` — directory-shaped, per the [Agent Skills](https://agentskills.io) standard, with `name` and `description` frontmatter. `make sync-skills` mirrors the tree into the `cmd/mxcli/skills/` embed dir (`//go:embed all:skills`), and `mxcli init` writes it into the project **twice**: `.ai-context/skills/` for every tool, and `.claude/skills/` — the only path Claude Code scans — when the project is set up for Claude. **Edit the `mendix/` source, not the embed dir** (it is regenerated, and the sync is `rsync --delete`). The `description` is the routing mechanism; the table in the generated CLAUDE.md is a shortcut, not the index. Upgrading a project retires the flat `.md` files older mxcli versions wrote, but never a skill the user added. The top-level `.claude/skills/*.md` are contributor/dev skills and are **not** synced. -- Commands: `.claude/commands/mendix/` (the `mxcli-dev/` folder is **not** synced) -- VS Code extension: `vscode-mdl/vscode-mdl-*.vsix` - -Build-time sync: `make build` syncs everything automatically. Individual targets: `make sync-skills`, `make sync-commands`, `make sync-vsix`. - -### VS Code Extension - -The `vscode-mdl` extension provides MDL language support: syntax highlighting, parse/semantic diagnostics, completion, symbols, folding, hover, go-to-definition, clickable terminal links, and context menu commands. The extension spawns `mxcli lsp --stdio` as the language server. Build with `make vscode-ext` (requires bun). - -### ANTLR4 Parser - -Regenerate after modifying `MDLLexer.g4`, `MDLParser.g4`, or any `domains/*.g4` file: `make grammar`. Generated files in `mdl/grammar/parser/` are **not** committed to git. See `docs/03-development/MDL_PARSER_ARCHITECTURE.md` for design details. - -## IMPORTANT: Before Writing MDL Scripts or Working with Data - -**Read the relevant skill files FIRST before writing any MDL, seeding data, or doing database/import work:** -- `.claude/skills/version-awareness.md` - **CHECK project version first** - Run `show features` before using version-gated syntax -- `.claude/skills/design-mdl-syntax.md` - **READ before designing new MDL syntax** - Design principles, decision framework, anti-patterns, checklist -- `.claude/skills/write-microflows.md` - Microflow syntax, common mistakes, validation checklist -- `.claude/skills/write-nanoflows.md` - Nanoflow syntax, restrictions, disallowed activities, validation checklist -- `.claude/skills/mendix/project-brain/SKILL.md` - **Project brain** (`mxcli brain`): the opt-in store for what mxcli cannot compute; why anything derivable from the model must never be written there, how anchors route an entry to its shard, and which `check` outcomes are failures -- `.claude/skills/mendix/write-rules.md` - **Rules** (CREATE/LIST/DESCRIBE/DROP/MOVE RULE): a rule returns Boolean or an enumeration and is callable only from a decision; what its body may not contain and the CE numbers behind each refusal; why there is no `grant execute on rule` -- `.claude/skills/write-workflows.md` - **Workflow authoring** (CREATE/DROP/ALTER WORKFLOW): activities (user task, decision, parallel split, jump, wait, boundary events), header options, gotchas. Workflows are authorable, not read-only. -- `.claude/skills/create-page.md` - Page/widget syntax reference -- `.claude/skills/mendix/write-layouts/SKILL.md` - **Layouts** (CREATE/DESCRIBE LAYOUT): the frame a page renders inside — scroll-container regions, the navigation tree, the placeholders pages bind to; why Atlas_Core is refused and why `mainplaceholder:` does not exist -- `.claude/skills/alter-page.md` - ALTER PAGE/SNIPPET in-place modifications (SET, INSERT, DROP, REPLACE, SET Layout) -- `.claude/skills/overview-pages.md` - CRUD page patterns -- `.claude/skills/master-detail-pages.md` - Master-detail page patterns -- `.claude/skills/generate-domain-model.md` - Entity/Association syntax -- `.claude/skills/mendix/scheduled-events-and-queues.md` - **Scheduled events (Mendix's cron) and task queues**: the eight Repeat variants and which fields each one takes, why a queue does NOT throttle a scheduled event, and why rewriting a microflow with a queued call is refused -- `.claude/skills/check-syntax.md` - Pre-flight validation checklist -- `.claude/skills/organize-project.md` - Folders, MOVE command, project structure conventions -- `.claude/skills/manage-security.md` - Security roles, access control, GRANT/REVOKE patterns -- `.claude/skills/manage-navigation.md` - Navigation profiles, home pages, menus, login pages -- `.claude/skills/demo-data.md` - **READ for any database/import work** - Mendix ID system, association storage, demo data insertion -- `.claude/skills/xpath-constraints.md` - XPath syntax in WHERE clauses, association paths, nested predicates, functions -- `.claude/skills/database-connections.md` - External database connections from microflows -- `.claude/skills/test-microflows.md` - **READ for testing work** - Test annotations, file formats, Docker setup requirement +**Read the matching skill first.** They are in `.claude/skills/` (contributor) and +`.claude/skills/mendix//SKILL.md` (synced to user projects). Each one's +frontmatter `description` says when to reach for it — that IS the index, so list the +directory rather than looking for a table here. ### Mendix Microflow/Nanoflow Idioms (MUST follow) @@ -489,60 +338,33 @@ These rules apply whenever generating microflow or nanoflow MDL. Violations are ./bin/mxcli check script.mdl -p app.mpr --references # With reference validation ``` -## MDL Syntax Quick Reference - -Full syntax tables for all MDL statements (microflows, pages, security, navigation, settings, business events, ALTER PAGE, reserved words) are in **[docs/01-project/MDL_QUICK_REFERENCE.md](docs/01-project/MDL_QUICK_REFERENCE.md)**. +## What mxcli Can Do — Ask the Tool -## What mxcli Can Do +**This file does not list features or syntax.** `./bin/mxcli syntax` enumerates every +MDL statement (`--json` for bulk), `./bin/mxcli help ` documents each command, +and `./bin/mxcli lint --list-rules` names every rule. A copy here is a transcription of +what those answer authoritatively, and it goes stale the next time anything ships. Full +syntax tables: [MDL_QUICK_REFERENCE.md](docs/01-project/MDL_QUICK_REFERENCE.md). -**This file does not list features.** `./bin/mxcli syntax` enumerates every MDL -statement (`--json` for bulk), `./bin/mxcli help ` documents each command, -and `./bin/mxcli lint --list-rules` names every rule. A list here is a transcription -of what those answer authoritatively, and it goes stale the next time anything ships. - -Per-doctype gotchas, CE numbers and the measurements behind them live in the skill -for that doctype (`.claude/skills/mendix//SKILL.md`) — loaded when you touch -that area rather than re-read into every session. Design rationale lives in -`docs/11-proposals/`; cross-cutting decisions in `docs/13-decisions/`. +Per-doctype gotchas, CE numbers and the measurements behind them live in the skill for +that doctype — loaded when you touch that area rather than re-read every session. +Design rationale is in `docs/11-proposals/`; cross-cutting decisions in +`docs/13-decisions/`. Still absent: 47 of 52 metamodel domains, delta/change tracking, runtime type reflection. -## Useful Files for Context - -- `README.md` - User documentation and API reference -- `api/api.go` - High-level fluent API entry point -- `api/domainmodels.go` - Entity/Association/Attribute builders -- `docs/01-project/SDK_EQUIVALENCE.md` - Detailed comparison with TypeScript SDK, gap analysis -- `modelsdk/codec/decoder.go` - BSON decoding (handles polymorphic types) -- `modelsdk/codec/encoder.go` - BSON encoding -- `mdl/backend/modelsdk/widget_pluggable_write.go` - Pluggable widget BSON, and the v1/v2 BSON driver conversion at the backend boundary -- `sdk/widgets/templates/` - Embedded widget templates for pluggable widgets (ComboBox, DataGrid2, etc.) -- `sdk/widgets/templates/README.md` - **Critical**: Template extraction requirements (must include both `type` AND `object`) -- `generated/metamodel/enums.go` - All Mendix enumeration types -- `modelsdk/meta/system_module.go` - The virtual System module's entities, attributes and associations. String lengths are **measured**, from the System module's domain model inside a built `deployment/model/model.mdp` (a BSON document stream, one `mxbuild --target=deploy` for all 115 at once) — not from the Model SDK, which describes metamodel types and does not contain them. `modelsdk/meta/testdata/system_string_lengths.txt` is the measurement and `TestSystemStringLengths` holds the table to it; a `Length` of 0 is Mendix's "unlimited", never "unmeasured". Measured identical across 10.24.4 and 11.14.0, which is why there is one table and not a per-version registry -- `mdl/grammar/MDL.g4` - ANTLR4 grammar for MDL syntax (production) -- `mdl/executor/executor.go` - MDL statement execution logic -- `reference/mdl-grammar/` - Comprehensive MDL grammar reference -- `reference/mendixmodellib/reflection-data/` - Type definitions with storage names and default values -- `docs/03-development/MDL_PARSER_ARCHITECTURE.md` - ANTLR4 parser design documentation -- `docs/03-development/MODELSDK_ENGINE_ARCHITECTURE.md` - **Read before extending the modelsdk engine**: layers, the canonical write/read/ALTER patterns, codec mechanisms (TypeDefaults, list markers, storage-name overrides), the engalar harvest rule, and the add-a-document-type recipe -- `docs/03-development/PAGE_BSON_SERIALIZATION.md` - Page/widget BSON format, type mappings, required defaults -- `docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md` - What's version-resilient vs version-fragile in widget BSON output, and how to onboard a new Mendix minor (e.g. 11.10) -- `.claude/skills/debug-bson.md` - Workflow for debugging BSON serialization issues with `mx` tool (includes the "Studio Pro Update Widget" diff methodology that closed CE0463) -- `.claude/skills/diagnose-ce0463.md` - **Read first for any CE0463 report**: the elimination order, the two controls that separate "the user upgraded a widget package" (not our bug) from a real mxcli defect, and the measurement traps that make CE0463 investigations go wrong -- `.claude/skills/verify-in-runtime.md` - Proving a fix in a real app in a real browser (`run --local` + Playwright). For symptoms that only exist at render time, where valid-looking BSON and a clean `mx check` prove nothing — see #812 -- `cmd/mxcli/lsp.go` - LSP server implementation (hover, definition, diagnostics, completion, symbols) -- `cmd/mxcli/init.go` - `mxcli init` command (project initialization + VS Code extension install) -- `cmd/mxcli/docker/oql.go` - OQL query execution against running Mendix runtime via M2EE admin API -- `sql/connection.go` - External SQL connection manager (credential isolation) -- `sql/config.go` - DSN resolution (env vars, YAML config) -- `sql/import.go` - IMPORT pipeline (batch insert, Mendix ID generation, sequence tracking) -- `sql/generate.go` - Database Connector MDL generation from external schema -- `sql/typemap.go` - SQL → Mendix type mapping, DSN → JDBC URL conversion -- `sql/mendix.go` - Mendix DB helpers (DSN builder, table/column name conversion) -- `cmd/mxcli/cmd_sql.go` - `mxcli sql` CLI subcommand -- `mdl/executor/cmd_sql.go` - SQL statement executor handlers -- `mdl/executor/cmd_import.go` - IMPORT statement executor (auto-connects to Mendix DB) -- `vscode-mdl/src/extension.ts` - VS Code extension entry point -- `vscode-mdl/package.json` - VS Code extension manifest (commands, menus, settings) +## Where to Look First + +Only the routing that reading the filename does not give you: + +| Before you | Read | +|---|---| +| extend the modelsdk engine | `docs/03-development/MODELSDK_ENGINE_ARCHITECTURE.md` | +| change the parser or grammar | `docs/03-development/MDL_PARSER_ARCHITECTURE.md` | +| write or debug widget BSON | `docs/03-development/PAGE_BSON_SERIALIZATION.md`, `WIDGET_BSON_VERSION_COMPATIBILITY.md` | +| act on a CE0463 report | `.claude/skills/diagnose-ce0463.md` — **read first**, it has the two controls that separate a user's widget upgrade from an mxcli defect | +| debug any other BSON issue | `.claude/skills/debug-bson.md` | +| prove a fix in the running app | `.claude/skills/verify-in-runtime.md` — for symptoms where valid BSON and a clean `mx check` prove nothing (#812) | +| add a pluggable widget template | `sdk/widgets/templates/README.md` — a template needs **both** `type` and `object` | +| rely on a System-module attribute's length | `modelsdk/meta/system_module.go` — the lengths are **measured** from a built `model.mdp`, not taken from the Model SDK, which does not contain them; `Length` 0 is Mendix's "unlimited", never "unmeasured" | diff --git a/docs-site/src/tools/domain-model-layout.md b/docs-site/src/tools/domain-model-layout.md index 0b38d18c61..4e776cc0c9 100644 --- a/docs-site/src/tools/domain-model-layout.md +++ b/docs-site/src/tools/domain-model-layout.md @@ -82,3 +82,7 @@ Two things to know if you place entities yourself: - An entity created with no position takes the next slot in a wrapping grid. That is a default, not a layout: it keeps a large model on screen and stops boxes overlapping, but it knows nothing about which entities are related. + +## Default position for an entity with no `@Position` + +A wrapping grid (`mdl/dmlayout`), not the single 6,000px row it used to be. From df9868dbb1f7c0b4a817f87c3d2878da4bc95a89 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 15:08:18 +0000 Subject: [PATCH 31/38] test: hold this repo's CLAUDE.md to a context budget (#611) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last item on #611. init_claudemd_budget_test.go argued the case for a user's project — a file re-read into every context is a per-session tax rather than a one-off — and then applied it outward only, while the repo's own CLAUDE.md grew to 18x that budget in the same package. Set at 28,000, just above the current 25,514. Higher than the generated file's 6,000 because this one legitimately carries more: the invariants whose violation is silent and unrecoverable, and the evidence bar for a change. Deliberately not generous — enough to edit within, not enough to regrow into, so adding something means taking something out. The test is mutation-checked rather than merely green: lowered to 20,000 it fails naming the real size, so it is measuring the file and not passing vacuously. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- cmd/mxcli/claudemd_repo_budget_test.go | 41 ++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 cmd/mxcli/claudemd_repo_budget_test.go diff --git a/cmd/mxcli/claudemd_repo_budget_test.go b/cmd/mxcli/claudemd_repo_budget_test.go new file mode 100644 index 0000000000..c44b1f1c55 --- /dev/null +++ b/cmd/mxcli/claudemd_repo_budget_test.go @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "os" + "testing" +) + +// This repo's own CLAUDE.md is re-read into every context started here, exactly +// as the generated one is for a user's project — so its size is a per-session +// tax, and the reasoning in init_claudemd_budget_test.go applies to it too. +// +// It was not applied. The file reached 108,761 B (~27k tokens), 18x the 6,000 +// enforced on users' projects, while that budget test sat in the same package +// (ako/mxcli#611). Four slices took it to ~25.5k by moving per-subsystem detail +// into the skill or doc that owns it, and deleting what `mxcli syntax`, +// `mxcli help` and a directory listing answer authoritatively. +// +// The budget is higher than the generated file's because this one legitimately +// carries more: the invariants whose violation is silent and unrecoverable +// (GUID-as-database-identity, conditional writes, storage names) plus the +// evidence bar for a change. It is set just above the current size on purpose — +// enough to edit within, not enough to regrow into. Adding something here means +// taking something out, which is the decision the budget exists to force. +const repoClaudeMDBudgetBytes = 28000 + +func TestRepoClaudeMDStaysWithinItsContextBudget(t *testing.T) { + const path = "../../CLAUDE.md" + info, err := os.Stat(path) + if err != nil { + t.Fatalf("cannot stat %s: %v", path, err) + } + if info.Size() > repoClaudeMDBudgetBytes { + t.Errorf("CLAUDE.md is %d bytes (~%d tokens), over the %d-byte budget.\n"+ + "It is re-read into every context started in this repo. Anything needed only when\n"+ + "touching one subsystem belongs in that subsystem's skill or doc; anything `mxcli\n"+ + "syntax`, `mxcli help` or `ls` answers belongs nowhere. See ako/mxcli#611.", + info.Size(), info.Size()/4, repoClaudeMDBudgetBytes) + } +} From 37b03e3aac3ffd163a785d9c39eef834c56ef21a Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 18:02:25 +0000 Subject: [PATCH 32/38] feat(run): --page-check answers in text what a screenshot answers in pixels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #614. Most screenshotting during agent work answers a textual question — did the page render, is there an error banner, did the grid get rows. A PNG costs ~1,500 tokens to read against ~100 for a verdict, and the cost is not a one-off: an image read into a conversation is re-read by every later model call, so one PNG early in a long session is charged hundreds of times. It is also the weaker instrument. Measured against two real rendered pages: page /p/customers title="Customers" h="Customer overview" rows=2 text=21 console-errors=0 page /p/customers title="Customers" NO VISIBLE TEXT console-errors=1 ERR Cannot read properties of undefined (reading 'items') The second is the blank-page symptom WITH ITS CAUSE NAMED. A console error is the commonest reason a page renders blank and it does not appear in a picture at all — that class of bug took ~40 calls to trace in the session behind PROPOSAL_agent_loop_efficiency.md, every one of them looking at pixels that could not show it. --page-check prints the verdict and no PNG; --screenshot now prints it alongside the PNG, so the cheap signal arrives whether or not it was asked for and the image need not be opened. No new dependency: the Playwright CLI has no text-dump subcommand, so the probe runs under the node Playwright already needs, resolving the package via `npm root -g`. It never throws on a bad page — a page that fails to render is the thing being measured, not an error in measuring it. Brevity is load-bearing, so everything unbounded is clamped and a test holds the verdict under 600 B on a page with 5,000 rows and 40 console errors: a verdict that grows with the page defeats its own purpose. Also adds the routing rule to verify-in-runtime.md, which had nothing about cost (0 mentions) — the proposal stated it and the skill never got it, so nothing steered an agent from pixels to text. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .claude/skills/verify-in-runtime.md | 41 ++++++ cmd/mxcli/cmd_run.go | 3 + cmd/mxcli/docker/pagecheck.go | 186 ++++++++++++++++++++++++++++ cmd/mxcli/docker/pagecheck_test.go | 77 ++++++++++++ cmd/mxcli/docker/runlocal.go | 29 +++++ 5 files changed, 336 insertions(+) create mode 100644 cmd/mxcli/docker/pagecheck.go create mode 100644 cmd/mxcli/docker/pagecheck_test.go diff --git a/.claude/skills/verify-in-runtime.md b/.claude/skills/verify-in-runtime.md index 9727aeb349..e7f21cb605 100644 --- a/.claude/skills/verify-in-runtime.md +++ b/.claude/skills/verify-in-runtime.md @@ -22,6 +22,47 @@ Use the layer where the symptom actually lives: we write, no unit or BSON test can prove the fix.** A page can serialize to perfectly correct-looking BSON and still render wrong. +## Assert in Text; Screenshot Only When the Question Is Visual + +Reaching this skill says the *running app* is the layer. It does not say the answer +has to be a picture, and usually it should not be. + +A screenshot costs roughly **1,500 tokens** to read against about **100** for a text +verdict, and the cost is not a one-off: an image read into a conversation is re-read +by every later model call, so one PNG early in a long session is charged hundreds of +times (`docs/11-proposals/PROPOSAL_agent_loop_efficiency.md`). + +It is also the *weaker* instrument for the usual question. "Did the page render?", +"is there an error banner?", "did the grid get rows?" are textual, and the commonest +cause of a page that renders blank — a console error — **does not appear in a +picture at all**. + +```bash +./mxcli run --local --page-check -p app.mpr # verdict, no PNG +./mxcli run --local --screenshot -p app.mpr # PNG *and* verdict +./mxcli playwright verify tests/ -p app.mpr # assertions; shoots only on failure +``` + +``` +page /p/customers title="Customers" h="Customer overview" rows=12 text=812 console-errors=0 +page /p/customers title="Customers" NO VISIBLE TEXT console-errors=1 + ERR Cannot read properties of undefined (reading 'items') +``` + +The second line is the blank-page symptom **with its cause named**. That class of bug +took ~40 calls to trace in the session behind the proposal, every one of them looking +at pixels that could not show it. + +So: + +- **Default to `--page-check` or `playwright verify`.** They answer the question and + leave the PNG unread. +- **Take a screenshot when the question is genuinely about appearance** — layout, + spacing, colour, "does this look right" — and then take it **once**, at the end. +- **Never screenshot to confirm something a verdict already reported.** If + `console-errors=0`, `rows=12` and the heading is right, the page rendered; a picture + adds cost and no information. + Worked example — mendixlabs/mxcli#812. Every popup opened by an mxcli-authored button showed a blank caption. The BSON was structurally valid, `mx check` reported 0 errors, and MxBuild completed. Nothing below the browser could see the defect, because the diff --git a/cmd/mxcli/cmd_run.go b/cmd/mxcli/cmd_run.go index 02919d8467..aec8758042 100644 --- a/cmd/mxcli/cmd_run.go +++ b/cmd/mxcli/cmd_run.go @@ -156,6 +156,7 @@ Examples: dbUser, _ := cmd.Flags().GetString("db-user") dbPassword, _ := cmd.Flags().GetString("db-password") screenshot, _ := cmd.Flags().GetBool("screenshot") + pageCheck, _ := cmd.Flags().GetBool("page-check") screenshotPath, _ := cmd.Flags().GetString("screenshot-path") screenshotURLs, _ := cmd.Flags().GetStringArray("screenshot-url") screenshotUser, _ := cmd.Flags().GetString("screenshot-user") @@ -210,6 +211,7 @@ Examples: EnsureDB: ensureDB, SetupOnly: setupOnly, Screenshot: screenshot, + PageCheck: pageCheck, ScreenshotPath: screenshotPath, ScreenshotURLs: screenshotURLs, ScreenshotUser: screenshotUser, @@ -324,6 +326,7 @@ func init() { runCmd.Flags().String("db-user", "", "Database user (default mendix)") runCmd.Flags().String("db-password", "", "Database password (default mendix)") runCmd.Flags().Bool("screenshot", false, "Capture a Playwright screenshot after boot and each applied change") + runCmd.Flags().Bool("page-check", false, "Print a text verdict for each page (title, headings, error banners, row count, console errors) instead of reading a screenshot — far cheaper, and it reports console errors a PNG cannot show") runCmd.Flags().String("screenshot-path", "", "Screenshot output PNG (default /.mxcli/run-local.png)") runCmd.Flags().StringArray("screenshot-url", nil, "Page to screenshot: a full URL or a path relative to the app root, e.g. /p/customers (default the app root). Repeat for a multi-page set.") runCmd.Flags().String("screenshot-user", "", "Log in with this user before screenshotting (for pages behind login)") diff --git a/cmd/mxcli/docker/pagecheck.go b/cmd/mxcli/docker/pagecheck.go new file mode 100644 index 0000000000..b3913a284f --- /dev/null +++ b/cmd/mxcli/docker/pagecheck.go @@ -0,0 +1,186 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "strings" + "time" +) + +// pagecheck.go answers, in text, the question a screenshot is usually taken to +// answer: did this page render, and is anything wrong with it. +// +// A PNG costs roughly 1,500 tokens to read and a verdict about 100 — and the +// PNG's cost is not a one-off, because an image read into a conversation is +// re-charged on every later model call (ako/mxcli#614, and +// docs/11-proposals/PROPOSAL_agent_loop_efficiency.md). The picture is also +// strictly *less* informative for this question: a console error, which is the +// usual cause of a page that renders blank, does not appear in one. +// +// It uses the Playwright already required by --screenshot, driven through node +// rather than the CLI, because the Playwright CLI has no text-dump subcommand. + +// PageSignals is what one page load tells us. +type PageSignals struct { + Title string `json:"title"` + Headings []string `json:"headings"` + Alerts []string `json:"alerts"` // visible Mendix error/warning banners + TextLen int `json:"textLen"` // body innerText length; 0 is the blank-page tell + Rows int `json:"rows"` // grid / list rows rendered + ConsoleErrors []string `json:"consoleErrors"` // what a screenshot cannot show +} + +// pageProbeJS runs in node with the global Playwright. It reports signals and +// never throws for a bad page — a page that fails to render is the thing being +// measured, not an error in measuring it. +const pageProbeJS = ` +const { chromium } = require('playwright'); +(async () => { + const [url, storage, waitMs] = [process.argv[2], process.argv[3], parseInt(process.argv[4] || '4000', 10)]; + const b = await chromium.launch(); + const ctx = await b.newContext(storage ? { storageState: storage } : {}); + const p = await ctx.newPage(); + const consoleErrors = []; + p.on('console', m => { if (m.type() === 'error') consoleErrors.push(m.text()); }); + p.on('pageerror', e => consoleErrors.push(String(e && e.message || e))); + const out = { title: '', headings: [], alerts: [], textLen: 0, rows: 0, consoleErrors }; + try { + await p.goto(url, { waitUntil: 'domcontentloaded' }); + await p.waitForTimeout(waitMs); + out.title = await p.title(); + out.headings = await p.$$eval('h1,h2', ns => ns.map(n => (n.innerText||'').trim()).filter(Boolean).slice(0, 3)); + out.alerts = await p.$$eval('.alert-danger,.alert-warning,.mx-validation-message', + ns => ns.map(n => (n.innerText||'').trim()).filter(Boolean).slice(0, 3)); + out.textLen = (await p.$eval('body', n => n.innerText || '')).trim().length; + out.rows = await p.$$eval('.mx-datagrid tbody tr, .mx-listview > ul > li, [role="row"]', ns => ns.length); + } catch (e) { + out.consoleErrors.push('probe: ' + String(e && e.message || e)); + } + console.log(JSON.stringify(out)); + await b.close(); +})(); +` + +// CheckPage loads url and returns its signals. storage is an optional Playwright +// storage-state file, for pages behind a login. +func CheckPage(url, storage string, waitMs int, timeout time.Duration) (PageSignals, error) { + var sig PageSignals + + node, err := exec.LookPath("node") + if err != nil { + return sig, fmt.Errorf("node not found; the page check needs the same Node that Playwright uses") + } + f, err := os.CreateTemp("", "mxcli-pagecheck-*.js") + if err != nil { + return sig, err + } + defer os.Remove(f.Name()) + if _, err := f.WriteString(pageProbeJS); err != nil { + f.Close() + return sig, err + } + f.Close() + + if waitMs == 0 { + waitMs = 4000 + } + if timeout == 0 { + timeout = 90 * time.Second + } + cmd := exec.Command(node, f.Name(), url, storage, fmt.Sprint(waitMs)) + cmd.Env = append(os.Environ(), "NODE_PATH="+globalNodeModules()) + out := &syncBuffer{} + cmd.Stdout, cmd.Stderr = out, out + + done := make(chan error, 1) + if err := cmd.Start(); err != nil { + return sig, fmt.Errorf("launching the page check: %w", err) + } + go func() { done <- cmd.Wait() }() + select { + case err := <-done: + if err != nil { + return sig, fmt.Errorf("page check failed: %w\n%s", err, out.String()) + } + case <-time.After(timeout): + _ = cmd.Process.Kill() + <-done + return sig, fmt.Errorf("page check timed out after %s", timeout) + } + + // The probe prints one JSON line; Playwright may print noise before it. + lines := strings.Split(strings.TrimSpace(out.String()), "\n") + last := lines[len(lines)-1] + if err := json.Unmarshal([]byte(last), &sig); err != nil { + return sig, fmt.Errorf("page check produced no verdict: %w\n%s", err, out.String()) + } + return sig, nil +} + +// globalNodeModules resolves the global node_modules so `require('playwright')` +// works from a temp file. `npm root -g` is authoritative; the common install +// path is the fallback when npm is absent. +func globalNodeModules() string { + if npm, err := exec.LookPath("npm"); err == nil { + if out, err := exec.Command(npm, "root", "-g").Output(); err == nil { + if p := strings.TrimSpace(string(out)); p != "" { + return p + } + } + } + return "/usr/lib/node_modules" +} + +// formatPageVerdict renders signals as the line that replaces the screenshot. +// +// Brevity is the point, so everything unbounded is clamped: the verdict must +// stay far cheaper than the image even on a page with hundreds of rows and a +// wall of console noise, or there is no reason to prefer it. +func formatPageVerdict(label string, s PageSignals) string { + var b strings.Builder + fmt.Fprintf(&b, "page %s", label) + if s.Title != "" { + fmt.Fprintf(&b, " title=%q", clip(s.Title, 60)) + } + if len(s.Headings) > 0 { + fmt.Fprintf(&b, " h=%q", clip(s.Headings[0], 60)) + } + if s.Rows > 0 { + fmt.Fprintf(&b, " rows=%d", s.Rows) + } + if s.TextLen == 0 { + b.WriteString(" NO VISIBLE TEXT") + } else { + fmt.Fprintf(&b, " text=%d", s.TextLen) + } + fmt.Fprintf(&b, " console-errors=%d\n", len(s.ConsoleErrors)) + + // Detail lines, capped. These are what make the verdict actionable rather + // than merely cheap, and the console error is the one a picture cannot give. + for _, a := range firstN(s.Alerts, 2) { + fmt.Fprintf(&b, " ALERT %s\n", clip(a, 140)) + } + for _, e := range firstN(s.ConsoleErrors, 2) { + fmt.Fprintf(&b, " ERR %s\n", clip(e, 140)) + } + return b.String() +} + +func firstN(ss []string, n int) []string { + if len(ss) > n { + return ss[:n] + } + return ss +} + +func clip(s string, n int) string { + s = strings.ReplaceAll(strings.TrimSpace(s), "\n", " ") + if len(s) > n { + return s[:n] + "…" + } + return s +} diff --git a/cmd/mxcli/docker/pagecheck_test.go b/cmd/mxcli/docker/pagecheck_test.go new file mode 100644 index 0000000000..09072b7662 --- /dev/null +++ b/cmd/mxcli/docker/pagecheck_test.go @@ -0,0 +1,77 @@ +// SPDX-License-Identifier: Apache-2.0 + +package docker + +import ( + "strings" + "testing" +) + +// A screenshot answers a textual question with an image. The PNG on disk is +// free; reading it into context is the whole cost, and from then on it is +// re-charged on every later model call (ako/mxcli#614). +// +// These cover the formatting, which is where the value is: the verdict has to +// be short enough to beat the picture and specific enough to replace it. + +func TestVerdictIsOneLineWhenThePageIsHealthy(t *testing.T) { + got := formatPageVerdict("/p/customers", PageSignals{ + Title: "Customers", Headings: []string{"Customer overview"}, + TextLen: 812, Rows: 12, + }) + if n := strings.Count(strings.TrimRight(got, "\n"), "\n") + 1; n != 1 { + t.Errorf("healthy page rendered %d lines, want 1:\n%s", n, got) + } + for _, want := range []string{"/p/customers", "Customers", "rows=12"} { + if !strings.Contains(got, want) { + t.Errorf("verdict omits %q:\n%s", want, got) + } + } +} + +// The blank-page tell. This is the symptom that cost ~40 calls in the session +// the proposal came from, and no screenshot shows WHY — the console error does. +func TestBlankPageIsCalledOutWithItsConsoleError(t *testing.T) { + got := formatPageVerdict("/p/customers", PageSignals{ + Title: "Customers", + TextLen: 0, + ConsoleErrors: []string{ + "TypeError: Cannot read properties of undefined (reading 'items')", + }, + }) + if !strings.Contains(got, "NO VISIBLE TEXT") { + t.Errorf("a page with no body text is not flagged:\n%s", got) + } + if !strings.Contains(got, "Cannot read properties of undefined") { + t.Errorf("the console error is not surfaced; it is the thing a screenshot cannot show:\n%s", got) + } +} + +// An error banner is visible in a screenshot, so the verdict must not be worse +// than the picture it replaces. +func TestVisibleErrorBannersAreReported(t *testing.T) { + got := formatPageVerdict("/", PageSignals{ + Title: "App", TextLen: 200, + Alerts: []string{"An error occurred, please contact your system administrator"}, + }) + if !strings.Contains(got, "contact your system administrator") { + t.Errorf("a visible alert banner is not reported:\n%s", got) + } +} + +// THE CONTROL on brevity. A verdict that grows with the page defeats its own +// purpose — it has to stay cheap on a page with a hundred rows and a long body. +func TestVerdictStaysShortOnALargePage(t *testing.T) { + long := make([]string, 40) + for i := range long { + long[i] = "a heading that is quite long and would bloat the verdict badly" + } + got := formatPageVerdict("/p/big", PageSignals{ + Title: "Big", Headings: long, TextLen: 400000, Rows: 5000, + ConsoleErrors: long, + }) + if len(got) > 600 { + t.Errorf("verdict is %d bytes on a large page; it must stay far cheaper than a "+ + "screenshot (~1,500 tokens) or there is no point:\n%s", len(got), got) + } +} diff --git a/cmd/mxcli/docker/runlocal.go b/cmd/mxcli/docker/runlocal.go index 1c5bc54d9e..433e61142f 100644 --- a/cmd/mxcli/docker/runlocal.go +++ b/cmd/mxcli/docker/runlocal.go @@ -102,6 +102,12 @@ type LocalRunOptions struct { // Screenshot, when set, captures a PNG of the app after boot and after each // applied change (requires the Playwright CLI + a browser). Screenshot bool + + // PageCheck prints a text verdict for each target page instead of (or as + // well as) capturing a PNG. It answers the question a screenshot is usually + // taken to answer at a fraction of the tokens, and reports console errors, + // which a picture cannot show. See pagecheck.go and ako/mxcli#614. + PageCheck bool // ScreenshotPath is where the PNG is written (default /.mxcli/run-local.png). // With multiple ScreenshotURLs, it is the base name and each page gets a // per-page suffix (run-local-.png). @@ -953,6 +959,13 @@ func runtimeStoppedError(rt *LocalRuntime) error { // maybeScreenshot captures the app (best-effort) when --screenshot is set. A // failure is reported but never aborts the loop — the app is still running. func maybeScreenshot(opts LocalRunOptions, rt *LocalRuntime) { + // The text verdict is printed whenever either flag asks for it: with + // --page-check alone it is the whole output, and alongside --screenshot it + // means the PNG does not have to be opened to learn whether the page + // rendered (ako/mxcli#614). + if opts.PageCheck { + reportPageChecks(opts, rt) + } if !opts.Screenshot { return } @@ -1384,3 +1397,19 @@ func declaredJarDependencies(reader backend.FullBackend) []JarDependencyRef { } return out } + +// reportPageChecks prints one verdict line per target page. +func reportPageChecks(opts LocalRunOptions, rt *LocalRuntime) { + targets := opts.ScreenshotURLs + if len(targets) == 0 { + targets = []string{""} + } + for _, t := range targets { + sig, err := CheckPage(resolveScreenshotURL(rt.AppURL(), t), opts.screenshotStorage, 4000, 0) + if err != nil { + fmt.Fprintf(opts.Stderr, " page check skipped (%s): %v\n", pageLabel(t), err) + continue + } + fmt.Fprint(opts.Stdout, " "+formatPageVerdict(pageLabel(t), sig)) + } +} From cb5cd2da25949ea5e859b324d04bea3f7caa50e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 18:03:55 +0000 Subject: [PATCH 33/38] =?UTF-8?q?docs:=20correct=20lever=205=20=E2=80=94?= =?UTF-8?q?=20four=20of=20five=20were=20already=20done,=20the=20fifth=20wa?= =?UTF-8?q?s=20a=20bug?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The proposal listed the five Mendix limitations the cost report named and proposed turning each into a check diagnostic or skill. That was written from the report's framing without verifying any of them. Checking all five afterwards: inputs in lists read-only an MXCLI BUG, fixed 2026-09-06 — List View has its own Editable (default No) that wins over the textbox's; buildListViewV3 never read it and the writer wrote false popup styling covered in theme-styling/SKILL.md scripted login input covered in test-app/SKILL.md, with the playwright-cli eval workaround wrong Java version handled in docker/javaversion.go (11.14 is the first wanting 25 rather than 21) stale sidebar the only one still open, and it is runtime refresh behaviour with no static signal So the lever as written would have shipped one diagnostic that is now actively wrong — inputs in lists ARE editable, and saying otherwise sends a reader back to a pop-up workaround they no longer need — and three that duplicate skills that already exist. What the correction reveals is a ROUTING problem rather than a knowledge one: the knowledge was there, in the skill whose description is supposed to surface it, and the session hit the wall anyway. Same failure as the skill table drifting to 12 of 68 (#906) — an index that does not route is indistinguishable from missing content. The principle survives; the lesson measuring adds is the prior step. Check whether it is already known, and whether it is even true, before encoding it. A platform limitation that was really a bug outlives the bug. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MgcYSQrLLbUcnAMaCHpyqQ --- .../PROPOSAL_agent_loop_efficiency.md | 44 +++++++++++++------ 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md index 8856b0593b..e17ae38eb5 100644 --- a/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md +++ b/docs/11-proposals/PROPOSAL_agent_loop_efficiency.md @@ -436,23 +436,39 @@ skills should say so with a trigger rather than a preference: **a diagnosis expected to take more than ~5 probes is delegated, not run inline.** The same applies to log spelunking and "which of these 30 files mentions X". -## Lever 5 — turn each discovered workaround into tool knowledge +## Lever 5 — turn each discovered workaround into tool knowledge (mostly already done) -Five Mendix limitations each cost an investigation and a rework in that session: +The first draft listed the five Mendix limitations the cost report named and proposed +turning each into a `check` diagnostic or a skill. **That was written from the +report's framing without verifying any of them, and checking all five afterwards +found that four were already covered and the fifth was not a Mendix limitation at +all.** -| Discovered the hard way | Where it should live instead | +| Reported as a Mendix limitation | What it actually is | |---|---| -| inputs inside lists are read-only → admin editing must be pop-ups | `mxcli check` diagnostic on an input widget in a list/gallery context | -| the sidebar went stale after actions | `create-page` / `patterns-crud` skill, refresh guidance | -| pop-up styling breaks (pop-ups sit outside the styled area) | `theme-styling` skill | -| login fields did not register scripted input | a `mxcli playwright login` helper that does it correctly | -| the Docker image had the wrong Java version | `mxcli docker check` preflight | - -This is the highest-leverage lever on any horizon longer than one session, -because it converts a cost paid **once per session per user** into one paid -**once, by us**. It is also exactly the repo's existing instinct — findings -files, lint rules, check diagnostics — applied to a class of knowledge that has -so far only been rediscovered. +| inputs inside lists are read-only, so admin editing became pop-ups | **An mxcli bug, fixed 2026-09-06.** Mendix's List View has its *own* `Editable` (default No) which wins over the textbox's; the parser accepted `Editable: true`, `buildListViewV3` never read it and the writer wrote false. See the finding in `mdl-executor.jsonl` | +| pop-up styling breaks outside the app's styled area | covered in `theme-styling/SKILL.md` — the class lands on ``, so popups rendered at `` follow it | +| login fields did not register scripted input | covered in `test-app/SKILL.md`, with the `playwright-cli eval` workaround, and it says the fill fails *silently* | +| the Docker image had the wrong Java version | handled in code — `docker/javaversion.go` knows 11.14 is the first version wanting Java 25 rather than 21 | +| the sidebar went stale after actions | the only one still open, and it is runtime refresh behaviour with no static signal to check for | + +So the lever as originally written would have produced one diagnostic that is now +**actively wrong** (inputs in lists are editable, and saying otherwise sends a reader +back to the pop-up workaround they no longer need) and three that duplicate existing +skills. + +**What the correction actually reveals is a routing problem, not a knowledge problem.** +The knowledge existed, in the skill whose `description` is supposed to surface it, and +the session hit the wall anyway. That is the same failure as the skill table drifting +to 12 of 68 (#906): an index that does not route is indistinguishable from missing +content. Effort here belongs in making skills *findable* at the moment of need, not in +writing more of them. + +The general principle survives and is worth keeping: a workaround discovered in a +session is a cost paid once per session per user until it becomes a diagnostic, a +refusal or a skill. The lesson added by measuring is the prior step — **check whether +it is already known, and whether it is even true, before encoding it.** Encoding a +platform limitation that was really a bug outlives the bug. ## Lever 6 — measure it, then claim it From c2dcb28fd2a203a07edae2948f41d94d9ac669f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 18:26:45 +0000 Subject: [PATCH 34/38] fix(examples): gate the workflow-group examples to Mendix 11.2+ The nightly failed on Mendix 10.24 in TestMxCheck_DoctypeScripts: alter settings workflows add group '' requires Mendix 11.2.0+ (project is 10.24.24.119349) Examples 4.3-4.7 in 14-project-settings-examples.mdl were added ungated, though Settings$WorkflowGroup is 11.2 metamodel and the executor rightly refuses it on an older project. Wrap them in a `-- @version: 11.2+` section, with the directive above the doc comment so it isn't orphaned. Reproduced against mxbuild 10.24.24.119349: the test fails unfixed with the nightly's exact error and passes with the gate. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01NcJgoFWn5vFoQ2Zw8dp3DE --- .claude/skills/fix-issue/findings/other.jsonl | 1 + mdl-examples/doctype-tests/14-project-settings-examples.mdl | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/.claude/skills/fix-issue/findings/other.jsonl b/.claude/skills/fix-issue/findings/other.jsonl index 08e5771671..314922aa01 100644 --- a/.claude/skills/fix-issue/findings/other.jsonl +++ b/.claude/skills/fix-issue/findings/other.jsonl @@ -18,3 +18,4 @@ {"area":".claude/skills/packs","date":"2026-09-21","symptom":"A URL-fed Vega-Lite chart in the mendix-vega-charts pack drew its axes and a FULL legend with zero data points, no console error and no Vega warning. The skill stated that \"same-origin requests carry the session cookie, so an endpoint authenticated by session is reachable ... without any token handling\".","cause":"Mendix refuses a session-authenticated request without the session's CSRF token on READS too, not just writes and not just /xas/. The cookie is sent; it is not sufficient. Vega's loader read the 401 body as an empty dataset, so the failure surfaced as a plausible-looking empty chart rather than as an error.","file":".claude/skills/packs/mendix-vega-charts/widget/src/csrf.ts, .../SKILL.md, cmd/mxcli/skillpacks_test.go","insight":"THE LEGEND IS THE TELL: it is built from the spec's scales, not from rows, so a chart with a complete legend and no marks has had its DATA refused, while a chart with a broken legend has a spec problem. That one distinction separates the two hypotheses before any measurement. Two things then send the diagnosis the wrong way and cost the time: document.cookie shows only originURI=/login.html (XASSESSIONID and xasid are httpOnly, so the browser IS sending them and JavaScript cannot see them) and basic auth on the same URL returns the data, which reads as proof the endpoint is fine and the chart is broken. Isolate on the HEADER, not the URL: two requests, one added header, everything else equal -- 401 vs 200, measured on a fresh 11.14.0 app. Skip the plausible wrong turn of adding the header unconditionally: the issue's own suggested loader tests the URI with /^[a-z][a-z0-9+.-]*:\\/\\//i, which passes //elsewhere.example/rows.json (no scheme, another host) and hands that host a working session credential. Resolve with new URL(uri, base) and compare origins instead -- it also gets the converse right, an absolute URL naming the app's own origin IS the app. End-to-end control through vega's real loader against the running app: 0 marks / 3 axes / no error without the token, 1 mark with it, which reproduces the reported symptom exactly.","refs":["ako/mxcli#574"],"ce":[],"mendix":"11.14.0"} {"area": "skills", "date": "2026-09-22", "symptom": "A workflow inbox over System.WorkflowUserTask, re-sourced from a MICROFLOW to get past the System-module ceiling, drew the right number of cards and every card was COMPLETELY BLANK — no CE code, no console warning, and mxcli check, lint, report and docker check all at 0 errors. The manage-security and system-module skills had offered exactly that microflow data source as the way past the ceiling (ako/ChipCoV4, Mendix 11.14.0; ako/mxcli#587).", "cause": "Not an mxcli defect — a Mendix rule both skills stated as a workaround without having measured it. A microflow does not apply entity access, so its retrieve returns every row, but the runtime RE-APPLIES entity access when it serializes those objects to the client, XPath constraint included: a row the role may not read arrives with every member empty. Both skills now state the rule (\"a microflow data source moves the ROWS, not the MEMBERS\") with the measurement, and point at reading the member inside the microflow and returning a module-owned object.", "file": "`.claude/skills/mendix/manage-security/SKILL.md` (The System-module ceiling); `.claude/skills/mendix/system-module/SKILL.md`; measurement `mdl-examples/bug-tests/security-587-system-member-access.mdl`", "insight": "The probe that settles this in one page: TWO microflow-sourced lists over the SAME retrieve — one over the System objects, one over a module-owned copy whose attribute was read inside the microflow — opened by an Administrator and by a plain User. The row COUNTS are the discriminator and they are equal in all four cells (2 and 2), which is what proves the microflow moved the rows and isolates the loss to serialization; only the System list loses values, and only for the non-admin. Use System.User rather than System.WorkflowUserTask for the probe: it needs no workflow and it shows the mechanism MORE sharply, because System.User's own rule reads [id = '[%CurrentUser%]'] so the non-admin sees exactly one populated row and one blank — per-OBJECT blanking, not per-attribute. That also explains why a user picker 'lists the current user only' and a workflow inbox is entirely blank: same rule, different constraint. The control here is the ROLE, not a before/after build — same binary, same model, two logins — so no A/B rebuild is needed. Two traps in the probe itself: dynamictext content is a static template and renders '[%Name%]' literally, so bind the attribute with a TEXTBOX; and `grant on System.User` PASSES `mxcli check --references` and is refused only by `exec`, so a script carrying one checks clean and then stops part-way."} {"area": "ci", "date": "2026-09-22", "symptom": "A failing CI job's log held nothing but `##[error]Process completed with exit code 1` — no test name, no failure message. On `windows-process-regression` that made a red check impossible to diagnose from the log alone: the only evidence of WHICH test had failed was the runner's own `Terminate orphan process: pid (2760) (PING)` cleanup line, absent from the green run on main.", "cause": "The step captured the command into a variable — `out=$(go test -v -run '…' ./cmd/mxcli/docker/)` — and echoed it on the NEXT line. The runner's shell is `bash --noprofile --norc -e -o pipefail`, so a non-zero `go test` aborts the step AT THE ASSIGNMENT and the `echo \"$out\"` never runs. The capture existed only to count `--- PASS:` lines (a `-run` filter passes vacuously if the tests are renamed away). Replaced with `go test … 2>&1 | tee go-test-output.txt`, `status=${PIPESTATUS[0]}`, then grep the file — output streams as it is produced, the vacuous-run guard still counts, and a real failure is reported with its own exit status. Same pattern was in `tunnel-seam-cross-platform`; both fixed.", "file": "`.github/workflows/push-test.yml` (tunnel-seam-cross-platform, windows-process-regression)", "insight": "**A CI step that captures output to echo it later loses exactly the runs you need it for** — it prints on success, where nobody reads it, and prints nothing on failure. `set -e` is what makes it silent, so it looks fine in local testing without `-e`. Grep workflows for `=$(` around a build/test command before trusting a bare exit code. The measurement that settles it costs a minute and needs no CI: extract the step's `run:` block straight out of the YAML (`yaml.safe_load`), put a stub `go` on PATH that exits 1 with realistic output, and run the block under `bash --noprofile --norc -e -o pipefail` — the old body prints zero lines. Exercise the vacuous-`-run` case too, or the fix quietly disables the guard the capture was there for.", "refs": ["ako/mxcli#594"]} +{"area": "examples/doctype-tests", "date": "2026-09-22", "symptom": "The nightly fails on Mendix 10.24 only, in `TestMxCheck_DoctypeScripts/14-project-settings-examples.mdl`, with `Execution error: alter settings workflows add group '' requires Mendix 11.2.0+ (project is 10.24.24.119349)`. Push CI (single, newer version) and `TestDoctypeScriptsParseAfterVersionFiltering` both pass", "cause": "The workflow-groups feature (832e9c80) added Examples 4.3-4.7 to the doctype script ungated, although its executor correctly refuses the statement below 11.2 (`Settings$WorkflowGroup` is 11.2 metamodel). Third time this class has landed (Atlas building block in 15c, `DecimalScale`, now workflow groups)", "file": "`mdl-examples/doctype-tests/14-project-settings-examples.mdl`", "insight": "Wrap the examples in `-- @version: 11.2+` ... `-- @version: any`, with the directive ABOVE the first `/** */` comment. The parse-only guard cannot catch this: it proves the filtered script parses, not that the executor accepts every statement on that version, so a version-refused statement only surfaces in the nightly's 10.24 job. When a feature adds a version check to the executor, gate its doctype example in the same change. Reproduce locally in about 30s: `mxcli setup mxbuild --version 10.24.24.119349`, then `go test -tags integration ./mdl/executor/ -run 'TestMxCheck_DoctypeScripts/