Skip to content

Parse EXTENDS/RESTRICTS with shape references; fix chardet decode crash - #4

Merged
melonora merged 2 commits into
linkml:mainfrom
ericprud:extends-shaperef
Sep 25, 2026
Merged

Parse EXTENDS/RESTRICTS with shape references; fix chardet decode crash#4
melonora merged 2 commits into
linkml:mainfrom
ericprud:extends-shaperef

Conversation

@ericprud

@ericprud ericprud commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What

  • Grammar: extension and restrictions take shapeRef (EXTENDS @<label>), matching usage everywhere (shexTest, shex.js, Apache Jena, rudof). The bare shapeExprLabel form was never correct and never deployed, so it is not accepted. The parser is regenerated with ANTLR 4.9.3, matching the antlr4-python3-runtime~=4.9.3 pin. The companion ShExDoc.g4 change is merged at extension/restrictions accept shape references (EXTENDS @<label>) shexSpec/grammar#17.
  • Serializer: ShExC emits the @-form for EXTENDS/RESTRICTS so its output re-parses under the strict grammar.
  • Visitors: visitQualifier (extension) and shapeExprDecl restrictions route shapeRef through the existing shapeRef_to_iriref.
  • Bug fix: generate_shexj.load_shex_file crashed with decode() argument 'encoding' must be str, not None when chardet.detect returned {'encoding': None} — this aborted the shexTest ShExC manifest on its first schema.
  • A follow-up commit resolves relative IRIs per RFC 3986 (urljoin) instead of string concatenation.

Verification

Repo test suite: 16 passed, 1 skipped. Via linkml/PyShEx against the shexSpec/shexTest validation manifest from ShExC: 1184 tests = 1180 passed / 0 failed / 4 skips (schemas pyshexc cannot parse).

🤖 Generated with Claude Code

This was referenced Jul 12, 2026
Merged
Merged
ericprud pushed a commit to ericprud/PyShEx that referenced this pull request Jul 28, 2026
Eric Prud'hommeaux @claude
EXTENDS validation with feasibility-pruned partition search
9b089d0 
Implements ShEx 2.1 EXTENDS/ABSTRACT/RESTRICTS evaluation:

- satisfiesExtendedShape: neighbourhood triples are allocated to the triple
  constraints of the extension targets (through references, AND/OR branches and
  extends chains); a constraint reached through several chains (diamond
  inheritance) allocates its triples to every one of them, so a triple cannot
  evade one chain's constraints by hiding in another. Extension targets are
  evaluated through ordinary satisfies() against their allocated part, so
  ShapeOr / ShapeAnd / ShapeNot parents work uniformly.
- Node-scoped restriction stack: extension evaluation sees only its allocated
  part while value-expression recursion on other nodes sees the whole graph;
  the (node, shape) evaluation cache is bypassed under a restriction (the same
  pair can genuinely differ between parts).
- satisfiesShapeDecl: restricts are conjuncts; an abstract declaration is
  satisfied only through a non-abstract descendant (except as an EXTENDS
  target); labeled shapes with registered extensions get the same descendant
  fallback. Handles both ShExJ 2.1 ShapeDecl and legacy inline-id flavours.
- pyshex/utils/feasibility.py: a sound refutation test over partial bags of
  TripleConstraint matches, used to prune hopeless neighbourhoods before
  partition enumeration and to pre-filter EXTRA permutations. Ported from the
  same analysis implemented for Apache Jena, rudof and shex.js.

shexSpec/shexTest validation manifest:
  ShExJ:  1182 tests = 1093 passed, 0 failed, 89 trait-skips
  ShExC:  1184 tests = 1091 passed, 0 failed, 93 trait-skips
  (baseline could not run: NotImplementedError on the first ShapeDecl)

pyshexc is temporarily sourced from the EXTENDS-grammar PR branch
(linkml/grammar-python-antlr-linkml#4); repoint to PyPI when released.
Requires tests/data/shexTest (git submodule) for the manifest tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eric Prud'hommeaux and others added 2 commits July 28, 2026 02:52
Eric Prud'hommeaux @claude
Parse EXTENDS/RESTRICTS with shape references, fix chardet decode crash
e00777a 
- ShExDoc grammar: extension and restrictions take shapeRef ('EXTENDS @<label>'),
  matching usage everywhere (shexTest, shex.js, Apache Jena, rudof).  The bare
  shapeExprLabel form was never correct and never deployed, so it is not accepted.
  Parser regenerated with ANTLR 4.9.3 to match the antlr4-python3-runtime~=4.9.3 pin.
  (Companion grammar change PRed at shexSpec/grammar#17.)
- visitQualifier / shapeExprDecl restrictions route shapeRef through shapeRef_to_iriref.
- ShExC serializer emits the @-form for EXTENDS/RESTRICTS so output re-parses.
- generate_shexj.load_shex_file: chardet.detect can return {'encoding': None}; never
  pass None to bytes.decode (crashed on the shexTest ShExC manifest).

Verified against shexSpec/shexTest validation manifest via linkml/PyShEx:
1184 tests = 1180 passed, 0 failed, 4 skips (schemas pyshexc cannot parse).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eric Prud'hommeaux @claude
Resolve relative IRIs per RFC 3986 instead of concatenation
7e3a6b5 
<S1> against base .../validation/1dot-relative.shex resolved to
.../1dot-relative.shexS1; urljoin gives .../validation/S1
(shexTest 1dot-relative_pass-*).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericprud pushed a commit to ericprud/PyShEx that referenced this pull request Jul 28, 2026
Eric Prud'hommeaux @claude
pyshexc: shapeRef-only EXTENDS/RESTRICTS grammar (bump pin to 7e3a6b5)
8755309 
shexSpec/grammar#17 and linkml/grammar-python-antlr-linkml#4 dropped the
never-deployed bare shapeExprLabel form; EXTENDS/RESTRICTS now require a
shape reference (EXTENDS @<label>).  All shexTest EXTENDS schemas already
use the @-form, so results are unchanged: 129 passed, 21 skipped, 1 xfailed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericprud pushed a commit to ericprud/PyShEx that referenced this pull request Jul 28, 2026
Eric Prud'hommeaux @claude
EXTENDS validation with feasibility-pruned partition search
d6b51f5 
Implements ShEx 2.1 EXTENDS/ABSTRACT/RESTRICTS evaluation:

- satisfiesExtendedShape: neighbourhood triples are allocated to the triple
  constraints of the extension targets (through references, AND/OR branches and
  extends chains); a constraint reached through several chains (diamond
  inheritance) allocates its triples to every one of them, so a triple cannot
  evade one chain's constraints by hiding in another. Extension targets are
  evaluated through ordinary satisfies() against their allocated part, so
  ShapeOr / ShapeAnd / ShapeNot parents work uniformly.
- Node-scoped restriction stack: extension evaluation sees only its allocated
  part while value-expression recursion on other nodes sees the whole graph;
  the (node, shape) evaluation cache is bypassed under a restriction (the same
  pair can genuinely differ between parts).
- satisfiesShapeDecl: restricts are conjuncts; an abstract declaration is
  satisfied only through a non-abstract descendant (except as an EXTENDS
  target); labeled shapes with registered extensions get the same descendant
  fallback. Handles both ShExJ 2.1 ShapeDecl and legacy inline-id flavours.
- pyshex/utils/feasibility.py: a sound refutation test over partial bags of
  TripleConstraint matches, used to prune hopeless neighbourhoods before
  partition enumeration and to pre-filter EXTRA permutations. Ported from the
  same analysis implemented for Apache Jena, rudof and shex.js.

shexSpec/shexTest validation manifest:
  ShExJ:  1182 tests = 1093 passed, 0 failed, 89 trait-skips
  ShExC:  1184 tests = 1091 passed, 0 failed, 93 trait-skips
  (baseline could not run: NotImplementedError on the first ShapeDecl)

pyshexc is temporarily sourced from the EXTENDS-grammar PR branch
(linkml/grammar-python-antlr-linkml#4); repoint to PyPI when released.
Requires tests/data/shexTest (git submodule) for the manifest tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ericprud pushed a commit to ericprud/PyShEx that referenced this pull request Sep 24, 2026
Eric Prud'hommeaux @claude Eric Prud'hommeaux
EXTENDS validation with feasibility-pruned partition search
f7fbe4b 
Implements ShEx 2.1 EXTENDS/ABSTRACT/RESTRICTS evaluation:

- satisfiesExtendedShape: neighbourhood triples are allocated to the triple
  constraints of the extension targets (through references, AND/OR branches and
  extends chains); a constraint reached through several chains (diamond
  inheritance) allocates its triples to every one of them, so a triple cannot
  evade one chain's constraints by hiding in another. Extension targets are
  evaluated through ordinary satisfies() against their allocated part, so
  ShapeOr / ShapeAnd / ShapeNot parents work uniformly.
- Node-scoped restriction stack: extension evaluation sees only its allocated
  part while value-expression recursion on other nodes sees the whole graph;
  the (node, shape) evaluation cache is bypassed under a restriction (the same
  pair can genuinely differ between parts).
- satisfiesShapeDecl: restricts are conjuncts; an abstract declaration is
  satisfied only through a non-abstract descendant (except as an EXTENDS
  target); labeled shapes with registered extensions get the same descendant
  fallback. Handles both ShExJ 2.1 ShapeDecl and legacy inline-id flavours.
- pyshex/utils/feasibility.py: a sound refutation test over partial bags of
  TripleConstraint matches, used to prune hopeless neighbourhoods before
  partition enumeration and to pre-filter EXTRA permutations. Ported from the
  same analysis implemented for Apache Jena, rudof and shex.js.

shexSpec/shexTest validation manifest:
  ShExJ:  1182 tests = 1093 passed, 0 failed, 89 trait-skips
  ShExC:  1184 tests = 1091 passed, 0 failed, 93 trait-skips
  (baseline could not run: NotImplementedError on the first ShapeDecl)

pyshexc is temporarily sourced from the EXTENDS-grammar PR branch
(linkml/grammar-python-antlr-linkml#4); repoint to PyPI when released.
Requires tests/data/shexTest (git submodule) for the manifest tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ericprud

ericprud commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

@melonora could you review this when you have a moment? PyShEx master (linkml/PyShEx#105) now depends on it through a git source, and its release workflow refuses to publish to PyPI until a pyshexc release with these changes is available. CI here is green on Ubuntu and Windows, Python 3.10–3.14.

@melonora

melonora commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

Will review today

melonora
melonora approved these changes Sep 25, 2026
View reviewed changes

@melonora melonora left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not see anything wrong with this, also tested on mac.

@melonora
melonora merged commit 815380f into linkml:main Sep 25, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@melonora melonora melonora approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ericprud @melonora