Conversation
A pure JVM SentencePiece tokenizer implementing the SubwordTokenizer interface from opennlp-api. Unigram and BPE models, the SentencePiece normalizer, byte fallback, and a double-array trie for piece lookup. The module follows the repository conventions: group org.apache.opennlp.addons, an artifact id without a suffix, and a description. RAT skips the generated test fixtures, which are TSV and plain text formats with no comment syntax. This needs SubwordTokenizer, which landed in core after 3.0.0-M5, so the module resolves the OpenNLP snapshot from the ASF repository and remains a draft until a release with that interface is published. 66 tests, one skipped: the real-model evaluation needs a model that is not part of the repository.
krickert
marked this pull request as ready for review
September 10, 2026 00:28
Contributor
Author
|
The add-ons build is a 3.0.0-SNAPSHOT, so a snapshot dependency on core is consistent. Snapshot-to-snapshot resolution is ordinary Maven practice; the restriction applies at release time, where maven-release-plugin rejects non-released dependencies. The core snapshot resolves from repository.apache.org through the Only |
krickert
marked this pull request as draft
September 10, 2026 00:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #178, which GitHub closed when the base branch was removed on the #183 merge. This one is based on
mainand headed fromapache/opennlp-addons, so committers can push to it.The first add-on under the layout #183 established.
What it is
A pure JVM SentencePiece tokenizer implementing
SubwordTokenizerfromopennlp-api. Unigram and BPE models, the SentencePiece normalizer, byte fallback, and a double-array trie for piece lookup. No native library and no JNI.Conventions from #183
Group
org.apache.opennlp.addons, artifact idsubwordwith no suffix, and a description. RAT skips the generated test fixtures, which are TSV and plain text with no comment syntax; the.modelbinaries need no entry, because RAT identifies them as binary.Why this is a draft
It needs
SubwordTokenizer, which reached core after 3.0.0-M5. The module therefore resolves3.0.0-SNAPSHOTfrom the ASF snapshot repository, already configured in the root POM. Per the manual's "Contributing an Add-on" chapter, a contribution that needs an unreleased core API remains a draft until it builds with a published artifact. Ready for review once a release with that interface exists.No extra setup is required to build it now, since the snapshot resolves from
repository.apache.org.Verification
Every module passes. The subword module runs 66 tests with 0 failures and 1 skipped, that being the real-model evaluation, which needs a model outside the repository. RAT clean, and the dependency license report is generated.