TinySegmenter.nvim is a plugin that provides a Lua port of TinySegmenter, an ultra-minimal Japanese tokenizer. Since the plugin depends only on the Lua standard library, it can be executed anywhere that supports Lua 5.1 or later.
-- tani/vim-jetpack
Jetpack "sirasagi62/tinysegmenter.nvim"local tinysegmenter = require("tinysegmenter")
-- return table like {"γγ","γ―","ζη« ","γ§γ","γ"}
local parsed_text = tinysegmenter.segment("γγγ―ζη« γ§γγ")
-- γγ|γ―|ζη« |γ§γ|γ
print(table.concat(parsed_text,"|"))
Returns a table of Japanese sentences split into words.
This program is provided under BSD-3-Clause.
Copyright information is here: ./lua/tinysegmenter.lua
This program is originally created by Taku Kudo in 2008, modified for ES module by Taisuke Fukuno in 2022.
Also, utf8.lua is provided under CC0 by NAKAI Tsuyoshi.