A lightning-fast, multi-language Concrete & Abstract Syntax Tree (CST/AST) generator and syntax highlighter. Architected with a high-performance C core engine powered by PCRE2 and JSON-C, textparser provides standalone native ports for Rust, Python, and Java.
It serves as a robust foundation for building linters, static analysis tools, compilers, and terminal utilities like the built-in ccat clone.
- 🌐 Massive Language Support: Rich declarative and regex-based grammars for modern and classic languages.
- ⚡ High Performance: Core tokenization, CST/AST construction, and Pratt expression parsing written in highly optimized C.
- ♻️ Incremental, CST/AST-Aware Re-parsing: Sub-millisecond re-parsing that only re-lexes and splices modified regions, preserving node identity, tree structure, and memoized state across keystrokes.
- 🧬 Multi-Language Ecosystem: Standalone native ports (Rust, Python, Java) with memory safety and full behavioral parity.
- 🎨 Built-in
ccatUtility: High-performance, syntax-highlighted terminal alternative to the standardcatcommand.
textparser provides two command-line utilities:
The primary CLI tool for parsing source files, inspecting token streams, and generating syntax trees (CST/AST).
- View colored syntax tree:
textparser main.c
- Generate structured JSON representation:
textparser main.c --json
- Export token stream with line/column positions:
textparser main.c --tokens
- Test against a custom language definition:
textparser main.c --definition custom_def.json
A lightning-fast, colorized replacement for the standard cat command that renders syntax-highlighted source code directly in the terminal.
- View syntax-highlighted source code:
ccat main.rs
- System & General: C, C++, C#, Java, Go, Rust, Swift, Zig, C3, Jai, Ada, Assembly (x86/ARM), Pascal, Visual Basic, Fortran
- Web & Data: HTML, CSS, JavaScript, TypeScript, CFML (ColdFusion), JSON, SQL, Markdown
- Scripting & Analytics: Python, PHP, Bash/Shell, Perl, MATLAB, R, Scratch
yay -S textparser
# Or build from the latest git commit
yay -S textparser-gitsudo add-apt-repository ppa:bbarbulovski-gmail/textparser
sudo apt-get update
sudo apt-get install textparserPull and run the pre-configured container instantly:
docker pull bokic78/textparser
docker run --rm -v \$(pwd):/workspace bokic78/textparser --file /workspace/main.cEnsure you have cmake, ninja (optional but recommended), libpcre2, and libjson-c installed on your system.
git clone https://github.com/bokic/textparser.git
cd textparser
./build.shtextparser is cross-platform and fully supports both little-endian (e.g. x86_64, aarch64) and big-endian architectures (e.g. IBM s390x).
The Windows scripts use LLVM/Clang and Ninja. Build dependencies and the
project for one architecture, or omit the architecture to build both x64
and arm64:
cd windows
build_deps.bat [x64|arm64]
build.bat [x64|arm64]
build_zip.bat [x64|arm64]Architecture-specific binaries and ZIP archives are written separately, for
example to bin\arm64 and windows\textparser-<version>-arm64.zip.
For in-depth architectural details, parser specifications, grammar definitions, and porting contracts, see:
- ARCHITECTURE.md — Internal engine design, execution pipelines, schema-v2 grammar specification, and porting guidelines.
- ROADMAP.md — Current development roadmap and upcoming features.
Contributions are what make the open-source community an amazing place to learn, inspire, and create.
- Fork the project.
- Create your feature branch (
git checkout -b feature/AmazingFeature). - Commit your changes (
git commit -m 'Add some AmazingFeature'). - Push to the branch (
git push origin feature/AmazingFeature). - Open a Pull Request.
This project is licensed under the MIT License - see the LICENSE file for details.
Developed with ❤️ by Boris Barbulovski (bokic).