Skip to content

Comments

feat: Initial tokenizer.json support#41

Draft
HavenDV wants to merge 1 commit intomainfrom
tokenizer
Draft

feat: Initial tokenizer.json support#41
HavenDV wants to merge 1 commit intomainfrom
tokenizer

Conversation

@HavenDV
Copy link
Contributor

@HavenDV HavenDV commented Jul 10, 2024

Closes #40

@HavenDV HavenDV changed the title feat: Initial work. feat: Initial tokenizer.json support Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate/load Encoder from tokenizer.json file

1 participant