Skip to content

Add note on Polish language token overhead and AI misinformation #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mkaput merged 5 commits into from
Apr 2, 2026
Merged

Add note on Polish language token overhead and AI misinformation #10

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
658b44d
add note on Polish language token overhead and AI misinformation
GrzywN Mar 31, 2026
ce4fb7b
Apply suggestion from @mkaput
GrzywN Apr 2, 2026
93c9b63
Apply suggestion from @mkaput
GrzywN Apr 2, 2026
28f678c
Apply suggestion from @mkaput
GrzywN Apr 2, 2026
8febcdd
fix: move `Common subscription options` to the `Subscriptions` section
GrzywN Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions src/content/docs/expanding-horizons/model-pricing.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,24 @@ Common subscription options:
- <ExternalLink href="https://cursor.com/pricing" />
- <ExternalLink href="https://factory.ai/pricing" />
- <ExternalLink href="https://opencode.ai/docs/go/" />

## Language choice and token cost

Token counts vary significantly by language.
Polish, for example, requires roughly 40% more tokens than English for equivalent content — you can verify this yourself with <ExternalLink href="https://tiktokenizer.vercel.app/"/>.

| Text | Tokens |
|------|--------|
| *"Jestem Adam i sprawdzam, ile tokenów potrzeba do zapisania tego tekstu. Czy język polski faktycznie wymaga ich więcej niż angielski?"* | 38 |
| *"My name is Adam, and I'm checking how many tokens are needed to write this text. Does Polish actually require more tokens than English?"* | 27 |

This is worth keeping in mind if you work in a non-English language — more tokens means higher cost and slower inference at scale.

It's also a good occasion to debunk a myth that spread in Polish media in early 2025.
Several articles claimed that Polish is the "best language for AI", citing a real paper: <ExternalLink href="https://arxiv.org/pdf/2503.01996"/>.
Marzena Karpińska from Microsoft, a co-author of the paper, addressed this directly:

> No. We didn't study that at all.
> We created a tool for diagnosing language models, checking how well they are able to extract information from very long texts.

A neat example of a fake news cycle born from real research — likely fueled by misunderstood patriotism.
2 changes: 2 additions & 0 deletions src/data/links.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ https://ampcode.com/threads/T-019cafee-1be2-72ab-bcec-28c6d41b753b,Reproduce fuz
https://ampcode.com/threads/T-d0d0c3c4-4994-4574-9e56-e7f97e88bc33,Implement file mentions in command palette,Nicolay Gerold (via Amp),2025-10-31,2026-03-04
https://ampcode.com/threads/T-f02e59f8-e474-493d-9558-11fddf823672,Tmux control mode protocol documentation,Mitchell Hashimoto (via Amp),2025-12-01,2026-03-04
https://app.devin.ai/review,Devin Review,,,2026-03-05
https://arxiv.org/pdf/2503.01996,One ruler to measure them all: Benchmarking multilingual long-context language models,Alveera Ahsan et al.,2025-03-03,2026-03-31
https://bits.logic.inc/p/ai-is-forcing-us-to-write-good-code,AI Is Forcing Us To Write Good Code,Steve Krenzel,2025-12-29,2026-03-04
https://claude.com/blog/category/claude-code,Claude Code Blog,,,2026-03-04
https://claude.com/chrome,Claude in Chrome,,,2026-03-04
Expand Down Expand Up @@ -98,6 +99,7 @@ https://support.apple.com/guide/mac-help/mh40584/mac,Dictate messages and docume
https://support.microsoft.com/en-us/windows/use-voice-typing-to-talk-instead-of-type-on-your-pc-fec94565-c4bd-329d-e59a-af033fa5689f,Use voice typing to talk instead of type on your PC - Microsoft Support,,,2026-03-10
https://swmansion.com/,Software Mansion,,,2026-03-04
https://tidewave.ai/,Tidewave,,,2026-03-04
https://tiktokenizer.vercel.app/,Tiktokenizer,,,2026-03-31
https://warden.sentry.dev/,Warden,,,2026-03-05
https://www.catonetworks.com/blog/cato-ctrl-weaponizing-claude-skills-with-medusalocker/,Weaponizing Claude Skills with MedusaLocker,Inga Cherny,2025-12-02,2026-03-18
https://www.figma.com/blog/introducing-claude-code-to-figma/,From Claude Code to Figma: Turning Production Code into Editable Figma Designs,Figma,2026-02-17,2026-03-04
Expand Down