rtf: a new format, with characters, text_encoding, elements and nesting

.github/workflows/ci.yml

@@ -1362,7 +1362,18 @@ jobs:
           set -euo pipefail
           sudo apt-get update
           sudo apt-get install -y --no-install-recommends \
-            p7zip-full ffmpeg poppler-utils inkscape python3-pil python3-yaml gdal-bin
+            p7zip-full ffmpeg poppler-utils inkscape python3-pil python3-yaml gdal-bin pandoc
+
+          # RTF is read by pandoc, the one of five readers measured that is
+          # an apt install here (docs/RTF-2026-10-15.md section 6.1). Its
+          # readings of good files - wrong letters in codepage, a dropped
+          # character after every letter written literal, hidden text left
+          # out - are held by the RTF guard as what the settings tell a user,
+          # and 3.1.3 and 3.7.0.2 read every file of that measurement alike.
+          # Asked for its version, so a pandoc that did not install stops this
+          # step rather than turning the RTF guard into a skip. Without a pipe:
+          # under pipefail a reader that stops early can fail a good install.
+          pandoc --version
 
           # GeoJSON is read by GDAL and its shapes judged by shapely, which is
           # GEOS underneath (docs/GEOJSON-2026-10-07.md). shapely goes in for

CHANGELOG.md

@@ -42,6 +42,27 @@ because it turns other people's test suites red.
 
 ### Added
 
+- **Documents: the `rtf` format.** A Rich Text Format document at the exact
+  size you ask for, read by LibreOffice, pandoc, the RichEdit control of
+  Windows, striprtf and Apache Tika. It holds headings, paragraphs with bold
+  and italic words, lists, tables and indented quotes, with the label as its
+  first paragraph and every line broken after a space - RTF ignores a line
+  end, so one between two letters would join two words in every reader.
+  `characters=international` adds paragraphs in Polish, Russian, Japanese and
+  Arabic, and one with emoji and the characters RTF escapes or names.
+  `text_encoding` writes their letters four ways, and readers do not agree on
+  two of them: pandoc shows `codepage` text in the wrong letters and drops
+  the character after every letter written `literal`. `elements=every` adds
+  every place a document keeps text a reader may show or leave out -
+  document properties, a page header and footer, a footnote, a comment,
+  hidden text, a link, a bookmark and groups a reader has to skip or does not
+  know - and each part says where it is. LibreOffice and striprtf returned the
+  same parts, and every other reader a set of its own, with two versions of
+  pandoc apart as well. `nesting` puts the content up to 10 000 000 groups deep. The
+  smallest document is 65 B, and the manifest records every setting and the
+  code page the header declares. A large document is a long one to read:
+  at 100 MiB pandoc took 95 s and 9.6 GB of memory, and LibreOffice had not
+  finished after 20 minutes.
 - **Films: the `webm` format.** A WebM video with an AV1 picture and no
   sound, at the exact size you ask for, like every other format. Its length
   is a setting of its own, independent of the size. The picture moves: every
@@ -338,6 +359,15 @@ because it turns other people's test suites red.
 
 ### Changed
 
+- **`empty-and-minimal` now includes `rtf`.** The preset builds the smallest
+  file of every format this build has, so with `formats` left at `all` it
+  gains an `rtf` target of 65 B, and the recipe `tfg preset eject
+  empty-and-minimal` prints is one target longer. A manifest from a run of
+  that preset records the hash of that recipe, so its `recipe_hash` differs
+  from one written before. Every file the preset wrote before is written as
+  it was, the `rtf` file is the only new one, and a run that names its
+  `formats` is untouched.
+
 - **The `eml` settings say what Thunderbird and MimeKit do with them.**
   Measured with Thunderbird 157.0.1 and MimeKit 4.18.1. `charset` says that
   MimeKit with no code page provider tries UTF-8 and then Latin-1, and reads

README.md

@@ -23,7 +23,7 @@
 
 **Testing Files Generator** is a tool for QA engineers and developers who need real
 files to test against - an upload form, a parser, anything that takes a file and
-has an opinion about it. You pick one of its 30 formats and the size you want,
+has an opinion about it. You pick one of its 31 formats and the size you want,
 and you get **exactly that**: ask for a 10 MB PDF and you get a PDF that a reader
 will open, at 10 MB to the byte. Every run also leaves a manifest saying **what
 your system should do with each file**, which is the part other generators leave
@@ -36,7 +36,7 @@ needs it finds out it exists.
 
 - **Hit an exact size, to the byte** - ask for 10485761 bytes and get exactly
   that, never a silently rounded file.
-- **Write 30 real formats** - a generated PNG opens in an image viewer, a DOCX
+- **Write 31 real formats** - a generated PNG opens in an image viewer, a DOCX
   opens in Word, a ZIP extracts. Not padded zeros with an extension.
 - **Say what should happen to each file** - the manifest carries an expected
   outcome, so your test reads the assertion instead of you writing it out.
@@ -174,7 +174,7 @@ owns it, at the exact size you asked for:
 
 | group | formats |
 |---|---|
-| 📄 **Documents** | `pdf`, `docx` (Word), `xlsx` (Excel), `pptx` (PowerPoint) |
+| 📄 **Documents** | `pdf`, `docx` (Word), `xlsx` (Excel), `pptx` (PowerPoint), `rtf` (Rich Text Format) |
 | 🖼️ **Images** | `png`, `jpg`, `bmp`, `gif`, `ico`, `svg`, `tiff`, `webp`, `avif`, `jxl` |
 | 📝 **Text and markup** | `txt`, `md`, `csv`, `json`, `xml`, `html`, `log` |
 | ⚙️ **Configuration** | `yaml`, `toml` |
@@ -627,6 +627,7 @@ recipe. `tfg formats <id>` prints the allowed range or list for each:
 | `docx` | `paragraphs` |
 | `xlsx` | `rows`, `columns` |
 | `pptx` | `slides` |
+| `rtf` | `characters`, `text_encoding`, `elements`, `nesting` |
 | `csv` | `delimiter`, `line_ending`, `header`, `quote_style`, `columns` |
 | `log` | `entry_format`, `timestamps`, `rate`, `methods`, `status_mix`, `level_mix`, `ip_version`, `line_ending` |
 | `txt`, `md`, `xml` | `encoding`, `bom` |

internal/format/all/all.go

@@ -26,6 +26,7 @@ import (
 	_ "github.com/donislawdev/TestingFilesGenerator/internal/format/pdf"
 	_ "github.com/donislawdev/TestingFilesGenerator/internal/format/png"
 	_ "github.com/donislawdev/TestingFilesGenerator/internal/format/pptx"
+	_ "github.com/donislawdev/TestingFilesGenerator/internal/format/rtf"
 	_ "github.com/donislawdev/TestingFilesGenerator/internal/format/svgfile"
 	_ "github.com/donislawdev/TestingFilesGenerator/internal/format/targz"
 	_ "github.com/donislawdev/TestingFilesGenerator/internal/format/tiff"

internal/format/rtf/blocks.go

@@ -0,0 +1,196 @@
+package rtf
+
+import (
+	// D11 promises the same bytes from the same seed, so a deliberate,
+	// reproducible generator is the product rather than a weakness. Nothing
+	// here ever makes a secret.
+	// nosemgrep: go.lang.security.audit.crypto.math_random.math-random-used
+	"math/rand/v2"
+	"strconv"
+
+	"github.com/donislawdev/TestingFilesGenerator/internal/core"
+)
+
+// The blocks a document is filled with, and the words of the paragraph that
+// closes it to the byte.
+//
+// Every block leaves the document the way it found it: its paragraph closed,
+// the paragraph properties reset with \pard, and every change to the letters
+// inside a group. So whatever follows starts from the defaults, and the closing
+// paragraph can be words and nothing else - a paragraph without \par at the end
+// of the document is shown by all five readers measured (docs/RTF-2026-10-15.md
+// section 3), which is what lets the closing record be as short as nothing.
+//
+// A line ends only after a space or after a control word, never between two
+// letters. RTF ignores a line end, so one between two letters joins two words
+// in every reader - measured in the same section.
+
+const (
+	// lineWidth is where a line of words is broken, well inside the 255
+	// characters the specification suggests.
+	lineWidth = 72
+	// endBlock closes a paragraph and resets its properties.
+	endBlock = `\par\pard` + "\r\n"
+	// lineBreak is what goes between two words where a line ends.
+	lineBreak = " \r\n"
+)
+
+// blocks is the record the document is filled with.
+type blocks struct{}
+
+// Append adds one whole block.
+func (blocks) Append(dst []byte, rng *rand.Rand) []byte {
+	switch rng.IntN(6) {
+	case 0:
+		// A heading by its outline level, which pandoc and LibreOffice both
+		// read as a heading without a style sheet (measured 2026-10-15), so the
+		// header stays as small as it can be.
+		dst = append(dst, `\outlinelevel1{\b\fs28 `...)
+		dst = appendTitle(dst, rng, 3)
+		return append(dst, "}"+endBlock...)
+	case 1:
+		return appendList(dst, rng)
+	case 2:
+		return appendTable(dst, rng)
+	case 3:
+		dst = append(dst, `\li720 `...)
+		dst = appendWords(dst, rng, 8, false)
+		return append(dst, endBlock...)
+	case 4:
+		dst = appendWords(dst, rng, 20+rng.IntN(20), true)
+		return append(dst, endBlock...)
+	default:
+		dst = appendWords(dst, rng, 20+rng.IntN(20), false)
+		return append(dst, endBlock...)
+	}
+}
+
+// AppendExact adds the words of the closing paragraph, exactly n bytes of them.
+func (blocks) AppendExact(dst []byte, _ *rand.Rand, n int64) []byte { return appendProse(dst, n) }
+
+// Shortest is nothing at all: the closing paragraph can be empty.
+func (blocks) Shortest() int64 { return 0 }
+
+// Discard has nothing to undo, since a block counts nothing.
+func (blocks) Discard() {}
+
+// appendList is three to five items, each with the bullet Word writes in front
+// of it for a reader that does not know lists.
+func appendList(dst []byte, rng *rand.Rand) []byte {
+	for i, n := 0, 3+rng.IntN(3); i < n; i++ {
+		dst = append(dst, `{\pntext\bullet\tab}{\*\pn\pnlvlblt\pnindent360{\pntxtb\bullet}}\fi-360\li720 `...)
+		dst = appendWords(dst, rng, 4+rng.IntN(4), false)
+		dst = append(dst, `\par`+"\r\n"...)
+	}
+	return append(dst, `\pard`+"\r\n"...)
+}
+
+// appendTable is a row of headings and two to four rows under it.
+func appendTable(dst []byte, rng *rand.Rand) []byte {
+	dst = appendRow(dst, "id", "field", "value")
+	for i, n := 0, 2+rng.IntN(3); i < n; i++ {
+		dst = appendRow(dst, strconv.Itoa(i+1), word(rng), word(rng))
+	}
+	return append(dst, `\pard`+"\r\n"...)
+}
+
+func appendRow(dst []byte, cells ...string) []byte {
+	dst = append(dst, `\trowd\cellx3000\cellx6000\cellx9000\pard\intbl `...)
+	for _, c := range cells {
+		dst = append(dst, c...)
+		dst = append(dst, `\cell `...)
+	}
+	return append(dst, `\row`+"\r\n"...)
+}
+
+// appendWords is n words, broken into lines at lineWidth, and with styled some
+// of them bold or italic, each in a group of its own.
+func appendWords(dst []byte, rng *rand.Rand, n int, styled bool) []byte {
+	col := 0
+	for i := 0; i < n; i++ {
+		w := word(rng)
+		if i > 0 {
+			dst, col = appendSpace(dst, col, len(w))
+		}
+		open, closing := styleOf(rng, styled)
+		dst = append(dst, open...)
+		dst = append(dst, w...)
+		dst = append(dst, closing...)
+		col += len(w)
+	}
+	return dst
+}
+
+// styleOf is the group around a styled word, one word in five: bold or italic.
+func styleOf(rng *rand.Rand, styled bool) (open, closing string) {
+	if !styled {
+		return "", ""
+	}
+	switch rng.IntN(10) {
+	case 0:
+		return `{\b `, "}"
+	case 1:
+		return `{\i `, "}"
+	}
+	return "", ""
+}
+
+// appendSpace is what goes in front of a word of length next at column col:
+// a space, or a space and a line end once the line is full.
+func appendSpace(dst []byte, col, next int) ([]byte, int) {
+	if col+1+next > lineWidth {
+		return append(dst, lineBreak...), 0
+	}
+	return append(dst, ' '), col + 1
+}
+
+// appendTitle is a phrase with each word capitalised, for a heading.
+func appendTitle(dst []byte, rng *rand.Rand, n int) []byte {
+	for i := 0; i < n; i++ {
+		if i > 0 {
+			dst = append(dst, ' ')
+		}
+		w := word(rng)
+		// The vocabulary is lower case ASCII, so the first byte is the first
+		// letter.
+		dst = append(dst, w[0]-'a'+'A')
+		dst = append(dst, w[1:]...)
+	}
+	return dst
+}
+
+// appendProse is n bytes of words, the lines broken the way appendWords breaks
+// them. The cut can land on a space or inside a line end, and both are fine in
+// front of the closing braces: RTF ignores the line end, and the space is the
+// end of the last word.
+func appendProse(dst []byte, n int64) []byte {
+	col := len(words[0])
+	return core.AppendFiller(dst, words, n, func(i int) string {
+		next := len(words[i%len(words)])
+		var sep []byte
+		sep, col = appendSpace(nil, col, next)
+		col += next
+		return string(sep)
+	})
+}
+
+func word(rng *rand.Rand) string { return words[rng.IntN(len(words))] }
+
+// words is the vocabulary: lower case ASCII, so no word holds a character RTF
+// escapes and no word starts a control word.
+var words = []string{
+	"account", "after", "amount", "answer", "before", "between", "branch",
+	"buffer", "build", "cache", "change", "client", "column", "config",
+	"content", "create", "default", "delete", "detail", "device", "domain",
+	"effect", "engine", "entry", "error", "export", "field", "filter",
+	"folder", "format", "handle", "header", "import", "index", "input",
+	"insert", "invoice", "layer", "length", "level", "limit", "loader",
+	"market", "member", "method", "module", "notice", "number", "object",
+	"option", "output", "packet", "parent", "parser", "period", "policy",
+	"prefix", "process", "public", "query", "reason", "record", "region",
+	"report", "result", "return", "sample", "schema", "script", "search",
+	"sender", "server", "signal", "single", "source", "status", "stream",
+	"string", "switch", "system", "target", "thread", "ticket", "toggle",
+	"token", "update", "upload", "vendor", "verify", "version", "volume",
+	"window", "worker", "writer",
+}

internal/format/rtf/catalogue.go

@@ -0,0 +1,34 @@
+package rtf
+
+// The parts elements=every adds - every place a document keeps text that a
+// reader may show, leave out or show where it should not. Each sentence says
+// where it is, so a test reading what an extractor returned knows which parts
+// it took without knowing RTF. Measured on 2026-10-15: six readers, five
+// different sets (docs/RTF-2026-10-15.md section 6.1 d).
+//
+// Two pieces, because RTF fixes where the first one goes: the document
+// properties come before any text, and a page header and footer open the
+// section they belong to. The rest are paragraphs after the label, each ending
+// the way every block does - its paragraph closed and the properties reset - so
+// whatever follows starts from the defaults.
+
+// catalogueHead is what goes in front of the label: the properties of the
+// document, and the header and footer of its pages.
+const catalogueHead = `{\info{\title This is the title of the document}{\author tfg}{\keywords catalogue rtf}` +
+	`{\doccomm This is the comment of the document}}` + "\r\n" +
+	`{\header\pard This sentence is in the page header.\par}` + "\r\n" +
+	`{\footer\pard This sentence is in the page footer.\par}` + "\r\n"
+
+// catalogueBody is the paragraphs after the label. The link goes to a domain
+// kept for examples (RFC 2606), and a field is never fetched by opening the
+// document - a reader that follows it is doing so on a click.
+const catalogueBody = `A paragraph with a footnote{\super\chftn}{\footnote\pard\plain{\super\chftn} This sentence is in a footnote.}.\par\pard` + "\r\n" +
+	`A paragraph with a comment{\*\atnid tfg}{\*\atnauthor tfg}\chatn{\*\annotation\pard\plain This sentence is in a comment.}.\par\pard` + "\r\n" +
+	`Before hidden text {\v This sentence is hidden text.}and after it.\par\pard` + "\r\n" +
+	`A link {\field{\*\fldinst HYPERLINK "https://example.com/tfg/link"}{\fldrslt This sentence is the text of a link.}}\par\pard` + "\r\n" +
+	`A bookmark {\*\bkmkstart tfg_bookmark}around these words{\*\bkmkend tfg_bookmark}.\par\pard` + "\r\n" +
+	`A group {\*\tfgunknown This sentence is in a group every reader has to skip.}that ends here.\par\pard` + "\r\n" +
+	`A group {\tfgword This sentence follows a word no reader knows, which is why it shows.} that ends here.\par\pard` + "\r\n" +
+	`Before a line break\line after it, before a tab\tab after it.\par\pard` + "\r\n" +
+	`This paragraph ends with a page break.\par\page\pard` + "\r\n" +
+	`This paragraph ends a section.\par\sect\sectd\pard` + "\r\n" //nolint:misspell // \sect is the RTF control word of a section break, read by misspell as a backslash and "ect"