feat: add the searchlist and documents commands

skills/fessctl/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: fessctl
-description: Operate Fess via fessctl CLI (admin API client). Use when managing webconfigs, file/data configs, schedulers, users/roles/groups, dictionaries, access tokens, or any Fess admin resource from Claude. Covers 34 resource types and cross-feature workflows like initial crawl setup.
+description: Operate Fess via fessctl CLI (admin API client). Use when managing webconfigs, file/data configs, schedulers, users/roles/groups, dictionaries, access tokens, or any Fess admin resource from Claude. Covers 36 resource types and cross-feature workflows like initial crawl setup.
 license: Apache-2.0
 version: 0.1.0
 ---
@@ -65,6 +65,8 @@ Each file documents one Fess admin feature: what it is, when to use it, fessctl
 | Plugins                  | references/features/plugin.md |
 | Object storage           | references/features/storage.md |
 | Dictionaries             | references/features/dict.md |
+| Indexed documents        | references/features/searchlist.md |
+| Bulk document indexing   | references/features/documents.md |
 | Key match                | references/features/keymatch.md |
 | Boost document           | references/features/boostdoc.md |
 | Elevate word             | references/features/elevateword.md |

skills/fessctl/references/features/documents.md

@@ -0,0 +1,53 @@
+# Bulk document indexing (fessctl `documents`)
+
+## What it is
+Index many documents in one request, straight into the search index, without a crawl. This is the admin `documents/bulk` API. It only indexes; to change or delete documents afterwards use `fessctl searchlist`.
+
+Available in every Fess version fessctl supports (14.19.2, 15.6.1, 15.7.0, 15.8.0, 15.9.0); the request body is identical in all of them.
+
+## Subcommand surface
+| Subcommand | Purpose | Required arguments |
+| --- | --- | --- |
+| `bulk` | Index the documents in a file, or on stdin when the argument is `-`. | `<file>` |
+
+The input is read as one of:
+- a JSON array of document objects,
+- an object `{"documents": [...]}`,
+- NDJSON: one document object per line (blank lines are ignored).
+
+Always reconfirm with `fessctl documents bulk --help`.
+
+## Document fields
+`url` and `title` are required. When a document leaves them out, the server fills `role` (the guest roles), `boost` (1.0), `content_length`, `click_count`, `favorite_count`, `last_modified` and `timestamp`. Any other index field (`title`, `content`, `label`, `lang`, `mimetype`, ...) is passed through. Arrays (`role`, `label`, `lang`, `anchor`, `virtual_host`) must be JSON arrays.
+```json
+{"url": "https://www.example.com/a.html", "title": "A page", "content": "text", "label": ["docs"]}
+```
+
+## Response shape
+```json
+{
+  "response": {
+    "status": 0,
+    "items": [
+      {"result": "CREATED", "id": "0ed4f2...e6e1"},
+      {"result": "BAD_REQUEST", "message": "mapper_parsing_exception ..."}
+    ]
+  }
+}
+```
+`items` lists one entry per submitted document, in order. `id` is the index `_id`, not the `doc_id` that `fessctl searchlist` uses.
+
+## Gotchas
+- Validation is all-or-nothing: a document without `url` or `title`, or with a value that does not convert (`boost` or `click_count` that is not a number, for example), fails the whole request with status 1 and nothing is indexed.
+- Failures at indexing time (a value the index mapping cannot parse) fail only that document. The other documents are indexed, the response has `status` 9, `fessctl` prints every item and exits with code 9.
+- The document `_id` is derived from the URL and the role list, so indexing the same URL again with the same roles replaces the document, while the same URL with different roles is a second document.
+- `bulk` sends everything in a single request with a short client timeout; split very large files into several calls.
+- Documents become searchable about a second later (index refresh).
+- There is no delete in this API. Remove what you indexed with `fessctl searchlist delete-by-query`.
+
+## Examples
+```bash
+fessctl documents bulk docs.json
+fessctl documents bulk docs.ndjson --output json
+cat docs.ndjson | fessctl documents bulk -
+```

skills/fessctl/references/features/searchlist.md

@@ -0,0 +1,68 @@
+# Indexed documents (fessctl `searchlist`)
+
+## What it is
+The admin document browser: search the documents that are in the Fess index, read one in full, and add, change or delete documents without a crawl. The Admin UI offers the same list as its Search List page. Every document is addressed by its `doc_id`; the search index `_id` shown in raw responses is a different value and is not accepted by `get`, `update` or `delete`.
+
+Available in every Fess version fessctl supports (14.19.2, 15.6.1, 15.7.0, 15.8.0, 15.9.0).
+
+## When to use
+- Check what a crawl indexed: `search "title:foo"`, then `get <doc_id>`.
+- Remove a document that should not be searchable: `delete <doc_id>`, or `delete-by-query` for a set of them.
+- Add or correct a single document by hand: `create` / `update`. For many documents use `fessctl documents bulk`.
+
+## Subcommand surface
+| Subcommand | Purpose | Required arguments |
+| --- | --- | --- |
+| `search` | Search the index. `--start` is the 0-based offset, `--size` the page size, `--sort`, `--lang` (repeatable). | none (a blank query matches all documents) |
+| `get` | Show one document. | `<doc_id>` |
+| `create` | Index a new document. | `--url`, `--title`, `--role` (repeatable); `--boost` defaults to 1.0 |
+| `update` | Change fields of a document; every field not given is kept. | `<doc_id>` and at least one field option |
+| `delete` | Delete one document. | `<doc_id>` |
+| `delete-by-query` | Delete every document matching a query. Asks for confirmation unless `--yes` is given. | `<query>` |
+
+`create` and `update` take the same field options: `--url`, `--title`, `--content`, `--role`, `--boost`, `--label`, plus `--field KEY=VALUE` (repeatable) for any other field, for example `--field mimetype=text/plain`. `--role` and `--label` replace the whole list. Always reconfirm with `fessctl searchlist <sub> --help`.
+
+## Resource JSON shape
+`get` returns the stored document under `response.doc`; `search` returns hits under `response.docs` with the paging fields beside them.
+```json
+{
+  "response": {
+    "status": 0,
+    "doc": {
+      "doc_id": "104d050992cb4992810ef340fba1c566",
+      "url": "https://www.example.com/a.html",
+      "title": "A page",
+      "content": "...",
+      "role": ["Rguest"],
+      "label": ["docs"],
+      "boost": 1.0,
+      "_id": "125cc2...",
+      "_version": 2
+    }
+  }
+}
+```
+`search` reports `record_count` (total hits), `page_size`, `page_number`, `start_record_number` and `end_record_number`. `create` answers `{"id": "<doc_id>", "created": true}`; `delete-by-query` answers `{"count": N}`.
+
+## Gotchas
+- The server requires `url`, `title`, `role` and `boost` on every create and update request (`index.admin.required.fields`). `update` reads the stored document first and sends those four back unchanged, so you only give what changes. A stored document that lacks one of them (for example one without a title) is refused unless you set it in the same call.
+- `--role` is stored as given: `Rguest` is the guest role, and a document is searchable by a user only when it shares a role with them. The document's index `_id` is derived from its URL and role list, so changing `--url` or `--role` moves the document to a new `_id`; the old one is removed and the `doc_id` stays the same.
+- `search` does not take `--page`; this API pages with `--start` and `--size`. The server caps `--size` at 100 (`paging.search.page.max.size`; 0 or less also means 100) and refuses an offset beyond `indexer.max.result.window.size` (10000 by default) with "Could not process the specified query".
+- `--sort` accepts only sortable index fields such as `last_modified.desc`, `created`, `timestamp`, `content_length` and `click_count`; `title.desc` is refused.
+- Documents are searchable about a second after they are written (index refresh). Poll `search` or `get` after `create`, `update` or a bulk index instead of expecting the result at once. The same delay applies to deletes: for a moment after `delete`, a second `delete` of the same `doc_id` can still report success.
+- `create` does not fill defaults: `last_modified`, `timestamp`, `content_length`, `click_count` and `favorite_count` are absent unless you pass them with `--field` (`documents bulk` does fill them).
+- Field values given with `--field` are sent as strings; the server converts the numeric and date fields it validates (`boost`, `click_count`, `expires`, ...) and refuses ones that do not convert.
+- `delete-by-query` takes a normal Fess query (`*:*` matches everything). `url` is matched whole, so use `url:"https://..."` with the exact URL; a wildcard on it matches nothing. A blank query is refused by the server. A confirmation is asked first; pass `--yes` in scripts.
+- Fess 15.8.0 and later refuse a `--content` change on a document whose content was split into chunks by the chunk-embedding pipeline, and drop `content_chunk_vector` and `content_chunk_status` given through `--field`; 15.9.0 also drops the tag field.
+- Fess 15.9.0 reports "not found" when `delete` matches no `doc_id`; 14.19.2 through 15.8.0 answer success without deleting anything.
+
+## Examples
+```bash
+fessctl searchlist search "title:release" --size 20
+fessctl searchlist search "*:*" --start 100 --size 100 --output json
+fessctl searchlist get DOC_ID
+fessctl searchlist create --url https://www.example.com/note --title "Note" --role Rguest --content "hello" --label notes
+fessctl searchlist update DOC_ID --title "New title" --boost 2
+fessctl searchlist delete DOC_ID
+fessctl searchlist delete-by-query 'url:"https://www.example.com/old/page.html"' --yes
+```

skills/fessctl/references/workflows.md

@@ -193,9 +193,3 @@ fessctl badword create --suggest-word internal-test
 ```
 
 **Caution:** several of these tunings take effect only after the suggest dictionary is rebuilt or the search index is reloaded. If changes do not appear in search results, trigger the matching maintenance scheduler job (inspect `fessctl scheduler list --output json | jq '.response.settings[] | select(.name | test("Suggest|Index"; "i"))'`).
-
----
-
-## Out of scope (today)
-
-These recipes intentionally restrict themselves to features fessctl currently wraps. Operations that depend on `searchlist` or `documents` are not represented here because fessctl 0.1.0 does not expose those subcommands. When fessctl adds them, update this file with the additional recipes.

src/fessctl/api/client.py

@@ -114,7 +114,7 @@ def send_request(
                         url, headers=headers, params=params, timeout=self.timeout
                     )
                 else:
-                    # httpx.delete cannot carry a body; some endpoints (plugin) bind it.
+                    # httpx.delete cannot carry a body; some endpoints (plugin, delete-by-query) bind it.
                     response = httpx.request(
                         "DELETE", url, headers=headers, params=params, json=json_data, timeout=self.timeout
                     )
@@ -873,6 +873,79 @@ def install_theme(self, name: str, version: str) -> dict:
         url = f"{self.base_url}/api/admin/theme"
         return self.send_request(Action.CREATE, url, json_data={"name": name, "version": version})
 
+    # Searchlist (document) APIs
+
+    def search_documents(
+        self,
+        query: Optional[str] = None,
+        start: Optional[int] = None,
+        size: Optional[int] = None,
+        sort: Optional[str] = None,
+        lang: Optional[List[str]] = None,
+    ) -> dict:
+        """
+        Searches the indexed documents. start is the 0-based offset of the first hit;
+        the server caps size at paging.search.page.max.size (100 by default).
+        """
+        url = f"{self.base_url}/api/admin/searchlist/docs"
+        params = {
+            key: value
+            for key, value in (
+                ("q", query),
+                ("start", start),
+                ("size", size),
+                ("sort", sort),
+                ("lang", lang),
+            )
+            if value is not None
+        }
+        return self.send_request(Action.LIST, url, params=params)
+
+    def get_document(self, doc_id: str) -> dict:
+        """
+        Retrieves an indexed document by its doc_id.
+        """
+        url = f"{self.base_url}/api/admin/searchlist/doc/{doc_id}"
+        return self.send_request(Action.GET, url)
+
+    def create_document(self, doc: dict) -> dict:
+        """
+        Indexes a new document. The server requires url, title, role and boost.
+        """
+        url = f"{self.base_url}/api/admin/searchlist/doc"
+        return self.send_request(Action.CREATE, url, json_data={"doc": doc})
+
+    def update_document(self, doc: dict) -> dict:
+        """
+        Updates an indexed document identified by doc["doc_id"]. The fields in doc are
+        merged over the stored document, but the server validates doc on its own, so
+        url, title, role and boost must be present.
+        """
+        url = f"{self.base_url}/api/admin/searchlist/doc"
+        return self.send_request(Action.EDIT, url, json_data={"doc": doc})
+
+    def delete_document(self, doc_id: str) -> dict:
+        """
+        Deletes an indexed document by its doc_id.
+        """
+        url = f"{self.base_url}/api/admin/searchlist/doc/{doc_id}"
+        return self.send_request(Action.DELETE, url)
+
+    def delete_documents_by_query(self, query: str) -> dict:
+        """
+        Deletes every indexed document matching the query.
+        """
+        url = f"{self.base_url}/api/admin/searchlist/query"
+        return self.send_request(Action.DELETE, url, json_data={"q": query})
+
+    # Documents APIs
+
+    def bulk_index_documents(self, documents: List[dict]) -> dict:
+        """
+        Indexes several documents in one request (POST on Fess 14, PUT on 15).
+        """
+        url = f"{self.base_url}/api/admin/documents/bulk"
+        return self.send_request(Action.EDIT, url, json_data={"documents": documents})
     # General APIs
 
     def get_general(self) -> dict:

src/fessctl/cli.py

@@ -13,7 +13,7 @@
 from fessctl.commands.crawlinginfo import crawlinginfo_app
 from fessctl.commands.dataconfig import dataconfig_app
 from fessctl.commands.dict import dict_app
-# from fessctl.commands.documents import documents_app
+from fessctl.commands.documents import documents_app
 from fessctl.commands.duplicatehost import duplicatehost_app
 from fessctl.commands.elevateword import elevateword_app
 from fessctl.commands.failureurl import failureurl_app
@@ -31,7 +31,7 @@
 from fessctl.commands.relatedcontent import relatedcontent_app
 from fessctl.commands.relatedquery import relatedquery_app
 from fessctl.commands.reqheader import reqheader_app
-# from fessctl.commands.searchlist import searchlist_app
+from fessctl.commands.searchlist import searchlist_app
 from fessctl.commands.scheduler import scheduler_app
 from fessctl.commands.stats import stats_app
 from fessctl.commands.storage import storage_app
@@ -52,7 +52,7 @@
 app.add_typer(crawlinginfo_app, name="crawlinginfo")
 app.add_typer(dataconfig_app, name="dataconfig")
 app.add_typer(dict_app, name="dict")
-# app.add_typer(documents_app, name="documents")
+app.add_typer(documents_app, name="documents")
 app.add_typer(duplicatehost_app, name="duplicatehost")
 app.add_typer(elevateword_app, name="elevateword")
 app.add_typer(failureurl_app, name="failureurl")
@@ -70,7 +70,7 @@
 app.add_typer(relatedcontent_app, name="relatedcontent")
 app.add_typer(relatedquery_app, name="relatedquery")
 app.add_typer(reqheader_app, name="reqheader")
-# app.add_typer(searchlist_app, name="searchlist")
+app.add_typer(searchlist_app, name="searchlist")
 app.add_typer(scheduler_app, name="scheduler")
 app.add_typer(stats_app, name="stats")
 app.add_typer(storage_app, name="storage")

src/fessctl/commands/documents.py

@@ -0,0 +1,74 @@
+import json
+from pathlib import Path
+
+import typer
+import yaml
+
+from fessctl.api.client import FessAPIClient
+from fessctl.config.settings import Settings
+from fessctl.utils import format_list_markdown, format_result_markdown
+
+documents_app = typer.Typer()
+
+
+def _parse_documents(text: str) -> list:
+    """
+    Accepts a JSON array of documents, {"documents": [...]}, a single document
+    object, or NDJSON (one document per line).
+    """
+    try:
+        data = json.loads(text)
+    except json.JSONDecodeError:
+        data = [json.loads(line) for line in text.splitlines() if line.strip()]
+    if isinstance(data, dict):
+        data = data["documents"] if "documents" in data else [data]
+    if not isinstance(data, list) or not all(isinstance(d, dict) for d in data):
+        raise ValueError("expected a JSON array of document objects")
+    return data
+
+
+@documents_app.command("bulk")
+def bulk_index(
+    file: str = typer.Argument(
+        ..., help="File with the documents (a JSON array, {\"documents\": [...]} or NDJSON), or - for stdin"),
+    output: str = typer.Option(
+        "text", "--output", "-o", help="Output format (text, json, yaml)"
+    ),
+):
+    """
+    Index many documents in one request.
+
+    Each document needs a url and a title; role, boost, the counters and the
+    timestamps are filled in by the server when omitted.
+    """
+    try:
+        text = typer.get_text_stream("stdin").read() if file == "-" else Path(file).read_text(encoding="utf-8")
+        documents = _parse_documents(text)
+    except (OSError, ValueError) as e:
+        typer.echo(format_result_markdown(False, f"Cannot read documents from '{file}'. {e}", "Document", "bulk"))
+        raise typer.Exit(code=1)
+
+    client = FessAPIClient(Settings())
+    result = client.bulk_index_documents(documents)
+    response = result.get("response", {})
+    status = response.get("status", 1)
+
+    if output == "json":
+        typer.echo(json.dumps(result, indent=2))
+    elif output == "yaml":
+        typer.echo(yaml.dump(result))
+    elif status == 0:
+        typer.echo(format_result_markdown(True, f"{len(response.get('items', []))} document(s) indexed.", "Document", "bulk"))
+    else:
+        message: str = response.get("message", "")
+        items = response.get("items", [])
+        if items:
+            typer.echo(format_list_markdown("Bulk results", [dict(item, index=i + 1) for i, item in enumerate(items)], [
+                ("#", "index"),
+                ("RESULT", "result"),
+                ("ID", "id"),
+                ("MESSAGE", "message"),
+            ]))
+        typer.echo(format_result_markdown(False, f"Failed to index documents. {message} Status code: {status}", "Document", "bulk"))
+    if status != 0:
+        raise typer.Exit(code=status)