indexd

indexd is a prototype Linux daemon that maintains a searchable LMDB index of file metadata and selected extended attributes. It performs an initial filesystem walk, then uses fanotify events to keep the index up to date as files change.

I developed it for my honours dissertation, with an emphasis on low steady-state resource use, human-scale update latency, and minimal interference with foreground applications. It is not a finished product, but in the spirate of free software, I decided to release it so that anyone can study, inspect, extend, adapt or otherwise make use of the code.

Design

The daemon has two threads:

• the main thread scans the filesystem, receives fanotify events, gathers metadata, and submits work through a bounded queue;
• a single index-writer thread owns all LMDB mutations.

The database stores canonical per-file state in two forward indexes:

• id_meta records fixed metadata such as size, modification time, and change time;
• id_xattr records configured extended attributes.

Reverse indexes map values back to file IDs. The built-in indexes cover basename, size, modification time, and change time, while each configured extended attribute receives its own reverse index. On every update, indexd diffs the new state against the forward records, removes obsolete reverse keys, inserts changed keys, and commits the complete per-file change in one LMDB transaction.

Keys are encoded to preserve useful bytewise ordering: unsigned integers use big-endian representation, strings use their raw UTF-8 bytes, and arbitrary extended-attribute values remain opaque.

Building

The build requires Linux, a C23 compiler, Meson, Ninja, and the LMDB development package.

meson setup build
meson compile -C build

This builds indexd and the small utilities under examples/.

Running

sudo ./build/indexd indexd.conf /path/to/database /path/to/filesystem-root

The arguments are the attribute configuration file, LMDB directory, and directory to scan. The watcher installs a filesystem-wide fanotify mark, and the file-handle operations require suitable privileges (normally running as root). Use a disposable test filesystem rather than pointing this prototype at important data.

indexd.conf contains an example list of extended attributes. Human-readable logs are written to standard error; lifecycle events used by the evaluation tools are emitted on standard output as EVENT <timestamp> <name>.

Supporting tools

• tools/generator/ creates deterministic synthetic directory hierarchies.
• tools/supervisor/ runs repeatable workloads, resource monitoring, and post-run correctness checks.
• tools/check_correctness.py compares the forward and expected reverse LMDB records with filesystem state.
• tools/pidstat_plot.py plots captured per-process resource measurements.
• examples/dump_index.c and examples/fanotify_filesystem.c are diagnostic utilities used during development.

The top-level Makefile isn’t for compiling the program; it manages disposable ext4 image files for evaluation.

Known limitations

• File identity currently relies on an ext4-style 32-bit inode file handle; Linux file handles are formally opaque and other formats are unsupported.
• Rename handling and hard-link semantics are incomplete.
• Path recovery and the gap between the initial scan and watcher setup leave race windows.
• Fanotify queue overflow is not repaired with a rescan.
• The on-disk indexes are implemented, but there is no end-user query API.

License

Copyright © 2025-2026 Michael Connor Buchan mikey@blindcomputing.org

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.