Skip to content

Commit b1fe693

Browse files
HowHsuHowHsu
committed
docs: add bilingual FUSE subsystem tutorial
1 parent 5c64014 commit b1fe693

24 files changed

Lines changed: 3386 additions & 1 deletion

‎index.md‎

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,33 @@ This blog is based on github pages, comments are welcome through github issues :
4343
* #### [mem initialization](./kernel/cloud-hypervisor/mem.md)
4444
* #### [device initialization](./kernel/cloud-hypervisor/device.md)
4545
* #### [fs initialization](./kernel/cloud-hypervisor/fs.md)
46-
* ### [fuse](./kernel/FUSE/fuse.md)
46+
* ### [FUSE subsystem tutorial / FUSE 子系统教程](./kernel/FUSE/fuse.md)
47+
* #### [中文目录](./kernel/FUSE/zh/index.md)
48+
* ##### 第一章：FUSE 子系统设计
49+
* [1.1 边界、分层与端到端架构](./kernel/FUSE/zh/chapter-1-design/01-boundary-and-architecture.md)
50+
* [1.2 核心对象、协议身份与能力协商](./kernel/FUSE/zh/chapter-1-design/02-object-model-and-protocol.md)
51+
* [1.3 缓存、I/O 模式与一致性模型](./kernel/FUSE/zh/chapter-1-design/03-caching-and-coherency.md)
52+
* [1.4 并发、背压、安全与失败模型](./kernel/FUSE/zh/chapter-1-design/04-concurrency-security-and-failure.md)
53+
* ##### 第二章：FUSE 代码实现分析
54+
* [2.1 源码地图、模块初始化、挂载与 FUSE_INIT](./kernel/FUSE/zh/chapter-2-implementation/01-source-map-mount-init.md)
55+
* [2.2 路径解析、目录操作、属性与引用记账](./kernel/FUSE/zh/chapter-2-implementation/02-namespace-and-metadata.md)
56+
* [2.3 OPEN、缓存读写、Direct I/O、Writeback 与 RELEASE](./kernel/FUSE/zh/chapter-2-implementation/03-open-read-write.md)
57+
* [2.4 请求分配、队列状态机与 /dev/fuse 传输](./kernel/FUSE/zh/chapter-2-implementation/04-request-transport.md)
58+
* [2.5 普通 mmap 与 virtio-fs DAX](./kernel/FUSE/zh/chapter-2-implementation/05-mmap-and-dax.md)
59+
* [2.6 卸载、Abort、调试方法与性能分析](./kernel/FUSE/zh/chapter-2-implementation/06-teardown-debug-performance.md)
60+
* #### [English contents](./kernel/FUSE/en/index.md)
61+
* ##### Chapter 1: FUSE subsystem design
62+
* [1.1 Boundaries, layers, and end-to-end architecture](./kernel/FUSE/en/chapter-1-design/01-boundary-and-architecture.md)
63+
* [1.2 Core objects, protocol identity, and capability negotiation](./kernel/FUSE/en/chapter-1-design/02-object-model-and-protocol.md)
64+
* [1.3 Caching, I/O modes, and coherency](./kernel/FUSE/en/chapter-1-design/03-caching-and-coherency.md)
65+
* [1.4 Concurrency, backpressure, security, and failure](./kernel/FUSE/en/chapter-1-design/04-concurrency-security-and-failure.md)
66+
* ##### Chapter 2: Linux implementation
67+
* [2.1 Source map, module initialization, mount, and FUSE_INIT](./kernel/FUSE/en/chapter-2-implementation/01-source-map-mount-init.md)
68+
* [2.2 Path walking, directory operations, attributes, and references](./kernel/FUSE/en/chapter-2-implementation/02-namespace-and-metadata.md)
69+
* [2.3 OPEN, cached I/O, direct I/O, writeback, and RELEASE](./kernel/FUSE/en/chapter-2-implementation/03-open-read-write.md)
70+
* [2.4 Request allocation, queue state machine, and /dev/fuse transport](./kernel/FUSE/en/chapter-2-implementation/04-request-transport.md)
71+
* [2.5 Ordinary mmap and virtio-fs DAX](./kernel/FUSE/en/chapter-2-implementation/05-mmap-and-dax.md)
72+
* [2.6 Teardown, abort, debugging, and performance analysis](./kernel/FUSE/en/chapter-2-implementation/06-teardown-debug-performance.md)
4773
* ### virtio
4874
* #### [virtio](./kernel/virtio/virtio.md)
4975
* #### [vhost](./kernel/virtio/vhost.md)

‎kernel/FUSE/en/chapter-1-design/01-boundary-and-architecture.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# 1.1 Boundary and Overall Architecture
2+
3+
[中文](../../zh/chapter-1-design/01-boundary-and-architecture.md) | **English** | [Contents](../index.md)
4+
5+
> Chapter 1: FUSE Subsystem Design · Article 1 of 4
6+
7+
FUSE is not simply “a filesystem implemented in user space.” It is a split filesystem architecture: the kernel keeps the VFS-facing half, while a user-space daemon implements filesystem policy. The protocol boundary between them is the central design fact from which performance, consistency, security, and failure behavior follow.
8+
9+
## 1. Learning goals
10+
11+
After this article, you should be able to:
12+
13+
- place FUSE correctly in the Linux VFS stack;
14+
- explain which responsibilities remain in the kernel and which move to the daemon;
15+
- trace a pathname lookup and a file read across the boundary;
16+
- distinguish classic `/dev/fuse`, io_uring transport, and virtio-fs;
17+
- identify the unavoidable and avoidable costs of the architecture.
18+
19+
## 2. The architectural boundary
20+
21+
A native filesystem usually resolves a VFS operation entirely inside the kernel. FUSE inserts a request/reply protocol between the VFS operation and the filesystem implementation:
22+
23+
```text
24+
application
25+
|
26+
| openat/read/write/stat/mmap/...
27+
v
28+
Linux VFS and page cache
29+
|
30+
| inode_operations / file_operations / address_space_operations
31+
v
32+
FUSE kernel client
33+
|
34+
| FUSE request/reply protocol
35+
v
36+
transport: /dev/fuse, io_uring, or virtio-fs
37+
|
38+
v
39+
user-space daemon or host-side backend
40+
|
41+
v
42+
backing store, remote service, archive, object store, or synthetic data
43+
```
44+
45+
The application still sees ordinary POSIX syscalls. It does not know whether the operation is serviced by ext4, a FUSE daemon, or a remote backend.
46+
47+
## 3. Responsibilities on each side
48+
49+
### 3.1 Kernel-side responsibilities
50+
51+
The kernel client must integrate with invariants that cannot be delegated:
52+
53+
- VFS pathname walking and object lifetime;
54+
- dentry, inode, page-cache, and mmap integration;
55+
- request construction, queueing, interruption, and completion;
56+
- credential and namespace context carried in requests;
57+
- cache validation and invalidation;
58+
- writeback, direct I/O, passthrough, and DAX dispatch;
59+
- mount teardown and behavior after a dead or disconnected daemon.
60+
61+
The kernel is therefore a stateful protocol client, not a thin syscall forwarder.
62+
63+
### 3.2 Daemon-side responsibilities
64+
65+
The daemon supplies filesystem policy and backend integration:
66+
67+
- mapping names to stable node identities;
68+
- returning attributes and directory entries;
69+
- implementing create, unlink, rename, permission, and xattr semantics;
70+
- creating open handles and processing data I/O;
71+
- maintaining backend consistency and persistence;
72+
- issuing invalidation notifications when data changes externally;
73+
- applying policy that is not already enforced by the kernel.
74+
75+
A daemon may use a local directory, a database, an RPC service, or no persistent storage at all.
76+
77+
## 4. A syscall is not always one FUSE request
78+
79+
The VFS works on objects and caches, while the protocol exchanges messages. Their boundaries do not match one-to-one.
80+
81+
One `openat()` may require several `LOOKUP` requests and an `OPEN`; or zero requests if dentries and attributes remain valid. One large `read()` may be split into several `READ` requests. A page fault may issue I/O without a new `read()` syscall. Conversely, readahead and writeback can create requests that have no direct syscall counterpart.
82+
83+
This is why diagnosing FUSE solely from syscall traces is incomplete. You must also observe the protocol and cache state.
84+
85+
## 5. Metadata path example
86+
87+
For `stat("/mnt/a/b")`, the conceptual path is:
88+
89+
1. VFS begins from the mount root.
90+
2. It checks the dentry cache for `a` and then `b`.
91+
3. For a missing or expired component, FUSE sends `FUSE_LOOKUP(parent_nodeid, name)`.
92+
4. The daemon returns a node ID, generation, attributes, and validity intervals.
93+
5. The kernel instantiates or updates the dentry and inode.
94+
6. If cached attributes expire later, `FUSE_GETATTR` may refresh them.
95+
96+
A lookup reply is therefore both a namespace result and a time-bounded cache lease.
97+
98+
## 6. Data path example
99+
100+
For a buffered `read(fd, buf, len)`:
101+
102+
1. VFS enters the FUSE file operations.
103+
2. The page cache satisfies already-cached ranges.
104+
3. Missing folios cause one or more `FUSE_READ` requests.
105+
4. The daemon fetches data and writes a reply.
106+
5. The kernel fills the page cache and copies data to the application.
107+
108+
Direct I/O bypasses the page cache and divides the user range into protocol requests. Passthrough can redirect operations to a backing kernel file. DAX maps file ranges through a finite device window and services faults through filesystem DAX helpers. These modes have different coherency and fallback constraints; they are not interchangeable optimizations.
109+
110+
## 7. Transport variants
111+
112+
### 7.1 Classic `/dev/fuse`
113+
114+
A daemon reads requests from a FUSE device file and writes replies back. The kernel maintains pending and processing queues, while each request carries a unique identifier for reply matching.
115+
116+
### 7.2 FUSE over io_uring
117+
118+
Newer kernels can negotiate an io_uring-based transport. It changes how buffers and completion are delivered, but not the VFS-facing architecture or the meaning of protocol operations.
119+
120+
### 7.3 virtio-fs
121+
122+
virtio-fs reuses the FUSE protocol across a virtio transport, normally between a guest kernel and a host backend such as `virtiofsd`. It can also expose a DAX window for shared mappings. This removes some copies and VM exits on suitable workloads, but introduces finite-window allocation, mapping recall, and guest/host coherency questions.
123+
124+
## 8. Cost model
125+
126+
Potential costs include:
127+
128+
- context switches or cross-VM transitions;
129+
- request allocation, queue synchronization, and wakeups;
130+
- serialization and validation of protocol structures;
131+
- data copies between application, kernel, daemon, and backend;
132+
- many metadata round trips during pathname walking;
133+
- lock hold time and queueing under daemon saturation.
134+
135+
The largest optimization opportunities usually come from reducing request count, increasing useful batching, selecting the right data path, and assigning cache validity intervals that match the consistency model.
136+
137+
## 9. Design questions to ask first
138+
139+
Before tuning or extending FUSE, answer these questions:
140+
141+
1. Which side is authoritative for names, attributes, and file contents?
142+
2. Can the backend change without going through this mount?
143+
3. Which objects may be cached, for how long, and who invalidates them?
144+
4. What happens when the daemon stalls, crashes, or replies late?
145+
5. Can the chosen data paths coexist without incoherent aliases?
146+
6. Which resources are bounded: requests, daemon workers, DAX mappings, or backing handles?
147+
7. Does a proposed fallback preserve the original I/O position and partial-completion semantics?
148+
149+
These questions are more useful than treating every latency spike as merely “user-space overhead.”
150+
151+
## 10. Summary
152+
153+
FUSE splits one filesystem into a kernel protocol client and a user-space policy engine. The kernel preserves VFS semantics and caches; the daemon defines the filesystem and accesses its backend. The protocol is the architectural seam, and all later topics—identities, leases, queueing, timeout, DAX, and teardown—must be understood relative to that seam.
154+
155+
[Next: Object Model and Protocol](02-object-model-and-protocol.md)

‎kernel/FUSE/en/chapter-1-design/02-object-model-and-protocol.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# 1.2 Object Model and Protocol
2+
3+
[中文](../../zh/chapter-1-design/02-object-model-and-protocol.md) | **English** | [Contents](../index.md)
4+
5+
> Chapter 1: FUSE Subsystem Design · Article 2 of 4
6+
7+
Correct FUSE implementations depend on understanding three different object systems at once: VFS objects, kernel FUSE objects, and daemon-owned identities. They overlap, but none is a one-to-one replacement for another.
8+
9+
## 1. Three layers of objects
10+
11+
### 1.1 VFS objects
12+
13+
The VFS presents familiar structures:
14+
15+
- `super_block`: a mounted filesystem instance;
16+
- `dentry`: a name-to-inode relationship, including negative entries;
17+
- `inode`: metadata and operation tables for a filesystem object;
18+
- `file`: one open file description;
19+
- `address_space`: cached file data and its I/O operations.
20+
21+
Their lifetimes are controlled by VFS references, not by FUSE protocol messages alone.
22+
23+
### 1.2 Kernel FUSE objects
24+
25+
The kernel adds protocol-specific state such as:
26+
27+
- the connection and mount objects;
28+
- per-inode FUSE metadata;
29+
- per-open-file state and daemon file handles;
30+
- request objects and input/output argument vectors;
31+
- pending, interrupt, and processing queues.
32+
33+
These objects bridge VFS lifetimes to message lifetimes.
34+
35+
### 1.3 Daemon identities
36+
37+
The daemon returns identifiers that have protocol meaning:
38+
39+
- `nodeid`: identity of a filesystem node within the connection;
40+
- `generation`: distinguishes reuse of a node ID;
41+
- `fh`: daemon-defined handle returned by `OPEN` or `OPENDIR`;
42+
- `unique`: identifier of one request, used to match its reply.
43+
44+
A `nodeid` is not a pointer, a Linux inode number, or an open handle. An `fh` is not a `nodeid`. A `unique` lasts for one protocol transaction only.
45+
46+
## 2. Namespace lifetime and `FORGET`
47+
48+
A successful `LOOKUP` gives the kernel a lookup reference on the returned node. The kernel can accumulate multiple references while dentries are cached. When it no longer needs them, it sends `FORGET` or `BATCH_FORGET`.
49+
50+
Important consequences:
51+
52+
- `FORGET` is normally one-way and has no ordinary reply;
53+
- the daemon must account for the `nlookup` decrement correctly;
54+
- forgetting a lookup identity does not imply that every open handle is closed;
55+
- an unlinked but open file may remain reachable through its `fh`;
56+
- node ID reuse requires correct generation handling.
57+
58+
A daemon that treats `FORGET` as `RELEASE` will eventually corrupt its lifetime model.
59+
60+
## 3. Protocol envelope
61+
62+
Every request begins with `struct fuse_in_header`, which includes message length, opcode, unique ID, node ID, caller UID/GID/PID, and related context. Most replies begin with `struct fuse_out_header`, which contains length, error, and the matching unique ID.
63+
64+
```text
65+
request = fuse_in_header + opcode-specific input + optional payload
66+
reply = fuse_out_header + opcode-specific output + optional payload
67+
```
68+
69+
The protocol is binary. Structure layout, feature negotiation, length validation, and compatibility rules matter as much as the logical operation.
70+
71+
## 4. Kernel request representation
72+
73+
The kernel does not need to flatten every message immediately. It describes input and output with argument vectors, together with request flags such as whether a reply is expected or output data is variable length. Transport code then copies or maps the described buffers.
74+
75+
This separation allows one logical request representation to support classic device I/O, io_uring delivery, and other channels.
76+
77+
## 5. Operation classes
78+
79+
The opcode space covers several classes:
80+
81+
- namespace: `LOOKUP`, `CREATE`, `MKNOD`, `MKDIR`, `UNLINK`, `RENAME`;
82+
- metadata: `GETATTR`, `SETATTR`, `ACCESS`, xattr operations;
83+
- open lifetime: `OPEN`, `OPENDIR`, `RELEASE`, `RELEASEDIR`, `FLUSH`;
84+
- data: `READ`, `WRITE`, fallocate, copy-file-range;
85+
- cache control: `FORGET`, invalidation notifications;
86+
- connection control: `INIT`, `DESTROY`, `INTERRUPT`;
87+
- specialized paths: mapping operations used by DAX and newer extensions.
88+
89+
Not every request expects a reply. That distinction must be explicit in both kernel and daemon logic.
90+
91+
## 6. `FUSE_INIT`: protocol negotiation
92+
93+
The mount is not ready for normal operation until the kernel and daemon exchange `FUSE_INIT`.
94+
95+
Negotiated information includes:
96+
97+
- protocol major and minor versions;
98+
- supported feature flags;
99+
- maximum write and readahead sizes;
100+
- background queue and congestion limits;
101+
- time granularity and alignment constraints;
102+
- support for writeback cache, parallel directory operations, DAX, passthrough, security context, io_uring transport, and other version-dependent capabilities.
103+
104+
Feature flags are a contract. A daemon must not advertise a capability and then implement only its happy path.
105+
106+
## 7. Example: `LOOKUP` reply
107+
108+
A successful `LOOKUP` returns a `fuse_entry_out` containing:
109+
110+
- `nodeid` and `generation`;
111+
- entry validity duration;
112+
- attribute validity duration;
113+
- the `fuse_attr` payload.
114+
115+
The kernel uses these values to instantiate or refresh both namespace and inode state. A negative lookup can also be cached when the daemon returns `ENOENT` with a configured validity interval.
116+
117+
## 8. Example: open identity versus node identity
118+
119+
Suppose two processes open the same FUSE inode. The VFS may use one inode object, while the daemon returns two different `fh` values. Per-open flags—such as direct I/O or keep-cache behavior—may differ. Later `READ`, `WRITE`, `FLUSH`, and `RELEASE` operations can carry the relevant `fh` even though their `nodeid` is identical.
120+
121+
This distinction is essential for backends with sessions, credentials, remote descriptors, or per-open locks.
122+
123+
## 9. Protocol invariants
124+
125+
A robust implementation preserves at least these invariants:
126+
127+
1. Every reply is matched to exactly one live request by `unique`.
128+
2. The reply opcode payload and length agree with the original request.
129+
3. Node identity remains stable for the advertised lifetime.
130+
4. Lookup counts and open-handle counts are accounted independently.
131+
5. Negotiated limits are honored on both input and output.
132+
6. Late replies, interrupted requests, and connection teardown cannot complete the same request twice.
133+
7. User-controlled lengths and offsets are validated before copying or mapping.
134+
135+
## 10. Summary
136+
137+
VFS objects, FUSE kernel state, and daemon identifiers form three related lifetime systems. `nodeid`, `generation`, `fh`, and `unique` answer different questions. The binary protocol joins those systems, while `INIT` defines which optional behaviors are legal for the lifetime of the connection.
138+
139+
[Previous: Boundary and Overall Architecture](01-boundary-and-architecture.md) | [Next: Caching and Coherency](03-caching-and-coherency.md)

0 commit comments

Comments
 (0)