cffi-callback-closures

Runtime-created, C-callable callbacks for Common Lisp, built on libffi closures. It adds a parallel callback API to CFFI without touching — or changing the behavior of — cffi:defcallback.

See it in action: lispfs is a FUSE filesystem built entirely on this library — every struct fuse_operations entry is a Lisp closure, so each filesystem syscall dispatches into Lisp.

Why

cffi:defcallback delegates to each Lisp implementation's native callback facility. That is the right design for the common case (it integrates with the host's threads, GC, and unwinding), but it has two limits:

1. No runtime-parameterized callbacks. defcallback defines a named, top-level callback. You can't cheaply mint N distinct C function pointers at runtime, each closing over different Lisp data.
2. No struct-by-value in callbacks on most backends.

libffi's closure API (ffi_closure_alloc / ffi_prep_closure_loc) addresses both — it's the symmetric counterpart to the ffi_call path that cffi-libffi already uses for structures-by-value.

defcallback vs. make-foreign-callback

In one line: cffi:defcallback is static, named, and native — the right default; make-foreign-callback is runtime, anonymous, and closure-backed — for when the callback must carry per-instance data.

	cffi:defcallback	make-foreign-callback (this lib)
Created	at compile/load time	at runtime, on demand
Identity	a named, top-level definition	an anonymous value (a pointer you hold)
Closes over data?	no — body sees only its args + globals	yes — any Lisp closure
How many?	one per name	as many distinct ones as you like
Lifetime	whole image; no free	you free-foreign-callback (or with-foreign-callback)
Mechanism	each Lisp's native callback	libffi closure → one shared native dispatcher
Dependencies	core CFFI only	needs cffi-libffi
Per-call cost	specialized, no consing	conses to marshal args (slower)
Image save/restore	survives	named ones rebuilt; runtime ones forgotten
Calling convention	:convention :cdecl/:stdcall	libffi :abi (currently :default-abi)
Thread / GC / unwind	host runtime handles it	same (the bridge is a defcallback)

The decisive difference

Make C-callable adders where adder_k(x) = x + k, with k chosen at runtime.

defcallback can't parameterize — you get one callback and must smuggle state through a global, so you can't have two live at once:

(defvar *k* 0)
(cffi:defcallback adder :int ((x :int)) (+ x *k*))
;; one callback, one shared *k*. adder-5 AND adder-100 live together? Can't.

make-foreign-callback closes over k — each call mints a distinct, independent C function pointer:

(defun make-adder (k)
  (make-foreign-callback (lambda (x) (+ x k)) :int '(:int)))

(defparameter *add5*   (make-adder 5))
(defparameter *add100* (make-adder 100))
;; two real C function pointers, each carrying its own k — both live at once

The same shape appears everywhere real: a qsort comparator parameterized by a runtime sort order, a write callback closing over this request's buffer, a table of N opcode handlers — natural with closures, awkward-to-impossible with defcallback's single-global-state model.

When to use which

• defcallback for a fixed, singular C hook known at compile time (a signal handler, one well-known library callback). Simpler, faster, survives image dumps cleanly, needs no libffi.
• make-foreign-callback when the callback must close over runtime data, when you need many distinct callbacks, or when you build them dynamically. It's a parallel API for the cases defcallback can't express — not a replacement.

How it works

There is exactly one native cffi:defcallback in the image — the generic dispatcher %dispatch. It is the only point where control crosses from C into Lisp, so it inherits the host Lisp's thread/GC/unwind handling. Every logical callback is a libffi closure whose handler is %dispatch and whose user_data is a small integer index into a registry:

C caller ─▶ libffi trampoline (per closure; ABI-correct marshalling)
              │  demux via user_data
              ▼
       %dispatch  ── the single native callback (safe Lisp entry)
              │
              ▼
       registry[index] → (function, arg-types, return-type)

This sidesteps the "libffi can't safely enter Lisp" problem: libffi does the per-signature ABI work, the host Lisp does the world-entry, exactly once.

Compatibility

This is a separate ASDF system. It sets no global CFFI state (unlike cffi-libffi, which installs *foreign-structures-by-value*). It is inert until you call its functions, and it does not redefine defcallback, callback, or any CFFI symbol. Code that uses cffi:defcallback is unaffected whether or not this system is loaded.

It reuses a few cffi-libffi internals (cffi::make-libffi-cif, cffi::free-libffi-cif) rather than re-groveling libffi, so it depends on cffi-libffi.

Usage

(require :cffi-callback-closures)
(use-package :cffi-callback-closures)

;; Runtime callback closing over data:
(let ((cb (make-foreign-callback (lambda (a b) (+ a b)) :int '(:int :int))))
  (unwind-protect
       (cffi:foreign-funcall-pointer cb () :int 3 :int 4 :int) ; => 7
    (free-foreign-callback cb)))

;; Scoped:
(with-foreign-callback (cb (lambda (x) (* x x)) :int '(:int))
  (cffi:foreign-funcall-pointer cb () :int 9 :int))            ; => 81

;; Named (analogous to defcallback, separate namespace):
(define-foreign-callback my-mul :int ((a :int) (b :int)) (* a b))
(cffi:foreign-funcall-pointer (foreign-callback my-mul) () :int 6 :int 7 :int)

Status / TODO

This is a scaffold. The scalar / pointer / float path works; the following are marked in the source and not yet complete:

• Struct-by-value arguments and returns (the descriptor machinery is reused from cffi-libffi, but %dispatch/write-return don't yet treat a struct slot specially).
• Alternate ABIs (:stdcall, :win64, …): cffi-libffi only surfaces :default-abi in its abi enum, so only the default convention is selectable.
• Image save/restore is handled (see below) for named callbacks. Runtime pointers from make-foreign-callback still cannot be replayed automatically — recreate them after startup.
• Return widening covers the standard integer keywords; enums and typedef'd integer signedness need more care.
• Registry reads take a lock on every invocation; a lock-free read path is a possible optimization.

Image save/restore

A libffi closure is a malloc'd cif plus an mmap'd executable trampoline, and the pointer handed to C is an address in this process — none of it survives save-lisp-and-die and a restart. The system handles this with three cooperating mechanisms (in src/closures.lisp):

• Dump hook (uiop:register-image-dump-hook) frees all live C resources and clears the live tables before the image is written, so the saved heap holds no dangling foreign state.
• Restore hook (uiop:register-image-restore-hook) runs in the resumed process: it bumps an epoch, drops the stale bookkeeping, reloads libffi, and rebuilds every named callback from its retained recipe. So a callback defined with define-foreign-callback just works after restore.
• Epoch guard: each closure records the image epoch it was built in, so a stale pointer is detectable (foreign-callback-live-p is NIL; get-foreign-callback transparently rebuilds) rather than silently dispatching into freed memory.

Limits: a runtime callback from make-foreign-callback has no replayable definition — it is forgotten on restore and must be recreated. And if a C library cached your pointer before the dump, you must re-register with that library after restore (inherent to image dumping with any FFI).

Tests

(asdf:test-system :cffi-callback-closures)

run includes an in-process simulation of the dump/restore hooks. A genuine end-to-end test that dumps a standalone executable with save-lisp-and-die, runs it, and checks the named callback still works is available separately (SBCL; builds a multi-megabyte core, so it's not in run):

(asdf:load-system :cffi-callback-closures/test)
(cffi-callback-closures-tests:run-image-test)   ; => verifies survivor(6) = 42

Demo

examples/demo.lisp contrasts libffi closures with cffi:defcallback using C qsort (a real callback API with no user_data slot):

(asdf:load-system :cffi-callback-closures/examples)
(cffi-cc-demo:run-all)

It shows four things closures give you and defcallback can't, or can only fake with globals:

1. Parameterized + reentrant comparators — a fresh qsort comparator per call, each closing over its order function and a private comparison counter.
2. The defcallback workaround — the same sort, forced through a global *cmp-less* + global counter, because there is only one named callback.
3. A callback factory — a table of N C function pointers built at runtime, each capturing its own index. Impossible with defcallback (N callbacks, compile-time-named).
4. Per-instance mutable state — two accumulator callbacks with independent running totals.

examples/filewalk.lisp is a richer showcase: it drives POSIX ftw(3) — a real OS tree-walker whose callback has no user_data slot — through a Lisp closure that builds a full report (file/dir counts, total bytes, deepest path, and a bar chart of file extensions). Each walk gets its own callback bound to its own report, so multiple trees are walked with zero shared state.

(asdf:load-system :cffi-callback-closures/examples)
(cffi-cc-filewalk:run)

whole project: …/cffi-callback-closures/
  files   268   dirs   37   other   0
  total size : 1.5 MB
  top file types:
    lisp     153 ########################
    md        64 ##########
    asd       12 ##

examples/threads.lisp runs a Lisp closure as the body of a real OS thread via pthread_create — the start routine void *(*)(void *) is a Lisp lambda. It also stress-tests the closure path from the hard direction: the thread that calls our dispatcher was created by C, so every call crosses the host Lisp's foreign-thread registration (works because there's exactly one native defcallback underneath). Verified on SBCL.

(asdf:load-system :cffi-callback-closures/examples)
(cffi-cc-threads:run)
;;   i=5  i^2=25  ran on pthread #x16D4FF000
;;   -> 6 distinct OS threads did the work.

examples/sqlite.lisp extends SQLite three ways, all with Lisp closures as C function pointers (libsqlite3 ships on macOS):

• scalar functions (sqlite3_create_function) — a Lisp lambda is the body of a SQL function, invoked inside SQLite's VM once per row, usable in SELECT/WHERE/ORDER BY;
• aggregates (xStep/xFinal) — a Lisp accumulator folded across rows;
• authorizer (sqlite3_set_authorizer) — a Lisp closure decides, per access, what SQL is allowed (here it hides a column).

(cffi-cc-sqlite:run)
;; WHERE fib(n) > 10 ORDER BY fib(n) DESC:   13, 10, 8
;; product(n) where n>0 => 31200     median(n) => 4
;; authorizer hides users.secret:    alice | <hidden>

examples/xml.lisp builds a SAX parser: expat's start-tag, end-tag, and character-data handlers are three Lisp closures sharing one closed-over parse stack, cooperatively turning XML into an s-expression tree. This is the "framework calls you" pattern, and making three distinct, state-sharing C callbacks at runtime is exactly what defcallback can't do. Uses libexpat (ships on macOS / most Linuxes).

(cffi-cc-xml:run)
;; <a><b>hi</b><b>yo</b></a>           => (:A (:B "hi") (:B "yo"))
;; <p>libffi <em>closures</em> rock</p> => (:P "libffi " (:EM "closures") " rock")

examples/http.lisp fetches a URL with libcurl; the CURLOPT_WRITEFUNCTION write callback is a Lisp closure that streams each response chunk into a closed-over buffer. It also hits — and fixes — a real ABI bug: curl_easy_setopt is variadic, and on Apple arm64 a plain foreign-funcall mis-passes the trailing argument (variadic args go on the stack, not in registers), so curl rejects the URL. The demo routes setopt through libffi's ffi_prep_cif_var, so it exercises two libffi features at once — closures and variadic calls.

(cffi-cc-http:run)
;; https://example.com
;;   528 bytes in 1 chunk; first line: "<!doctype html>...Example Domain..."

examples/scandir.lisp filters a directory with C scandir(), whose per-entry filter int (*)(const struct dirent *) has no user_data — so the selection logic comes from a Lisp closure over a runtime-chosen predicate. It grovels struct dirent to read d_name portably, then reads back and frees the C-allocated result array.

(cffi-cc-scandir:run)
;; *.md files:              README.md
;; names containing 'c':    COPYRIGHT  cffi-callback-closures.asd  ocicl  src

examples/vm.lisp is the table-of-callbacks pattern: a bytecode stack VM whose opcodes are Lisp closures stored in a C jump table (a void(*[])(int)). Running a program means reading table[opcode] out of C memory and calling it — a real C indirect dispatch into a Lisp closure. The table is built and sized at runtime (the demo even adds a new :sq opcode on the fly), and each slot closes over the shared VM stack — neither of which defcallback can express.

(cffi-cc-vm:run)
;; (:PUSH 6 :PUSH 7 :MUL :PRINT)            => 42
;; (:PUSH 5 :DUP :MUL :PRINT)               => 25
;; adding opcode :sq at runtime... (:PUSH 9 :SQ :PRINT) => 81

examples/tree.lisp builds a binary search tree with libc tsearch/twalk: the tree's order comes from a Lisp comparator closure and an in-order visitor closure collects the keys — two cooperating callbacks driving a C data structure.

(cffi-cc-tree:run)
;; alphabetical: apple banana cherry date fig kiwi pear
;; by length:    fig date kiwi pear apple banana cherry

examples/complete.lisp does tab completion powered by a Lisp closure: libedit's rl_completion_matches repeatedly calls a generator callback (char *(*)(const char *, int)) to enumerate matches; here the generator is a Lisp closure over a word list. It returns malloc'd strings for libedit to free.

(cffi-cc-complete:run)
;; complete "fo"   -> food foot football force foreign format   [common: "fo"]
;; complete "ba"   -> banana bar baz                            [common: "ba"]

examples/pcre2.lisp runs a Lisp predicate inside the regex engine. PCRE2 calls a callout function at (?C1) points mid-match; here the callout is a Lisp closure that reads the text matched so far and returns accept/reject, so the regex's acceptance is decided by Lisp. (Grovels pcre2_callout_block for the field offsets; uses libpcre2-8.)

(cffi-cc-pcre2:run)
;; pattern \b\d++(?C1) over "12 7 100 33 8 250 17 4 99 23"
;; numbers (even):  12 100 8 250 4
;; numbers (prime): 7 17 23
;; numbers (> 50):  100 250 99

examples/tk.lisp registers Lisp closures as Tcl/Tk commands (Tcl_CreateCommand), so a button's -command handler is a Lisp closure over app state — a tiny GUI with no user_data anywhere. The closure-as-command mechanism needs no display and is verified headlessly by run-headless; run opens an actual window (needs a display, and on macOS the process main thread).

(cffi-cc-tk:run-headless)
;; Tcl: for i in 0..10 -> [fib $i] = 0 1 1 2 3 5 8 13 21 34 55
;; Tcl: [greet World] = Hello, World, from Lisp!
(cffi-cc-tk:run)         ; opens a Tk window; buttons call Lisp closures

examples/atexit.lisp registers Lisp closures as C shutdown hooks with libc atexit — the zero-argument callback case, invoked by the C runtime at process teardown (LIFO). run spawns a child that registers three hooks and exits, showing them fire:

(cffi-cc-atexit:run)
;; hook 3 fired (Lisp closure over k=3)
;; hook 2 fired (Lisp closure over k=2)
;; hook 1 fired (Lisp closure over k=1)

examples/audio.lisp is the limits example. A PortAudio stream callback is a synth voice as a Lisp closure (an arpeggio over a closed-over phase). But the callback runs on a real-time foreign thread, and a GC'd runtime there is the classic anti-pattern: our dispatcher conses, sample writes box floats, and entering SBCL on PortAudio's RT thread traps hard here (a trap kills the process — it can't be caught). So run builds the closure and enumerates devices but does not start the stream by default; (run :play t) opts in knowingly. It's included to show the boundary honestly, not as good practice — real-time audio wants an allocation-free, non-GC'd callback path.

(cffi-cc-audio:run)
;; PortAudio: V19.7.0 ...   devices: 4   default output: 1
;; built the synth: a Lisp closure at #x...
;; Not starting the stream by default (real-time thread + GC'd runtime).

License

MIT. See COPYRIGHT.