curlcl — a libcurl binding for Common Lisp

libcurl is installed everywhere already, make use of it

A comprehensive Common Lisp binding to libcurl — the easy interface with all 308 of its options and 78 info values, multi, share, the URL parser, MIME, the header API and websockets — plus an HTTP client built on top of it.

lispnik.github.io/curlcl shows each libcurl C call beside the Lisp that does the same thing, which is the quickest way in if you already know libcurl.

Installation

As a library:

ocicl install

Then:

(asdf:load-system :curlcl)

Needs libcurl. Nothing here is groveled or compiled, so no C toolchain is required to build it. On macOS the binding prefers Homebrew's libcurl over the one in the dyld shared cache, because the system build has no websocket support; set LIBCURL_LIBRARY to pin a specific one.

libcurl 7.83 or newer. The client reads response headers through curl_easy_header/curl_easy_nextheader, which arrived in 7.83, and unlike the other version-gated functions those are declared rather than resolved at load time — so an older libcurl loads and reports its version quite happily and then fails on the first transfer with "the alien function curl_easy_nextheader is undefined". Ubuntu 22.04's 7.81 is the one still in the wild worth naming.

Just the curlcl command, with no Lisp toolchain — on macOS, which also pulls in a libcurl that can speak ws://:

brew tap lispnik/curlcl
brew trust lispnik/curlcl
brew install curlcl

The brew trust line is required and is not particular to this tap: Homebrew refuses to load a formula from any third-party tap until told to, and without it the install stops at "Refusing to load formula … from untrusted tap".

Or take a binary from the releases, built for Linux and macOS on both architectures and for Windows on x86-64. Each archive carries the libraries that are not part of a stock system, and each is checked in CI by running it with the build toolchain taken away.

Unpack and put curlcl on your PATH — on Windows it is bin\curlcl.exe, and the archive's layout matters: MinGW's libcurl looks for its CA bundle relative to its own DLL, so bin and etc have to keep their positions or HTTPS stops working while plain HTTP carries on.

libcurl itself is not bundled, on purpose — curlcl opens it at startup rather than linking it, so it uses the one already on your machine and gets that build's capabilities. curlcl -V prints which library it found and what that library can speak. On macOS the binaries are signed ad-hoc rather than notarised, so Gatekeeper wants xattr -d com.apple.quarantine curlcl first.

Usage

The package is curlcl, nicknamed curl.

(curl:http-get "https://example.com/")
;; => #<RESPONSE 200 https://example.com/ 559 bytes>

A non-2xx status is a response, not a condition. Only transport failures signal, because only then is there nothing to return.

(let ((response (curl:http-get "https://api.example.com/thing")))
  (case (curl:response-status response)
    (200 (curl:response-text response))
    (404 nil)
    (t (warn "unexpected ~D" (curl:response-status response)))))

Bodies are decoded when the Content-Type says they are text and names a charset we know; otherwise they arrive as octets. A charset we do not recognise gives octets rather than a guess.

(curl:http-post "https://example.com/form"
                :content '(("name" . "a value") ("other" . "x&y")))

(curl:http-post "https://example.com/upload"
                :multipart '((:name "field" :data "value")
                             (:name "file" :file #p"/tmp/report.pdf"
                              :content-type "application/pdf")))

Headers are accepted as an alist, a plist, or a list of strings, because all three appear in real code:

(curl:http-get "https://example.com/" :headers '(("Accept" . "application/json")))
(curl:http-get "https://example.com/" :headers '(:accept "application/json"))
(curl:http-get "https://example.com/" :headers '("Accept: application/json"))

Duplicated response headers are kept, which is the only representation that can be right for Set-Cookie:

(curl:response-header-values response "set-cookie")   ; => ("a=1" "b=2")

Streaming

Nothing is buffered when you give it somewhere to go — in either direction:

(curl:download "https://example.com/big.iso" #p"/tmp/big.iso")

(curl:http-get "https://example.com/big.iso"
               :on-data (lambda (octets) (process octets)))

;; :INPUT is the request-side counterpart of :OUTPUT.  A pathname, a stream,
;; or a reader function; the source never has to fit in memory.
(curl:http-put "https://example.com/big.iso" :input #p"/tmp/big.iso")

A size is declared when it can be known, so the request carries a Content-Length; when it cannot — a pipe, a generator — the body goes out chunked. A file also gets a seek callback, so libcurl can rewind it to repeat the request for a redirect or an authentication challenge.

Streaming and :retry interact, because a retried transfer delivers its body from the beginning. Who owns the destination decides whether that is a problem:

;; Fine: the pathname is ours, so each attempt reopens and truncates the file.
(curl:download "https://example.com/big.iso" #p"/tmp/big.iso" :retry 3)

;; Signals UNSAFE-RETRY: the stream is yours, and rewinding or truncating it is
;; not this library's to do -- so the failed attempt's bytes would sit in front
;; of the successful attempt's, with nothing to say so.
(with-open-file (out #p"/tmp/big.iso" :direction :output
                                      :element-type '(unsigned-byte 8))
  (curl:http-get url :output out :retry 3))

;; Retry it anyway, having said out loud that seeing part of the body twice is
;; acceptable.
(curl:http-get url :on-data #'process :retry 3 :retry-streamed t)

The refusal happens before the first attempt rather than on the retry that would have corrupted the file.

Sessions

A session pools easy handles over a share, so connections, DNS answers, TLS sessions and cookies are common to a run of requests:

(curl:with-session (session)
  (curl:http-post (test-url "/login") :content credentials :session session)
  ;; The cookie set by the login is sent with this one.
  (curl:http-get "https://example.com/account" :session session))

Retry

libcurl has no retry logic; this does. The defaults are narrow on purpose — transport failures that say nothing was processed, plus the statuses that explicitly mean "try again" — and POST is not retried unless you say so, because only you know whether repeating one duplicates an order.

(curl:http-get "https://flaky.example.com/"
               :retry '(:max-attempts 5 :initial-delay 0.5))

Several at once

(curl:request-many (list "https://a.example.com/"
                         (list "https://b.example.com/" :method :post
                                                        :content "body")))
;; => (#<RESPONSE 200 ...> #<RESPONSE 201 ...>)

A failure sits in its own slot as a condition rather than aborting the batch.

Retries work here too, and are scheduled rather than sequential — a request waits out its backoff while the rest of the batch keeps transferring:

(curl:request-many urls
                   :retry '(:max-attempts 4 :initial-delay 0.4)
                   :on-complete (lambda (index outcome) (report index outcome)))

:on-complete fires as each request reaches its final outcome, so it can drive a progress display; an individual request can override the batch policy, or opt out with :retry nil.

The binding underneath

The client is a thin layer; the whole C API is there if you want it.

(curl:with-easy (handle)
  (setf (curl:callback-function handle :write)
        (lambda (octets) (write-sequence octets *standard-output*) t))
  (curl:setopts handle :url "https://example.com/" :followlocation t)
  (curl:perform handle)
  (curl:getinfo handle :response-code))

Options are keywords derived mechanically from the C names — drop CURLOPT_, downcase, underscores to hyphens — so CURLOPT_SSL_VERIFYPEER is :ssl-verifypeer. An option the loaded libcurl does not have is reported by name rather than as CURLE_UNKNOWN_OPTION.

The client takes the same keywords through :setopts, applied after everything else and so able to override it. The named arguments cover what a client usually wants; this is the way to the remaining three hundred without dropping down to with-easy and rebuilding the request:

(curl:http-get "https://example.com/"
               :setopts '(:interface "en0"
                          :max-recv-speed-large 65536
                          :resolve ("example.com:443:127.0.0.1")))

The command-line driver

make build produces bin/curlcl, a curl(1) workalike built on the library. Option names, defaults, output destinations and exit codes follow curl, so most curl command lines work unchanged and scripts that check the exit status keep working — the codes are libcurl's own CURLcode values.

$ curlcl -s -o /dev/null -w '%{http_code} %{size_download}b in %{time_total}s\n' https://example.com/
200 559b in 0.086570s

$ curlcl -s -L -H 'Accept: application/json' https://api.example.com/thing
$ curlcl -s -F 'file=@report.pdf;type=application/pdf' https://example.com/upload
$ curlcl -sZ -o a.html -o b.html https://a.example/ https://b.example/   # parallel
$ curlcl --retry 3 https://flaky.example/                                # scheduled backoff

$ curlcl -s -d @payload.json -H 'Content-Type: application/json' https://api.example/
$ curlcl -s --data-urlencode 'q=a b&c' https://example.com/search
$ curlcl -s -D headers.txt -o body.html https://example.com/

Holding to curl's behaviour is the point: it forces the library to cover what a real client needs rather than what is convenient to expose. -Z goes through request-many, --retry through the scheduled backoff, -F through curl_mime_*, -w through getinfo.

One deliberate difference, noted in --help: there is no progress meter unless --progress-bar (or -#) is given.

A ws:// or wss:// URL opens a websocket instead, standard input to frames and frames to standard output. This is the one place the driver goes beyond curl rather than following it — curl accepts the scheme but has no interactive mode for it, and the library underneath has the whole API:

$ printf 'hello\nagain\n' | curlcl ws://echo.example/
hello
again

$ curlcl --ws-binary ws://echo.example/ < payload.bin > reply.bin

A line is one text frame, with the newline treated as the terminator it is; --ws-binary sends raw blocks and adds nothing on the way out.

curlcl -V also reports which libcurl it loaded, which curl has no need to do — this binding can load any of several, and on macOS they differ in version, TLS backend and protocol support.

Running the tests

make test

Integration tests run against an HTTP server started inside the image on an ephemeral loopback port, so the suite is hermetic and can serve responses no public endpoint will produce on demand — a truncated body, a redirect loop, a slow trickle, a route that fails exactly as many times as it is asked to. There is a websocket echo server too.

CURL_LIVE_TESTS=1 make test

adds a small suite that uses the real network, for the things a local server cannot stand in for: a real certificate chain, a rejected expired one, real DNS, HTTP/2, and connection reuse observed through timing.

Implementation notes

• The option and info tables are generated from the curl headers by generator/generate-tables.lisp and committed, so a build needs no headers and no C compiler. Parsing C with regular expressions is only defensible because libcurl can describe its own options at runtime: the suite checks every entry against curl_easy_option_by_name on the library actually loaded, which also catches the table and the library having drifted apart. Run make tables to regenerate.

• The spelled type matters. CURLOPT_URL, CURLOPT_HTTPHEADER, CURLOPT_WRITEDATA and CURLOPT_POSTFIELDS are all "10000 plus something", but one is a string libcurl copies, one is an slist the caller must keep alive, one is opaque callback data, and one is a buffer libcurl explicitly does not copy. The generator keeps the nine spelled types rather than the five numeric bases, because that distinction decides who owns the memory.

• :postfields is routed to CURLOPT_COPYPOSTFIELDS, so libcurl owns the copy. CURLOPT_COPYPOSTFIELDS exists precisely because CURLOPT_POSTFIELDS does not copy, and making callers reason about that is not worth the one avoided memcpy.

• There are no finalizers, deliberately. The callback registry must hold a handle's state strongly for as long as libcurl might call into it, which would keep the handle alive and stop a finalizer firing anyway. with-easy and with-session are the contract; the client layer wraps everything in unwind-protect, so ordinary use cannot leak. live-callback-count makes a leak assertable.

• curl_easy_duphandle copies every option value, including CURLOPT_WRITEDATA and CURLOPT_ERRORBUFFER — so a duplicate points at the original's registry key and error buffer. Both are re-pointed.

• Clearing a callback keeps the trampoline installed, with a nil closure acting as a no-op sink. "Never set" and "set to NULL" are different states in libcurl: CURLOPT_READDATA set to NULL makes the built-in read callback call fread on a null FILE*. For the same reason the write and read trampolines are installed from the start — libcurl's built-in ones write to stdout and read from stdin.

• Websockets are feature-gated at runtime, since whether ws:// works is a property of the loaded library. macOS ships the headers for a libcurl built without it. libcurl marks this API experimental; that caveat is passed on.

Implementations

Developed and tested on SBCL, which is what CI runs on macOS, Linux and Windows — one workflow per platform, so each badge above reports its own platform. That is not cosmetic: GitHub's badge endpoint reports a whole workflow and silently ignores a ?job= parameter, so three badges pointed at one matrix would all have shown the same status no matter which platform broke. The three are not redundant either — macOS is where a variadic argument goes on the stack, Linux is where it goes in a register, and Windows is where curl_socket_t is eight bytes wide.

Linux runs on both x86-64 and arm64, since those answer different questions. Apple's arm64 ABI is the one that puts variadic arguments on the stack; AAPCS64 keeps them in registers like named arguments. So the Linux arm64 leg is what tells an instruction-set problem apart from a Darwin calling-convention one — a failure on arm64 everywhere means the former, a failure on macOS alone means the latter. It also tests the architecture the release workflow has been shipping a linux-arm64 binary for all along.

The library itself is close to portable: the only implementation-specific code in the library proper is a bulk octet copy in memory.lisp, which has a portable fallback.

ECL was made to load the system and perform real HTTPS requests at one point, and the suite was seen to stall partway through, apparently because the in-process test server spawns a thread per keep-alive connection and never reaps them. Treat that as hearsay: nothing in CI covers ECL, the last attempt to reproduce it did not get as far as loading the system, and the build has changed since — cffi-libffi is no longer a dependency. ECL is unverified, not supported.

bin/curlcl needs a byte stream on standard input and output, which the standard has no way to ask for. There are clauses for SBCL, ECL, CCL and CLISP, and a /dev/fd fallback for any other Unix implementation. The descriptor comes from the implementation rather than being written as 0 or 1 — see standard-descriptor in src/cli.lisp for why that distinction is not academic.

Windows

Supported on SBCL, in CI, running the same suite as the Unix jobs, green. A number of tests skip there rather than run: most are the opt-in network suite, the rest are the websocket tests — the libcurl the runner picks up is built without ws:// — and one needs a Unix shell. No count is given here on purpose; it changes with every commit, and the badge above is the claim that keeps itself true.

Four things had to be true for this to work, and three of them were broken when it was first tried:

• Line endings. FORMAT's continuation directive is ~ followed by a newline; with CRLF the next character is a Return, which is not a directive, and every format string that wraps stops compiling. .gitattributes pins *.lisp and *.asd to LF.
• curl_socket_t. A file descriptor on Unix, a Win32 SOCKET — UINT_PTR, 8 bytes — on Win64. Declaring it :int would have had CURLINFO_ACTIVESOCKET write 8 bytes into 4, passed half a socket to curl_multi_socket_action, and made struct curl_waitfd the wrong size for curl_multi_wait to read as an array.
• Standard descriptors. SBCL on Windows keeps an OS handle where Unix keeps a descriptor, so curlcl could not write a response body until it stopped assuming 1 meant standard output.
• cffi-libffi, which needed a C toolchain and libffi. MSYS2's MinGW-w64 packages supplied both; this was the part expected to be the obstacle, and it was the one thing that worked first time. It is no longer a dependency — CFFI's own foreign-funcall-varargs does the job — so Windows now needs nothing from MSYS2 but libcurl itself.

C long is 4 bytes on Windows and 8 on LP64 Unix. The binding passes CFFI's :LONG, which already tracks that, so nothing needed changing — but the test that claimed to check it had hard-coded 8, and now measures the width by asking libcurl to write a CURLINFO_LONG into a poisoned buffer instead.

What is not bound, and why

79 of the 100 curl_* symbols the library exports are bound. The rest are left out deliberately:

Not bound	Why
curl_formadd, curl_formfree, curl_formget	Deprecated on every enumerator since 7.56.0, and variadic with a sentinel-terminated option list. curl_mime_* replaces it.
curl_escape, curl_unescape	Deprecated forms that take no handle; curl_easy_escape is bound.
curl_multi_socket, curl_multi_socket_all	Deprecated; curl_multi_socket_action is bound.
curl_mprintf and its nine relatives	Lisp has format.
curl_strequal, curl_strnequal, curl_getenv	string-equal and uiop:getenv.
curl_global_init_mem	Bindable, but a Lisp allocator called from libcurl's resolver threads is a GC-deadlock foothold. Left out rather than offered as a trap.

Version-gated functions — curl_ws_start_frame, curl_multi_get_offt, curl_multi_notify_enable/disable, curl_easy_ssls_import/export — are resolved at load time rather than declared, so an older libcurl reports the absence through unsupported-feature instead of failing at the first call.

License

MIT. See LICENSE.