From ed4f169bc64647ece433b69227a82c9e6f4a2654 Mon Sep 17 00:00:00 2001 From: Kyle S Passarelli Date: Sun, 16 Aug 2026 16:02:23 -0600 Subject: [PATCH 1/5] Implement sessions and async --- README.md | 80 +++++++++- decisions/02-async-sessions.org | 27 ++++ devops-drift.el | 47 +++--- devops-test.el | 224 ++++++++++++++++++++++++++++ devops.el | 249 ++++++++++++++++++++++++++++---- 5 files changed, 577 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index 98f0680..7ae03ec 100644 --- a/README.md +++ b/README.md @@ -272,8 +272,84 @@ If you use [cljbang.el](https://github.com/borkdude/cljbang.el), the drift API r ## Long-running commands -Long commands such as `apt-get update` can lock up emacs. There are several worakrounds, from -using sessions and `:async yes` (built in), to [ob-async](https://github.com/astahlman/ob-async), and probably others. YMMV. +Long commands such as `apt-get update` can lock up emacs. By default a block +runs in a synchronous subprocess and emacs waits; on a remote target every +command also pays connection latency, and an unexpected prompt (`sudo`, an ssh +host key confirmation, `apt` asking a question) leaves emacs waiting on a +process that will never finish on its own. + +Org's answer is `:async` — but only inside a `:session`, since a sessionless +shell block has no comint buffer to attach a filter to. A target tag is already +a name for "which machine is this block talking to", so it can name the session +too: + +```elisp +(setq devops-enable-session-async t) +``` + +With that on, a block under a target-tagged heading is executed as if you had +written the session and async headers yourself: + +``` +#+begin_src sh :dir /ssh:example.com: :session devops:example :async yes +apt-get update +#+end_src +``` + +Emacs returns immediately with a placeholder and the output replaces it when +the command finishes. A command that asks a question no longer hangs emacs +either: the prompt waits in the session buffer, and `devops-goto-session` takes +you there to answer it. + +This is off by default because it makes blocks stateful. `cd`, `export`, an +activated virtualenv and `ssh-agent` now survive from one block to the next +under the same heading, which is useful, but it costs idempotency: blocks become +order-dependent, and a block that passed in a dirty session may fail in a fresh +one. Prefer blocks that do not depend on the ones above them, and use +`devops-restart-session` to get back to a clean shell. + +Two tags on the same host get two sessions, because two tags mean two +directories — a shell session's working directory is fixed when its buffer is +created, and later blocks do not `cd` to their `:dir`. + +| Command | What it does | +|--------------------------|--------------------------------------------------| +| `devops-goto-session` | Pop to the heading's session buffer | +| `devops-restart-session` | Kill it, so the next block starts a fresh shell | + +### Per-block escape hatches + +Whatever the block itself says wins: + +| Header | Effect | +|-----------------|-----------------------------------------------------------| +| `:async no` | Run this one block synchronously, in the same session | +| `:session none` | No session, no async, no state shared with other blocks | +| `:session foo` | Attach to a session of your choosing | +| `:target nil` | No target, and so no session either | + +A block that names its own `:dir` is running somewhere devops did not choose, +so it gets no session either. + +Only languages in `devops-async-session-languages` (`sh`, `bash`, `shell`, +`python`) are affected. `:session` is not a neutral header argument — for +`emacs-lisp` it means an ielm buffer, and for the non-executable blocks that +carry `:tangle` it is meaningless — so everything else keeps getting `:dir` and +nothing more. + +Tangling and drift checks stay synchronous, because they need real values: a +noweb reference that executes a block (`<>`) would otherwise resolve +to the placeholder, and the placeholder is what would get written to the file on +the server. Anything else that reads the return value of +`org-babel-execute-src-block` should wrap it in `devops-with-sync`. + +Session names come from `devops-session-name-function`, which defaults to +`devops:`. A session name is a buffer name in a single global namespace, so +if two org files use the same tag for different hosts, set this to a function +that also folds in the target or the buffer name. + +If you would rather not use sessions at all, there is also +[ob-async](https://github.com/astahlman/ob-async), and probably others. YMMV. [ob-screen]: https://howardism.org/Technical/Emacs/literate-devops.html#fnr.4 diff --git a/decisions/02-async-sessions.org b/decisions/02-async-sessions.org index 72beb2b..bbe4279 100644 --- a/decisions/02-async-sessions.org +++ b/decisions/02-async-sessions.org @@ -207,3 +207,30 @@ status probe to the body and parsing it back out — a separate decision. target. Session-per-tag makes running both concurrently a natural next step, one session each, results merged. Keeping the prompt for now; the naming scheme is chosen so this does not need to be revisited. + +* Implementation notes + +Two things the decisions above did not anticipate. + +** =:session none= is org's default, not an opt-out signal + +=org-babel-default-header-args= contains =(:session . "none")=, so /every/ +block arrives at the advice already carrying =:session none=. Reading the +merged header arguments cannot tell a block that opted out of sessions from one +that never mentioned them, which is what decision 4 assumed. + +=devops--user-header-args= resolves it: re-read the block with +=org-babel-default-header-args= and its language-specific counterpart bound to +nil, and whatever =:session= survives was written by the user. Any value other +than ="none"= needs no such check — no default produces one. + +Where the block cannot be re-read, "declared" is the safe answer. A =#+call:= +line executes with an =INFO= built from the Library of Babel while point is not +on a src block, so it gets no session: losing async on call lines is a smaller +error than attaching a block to shared state it never asked for. + +** =:async yes=, not =:async t= + +=ob-shell= and =ob-python= both declare the header as =(async . ((yes no)))=, +and =org-babel-comint-use-async= only tests against ="no"=. Injecting ="yes"= +matches what the languages document; a hand-written =:async t= still works. diff --git a/devops-drift.el b/devops-drift.el index 5692a85..4bc7510 100644 --- a/devops-drift.el +++ b/devops-drift.el @@ -179,28 +179,33 @@ heading at point (mirroring `devops-tangle'). Return (LOCAL-ROOT . ENTRIES) where ENTRIES are the plists of `devops--drift-tangle-heading', each with :status and :detail added. Unlike tangling to a remote, a drift check is read-only, so a failing target yields an `error' entry instead of aborting. -The caller owns LOCAL-ROOT and must delete it." +The caller owns LOCAL-ROOT and must delete it. + +Runs under `devops-with-sync', for the same reason tangling does: a +comparison needs the real output of an executed noweb block, not the +placeholder an async evaluation returns." (with-current-buffer source-buf - (let ((spec (devops--tangle-spec all)) - (root (make-temp-file "devops-drift-" t)) - (entries nil)) - (dolist (e spec) - (setq entries - (append entries - (devops--drift-tangle-heading - source-buf (plist-get e :heading-pos) - (plist-get e :tag) (plist-get e :target) root)))) - ;; Several blocks may append to one tangled file; one entry each. - (setq entries (seq-uniq entries - (lambda (a b) - (equal (plist-get a :local) - (plist-get b :local))))) - (dolist (entry entries) - (let ((status (devops--drift-status (plist-get entry :local) - (plist-get entry :remote)))) - (plist-put entry :status (car status)) - (plist-put entry :detail (cdr status)))) - (cons root entries)))) + (devops-with-sync + (let ((spec (devops--tangle-spec all)) + (root (make-temp-file "devops-drift-" t)) + (entries nil)) + (dolist (e spec) + (setq entries + (append entries + (devops--drift-tangle-heading + source-buf (plist-get e :heading-pos) + (plist-get e :tag) (plist-get e :target) root)))) + ;; Several blocks may append to one tangled file; one entry each. + (setq entries (seq-uniq entries + (lambda (a b) + (equal (plist-get a :local) + (plist-get b :local))))) + (dolist (entry entries) + (let ((status (devops--drift-status (plist-get entry :local) + (plist-get entry :remote)))) + (plist-put entry :status (car status)) + (plist-put entry :detail (cdr status)))) + (cons root entries))))) ;;; Noninteractive API diff --git a/devops-test.el b/devops-test.el index add3171..94742fb 100644 --- a/devops-test.el +++ b/devops-test.el @@ -650,6 +650,230 @@ targets land next to the org file rather than in the system temp dir." (let ((org-confirm-babel-evaluate nil)) (should-error (org-babel-execute-src-block) :type 'user-error))))) +;;; Async sessions (decision 2: async execution in per-target sessions) + +(defun devops-test--executor-params (lang) + "Execute the src block at point with `org-babel-execute:LANG' stubbed out. +Return the header arguments the executor was handed, so an injected +`:session' or `:async' can be read without starting a shell." + (let ((fn (intern (concat "org-babel-execute:" lang))) + (seen nil)) + (cl-letf (((symbol-function fn) + (lambda (_body params) (setq seen params) ""))) + (let ((org-confirm-babel-evaluate nil)) + (org-babel-execute-src-block))) + seen)) + +(defmacro devops-test--with-session-org (header &rest body) + "Run BODY on a sh block carrying HEADER, under a heading tagged `:local:'." + (declare (indent 1)) + `(devops-test--with-org + (concat "#+TARGET: /srv/app/ (local)\n\n" + "* Run\t\t:local:\n\n" + "#+begin_src sh " ,header "\npwd\n#+end_src\n") + (goto-char (point-min)) + (re-search-forward "begin_src") + ,@body)) + +(ert-deftest devops-session-async-off-by-default-test () + "Without `devops-enable-session-async', no session or async is injected." + (devops-test--with-session-org "" + (let ((params (devops-test--executor-params "sh"))) + (should (equal (cdr (assq :dir params)) "/srv/app/")) + (should (equal (cdr (assq :session params)) "none")) + (should-not (assq :async params))))) + +(ert-deftest devops-session-async-injects-session-and-async-test () + "With the option on, a block gets `:session devops:TAG' and `:async yes'." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org "" + (let ((params (devops-test--executor-params "sh"))) + (should (equal (cdr (assq :dir params)) "/srv/app/")) + (should (equal (cdr (assq :session params)) "devops:local")) + (should (equal (cdr (assq :async params)) "yes")))))) + +(ert-deftest devops-session-name-function-test () + "`devops-session-name-function' decides the session name." + (let ((devops-enable-session-async t) + (devops-session-name-function + (lambda (tag target) (format "%s@%s" tag target)))) + (devops-test--with-session-org "" + (should (equal (cdr (assq :session (devops-test--executor-params "sh"))) + "local@/srv/app/"))))) + +(ert-deftest devops-session-async-explicit-session-wins-test () + "A `:session' on the block is not overwritten by the tag's session." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org ":session other" + (let ((params (devops-test--executor-params "sh"))) + (should (equal (cdr (assq :session params)) "other")) + (should (equal (cdr (assq :async params)) "yes")))))) + +(ert-deftest devops-session-async-session-none-test () + "`:session none' opts a block out of both the session and async." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org ":session none" + (let ((params (devops-test--executor-params "sh"))) + (should (equal (cdr (assq :session params)) "none")) + (should-not (assq :async params)))))) + +(ert-deftest devops-session-async-explicit-async-no-test () + "`:async no' runs one block synchronously while keeping its session." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org ":async no" + (let ((params (devops-test--executor-params "sh"))) + (should (equal (cdr (assq :session params)) "devops:local")) + (should (equal (cdr (assq :async params)) "no")))))) + +(ert-deftest devops-session-async-from-property-test () + "A `:session' inherited from a `header-args' property counts as explicit." + (let ((devops-enable-session-async t)) + (devops-test--with-org + (concat "#+TARGET: /srv/app/ (local)\n\n" + "* Run\t\t:local:\n" + ":PROPERTIES:\n" + ":header-args: :session other\n" + ":END:\n\n" + "#+begin_src sh\npwd\n#+end_src\n") + (goto-char (point-min)) + (re-search-forward "begin_src") + (should (equal (cdr (assq :session (devops-test--executor-params "sh"))) + "other"))))) + +(ert-deftest devops-session-async-language-restricted-test () + "Languages outside `devops-async-session-languages' get :dir and nothing else." + (let ((devops-enable-session-async t)) + (devops-test--with-org + (concat "#+TARGET: /srv/app/ (local)\n\n" + "* Run\t\t:local:\n\n" + "#+begin_src emacs-lisp\n\"hi\"\n#+end_src\n") + (goto-char (point-min)) + (re-search-forward "begin_src") + (let ((params (devops-test--executor-params "emacs-lisp"))) + (should (equal (cdr (assq :dir params)) "/srv/app/")) + (should (equal (cdr (assq :session params)) "none")) + (should-not (assq :async params)))))) + +(ert-deftest devops-session-async-explicit-dir-test () + "A block that names its own :dir gets no session either." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org ":dir /srv/other/" + (let ((params (devops-test--executor-params "sh"))) + (should (equal (cdr (assq :dir params)) "/srv/other/")) + (should (equal (cdr (assq :session params)) "none")) + (should-not (assq :async params)))))) + +(ert-deftest devops-session-async-target-nil-test () + "`:target nil' opts a block out of the session along with the target." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org ":target nil" + (let ((params (devops-test--executor-params "sh"))) + (should-not (assq :dir params)) + (should (equal (cdr (assq :session params)) "none")) + (should-not (assq :async params)))))) + +(ert-deftest devops-with-sync-inhibits-async-test () + "`devops-with-sync' suppresses injection even with the option on." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org "" + (let ((params (devops-with-sync (devops-test--executor-params "sh")))) + (should (equal (cdr (assq :dir params)) "/srv/app/")) + (should (equal (cdr (assq :session params)) "none")) + (should-not (assq :async params)))))) + +(ert-deftest devops-session-async-tangle-noweb-test () + "Tangling resolves an executing noweb reference to output, not a placeholder. +Under `:async' the return value of a block is a UUID, which is what would +land in the tangled file on the server." + (let ((devops-enable-session-async t)) + (devops-test--with-local-target target + (devops-test--with-org + (format (concat "#+TARGET: %s (local)\n\n" + "* Deploy\t\t:local:\n\n" + "#+name: SECRET\n" + "#+begin_src sh\nprintf s3cret\n#+end_src\n\n" + "#+begin_src yaml :tangle config.yaml :noweb yes\n" + "api-key: <>\n#+end_src\n") + target) + (let ((org-confirm-babel-evaluate nil)) + (devops-tangle-headline (current-buffer) "Deploy")) + (with-temp-buffer + (insert-file-contents (concat target "config.yaml")) + (should (search-forward "api-key: s3cret" nil t))))))) + +(ert-deftest devops-session-async-executes-in-session-test () + "A block returns a placeholder, then its output arrives from the session. +The end-to-end path: the heading's tag names a shell session, the block +runs there at the target's directory, and `org-babel-comint-async-filter' +replaces the placeholder in the buffer when the command finishes." + (let ((devops-enable-session-async t)) + (devops-test--with-local-target target + (unwind-protect + (devops-test--with-org + (format (concat "#+TARGET: %s (local)\n\n" + "* Run\t\t:local:\n\n" + "#+begin_src sh\npwd\n#+end_src\n") + target) + (goto-char (point-min)) + (re-search-forward "begin_src") + (let* ((org-confirm-babel-evaluate nil) + (uuid (org-babel-execute-src-block)) + (deadline (+ (float-time) 30))) + (should (get-buffer "devops:local")) + (should (string-match-p "\\`[0-9a-f-]+\\'" uuid)) + (while (and (< (float-time) deadline) + (save-excursion + (goto-char (point-min)) + (search-forward uuid nil t))) + (accept-process-output nil 0.2)) + (goto-char (point-min)) + (should-not (search-forward uuid nil t)) + (should (re-search-forward "^: \\(.+\\)$" nil t)) + (should (equal (file-name-as-directory + (file-truename (org-trim (match-string 1)))) + (file-name-as-directory (file-truename target)))))) + (when-let* ((buf (get-buffer "devops:local"))) + (let ((kill-buffer-query-functions nil)) + (kill-buffer buf))))))) + +(ert-deftest devops-goto-session-test () + "`devops-goto-session' pops to the heading's session buffer." + (let ((devops-enable-session-async t) + (buf (get-buffer-create "devops:local"))) + (unwind-protect + (devops-test--with-session-org "" + (save-window-excursion + (devops-goto-session) + (should (eq (current-buffer) buf)))) + (kill-buffer buf)))) + +(ert-deftest devops-goto-session-without-buffer-errors-test () + "`devops-goto-session' says so when the session has not been started." + (let ((devops-enable-session-async t)) + (devops-test--with-session-org "" + (should-error (devops-goto-session) :type 'user-error)))) + +(ert-deftest devops-restart-session-test () + "`devops-restart-session' kills the heading's session buffer." + (let ((devops-enable-session-async t) + (buf (get-buffer-create "devops:local"))) + (unwind-protect + (devops-test--with-session-org "" + (devops-restart-session) + (should-not (buffer-live-p buf))) + (when (buffer-live-p buf) (kill-buffer buf))))) + +(ert-deftest devops-session-no-target-errors-test () + "The session commands error on a heading with no target tag." + (devops-test--with-org + (concat "#+TARGET: /srv/app/ (local)\n\n" + "* Run\n\n" + "#+begin_src sh\npwd\n#+end_src\n") + (goto-char (point-min)) + (re-search-forward "begin_src") + (should-error (devops-goto-session) :type 'user-error) + (should-error (devops-restart-session) :type 'user-error))) + ;;; Multi-target tangling (README: same file to several servers) (ert-deftest devops-tangle-multi-target-test () diff --git a/devops.el b/devops.el index 78b063e..f014b33 100644 --- a/devops.el +++ b/devops.el @@ -37,6 +37,65 @@ :type '(choice (const ghostty)) :group 'devops) +(defcustom devops-enable-session-async nil + "When non-nil, run blocks under a target-tagged heading in an async session. +A src block then gets `:session' and `:async' injected alongside its +`:dir', so emacs returns immediately with a placeholder and the output +lands in the results block when the command finishes. A command that +asks a question waits in the session buffer instead of hanging emacs; +`devops-goto-session' goes there. + +Off by default, because a session makes blocks stateful: `cd', `export', +an activated virtualenv and `ssh-agent' survive from one block to the +next, which is useful but costs idempotency. A block that passed in a +dirty session may fail in a fresh one, so prefer blocks that do not +depend on the ones above them, and use `devops-restart-session' to get +back to a known state." + :type 'boolean + :group 'devops) + +(defcustom devops-session-name-function + (lambda (tag _target) (format "devops:%s" tag)) + "Function mapping a target TAG and TARGET to a session name. +A shell session name is a buffer name in a single global namespace, so a +bare tag like \"web\" would collide with anything else that picked the +same word — including another org file whose \"web\" tag points at a +different host. Sending commands to the wrong machine is the worst +failure this package can have, so the default prefixes `devops:'. + +That removes accidental collisions, not deliberate ones. If two org +files reuse a tag for different hosts, set this to a function that also +folds in the target or the buffer name." + :type 'function + :group 'devops) + +(defcustom devops-async-session-languages '("sh" "bash" "shell" "python") + "Languages that get a `:session' and `:async' injected. +`:session' is not a neutral header argument: for `emacs-lisp' it means an +ielm buffer, and for the non-executable blocks that carry `:tangle' it is +meaningless. Only languages that support `org-babel-comint-async-register' +belong here; blocks in any other language keep getting `:dir' and nothing +else." + :type '(repeat string) + :group 'devops) + +(defvar devops--inhibit-async nil + "When non-nil, do not inject `:session' or `:async'. +Bound by `devops-with-sync' around tangling and drift checks.") + +(defmacro devops-with-sync (&rest body) + "Run BODY with async injection inhibited. +Async breaks the contract that the return value of +`org-babel-execute-src-block' is the block's result: under `:async' it is +a UUID placeholder. Anything that reads that value — a noweb reference +that executes a block, `devops-tangle-headline', a drift check — needs +the real thing, so it runs inside this macro. Interactive \\[org-ctrl-c-ctrl-c] +gets async; everything scripted gets synchronous evaluation unless it +asks otherwise." + (declare (indent 0) (debug t)) + `(let ((devops--inhibit-async t)) + ,@body)) + (defun devops--parse-target-keyword (value) "Parse a #+TARGET value like \"target1 (source)\" into (TAG . TARGET)." (when (string-match "\\`\\([^ ]+\\) +(\\([^)]+\\))\\'" value) @@ -65,25 +124,33 @@ Searches heading's tags against all #+TARGET keywords." (cons tag target))) tags)))) -(defun devops--heading-target-dir () - "Return :dir from the current heading's tags and #+TARGET mappings. -If there is more than one target, use completing-read, allowing the -user to select one." - (interactive) +(defun devops--heading-target () + "Return the (TAG . TARGET) in effect for the current heading, or nil. +If more than one of the heading's tags names a target, use +completing-read, allowing the user to select one. The tag is kept +alongside the target because it, not the directory, is what names the +session: a tag maps 1:1 to a target, and two tags on the same host mean +two directories, hence two sessions." (let ((matches (devops--heading-target-tags))) (cond ((null matches) nil) ((= 1 (length matches)) - (cdr (car matches))) + (car matches)) (t (let* ((options (mapcar (lambda (pair) (cons (format "%s: %s" (car pair) (cdr pair)) - (cdr pair))) + pair)) matches)) (selected (completing-read "Choose target: " (mapcar #'car options) nil t))) (cdr (assoc selected options))))))) +(defun devops--heading-target-dir () + "Return :dir from the current heading's tags and #+TARGET mappings. +If there is more than one target, use completing-read, allowing the +user to select one." + (cdr (devops--heading-target))) + (defun devops-set-header-args-from-tags () "Set :header-args: :dir from the current heading's tag and #+TARGET mappings." (interactive) @@ -96,15 +163,21 @@ Org reads a header value as a string, so a block written `:target nil' arrives as \"nil\". A genuine nil is accepted too, for params passed to `org-babel-execute-src-block' from Lisp.") +(defun devops--block-info (info) + "Return the src block info for the block being executed. +INFO is the info given to `org-babel-execute-src-block', or nil when +point is on the block." + (or info + (ignore-errors + (org-babel-get-src-block-info 'no-eval)))) + (defun devops--block-params (info) "Return the header arguments of the src block being executed. INFO is the src block info given to `org-babel-execute-src-block', or nil when point is on the block. Covers header arguments on the block itself, on a #+header: line, inherited from a `header-args' property, and the defaults in `org-babel-default-header-args'." - (nth 2 (or info - (ignore-errors - (org-babel-get-src-block-info 'no-eval))))) + (nth 2 (devops--block-info info))) (defun devops--header-cell (key params block-params) "Return the (KEY . VALUE) header argument in effect, or nil. @@ -147,23 +220,139 @@ block's header from any other." (when (and (>= pos (car region)) (< pos (cdr region))) (throw 'hit t))))) +(defun devops--session-name (tag target) + "Return the session name for TAG and TARGET." + (funcall devops-session-name-function tag target)) + +(defun devops--user-header-args (lang) + "Return the header arguments written on the src block at point. +Org's defaults are unbound while the block is read, so a `:session none' +in the result is one the user wrote rather than the one +`org-babel-default-header-args' hands to every block. LANG names the +language-specific defaults to suppress along with the global ones. +Returns nil when point is not on a src block." + (let* ((sym (and lang (intern-soft + (concat "org-babel-default-header-args:" lang)))) + (lang-default (and sym (boundp sym) sym)) + (saved (and lang-default (symbol-value lang-default))) + (org-babel-default-header-args nil)) + (unwind-protect + (progn + (when lang-default (set lang-default nil)) + (nth 2 (ignore-errors (org-babel-get-src-block-info 'no-eval)))) + (when lang-default (set lang-default saved))))) + +(defun devops--session-declared-p (params block-params lang) + "Non-nil when the block, not org, decided its `:session'. +`org-babel-default-header-args' gives every block `:session none', so the +merged header arguments cannot tell a block that opted out of sessions +from one that never mentioned them. Any other value had to be written by +hand; \"none\" is re-checked against the block's own header arguments +\(see `devops--user-header-args'). + +Unreadable cases count as declared. A `#+call:' line, for instance, is +executed with an INFO built from the Library of Babel while point is not +on a src block, so nothing here can prove the block said nothing — and +attaching a block to a shared session it did not ask for is the error +worth avoiding." + (let ((cell (devops--header-cell :session params block-params))) + (and cell + (or (not (equal (cdr cell) "none")) + (assq :session params) + (let ((own (devops--user-header-args lang))) + (or (null own) (assq :session own))))))) + +(defun devops--async-session-cells (params block-params lang tag target) + "Return the :session and :async header cells to inject, or nil. +PARAMS and BLOCK-PARAMS are as in `devops--header-cell', LANG is the +block's language, and TAG and TARGET the heading's resolved target. +Nothing is injected unless `devops-enable-session-async' is on, LANG is +in `devops-async-session-languages', and we are executing on the user's +behalf rather than under `devops-with-sync'. + +What the block already says is left alone, so per-block escape hatches +need no new syntax: `:async no' runs one block synchronously in the +heading's session, `:session none' gives it neither a session nor async, +and `:session other' attaches it to a session of the user's choosing." + (when (and devops-enable-session-async + (not devops--inhibit-async) + (member lang devops-async-session-languages)) + (let* ((declared (devops--session-declared-p params block-params lang)) + (session (cdr (devops--header-cell :session params block-params)))) + (unless (and declared (equal session "none")) + (append + (unless declared + (list (cons :session (devops--session-name tag target)))) + (unless (devops--header-cell :async params block-params) + (list (cons :async "yes")))))))) + (defun devops--inject-header-args-from-tags (orig-fn &optional arg info params executor-type) "Advise org-babel-execute-src-block to inject :dir from #+TARGET tags. An explicit :dir wins: the heading's target is neither resolved nor prompted for when the block already carries one. `:target nil' opts the block out of the heading's target without naming a directory, leaving -:dir to org." - (let* ((block-params (devops--block-params info)) - (dir (unless (or (devops--target-opted-out-p params block-params) - (devops--header-cell :dir params block-params)) - (devops--heading-target-dir))) - (params (if dir - (cons (cons :dir dir) params) +:dir to org. + +When the heading's target is what supplies :dir, the same lookup can also +supply :session and :async; see `devops--async-session-cells'. A block +that named its own :dir is running somewhere devops did not choose, so it +gets no session either." + (let* ((block-info (devops--block-info info)) + (block-params (nth 2 block-info)) + (pair (unless (or (devops--target-opted-out-p params block-params) + (devops--header-cell :dir params block-params)) + (devops--heading-target))) + (params (if pair + ;; Ours first: within one alist `org-babel-merge-params' + ;; lets a later pair overwrite an earlier one, so an + ;; explicit PARAMS from the caller still wins. + (append (cons (cons :dir (cdr pair)) + (devops--async-session-cells + params block-params (nth 0 block-info) + (car pair) (cdr pair))) + params) params))) (apply orig-fn arg info params (and executor-type (list executor-type))))) (advice-add 'org-babel-execute-src-block :around #'devops--inject-header-args-from-tags) +(defun devops--heading-session-name () + "Return the session name for the current heading's target. +Signal a `user-error' if no tag on the heading names a target." + (let ((pair (or (devops--heading-target) + (user-error "No #+TARGET match for tags on current heading")))) + (devops--session-name (car pair) (cdr pair)))) + +;;;###autoload +(defun devops-goto-session () + "Pop to the session buffer for the current heading's target. +Under `devops-enable-session-async' a command that asks a question — sudo, +an ssh host key confirmation, apt — no longer freezes emacs: the prompt +sits in the session buffer waiting for an answer. This is how to get +there and answer it." + (interactive) + (let ((name (devops--heading-session-name))) + (pop-to-buffer + (or (get-buffer name) + (user-error "No session %s yet; run a block under this heading" name))))) + +;;;###autoload +(defun devops-restart-session () + "Kill the session buffer for the current heading's target. +The next block run under the heading starts a fresh shell, at the +target's directory and with none of the state — `cd', `export', an +activated virtualenv — that earlier blocks left behind." + (interactive) + (let* ((name (devops--heading-session-name)) + (buf (get-buffer name))) + (if (not buf) + (message "No session %s" name) + ;; A live comint process would otherwise ask for confirmation, which + ;; is the whole point of the command. + (let ((kill-buffer-query-functions nil)) + (kill-buffer buf)) + (message "Killed session %s" name)))) + (defun devops--specialize-noweb-blocks (tag) "Rewrite #+name: FOO (TAG) blocks for server-specific noweb resolution. Blocks matching TAG get renamed to #+name: FOO. @@ -318,16 +507,22 @@ Otherwise include only the current heading." "Tangle each entry of SPEC from SOURCE-BUF. SPEC is a list of plists as built by `devops--tangle-spec'. Return a list of (TAG TARGET N) results. Free of interaction and messaging, so it can be -driven noninteractively (e.g. from a pod or a test)." - (let ((results nil)) - (dolist (entry spec) - (let* ((tag (plist-get entry :tag)) - (target (plist-get entry :target)) - (heading-pos (plist-get entry :heading-pos)) - (n (devops--tangle-heading source-buf heading-pos tag target))) - (when n - (push (list tag target n) results)))) - (nreverse results))) +driven noninteractively (e.g. from a pod or a test). + +Runs under `devops-with-sync': a noweb reference that executes a block +must resolve to the block's output, and under `:async' it would resolve +to a UUID placeholder — which is then what gets written to the file on +the server." + (devops-with-sync + (let ((results nil)) + (dolist (entry spec) + (let* ((tag (plist-get entry :tag)) + (target (plist-get entry :target)) + (heading-pos (plist-get entry :heading-pos)) + (n (devops--tangle-heading source-buf heading-pos tag target))) + (when n + (push (list tag target n) results)))) + (nreverse results)))) (defun devops--tangle-report (results) "Format RESULTS from `devops--tangle-spec-execute' as a status string." From e01553390bbc73190cd77e1cd932c4fafd564833 Mon Sep 17 00:00:00 2001 From: Kyle S Passarelli Date: Sun, 16 Aug 2026 18:50:37 -0600 Subject: [PATCH 2/5] Try to fix CI for emacs 29.4 --- README.md | 37 +++++++++++++++++---------- decisions/02-async-sessions.org | 32 +++--------------------- devops-drift.el | 7 +++++- devops-lob.el | 30 +++++++++++++++++++++- devops-test.el | 38 ++++++++++++++++++++++++++++ devops.el | 44 ++++++++++++++++++++++++++++++--- 6 files changed, 141 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index 7ae03ec..c403414 100644 --- a/README.md +++ b/README.md @@ -278,16 +278,10 @@ command also pays connection latency, and an unexpected prompt (`sudo`, an ssh host key confirmation, `apt` asking a question) leaves emacs waiting on a process that will never finish on its own. -Org's answer is `:async` — but only inside a `:session`, since a sessionless -shell block has no comint buffer to attach a filter to. A target tag is already -a name for "which machine is this block talking to", so it can name the session -too: - -```elisp -(setq devops-enable-session-async t) -``` +Org's 9.7+ added an `:async` option for sessions. Emacs returns immediately with +a placeholder and the output replaces it when the command finishes. -With that on, a block under a target-tagged heading is executed as if you had +A block under a target-tagged heading is executed as if you had written the session and async headers yourself: ``` @@ -296,10 +290,27 @@ apt-get update #+end_src ``` -Emacs returns immediately with a placeholder and the output replaces it when -the command finishes. A command that asks a question no longer hangs emacs -either: the prompt waits in the session buffer, and `devops-goto-session` takes -you there to answer it. + + +a name for "which machine is this block talking to", so it can name the session +too: + +```elisp +(setq devops-enable-session-async t) +``` + +A command that asks a question no longer hangs emacs +either: the prompt waits in the session buffer, and +`devops-goto-session` takes you there to answer it. + + +Shell blocks need **Org 9.7 or newer**, `:async` — +Emacs 30.1 and later bundle it, Emacs 29 ships Org 9.6 and needs org from ELPA. +On an older org the option leaves shell blocks alone rather than putting them in +a session it cannot drive asynchronously, since a synchronous block in an unseen +comint buffer hangs worse than one without a session. + + This is off by default because it makes blocks stateful. `cd`, `export`, an activated virtualenv and `ssh-agent` now survive from one block to the next diff --git a/decisions/02-async-sessions.org b/decisions/02-async-sessions.org index bbe4279..9dd908b 100644 --- a/decisions/02-async-sessions.org +++ b/decisions/02-async-sessions.org @@ -1,4 +1,5 @@ -#+TITLE: Decision 2: Async execution in per-target sessions +#+TITLE: Decision 2: Async execution in ob-shell sessions +#+STATUS: Implemented #+DATE: 2026-07-30 #+LINK: ob-shell https://orgmode.org/worg/org-contrib/babel/languages/ob-doc-shell.html #+LINK: async-header https://orgmode.org/manual/Environment-of-a-Code-Block.html @@ -14,8 +15,9 @@ connection latency, and an unexpected prompt (=sudo=, an ssh host key confirmation, =apt= asking a question) leaves emacs waiting on a process that will never finish on its own. -Since Org 9.6 =ob-shell= supports the =:async= header argument, which +Since Org 9.7 =ob-shell= supports the =:async= header argument, which puts a placeholder in the buffer, and replaces it when the response is ready. +(=ob-python= has had it since 9.6) The implementation works as follows: @@ -208,29 +210,3 @@ target. Session-per-tag makes running both concurrently a natural next step, one session each, results merged. Keeping the prompt for now; the naming scheme is chosen so this does not need to be revisited. -* Implementation notes - -Two things the decisions above did not anticipate. - -** =:session none= is org's default, not an opt-out signal - -=org-babel-default-header-args= contains =(:session . "none")=, so /every/ -block arrives at the advice already carrying =:session none=. Reading the -merged header arguments cannot tell a block that opted out of sessions from one -that never mentioned them, which is what decision 4 assumed. - -=devops--user-header-args= resolves it: re-read the block with -=org-babel-default-header-args= and its language-specific counterpart bound to -nil, and whatever =:session= survives was written by the user. Any value other -than ="none"= needs no such check — no default produces one. - -Where the block cannot be re-read, "declared" is the safe answer. A =#+call:= -line executes with an =INFO= built from the Library of Babel while point is not -on a src block, so it gets no session: losing async on call lines is a smaller -error than attaching a block to shared state it never asked for. - -** =:async yes=, not =:async t= - -=ob-shell= and =ob-python= both declare the header as =(async . ((yes no)))=, -and =org-babel-comint-use-async= only tests against ="no"=. Injecting ="yes"= -matches what the languages document; a hand-written =:async t= still works. diff --git a/devops-drift.el b/devops-drift.el index 4bc7510..5fb7751 100644 --- a/devops-drift.el +++ b/devops-drift.el @@ -1,7 +1,10 @@ -;; devops-drift.el - Drift detection for devops.el -*- lexical-binding: t; -*- +;;; devops-drift.el --- Drift detection for devops.el -*- lexical-binding: t; -*- ;; Copyright (C) 2026 Kyle S Passarelli +;; Author: Kyle S Passarelli +;; URL: https://github.com/kpassapk/devops.el + ;; This package is free software; you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation; either version 3, or (at your option) @@ -546,3 +549,5 @@ With prefix ARG, check every target-tagged heading in the buffer." (pop-to-buffer buf))) (provide 'devops-drift) + +;;; devops-drift.el ends here diff --git a/devops-lob.el b/devops-lob.el index e9b2400..fba5fd0 100644 --- a/devops-lob.el +++ b/devops-lob.el @@ -1,4 +1,30 @@ -;; devops-lob.el -- Per-project tools.org Library-of-Babel management -*- lexical-binding: t; -*- +;;; devops-lob.el --- Per-project Library-of-Babel management -*- lexical-binding: t; -*- + +;; Copyright (C) 2026 Kyle S Passarelli + +;; Author: Kyle S Passarelli +;; URL: https://github.com/kpassapk/devops.el + +;; This package is free software; you can redistribute it and/or modify +;; it under the terms of the GNU General Public License as published by +;; the Free Software Foundation; either version 3, or (at your option) +;; any later version. + +;; This package is distributed in the hope that it will be useful, +;; but WITHOUT ANY WARRANTY; without even the implied warranty of +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;; GNU General Public License for more details. + +;; You should have received a copy of the GNU General Public License +;; along with GNU Emacs. If not, see . + +;;; Commentary: +;; +;; Load a project's `tools.org' into the Library of Babel, so its named +;; src blocks are callable from any org file in that project, and unload +;; them again when the project is left. + +;;; Code: (require 'ob-lob) (require 'project) @@ -100,3 +126,5 @@ Skips TRAMP remote paths." (remove-hook 'find-file-hook #'devops--lob-maybe-load-on-find-file))) (provide 'devops-lob) + +;;; devops-lob.el ends here diff --git a/devops-test.el b/devops-test.el index 94742fb..6fe2e50 100644 --- a/devops-test.el +++ b/devops-test.el @@ -675,6 +675,17 @@ Return the header arguments the executor was handed, so an injected (re-search-forward "begin_src") ,@body)) +(defun devops-test--skip-unless-shell-async () + "Skip the calling test unless org can run a shell block asynchronously. +`ob-shell' gained `:async' in Org 9.7. Under the Org 9.6 that Emacs 29 +bundles, `devops--lang-async-p' turns the injection off, so there is no +session and no placeholder to assert on. + +`ert-skip' rather than `skip-unless': the latter is gone on Emacs 31 and +its replacement `ert-skip-unless' does not exist on 29." + (unless (devops--lang-async-p "sh") + (ert-skip "ob-shell has no :async support (needs Org 9.7)"))) + (ert-deftest devops-session-async-off-by-default-test () "Without `devops-enable-session-async', no session or async is injected." (devops-test--with-session-org "" @@ -685,6 +696,7 @@ Return the header arguments the executor was handed, so an injected (ert-deftest devops-session-async-injects-session-and-async-test () "With the option on, a block gets `:session devops:TAG' and `:async yes'." + (devops-test--skip-unless-shell-async) (let ((devops-enable-session-async t)) (devops-test--with-session-org "" (let ((params (devops-test--executor-params "sh"))) @@ -692,8 +704,31 @@ Return the header arguments the executor was handed, so an injected (should (equal (cdr (assq :session params)) "devops:local")) (should (equal (cdr (assq :async params)) "yes")))))) +(ert-deftest devops-session-async-unsupported-language-test () + "An org whose `ob-shell' has no `:async' gets neither session nor async. +Injecting `:session' alone would leave the block running synchronously +inside a comint buffer the user never sees, so a prompt would hang emacs +worse than it does with no session at all." + (let ((devops-enable-session-async t)) + (cl-letf (((symbol-function 'devops--lang-async-p) #'ignore)) + (devops-test--with-session-org "" + (let ((params (devops-test--executor-params "sh"))) + (should (equal (cdr (assq :dir params)) "/srv/app/")) + (should (equal (cdr (assq :session params)) "none")) + (should-not (assq :async params))))))) + +(ert-deftest devops--lang-async-p-test () + "The probe answers for shell from `ob-shell', and yes for anything else." + (should (eq (devops--lang-async-p "sh") (devops--lang-async-p "bash"))) + (should (devops--lang-async-p "python")) + (should (equal (devops--lang-async-p "sh") + (and (require 'ob-shell nil t) + (boundp 'ob-shell-async-indicator) + t)))) + (ert-deftest devops-session-name-function-test () "`devops-session-name-function' decides the session name." + (devops-test--skip-unless-shell-async) (let ((devops-enable-session-async t) (devops-session-name-function (lambda (tag target) (format "%s@%s" tag target)))) @@ -703,6 +738,7 @@ Return the header arguments the executor was handed, so an injected (ert-deftest devops-session-async-explicit-session-wins-test () "A `:session' on the block is not overwritten by the tag's session." + (devops-test--skip-unless-shell-async) (let ((devops-enable-session-async t)) (devops-test--with-session-org ":session other" (let ((params (devops-test--executor-params "sh"))) @@ -719,6 +755,7 @@ Return the header arguments the executor was handed, so an injected (ert-deftest devops-session-async-explicit-async-no-test () "`:async no' runs one block synchronously while keeping its session." + (devops-test--skip-unless-shell-async) (let ((devops-enable-session-async t)) (devops-test--with-session-org ":async no" (let ((params (devops-test--executor-params "sh"))) @@ -806,6 +843,7 @@ land in the tangled file on the server." The end-to-end path: the heading's tag names a shell session, the block runs there at the target's directory, and `org-babel-comint-async-filter' replaces the placeholder in the buffer when the command finishes." + (devops-test--skip-unless-shell-async) (let ((devops-enable-session-async t)) (devops-test--with-local-target target (unwind-protect diff --git a/devops.el b/devops.el index f014b33..cd3ca40 100644 --- a/devops.el +++ b/devops.el @@ -1,7 +1,14 @@ -;; devops.el - Development target -*- lexical-binding: t; -*- +;;; devops.el --- Infrastructure as an org file -*- lexical-binding: t; -*- ;; Copyright (C) 2026 Kyle S Passarelli +;; Author: Kyle S Passarelli +;; Maintainer: Kyle S Passarelli +;; URL: https://github.com/kpassapk/devops.el +;; Version: 0.1.0 +;; Package-Requires: ((emacs "29.1")) +;; Keywords: tools, processes, outlines + ;; This package is free software; you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation; either version 3, or (at your option) @@ -19,6 +26,14 @@ ;; ;; `devops.el' offers utilities for running commands on local and remote ;; machines using org mode. +;; +;; The package itself needs nothing newer than the org bundled with Emacs +;; 29. `devops-enable-session-async' is the exception: running shell +;; blocks asynchronously needs the `:async' support `ob-shell' gained in +;; Org 9.7 (Emacs 30.1), and turns itself off under an older org rather +;; than putting blocks in a session it cannot drive. Hence no (org "9.7") +;; in Package-Requires: an optional feature should not force everyone to +;; replace their built-in org. ;;; Code: @@ -50,7 +65,11 @@ an activated virtualenv and `ssh-agent' survive from one block to the next, which is useful but costs idempotency. A block that passed in a dirty session may fail in a fresh one, so prefer blocks that do not depend on the ones above them, and use `devops-restart-session' to get -back to a known state." +back to a known state. + +Shell blocks need Org 9.7 or newer, where `ob-shell' learned `:async'; +under an older org this option leaves them alone rather than putting +them in a session it cannot run asynchronously." :type 'boolean :group 'devops) @@ -262,12 +281,26 @@ worth avoiding." (let ((own (devops--user-header-args lang))) (or (null own) (assq :session own))))))) +(defun devops--lang-async-p (lang) + "Non-nil when org can evaluate LANG asynchronously in a session. +`ob-shell' gained `:async' in Org 9.7. On older org the header argument +is not merely unsupported but silently ignored, and the block runs +synchronously in the comint session — worse than no session at all, +since a command that asks a question then blocks emacs inside a buffer +the user never sees. Probing the feature rather than the org version +also keeps Emacs 29 working once org is upgraded from ELPA." + (if (member lang '("sh" "bash" "shell")) + (and (require 'ob-shell nil t) + (boundp 'ob-shell-async-indicator)) + t)) + (defun devops--async-session-cells (params block-params lang tag target) "Return the :session and :async header cells to inject, or nil. PARAMS and BLOCK-PARAMS are as in `devops--header-cell', LANG is the block's language, and TAG and TARGET the heading's resolved target. Nothing is injected unless `devops-enable-session-async' is on, LANG is -in `devops-async-session-languages', and we are executing on the user's +in `devops-async-session-languages' and supported by the running org +\(see `devops--lang-async-p'), and we are executing on the user's behalf rather than under `devops-with-sync'. What the block already says is left alone, so per-block escape hatches @@ -276,7 +309,8 @@ heading's session, `:session none' gives it neither a session nor async, and `:session other' attaches it to a session of the user's choosing." (when (and devops-enable-session-async (not devops--inhibit-async) - (member lang devops-async-session-languages)) + (member lang devops-async-session-languages) + (devops--lang-async-p lang)) (let* ((declared (devops--session-declared-p params block-params lang)) (session (cdr (devops--header-cell :session params block-params)))) (unless (and declared (equal session "none")) @@ -726,3 +760,5 @@ In a src block, if the : copies body to clipboard and exports :var env vars." (devops--open-terminal-at-dir dir env-vars))) (provide 'devops) + +;;; devops.el ends here From 0639ac0a2f26d9a4a4ee6ebc1ad7d347cf091408 Mon Sep 17 00:00:00 2001 From: Kyle S Passarelli Date: Sun, 16 Aug 2026 20:00:06 -0600 Subject: [PATCH 3/5] More updates to README --- README.md | 204 +++++++++------------------------------- examples/1_commands.org | 1 + 2 files changed, 43 insertions(+), 162 deletions(-) diff --git a/README.md b/README.md index c403414..a011734 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # devops.el: Infrastructure as an org file -By following some conventions, this package helps you to manage infrastructure as a set of org files. Infrastructure here may be servers, containers, serverless functions, DNS... up to you. +By following some conventions, this package helps you to manage infrastructure with org mode. Infrastructure here may be servers, containers, serverless functions, DNS... up to you. ## Installation @@ -159,17 +159,18 @@ Given | `/etc/foo.txt` | `/ssh:example1.com:/etc/foo.txt` | | `~/foo.txt` | `/ssh:example1.com:~/foo.txt` | -## Disabling with :target nil +### Disabling with :target nil -Sometimes you may want to "turn off" the target for a single block. +Sometimes you may want to "turn off" the target for a single block. This is useful for locally +processing the output of the code block. -As an example, let's say we have a container `TARGET`: +For example, let's say we have a `TARGET` that points to a Podman container in a server: ``` #+TARGET: /ssh:server.com|podman:my-container: (container) ``` -We can filter the results locally by adding `target: nil` to a second block: +We can chain together a remote command and a local processing step as follows: ``` * Service status :container: @@ -191,15 +192,8 @@ This will work even if the `jq` command is not installed in the container :) ## Drift detection -Tangling pushes the org file out to its targets. `devops-drift` asks the -opposite question: does what is on the target still match what the org file -says? The heading is tangled to a local temp directory first — so noweb, including -per-server noweb, resolves exactly as a real tangle would — and each tangled file -is compared byte-for-byte with its counterpart on the target. Nothing is written -to a target: a drift check is read-only. - -`devops-drift` (`C-u` for the whole buffer) shows a `*Drift Report*` buffer, one -row per file: +`devops-drift` shows a `*Drift Report*` buffer, telling you whtether code blocks and their +tangle targets have identical content. ``` ok server1 /ssh:example1.com:~/foo.txt @@ -207,6 +201,8 @@ row per file: MISSING server2 /ssh:example2.com:~/app.conf ``` +The following keys are active in the diff report buffer: + | Key | Action | |-----------|-------------------------------------------| | `RET` | jump to the source block behind the row | @@ -214,155 +210,63 @@ row per file: | `d` | diff them | | `g` | re-run the check | -### Checking drift from a source block +There are also noninteractive variants - `devops-drift-{all|headline|custom-id}` +for scripting. -The report is a buffer you read and then lose. A check written into the org file -stays with the notes explaining it, and its result is part of the document: +## Long runnign commands -```org -#+begin_src emacs-lisp :results output -(princ (devops-drift-summary (devops-drift-headline "servers/box.org" "Caddy"))) -#+end_src +There are a few options for running background commands in emacs: -#+RESULTS: -: DRIFT server1 /ssh:example1.com:~/caddy_etc/Caddyfile -: -: --- /ssh:example1.com:~/caddy_etc/Caddyfile -: +++ ~/caddy_etc/Caddyfile (server1) -: @@ -78,6 +78,11 @@ -: ... -``` +1. [ob-async](https://github.com/astahlman/ob-async) +2. [ob-screen](https://howardism.org/Technical/Emacs/literate-devops.html#fnr.4) +3. [detached.el](https://sr.ht/~niklaseklund/detached.el) -| Function | Checks | -|---------------------------------------|-------------------------------------------------| -| `(devops-drift-all source)` | every target-tagged heading | -| `(devops-drift-headline source title)` | the subtree titled `title` | -| `(devops-drift-custom-id source id)` | the subtree whose `CUSTOM_ID` is `id` | - -`source` is an org buffer or a file name. All three return the same data: one -alist per tangled file, and no temp directory left behind. - -| Key | Value | -|-----------|--------------------------------------------------------------| -| `:status` | `:same`, `:drift`, `:missing` (no file there) or `:error` | -| `:tag` | the target tag this file was tangled for | -| `:path` | the block's `:tangle` value | -| `:remote` | where that path lands on the target | -| `:target` | the `#+TARGET` value | -| `:detail` | the error message, for `:error` | -| `:diff` | unified diff, target first, for `:drift` | - -`devops-drift-summary` formats that list as the text above, -`devops-drift-table` as an org table (for `:results table`), and -`devops-drift-ok-p` reduces it to a boolean — nil for an empty list, since a -check that compared nothing has shown nothing. - -### cljbang.el - -If you use [cljbang.el](https://github.com/borkdude/cljbang.el), the drift API returns maps: - -```clojure -(require '[devops-drift :as drift]) - -(->> (drift/all "servers/box.org") - (remove #(= (:status %) :same)) - (map (fn [{:keys [path status]}] [path status]))) -;; => (["app.conf" :drift]) -``` +To avoid any dependencies, -## Long-running commands +### Shell async sessions -Long commands such as `apt-get update` can lock up emacs. By default a block -runs in a synchronous subprocess and emacs waits; on a remote target every -command also pays connection latency, and an unexpected prompt (`sudo`, an ssh -host key confirmation, `apt` asking a question) leaves emacs waiting on a -process that will never finish on its own. +In Org's 9.7+, `ob-shell` sessions have an `async` option. -Org's 9.7+ added an `:async` option for sessions. Emacs returns immediately with -a placeholder and the output replaces it when the command finishes. +Executing source blocks with this option enabled will print a placeholder. +Once the background command finishes, the placehodler gets replaced with the output. -A block under a target-tagged heading is executed as if you had -written the session and async headers yourself: +When `devops-enable-session-async` is enabled, blocks are executed as if you had written +the `session` and `async` headers yourself. For example, ``` -#+begin_src sh :dir /ssh:example.com: :session devops:example :async yes -apt-get update +#+TARGET: /ssh:example.com: (example) + +* Update packages :example: + +#+begin_src sh :results output + apt-get update #+end_src ``` +injects these `dir`, `session` and `async` headers: +``` +#+begin_src sh :results output :dir /ssh:example.com: :session devops:example :async yes + apt-get update +#+end_src +``` -a name for "which machine is this block talking to", so it can name the session -too: +(Only works iwth `:results output`. You will probably want to set this at top of file. +See [](examples/1_commands.org)) + +This is off by default because it makes blocks stateful. Commands like `cd` now survive +from one block to the next. To enable, set ```elisp (setq devops-enable-session-async t) ``` -A command that asks a question no longer hangs emacs -either: the prompt waits in the session buffer, and -`devops-goto-session` takes you there to answer it. - - -Shell blocks need **Org 9.7 or newer**, `:async` — -Emacs 30.1 and later bundle it, Emacs 29 ships Org 9.6 and needs org from ELPA. -On an older org the option leaves shell blocks alone rather than putting them in -a session it cannot drive asynchronously, since a synchronous block in an unseen -comint buffer hangs worse than one without a session. - - - -This is off by default because it makes blocks stateful. `cd`, `export`, an -activated virtualenv and `ssh-agent` now survive from one block to the next -under the same heading, which is useful, but it costs idempotency: blocks become -order-dependent, and a block that passed in a dirty session may fail in a fresh -one. Prefer blocks that do not depend on the ones above them, and use -`devops-restart-session` to get back to a clean shell. - -Two tags on the same host get two sessions, because two tags mean two -directories — a shell session's working directory is fixed when its buffer is -created, and later blocks do not `cd` to their `:dir`. - -| Command | What it does | -|--------------------------|--------------------------------------------------| -| `devops-goto-session` | Pop to the heading's session buffer | -| `devops-restart-session` | Kill it, so the next block starts a fresh shell | - -### Per-block escape hatches - -Whatever the block itself says wins: - -| Header | Effect | -|-----------------|-----------------------------------------------------------| -| `:async no` | Run this one block synchronously, in the same session | -| `:session none` | No session, no async, no state shared with other blocks | -| `:session foo` | Attach to a session of your choosing | -| `:target nil` | No target, and so no session either | - -A block that names its own `:dir` is running somewhere devops did not choose, -so it gets no session either. - -Only languages in `devops-async-session-languages` (`sh`, `bash`, `shell`, -`python`) are affected. `:session` is not a neutral header argument — for -`emacs-lisp` it means an ielm buffer, and for the non-executable blocks that -carry `:tangle` it is meaningless — so everything else keeps getting `:dir` and -nothing more. - -Tangling and drift checks stay synchronous, because they need real values: a -noweb reference that executes a block (`<>`) would otherwise resolve -to the placeholder, and the placeholder is what would get written to the file on -the server. Anything else that reads the return value of -`org-babel-execute-src-block` should wrap it in `devops-with-sync`. - Session names come from `devops-session-name-function`, which defaults to `devops:`. A session name is a buffer name in a single global namespace, so if two org files use the same tag for different hosts, set this to a function that also folds in the target or the buffer name. -If you would rather not use sessions at all, there is also -[ob-async](https://github.com/astahlman/ob-async), and probably others. YMMV. - -[ob-screen]: https://howardism.org/Technical/Emacs/literate-devops.html#fnr.4 +### Terminal DWIM command I like using a separate terminal to run most commands, instead of emacs. This package provides a `devops-open-terminal-dwim` command, which opens the current source block in a terminal. (Only `ghostty` supported at the moment, but more terminals planned.) @@ -386,30 +290,6 @@ Any project with a `tools.org` at its root can expose named org-babel blocks as With `devops-lob-auto-mode` enabled, opening any file in a project that has `tools.org` automatically loads its named blocks into the org-babel Library of Babel. -### tools.org format - -```org -#+title: Tools - -#+name: deploy -#+begin_src sh :var env="staging" -./deploy.sh $env -#+end_src - -#+name: health-check -#+begin_src sh :var host="localhost" -curl -sf http://$host/health -#+end_src -``` - -Call tools from any org buffer: - -```org -#+call: deploy(env="production") - -#+call: health-check(host="app.example.com") -``` - ### tools.org commands | Command | Description | diff --git a/examples/1_commands.org b/examples/1_commands.org index b0c52ea..146006d 100644 --- a/examples/1_commands.org +++ b/examples/1_commands.org @@ -1,3 +1,4 @@ +#+PROPERTY: header-args:sh :results output #+TARGET: /ssh:example1.com: (server1) #+TARGET: /ssh:example2.com: (server2) #+TITLE: Running server commands From 40fd79133acb785e6ae9a97577bea742c1f056df Mon Sep 17 00:00:00 2001 From: Kyle S Passarelli Date: Sun, 16 Aug 2026 20:14:30 -0600 Subject: [PATCH 4/5] [skip ci] Update README --- README.md | 20 ++++++++++---------- examples/1_commands.org | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index a011734..92f7520 100644 --- a/README.md +++ b/README.md @@ -213,22 +213,20 @@ The following keys are active in the diff report buffer: There are also noninteractive variants - `devops-drift-{all|headline|custom-id}` for scripting. -## Long runnign commands +## Long running commands -There are a few options for running background commands in emacs: +There are seveeral options for running background commands in emacs asynchronously: 1. [ob-async](https://github.com/astahlman/ob-async) 2. [ob-screen](https://howardism.org/Technical/Emacs/literate-devops.html#fnr.4) 3. [detached.el](https://sr.ht/~niklaseklund/detached.el) -To avoid any dependencies, +To avoid dependnecies, `devops.el` uses built-in features and tries to make them more +convenient. ### Shell async sessions -In Org's 9.7+, `ob-shell` sessions have an `async` option. - -Executing source blocks with this option enabled will print a placeholder. -Once the background command finishes, the placehodler gets replaced with the output. +In recent org mode versions, (Org 9.7+), executing source blocks with `:session foo :async yes` will print a placeholder. Once the background command finishes, the placehodler gets replaced with the output. When `devops-enable-session-async` is enabled, blocks are executed as if you had written the `session` and `async` headers yourself. For example, @@ -268,10 +266,12 @@ that also folds in the target or the buffer name. ### Terminal DWIM command -I like using a separate terminal to run most commands, instead of emacs. This package provides a `devops-open-terminal-dwim` command, which opens the current source block in a terminal. (Only `ghostty` supported at the moment, but more terminals planned.) +I often like using a separate terminal to run most commands, instead of emacs. +This package provides a `devops-open-terminal-dwim` command, which opens the current source block in a terminal. (Only `ghostty` supported at the moment, but more terminals planned.) -Any `var` references become environment variables loaded into the (usually remote) remote shell. The source block content is copied to the clipboard, so you can do `devops-open-terminal-dwim`, then -paste, and you will be running the command at the correct location. +Any `var` references become environment variables loaded into the (usually remote) +shell. The source block content is copied to the clipboard, so you can do +`devops-open-terminal-dwim`, then paste, and you will be running the command at the correct location. ## devops-lob diff --git a/examples/1_commands.org b/examples/1_commands.org index 146006d..5464fed 100644 --- a/examples/1_commands.org +++ b/examples/1_commands.org @@ -32,7 +32,7 @@ With =devops.el=, we can run the same command by adding a tag to a parent headin * Try it Instructions: -1. Modify the server blocks at the top of file (see caveats) +1. Modify the server blocks at the top of file 2. Run =devops-set-header-args-from-tag= 3. You should see header-args set as follows From 1eb056edaa61a71aa62f1c66b54bc99d3228b692 Mon Sep 17 00:00:00 2001 From: Kyle S Passarelli Date: Sun, 16 Aug 2026 20:29:10 -0600 Subject: [PATCH 5/5] [skip ci] README --- README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 92f7520..9feaf82 100644 --- a/README.md +++ b/README.md @@ -59,14 +59,16 @@ There are some shortcomings and annoyances, however: 1. Long-running commands (like `apt-get update`) can lock up emacs for an extended period of time. In devops workflows, most of the work is remote, so the experience is... choppy. Even worse, if a command asks for input your emacs might become unresponsive. -2. When describing an actual production environment, it's easy to end up with duplicate `/ssh:someuser@someserver:somedirectory/...` `:dir` properties all over the file. This is extremely difficult to scan. +2. When describing an actual production environment, it's easy to end up with duplicate `/ssh:someuser@someserver:somedirectory/...` `:dir` properties all over the file. This is difficult to scan. 3. Each source block can only have a single `:dir`. This makes the following typical use cases difficult: - - Uploading the same content on multiple servers + - Uploading the same content to multiple servers - Running the same command on multiple servers -4. Tangling socpe is either too small or to wide. The `org-babel-tangle` function tangles the entire buffer by default, or alternatively a single source code block. Tangling an entire buffer might be risky, and tangling a single block gets very annoying. +4. Tangling ignores `:dir`. If you are uploading a file and then running a server command, now the server needs to go in two places. (`:dir` and `:tangle`) + +5. tangling scope is either too small or to wide. The `org-babel-tangle` function tangles the entire buffer by default, or alternatively a single source code block. Tangling an entire buffer might be risky, and tangling a single block gets very annoying. This library provides functionality to better support devops-like workflows. It does this by applying some conventions on top of org mode. @@ -249,7 +251,7 @@ injects these `dir`, `session` and `async` headers: #+end_src ``` -(Only works iwth `:results output`. You will probably want to set this at top of file. +(Only works with `:results output` I think. You will probably want to set this at top of file. See [](examples/1_commands.org)) This is off by default because it makes blocks stateful. Commands like `cd` now survive @@ -262,7 +264,7 @@ from one block to the next. To enable, set Session names come from `devops-session-name-function`, which defaults to `devops:`. A session name is a buffer name in a single global namespace, so if two org files use the same tag for different hosts, set this to a function -that also folds in the target or the buffer name. +that also folds in the project, target or buffer name. ### Terminal DWIM command