`clevis-decrypt-sss` fails to kill all child processes (and their respective children)

[m-ueberall]

For an example, see olastor/clevis-pin-fido2#3 (however, this is not a problem specific to this pin; clevis-decrypt-sss should always ensure that no (grand)child processes are left behind).
As demonstrated below (using two terminals), threshold t=1 is reached and the decryption succeeds, but one of the FIDO2 keys is still being queried until the key-specific timeout kicks in while clevis-decrypt-sss already terminated:

[2024-04-12T13:16:01+0200] root@ubuntu:/tmp# echo "Hello, world." | clevis encrypt sss '{"t": 1, "pins": {"fido2": [{"device": "/dev/input/by-id/yubikey_12345678"}, {"device": "/dev/input/by-id/yubikey_87654321"}]}}' >test01.jwe
clevis-encrypt-fido2: Please insert your specified FIDO2 token /dev/input/by-id/yubikey_12345678
clevis-encrypt-fido2: Please insert your specified FIDO2 token /dev/input/by-id/yubikey_87654321
[…]
[2024-04-12T13:17:31+0200] root@ubuntu:/tmp# clevis decrypt < test01.jwe
Hello, world.
[2024-04-12T13:17:55+0200] root@ubuntu:/tmp# fido2-assert: fido_dev_get_assert: FIDO_ERR_ACTION_TIMEOUT

[2024-04-12T13:17:33+0200] root@ubuntu:/tmp# ps axu | grep -E 'clevis|fido2'
root     1183736  0.0  0.0  18680  3580 pts/7    S+   13:17   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     1183741  0.0  0.0   7800  5084 pts/7    S+   13:17   0:00 /usr/bin/clevis-decrypt-sss
root     1183742  0.0  0.0  18680  3588 pts/7    S+   13:17   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     1183743  0.0  0.0  18680  3492 pts/7    S+   13:17   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     1183752  0.0  0.0  18680  3604 pts/7    S+   13:17   0:00 /bin/bash /usr/bin/clevis-decrypt-fido2
root     1183753  0.0  0.0  18680  3584 pts/7    S+   13:17   0:00 /bin/bash /usr/bin/clevis-decrypt-fido2
root     1183802  0.0  0.0  18680  1716 pts/7    S+   13:17   0:00 /bin/bash /usr/bin/clevis-decrypt-fido2
root     1183804  0.3  0.0   8072  5668 pts/7    S+   13:17   0:00 fido2-assert -G -h -t up=true -t pin=false /dev/input/by-id/yubikey_12345678
root     1183808  0.0  0.0  18680  1640 pts/7    S+   13:17   0:00 /bin/bash /usr/bin/clevis-decrypt-fido2
root     1183811  0.3  0.0   8072  5788 pts/7    S+   13:17   0:00 fido2-assert -G -h -t up=true -t pin=false /dev/input/by-id/yubikey_87654321
root     1183819  0.0  0.0  18040  2316 pts/5    S+   13:17   0:00 grep --color=auto -E clevis|fido2
[2024-04-12T13:17:46+0200] root@ubuntu:/tmp# pstree -p 1183741
clevis-decrypt-(1183741)─┬─clevis-decrypt(1183742)───clevis-decrypt-(1183752)───clevis-decrypt-(1183802)─┬─base64(1183807)
                         │                                                                               ├─fido2-assert(1183804)
                         │                                                                               ├─head(1183805)
                         │                                                                               ├─jose(1183809)
                         │                                                                               └─tail(1183806)
                         └─clevis-decrypt(1183743)───clevis-decrypt-(1183753)───clevis-decrypt-(1183808)─┬─base64(1183814)
                                                                                                         ├─fido2-assert(1183811)
                                                                                                         ├─head(1183812)
                                                                                                         ├─jose(1183815)
                                                                                                         └─tail(1183813)
[2024-04-12T13:17:52+0200] root@ubuntu:/tmp# ps axu | grep -E 'clevis|fido2'
root     1183753  0.0  0.0  18680  3584 pts/7    S    13:17   0:00 /bin/bash /usr/bin/clevis-decrypt-fido2
root     1183808  0.0  0.0  18680  1640 pts/7    S    13:17   0:00 /bin/bash /usr/bin/clevis-decrypt-fido2
root     1183811  0.0  0.0   8072  5788 pts/7    S    13:17   0:00 fido2-assert -G -h -t up=true -t pin=false /dev/input/by-id/yubikey_87654321
root     1183865  0.0  0.0  18040  2304 pts/5    S+   13:18   0:00 grep --color=auto -E clevis|fido2
[2024-04-12T13:18:03+0200] root@ubuntu:/tmp# pstree -c 1183753
clevis-decrypt-───clevis-decrypt-─┬─base64
                                  ├─fido2-assert
                                  ├─head
                                  ├─jose
                                  └─tail
[2024-04-12T13:18:09+0200] root@ubuntu:/tmp# 

[sarroutbi]

Hello @m-ueberall . Could you reproduce this issue with "no FIDO2" pin? I was not able to do so ... but I might be missing something in my scenario

[m-ueberall]

@sarroutbi : You should be able to reproduce the above with the tang pin contained in this repository as follows:

[2024-04-12T22:31:53+0200] root@ubuntu:/tmp# echo "Hello, world." | clevis encrypt sss '{"t": 1, "pins": {"tang": [{"url": "http://example.org:18888"}, {"url": "http://example.org:18889"}]}}' >test03.jwe
The advertisement contains the following signing keys:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX1

Do you wish to trust these keys? [ynYN] Y
The advertisement contains the following signing keys:

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2

Do you wish to trust these keys? [ynYN] Y
[2024-04-12T22:32:42+0200] root@ubuntu:/tmp#
[2024-04-12T22:34:58+0200] root@ubuntu:/tmp# clevis decrypt < test03.jwe
Hello, world.

The first run above worked as expected. Let's create a situation where one of the Tang servers is unable to respond in a timely manner, i.e., before the decryption has taken place using the other pin/server. To keep things as simple as possible, we just silently prevent it from receiving the request at all:

[2024-04-12T22:35:49+0200] root@example.org:/tmp# iptables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
[2024-04-12T22:36:32+0200] root@example.org:/tmp# ip6tables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
[2024-04-12T22:36:36+0200] root@example.org:/tmp#

Now we rerun the test (no pstree output this time, but it should be obvious that all the processes shown below are related to our second run on the client); the only challenge here is to select the correct value for the first call to sleep if you want to do this in a single terminal, because everything usually happens very fast if both client and servers are in the same local network:

[2024-04-12T22:41:13+0200] root@ubuntu:/tmp# clevis decrypt < test03.jwe & sleep 0.25; ps axu | grep -E 'clevis|curl'; sleep 5; echo "----"; ps axu | grep -E 'clevis|curl'
[1] 1244745
root     1244745  0.0  0.0   4416  3168 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     1244751  0.0  0.0   7800  5008 pts/5    S    22:41   0:00 /usr/bin/clevis-decrypt-sss
root     1244752  0.0  0.0   4416  3052 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     1244753  0.0  0.0   4416  3200 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     1244762  0.0  0.0   4416  3132 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     1244763  0.0  0.0   4416  3140 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     1244802  0.0  0.0   4416  1876 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     1244803  0.0  0.0   4416  1820 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     1244804  0.0  0.0  96720  9164 pts/5    S    22:41   0:00 curl -sfg -X POST -H Content-Type: application/jwk+json --data-binary @- http://example.org:18889/rec/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2
root     1244805  0.0  0.0  96720  9316 pts/5    S    22:41   0:00 curl -sfg -X POST -H Content-Type: application/jwk+json --data-binary @- http://example.org:18888/rec/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX1
root     1244809  0.0  0.0   3624  1620 pts/5    S+   22:41   0:00 grep --color=auto -E clevis|curl
Hello, world.
[1]+  Done                    clevis decrypt < test03.jwe
----
root     1244763  0.0  0.0   4416  3140 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     1244803  0.0  0.0   4416  1820 pts/5    S    22:41   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     1244805  0.0  0.0  96720  9316 pts/5    S    22:41   0:00 curl -sfg -X POST -H Content-Type: application/jwk+json --data-binary @- http://example.org:18888/rec/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX1
root     1244824  0.0  0.0   3624  1688 pts/5    S+   22:41   0:00 grep --color=auto -E clevis|curl
[2024-04-12T22:41:20+0200] root@ubuntu:/tmp# Error communicating with server http://example.org:18888

[2024-04-12T22:43:29+0200] root@ubuntu:/tmp#

In the next to last line above, I simply hit 'Enter' to get a current timestamp; you can see that it took more than 120 seconds for the remaining clevis-decrypt-tang instance and its children to terminate on its own instead of having been killed by clevis-decrypt[-sss].

Don't forget to undo the artificially introduced problem:

[2024-04-12T22:49:02+0200] root@example.org:/tmp# iptables -D INPUT 1
[2024-04-12T22:49:12+0200] root@example.org:/tmp# ip6tables -D INPUT 1
[2024-04-12T22:49:16+0200] root@example.org:/tmp#

[sergio-correia]

@m-ueberall: Could you check if #539 resolves the issue for you, please?

[m-ueberall]

@sergio-correia : I checked #539, but the proposed change currently still seems to contain a logic error:

(Commit/patch fe5cc93 applies cleanly to the current Debian/Ubuntu clevis source, but to be sure, I cloned this repository, explicitly checked out said commit and followed the steps in INSTALL.md inside a build container based on Ubuntu 24.04.)

Instead of following the test strategy in sss-child-process-cleanup, I locally started two working tangd instances on ports 18888, 18889.

• using the "stock" Ubuntu clevis (v20-1), the following command worked as expected:

  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# echo "Hello, world." | clevis encrypt sss '{"t": 1, "pins": {"tang": [{"url": "http://127.0.0.1:18888"}, {"url": "http://127.0.0.1:18889"}]}}' >test03.jwe.run01
  The advertisement contains the following signing keys:
  
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxnZ1k
  
  Do you wish to trust these keys? [ynYN] Y
  The advertisement contains the following signing keys:
  
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxtfoc
  
  Do you wish to trust these keys? [ynYN] Y
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ls -l test03.jwe.run01 
  -rw-r--r-- 1 root root 3760 Dec 18 14:39 test03.jwe.run01
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build#
  

• using the version of clevis including the above patch (installed in /usr/local/bin after explicitly removing the previously installed package clevis and starting a new instance of bash), however, the above encryption process hung and had to be killed manually after a couple of minutes:

  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# echo "Hello, world." | clevis encrypt sss '{"t": 1, "pins": {"tang": [{"url": "http://127.0.0.1:18888"}, {"url": "http://127.0.0.1:18889"}]}}' >test03.jwe.run02
  The advertisement contains the following signing keys:
  
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxnZ1k
  
  Do you wish to trust these keys? [ynYN] Y
  ^C
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build#
  

I did not really monitor child processes, use different pin types, or take a closer look at the source, but it seems as if the above pull request #539 currently prevents clevis encrypt from properly receiving the answers from all pins which is required to produce the intended .jwe output.
(While clevis decrypt worked for both the "stock" and the patched clevis using the test03.jwe.run01 file against the two working tangd instances, I did not test further at the moment.)

[sergio-correia]

Thanks for testing, I will investigate.

[sergio-correia]

I updated it to kill the process group only on the decryption path; if you could give it another try, I would appreciate it.

[m-ueberall]

@sergio-correia: I retested the new commit c34dd2b (checked out/compiled as before, installed in /usr/local/bin/) and compared it to the "stock" clevis package (installed in /usr/bin), and now almost everything worked as expected with different thresholds, reachable/non-reachable tangd instance combinations. Excerpt from the test runs:

• Encryption:

  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# echo "Hello, world." | /usr/local/bin/clevis encrypt sss '{"t": 0, "pins": {"tang": [{"url": "http://127.0.0.1:18888"}, {"url": "http://127.0.0.>
  Invalid threshold (required: 1 <= 0 <= 2)!
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# echo "Hello, world." | /usr/local/bin/clevis encrypt sss '{"t": 1, "pins": {"tang": [{"url": "http://127.0.0.1:18888"}, {"url": "http://127.0.0.>
  The advertisement contains the following signing keys:
  
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxnZ1k
  
  Do you wish to trust these keys? [ynYN] Y
  The advertisement contains the following signing keys:
  
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxtfoc
  
  Do you wish to trust these keys? [ynYN] Y
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# echo "Hello, world." | /usr/local/bin/clevis encrypt sss '{"t": 2, "pins": {"tang": [{"url": "http://127.0.0.1:18888"}, {"url": "http://127.0.0.>
  The advertisement contains the following signing keys:
  
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxnZ1k
  
  Do you wish to trust these keys? [ynYN] Y
  The advertisement contains the following signing keys:
  
  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxtfoc
  
  Do you wish to trust these keys? [ynYN] Y
  #### we prevent one tangd instance from answering in a timely fashion; for threshold=2, this cannot work:
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# echo "Hello, world." | /usr/local/bin/clevis encrypt sss '{"t": 2, "pins": {"tang": [{"url": "http://127.0.0.1:18888"}, {"url": "http://127.0.0.>
  Unable to fetch advertisement: 'http://127.0.0.1:18888/adv/'!
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ls -l test03.jwe.run1*
  -rw-r--r-- 1 root root    0 Dec 19 19:19 test03.jwe.run11
  -rw-r--r-- 1 root root 3760 Dec 19 19:19 test03.jwe.run12
  -rw-r--r-- 1 root root 3760 Dec 19 19:19 test03.jwe.run13
  -rw-r--r-- 1 root root    0 Dec 19 19:22 test03.jwe.run14
  #### cleanup (tangd instance will be reachable again afterwards):
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -D INPUT 1
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -D INPUT 1
  

• Decryption:

  #### we prevent one tangd instance from answering in a timely fashion; for threshold=1, this still should work:
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
  #### without the commit/patch, we still see child processes (one of them running into a timeout waiting for an answer):
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# /usr/bin/clevis decrypt < test03.jwe.run12
  Hello, world.
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
  root     2917106  0.0  0.0   4424  2432 pts/0    S    19:27   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
  root     2917148  0.0  0.0   4424  1456 pts/0    S    19:27   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
  root     2917307  0.0  0.0   3612  1280 pts/0    S+   19:27   0:00 grep clevis
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# sleep 300
  Error communicating with server http://127.0.0.1:18888
  #### with the commit/patch, all child processes get killed:
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# /usr/local/bin/clevis decrypt < ./test03.jwe.run12
  Hello, world.
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
  root     2929645  0.0  0.0   3612  1280 pts/0    S+   19:36   0:00 grep clevis
  #### threshold=2, but only one 'working' tang server--we expect to see a timeout after less than 600 seconds:
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# timeout 600 /usr/local/bin/clevis decrypt < ./test03.jwe.run13
  Error communicating with server http://127.0.0.1:18888
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# timeout 600 /usr/bin/clevis decrypt < ./test03.jwe.run13
  Error communicating with server http://127.0.0.1:18888
  #### cleanup (tangd instance will be reachable again afterwards):
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -D INPUT 1
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -D INPUT 1
  root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build#
  

A variant of the last test revealed an unexpected behaviour of clevis, though (at least for me): If it receives a SIGSTOP signal followed by a SIGCONT signal a couple of seconds later, it will terminate; with the commit/patch, it'll also clean the child processes in this case (good!). However, I would have expected that clevis either terminates immediately (upon SIGSTOP) or resumes—not sure whether this behaviour is specified anywhere:

#### one of the tangd instances is unreachable so this would run into a timeout eventually, but we intervene manually:
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# /usr/bin/clevis decrypt < ./test03.jwe.run13
^Z
[1]+  Stopped                    /usr/bin/clevis decrypt < ./test03.jwe.run13
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     2999669  0.0  0.0   4424  2432 pts/0    T    20:22   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     2999678  0.0  0.0  10476  4608 pts/0    T    20:22   0:00 /usr/bin/clevis-decrypt-sss
root     2999684  0.0  0.0   4424  2560 pts/0    T    20:22   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     2999697  0.0  0.0   4424  2560 pts/0    T    20:22   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     2999739  0.0  0.0   4424  1584 pts/0    T    20:22   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     2999824  0.0  0.0   3612  1280 pts/0    S+   20:22   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# fg
/usr/bin/clevis decrypt < ./test03.jwe.run13
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     2999697  0.0  0.0   4424  2560 pts/0    S    20:22   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     2999739  0.0  0.0   4424  1584 pts/0    S    20:22   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     2999981  0.0  0.0   3612  1280 pts/0    S+   20:22   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# killall clevis-decrypt-tang
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3000461  0.0  0.0   3612  1280 pts/0    S+   20:22   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# 

#### one of the tangd instances is unreachable so this would run into a timeout eventually, but we intervene manually:
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# /usr/local/bin/clevis decrypt < ./test03.jwe.run13
^Z
[1]+  Stopped                    /usr/local/bin/clevis decrypt < ./test03.jwe.run13
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3010855  0.0  0.0   4424  2560 pts/0    T    20:30   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt
root     3010860  0.0  0.0  10480  4608 pts/0    T    20:30   0:00 /usr/local/bin/clevis-decrypt-sss
root     3010861  0.0  0.0   4424  2560 pts/0    S    20:30   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt
root     3010872  0.0  0.0   4424  2432 pts/0    S    20:30   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt-tang
root     3010913  0.0  0.0   4424  1328 pts/0    S    20:30   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt-tang
root     3011113  0.0  0.0   3612  1280 pts/0    S+   20:30   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# fg
/usr/local/bin/clevis decrypt < ./test03.jwe.run13
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3011256  0.0  0.0   3612  1280 pts/0    S+   20:30   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# 

[m-ueberall]

Addendum: If I interpret the above correctly, the commit/patch will not "propagate" job control signals to the clevis-[en|decrypt]-sss child processes anymore (IMHO it would make sense to ensure that if possible).

[sergio-correia]

Thanks for retesting! So, quoting the last reply first:

Addendum: If I interpret the above correctly, the commit/patch will not "propagate" job control signals to the clevis-[en|decrypt]-sss child processes anymore (IMHO it would make sense to ensure that if possible).

Right. Before the patch, parent and children were in the same process group, so job control signals (SIGTSTP from ctrl+z, SIGCONT from fg) affected everyone. With the patch -- (setpgid(0,0)) --, each child creates its own process group and the parents' process group is separate from the children's, which mean job control signals sent to parent don't reach children.

Regarding job control propagation: adding it back would require setting up signal handlers to manually
forward SIGTSTP/SIGCONT to each child's process group, which adds complexity; we could address that in a follow-up, if it is deemed required.

A variant of the last test revealed an unexpected behaviour of clevis, though (at least for me): If it receives a SIGSTOP signal followed by a SIGCONT signal a couple of seconds later, it will terminate; with the commit/patch, it'll also clean the child processes in this case (good!). However, I would have expected that clevis either terminates immediately (upon SIGSTOP) or resumes—not sure whether this behaviour is specified anywhere:

What happens here is: SIGSTOP is sent and then clevis pauses; SIGCONT sent a few seconds later and clevis resumes, then exits.

The exit after SIGCONT is likely because while clevis was paused:

• child processes continued running (they weren't stopped)
• children completed or timed out

When clevis resumes, it finds children are done and exits normally.

[m-ueberall]

I managed to conduct a number of additional tests in the meantime.

What happens here is: …

The behaviour in the last test in conjunction with the patched clevis-decrypt-sss still seems to be inconsistent: Even if one of the two clevis-decrypt-tang children is finished when clevis receives the SIGCONT signal (which I sent less than 20 seconds after the ^Z/SIGSTOP signal using fg), the second child cannot have timed out because its call to curl will run for more than 120 seconds:

root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# date; curl -sfg -X POST -H "Content-Type: application/jwk+json" --data-binary @- "http://127.0.0.1:18888/test" <<< ""; date
Mon Dec 22 15:46:13 UTC 2025
Mon Dec 22 15:48:28 UTC 2025
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -D INPUT 1
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -D INPUT 1

This means that clevis-decrypt-sss would still find that one of its children is running.

If we only query a single tang server (without the need for SSS) which does not answer due to dropped packets and we also send SIGSTOP followed by SIGCONT, we see the expected timeout message:

root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# echo "Hello, world." | clevis encrypt tang '{"url": "http://127.0.0.1:18888"}' >test04.jwe
The advertisement contains the following signing keys:

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxnZ1k

Do you wish to trust these keys? [ynYN] Y
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ls -l test04.jwe
-rw-r--r-- 1 root root 1255 Dec 22 15:38 test04.jwe
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# clevis decrypt < test04.jwe
^Z
[1]+  Stopped                    clevis decrypt < test04.jwe
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3108971  0.0  0.0   4424  2304 pts/0    T    15:40   0:00 /bin/bash -e /usr/bin/clevis-decrypt
root     3108981  0.0  0.0   4424  2432 pts/0    T    15:40   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     3109007  0.0  0.0   4424  1456 pts/0    T    15:40   0:00 /bin/bash -e /usr/bin/clevis-decrypt-tang
root     3109158  0.0  0.0   3612  1152 pts/0    S+   15:40   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# pstree 3108971
clevis-decrypt---clevis-decrypt----clevis-decrypt----curl
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# fg
clevis decrypt < test04.jwe
Error communicating with server http://127.0.0.1:18888
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -D INPUT 1
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -D INPUT 1

Regarding job control propagation: adding it back would require setting up signal handlers to manually forward SIGTSTP/SIGCONT to each child's process group, which adds complexity; we could address that in a follow-up, if it is deemed required.

While stopping/continuing clevis might not be a common use case and the removal of unneeded/still-running subprocesses is definitely an improvement, the above inconsistency should IMHO at least be documented as part of the commit/patch for the time being.

However, the following additional tests showed that clevis-decrypt-sss does not manage to kill all of its (grand)children yet:

If you modify clevis-decrypt-tang as follows so that it will start a subprocess (thereby simulating a more complex type of clevis-decrypt-xxx pin that relies on child processes by itself running in the background) …

root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# diff -U2 /usr/local/bin/clevis-decrypt-tang.org /usr/local/bin/clevis-decrypt-tang
--- /usr/local/bin/clevis-decrypt-tang.org      2025-12-19 19:14:35.401046259 +0000
+++ /usr/local/bin/clevis-decrypt-tang  2025-12-22 15:58:01.846295233 +0000
@@ -100,4 +100,5 @@
 xfr="$(jose jwk exc -i '{"alg":"ECMR"}' -l- -r- <<< "$clt$eph")"
 
+$0_grandchild 1>/dev/null 2>&1 &
 rec_url="$url/rec/$kid"
 ct="Content-Type: application/jwk+json"
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# cat /usr/local/bin//clevis-decrypt-tang_grandchild 
#! /bin/bash
sleep 3600
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build#

… and again use JWE definitions that require clevis-decrypt to receive the correct answer from one or two (of two) working tang instances (using SSS), we see that the "grandchildren" will not explicitly be terminated:

root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3249011  0.0  0.0   3612  1280 pts/0    S+   16:47   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# clevis decrypt < test03.jwe.run12
Hello, world.
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3249423  0.0  0.0   4424  2304 pts/0    S    16:47   0:00 /bin/bash /usr/local/bin/clevis-decrypt-tang_grandchild
root     3249706  0.0  0.0   3612  1280 pts/0    S+   16:47   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# killall clevis-decrypt-tang_grandchild
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3250261  0.0  0.0   3612  1152 pts/0    S+   16:47   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# clevis decrypt < test03.jwe.run13
Hello, world.
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3250538  0.0  0.0   4424  2176 pts/0    S    16:47   0:00 /bin/bash /usr/local/bin/clevis-decrypt-tang_grandchild
root     3250570  0.0  0.0   4424  2304 pts/0    S    16:47   0:00 /bin/bash /usr/local/bin/clevis-decrypt-tang_grandchild
root     3250700  0.0  0.0   3612  1280 pts/0    S+   16:47   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# killall clevis-decrypt-tang_grandchild

To demonstrate that the children are indeed listed as descendants in the process table, we can again drop packets so that they don't reach one of the tang instances and immediately send a SIGSTOP signal to clevis manually:

root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# iptables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ip6tables -I INPUT -p tcp -m tcp --dport 18888 -j DROP
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# clevis decrypt < test03.jwe.run13
^Z
[1]+  Stopped                    clevis decrypt < test03.jwe.run13
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3206295  0.0  0.0   4424  2432 pts/0    T    16:27   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt
root     3206306  0.0  0.0  10480  4096 pts/0    T    16:27   0:00 /usr/local/bin/clevis-decrypt-sss
root     3206313  0.0  0.0   4424  2432 pts/0    S    16:27   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt
root     3206339  0.0  0.0   4424  2432 pts/0    S    16:27   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt-tang
root     3206396  0.0  0.0   4424  2304 pts/0    S    16:27   0:00 /bin/bash /usr/local/bin/clevis-decrypt-tang_grandchild
root     3206412  0.0  0.0   4424  2304 pts/0    S    16:27   0:00 /bin/bash /usr/local/bin/clevis-decrypt-tang_grandchild
root     3206413  0.0  0.0   4424  1332 pts/0    S    16:27   0:00 /bin/bash -e /usr/local/bin/clevis-decrypt-tang
root     3206554  0.0  0.0   3612  1280 pts/0    S+   16:27   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# pstree -a -l 3206306
clevis-decrypt-
  `-clevis-decrypt -e /usr/local/bin/clevis-decrypt
      `-clevis-decrypt- -e /usr/local/bin/clevis-decrypt-tang
          |-clevis-decrypt- /usr/local/bin/clevis-decrypt-tang_grandchild
          |   `-sleep 3600
          `-clevis-decrypt- -e /usr/local/bin/clevis-decrypt-tang
              `-curl -sfg -X POST -H Content-Type: application/jwk+json --data-binary @- http://127.0.0.1:18888/rec/yHyA9cLsciUZXYdyOqv8BfiH1nIRZOJWiqj4-LzWq4M
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# fg
clevis decrypt < test03.jwe.run13
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build# ps axu|grep clevis
root     3206396  0.0  0.0   4424  2304 pts/0    S    16:27   0:00 /bin/bash /usr/local/bin/clevis-decrypt-tang_grandchild
root     3207611  0.0  0.0   3612  1152 pts/0    S+   16:28   0:00 grep clevis
root@vserver35<kvm>(chroot:development-ubuntu2404-amd64):/build#

[m-ueberall]

To clarify: clevis decrypt < test04.jwe which only leads to the invocation of one clevis-decrypt-tang instance also leaves behind a clevis-decrypt-tang_grandchild process—this means that technically, the current commit/patch for clevis-decrypt-sss correctly kills child processes related to pins that are no longer needed. However, clevis{-encrypt,-decrypt} should probably also always kill all (remaining) descendants upon termination.