`docker cp` into a sysbox-runc container fails with "Could not find the file"

[ndeloof]

docker cp into a sysbox-runc container fails with "Could not find the file"

Description

When a container is run with --runtime=sysbox-runc, any docker cp into
that container fails — including paths that clearly exist in the image
and paths created at runtime. The same image works fine with the default
runc runtime, so the bug is in sysbox-runc, not in the image or in the
Docker daemon.

The container itself is healthy: docker exec can list, read and write
the same paths without issue. The mismatch suggests the daemon's view of
the container rootfs is no longer in sync with what the container
actually sees after sysbox-runc finishes setting up.

This breaks any tool that relies on the Docker CopyToContainer API.
The original report that led me here was docker compose watch, which
uses that API to sync source files into a running container — but the
issue is much more general.

Steps to reproduce

docker run -d --runtime=sysbox-runc --name repro alpine sleep 3600

# any docker cp into the container fails
echo hi > /tmp/x
docker cp /tmp/x repro:/x
# Error response from daemon: Could not find the file / in container repro

docker cp /tmp/x repro:/tmp/x
# Error response from daemon: Could not find the file /tmp in container repro

# control: same image, default runtime, works fine
docker rm -f repro
docker run -d --name repro alpine sleep 3600
docker cp /tmp/x repro:/x   # OK

Inside the container, the paths exist and are writable:

docker exec repro sh -c 'ls -la / ; echo > /from-exec && ls -la /from-exec'
# / and /from-exec both visible

Environment

OS:          Ubuntu 24.04, kernel 6.17.0-20-generic
Docker:      29.5.2 (server)
runc:        1.3.5
sysbox-runc: 0.7.0 CE (commit a4dd414f7b9b7455c0fbf0d5e5db7bcfe30645bc,
             built Tue Mar  3 21:51:47 UTC 2026)
containerd:  v2 with overlayfs snapshotter

Relevant Docker daemon config (needed so that sysbox-runc starts at all
on Docker 29.5.x, but unrelated to this bug):

{
  "runtimes": { "sysbox-runc": { "path": "/usr/bin/sysbox-runc" } },
  "features": { "time-namespaces": false }
}

Expected behavior

docker cp <src> <container>:<dest> should write the file to <dest>
inside the container, the same way it does with the default runc
runtime.

Actual behavior

The daemon's ContainerStatPath call returns "not found" for every
path in the container, including /. Nothing can be copied in.

[ctalledo]

Thanks for the clean repro @ndeloof, and for the cross-reference from docker/compose#13814.

This is a real docker cp / CopyToContainer bug with Sysbox, but its scope is narrower than "all sysbox-runc containers". It depends on how Sysbox sets up the container's filesystem to handle UID/GID mapping:

• shiftfs (kernels with the shiftfs module, e.g. Ubuntu): docker cp works correctly. The daemon and the container share the same on-disk rootfs and the daemon's ContainerStatPath view stays consistent, so copies in/out behave as with plain runc.
• ID-mapped mounts (idmapped mounts on newer kernels) or rootfs cloning + chown (the fallback Sysbox uses when neither shiftfs nor idmapped mounts are available): this is where the failure shows up. In these modes the rootfs the daemon stat's/copies into is not the same view the container ultimately runs against, so the daemon reports paths as missing / directory not empty even though they exist and are writable from docker exec inside the container.

So the trigger is the filesystem ID-mapping strategy, not the container image or the Docker daemon. On a host where Sysbox uses idmapped mounts or rootfs-clone+chown, docker cp into the container fails; on a shiftfs host the same scenario works.

The repro here (Ubuntu 24.04, kernel 6.17, no shiftfs) lands on the idmapped-mounts / clone path, which is why it reproduces. To confirm the boundary, the same docker run --runtime=sysbox-runc ... && docker cp sequence on a shiftfs-capable host should succeed.

We're tracking the Docker Desktop manifestation (Enhanced Container Isolation, which is built on Sysbox) internally as well. Will follow up here as we dig into the rootfs-view inconsistency in the idmapped / clone+chown paths.