No-volume implementation causing "Unexpected end of data" error when copying data to or from workflow pods

[vvanouytsel]

When using the 'no-volume' implementation we notice that the mechanism to copy data to or from the pod often fails.
I cannot reproduce it on demand, but it became very noticeable once our developers starting using these runners.

I suspect, but cannot prove, that the combination of tar and the Kubernetes exec API are responsible for the issue.
This issue prevents us from adopting the no-volume implementation in production because of the instability it currently brings.

##[debug]execPodStep response: {"metadata":{},"status":"Success"}
##[debug]Copying from job pod 'default-staging-2rrxt-runner-hl8ht-workflow' /__w/_temp to /home/runner/_work/_temp
##[debug]Copying from pod default-staging-2rrxt-runner-hl8ht-workflow /__w/_temp to /home/runner/_work/_temp
/home/runner/k8s/index.js:20421
    cb(this._finished ? null : new Error('Unexpected end of data'))
                               ^

Error: Unexpected end of data
    at Extract._final (/home/runner/k8s/index.js:20421:32)
    at WritableState.updateNonPrimary (/home/runner/k8s/index.js:18588:14)
    at WritableState.update (/home/runner/k8s/index.js:18577:72)
    at WritableState.updateWriteNT (/home/runner/k8s/index.js:18946:10)
    at node:internal/process/task_queues:140:7
    at AsyncResource.runInAsyncScope (node:async_hooks:206:9)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
Emitted 'error' event on Extract instance at:
    at WritableState.afterDestroy (/home/runner/k8s/index.js:18896:19)
    at Extract._destroy (/home/runner/k8s/index.js:20430:5)
    at WritableState.updateNonPrimary (/home/runner/k8s/index.js:18595:16)
    at WritableState.update (/home/runner/k8s/index.js:18577:72)
    [... lines matching original stack trace ...]
    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)

[zarko-a]

Curious which Kubernetes distro and version are you running.

We are on AWS EKS 1.33 and haven't noticed this specific issue once. I've switched all our runners to the custom image running the container hooks [with no-volume mode] from main branch. It's been working fine after those few bugs have been fixed in the last 2-3 weeks.

[vvanouytsel]

Curious which Kubernetes distro and version are you running.

We are on AWS EKS 1.33 and haven't noticed this specific issue once. I've switched all our runners to the custom image running the container hooks [with no-volume mode] from main branch. It's been working fine after those few bugs have been fixed in the last 2-3 weeks.

We are running on Azue AKS 1.32.9.
Are your working directories large or small?

I have a feeling it occurs when the working directory is pretty large, in our case with node projects as the node_modules directory can get quite large and is also copied over. Especially these large streams seem to give some issues when you pipe them to the Kubernetes exec API.

[vvanouytsel]

Alternatively, could this be because there are too many runners which call the exec API and thus basically overload the Kubernetes API?

[zarko-a]

Are your working directories large or small?

I haven't payed that much attention. But looking at the few now, they are <15Mb. That's the _work directory without repo, since repo is only checked out on the workflow pod and doesn't get transferred via k8s exec API. That's my understanding at least.

Alternatively, could this be because there are too many runners which call the exec API and thus basically overload the Kubernetes API?

This is possible. K8S has API rate limiting.

In our env we don't get more than 10-20 runners at the same time and even then, not all start at exactly same moment so they wouldn't be calling k8s exec API simultaneously.

[vvanouytsel]

Another example of an occurrence.

##[debug]Evaluating condition for step: 'Post Checkout repository'
##[debug]Evaluating: always()
##[debug]Evaluating always:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Post Checkout repository
##[debug]Loading inputs
##[debug]Evaluating: github.repository
##[debug]Evaluating Index:
##[debug]..Evaluating github:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'repository'
##[debug]=> 'TrendMiner/tm-actions-aft-execute-step-function'
##[debug]Result: 'TrendMiner/tm-actions-aft-execute-step-function'
##[debug]Evaluating: github.token
##[debug]Evaluating Index:
##[debug]..Evaluating github:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'token'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Post job cleanup.
##[debug]Running JavaScript Action with default external tool: node24
Run '/home/runner/k8s/index.js'
##[debug]/home/runner/externals/node20/bin/node /home/runner/k8s/index.js
##[debug]Copying /home/runner/_work/_temp to pod default-s-8d24f-runner-skchj-workflow at /__w/_temp
(node:44142) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
##[debug]internalExecOutput response: {"metadata":{},"status":"Success"}
##[debug]execPodStep response: {"metadata":{},"status":"Success"}
##[debug]Getting git version
/usr/bin/git version
git version 2.43.0
##[debug]0
##[debug]git version 2.43.0
##[debug]
##[debug]Set git useragent to: git/2.43.0 (github-actions-checkout)
::add-mask::***
Temporarily overriding HOME='/__w/_temp/96cd8a38-d243-4c67-a8d6-9acc9979e742' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
/usr/bin/git config --global --add safe.directory /__w/tm-actions-aft-execute-step-function/tm-actions-aft-execute-step-function
##[debug]0
##[debug]
Removing SSH command configuration
/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
##[debug]1
##[debug]
/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
##[debug]0
##[debug]
Removing HTTP extra header
/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
##[debug]1
##[debug]
/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
##[debug]0
##[debug]
Removing includeIf entries pointing to credentials config files
/usr/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
includeif.gitdir:/__w/tm-actions-aft-execute-step-function/tm-actions-aft-execute-step-function/.git.path
includeif.gitdir:/github/workspace/.git.path
##[debug]0
##[debug]includeif.gitdir:/__w/tm-actions-aft-execute-step-function/tm-actions-aft-execute-step-function/.git.path
##[debug]includeif.gitdir:/github/workspace/.git.path
##[debug]
/usr/bin/git config --local --get-all includeif.gitdir:/__w/tm-actions-aft-execute-step-function/tm-actions-aft-execute-step-function/.git.path
/__w/_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config
##[debug]0
##[debug]/__w/_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config
##[debug]
/usr/bin/git config --local --unset includeif.gitdir:/__w/tm-actions-aft-execute-step-function/tm-actions-aft-execute-step-function/.git.path /__w/_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config
##[debug]0
##[debug]
/usr/bin/git config --local --get-all includeif.gitdir:/github/workspace/.git.path
/github/runner_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config
##[debug]0
##[debug]/github/runner_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config
##[debug]
/usr/bin/git config --local --unset includeif.gitdir:/github/workspace/.git.path /github/runner_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config
##[debug]0
##[debug]
/usr/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
##[debug]0
##[debug]
Removing credentials config '/__w/_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config'
##[debug]Skipping removal of credentials config '/github/runner_temp/git-credentials-145a81f4-ae92-4c21-a9b6-6f58abd0da68.config' - not under RUNNER_TEMP
##[debug]Unsetting HOME override
##[debug]execPodStep response: {"metadata":{},"status":"Success"}
##[debug]Copying from job pod 'default-s-8d24f-runner-skchj-workflow' /__w/_temp to /home/runner/_work/_temp
##[debug]Copying from pod default-s-8d24f-runner-skchj-workflow /__w/_temp to /home/runner/_work/_temp
/home/runner/k8s/index.js:20421
    cb(this._finished ? null : new Error('Unexpected end of data'))
                               ^

Error: Unexpected end of data
    at Extract._final (/home/runner/k8s/index.js:20421:32)
    at WritableState.updateNonPrimary (/home/runner/k8s/index.js:18588:14)
    at WritableState.update (/home/runner/k8s/index.js:18577:72)
    at WritableState.updateWriteNT (/home/runner/k8s/index.js:18946:10)
    at node:internal/process/task_queues:140:7
    at AsyncResource.runInAsyncScope (node:async_hooks:206:9)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
Emitted 'error' event on Extract instance at:
    at WritableState.afterDestroy (/home/runner/k8s/index.js:18896:19)
    at Extract._destroy (/home/runner/k8s/index.js:20430:5)
    at WritableState.updateNonPrimary (/home/runner/k8s/index.js:18595:16)
    at WritableState.update (/home/runner/k8s/index.js:18577:72)
    [... lines matching original stack trace ...]
    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)

Node.js v20.19.5
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug]System.Exception: Executing the custom container implementation failed. Please contact your self hosted runner administrator.
##[debug] ---> System.Exception: The hook script at '/home/runner/k8s/index.js' running command 'RunScriptStep' did not execute successfully
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   --- End of inner exception stack trace ---
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.ExecuteHookScript[T](IExecutionContext context, HookInput input, ActionRunStage stage, String prependPath)
##[debug]   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.RunScriptStepAsync(IExecutionContext context, ContainerInfo container, String workingDirectory, String entryPoint, String entryPointArgs, IDictionary`2 environmentVariables, String prependPath)
##[debug]   at GitHub.Runner.Worker.Handlers.ContainerStepHost.ExecuteAsync(IExecutionContext context, String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, String standardInInput, CancellationToken cancellationToken)
##[debug]   at GitHub.Runner.Worker.Handlers.NodeScriptActionHandler.RunAsync(ActionRunStage stage)
##[debug]   at GitHub.Runner.Worker.ActionRunner.RunAsync()
##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
##[debug]Finishing: Post Checkout repository