CI: htmldoc artifacts and package

[hdiethelm]

This PR adds:

• Artifacts for htmldoc
• A published package for htmldocs Removed
  • Github doesn't support generic packages but containers can be used with oras
  • The package is published for 2.9 / master and any tag
  • Download with a persistent link: oras pull ghcr.io/linuxcnc/linuxcnc/doc-html:master
• Allow sid builds to fail and continue building other packages

If needed for the 2.9 branch, I can backport this.

The discussion started in: #4119

As much as I understand the github docs, only members are allowed to write packages. So PR's should not be able to create a package, even if one removes the if() to only run this stage always.

For testing, I run this stage in my github account: https://github.com/hdiethelm/linuxcnc-fork/actions/runs/27276506335
The package is here: https://github.com/hdiethelm/linuxcnc-fork
And can be downloaded with: oras pull ghcr.io/hdiethelm/linuxcnc/doc-html:ci_doc_build_test

@BsAtHome
Do you thing this does the job?
I will create a commit that removes the if() and see what happens. Then I will revert it again.

[hdiethelm]

And again, the CI broke because sid is broken. Might be we should make the sid package build allow to fail?

I tested it:

apt install python3-opencv
Solving dependencies... Error!  
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

Unsatisfied dependencies:
 python3-opencv : Depends: libopencv-calib3d410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-contrib410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-core410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-dnn410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-features2d410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-flann410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-highgui410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-imgcodecs410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-imgproc410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-ml410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-objdetect410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-photo410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-shape410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-stitching410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-video410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-videoio410 (>= 4.10.0+dfsg) but it is not going to be installed
                  Depends: libopencv-viz410 (>= 4.10.0+dfsg) but it is not going to be installed
Error: Unable to satisfy dependencies. Reached two conflicting assignments:
   1. libadios2-mpi-core-2.11:amd64 is selected for install because:
      1. python3-opencv:amd64=4.10.0+dfsg-7+b2 is selected for install
      2. python3-opencv:amd64 Depends libopencv-viz410 (>= 4.10.0+dfsg)
      3. libopencv-viz410:amd64 Depends libvtk9.5 (>= 9.5.2+dfsg4)
      4. libvtk9.5:amd64 Depends libadios2-mpi-c++-2.11 (>= 2.11.0+dfsg1)
      5. libadios2-mpi-c++-2.11:amd64 Depends libadios2-mpi-core-2.11 (>= 2.11.0+dfsg1)
   2. libadios2-mpi-core-2.11:amd64 Depends libadios2-mpi-plugins (= 2.11.0+dfsg1-7+b1)
      but none of the choices are installable:
      [no choices]

I guess it will be fixed soon in sid.

[BsAtHome]

I'm not sure that the webserver can use oras. Besides, I'm not sure you would want to involve yet another third party in this process.

The point is that github already has stored any built artifact (like in the deb package builds). The question is whether we can exploit that. We only need the link to the artifact.

[BsAtHome]

And, yes, a soft-fail on Debian:sid may be appropriate.
It is a rare occasion that sid breaks, but it is a blocker.

[hdiethelm]

I'm not sure that the webserver can use oras. Besides, I'm not sure you would want to involve yet another third party in this process.

It is a debian package, depending how it is set up, apt-get install oras is all it needs.

The point is that github already has stored any built artifact (like in the deb package builds). The question is whether we can exploit that. We only need the link to the artifact.

The only issue there is, that artifacts are linked to CI runs. So for an update you would have to download the file manually. Even wget doesn't work to download the file when you use right-click copy link due to you have to be logged in.
You can try it right now, the doc artifact is already enabled: https://github.com/LinuxCNC/linuxcnc/actions/runs/27279180672

Is this good enough?

Otherwise what I found so far:
https://gist.github.com/umohi/bfc7ad9a845fc10289c03d532e3d2c2f
I can try that and give you a command that should work. But you will need an access token.

[BsAtHome]

I'm not sure that the webserver can use oras. Besides, I'm not sure you would want to involve yet another third party in this process.

It is a debian package, depending how it is set up, apt-get install oras is all it needs.

That is a problem, right there... We can't install or sudo on the webserver.

The point is that github already has stored any built artifact (like in the deb package builds). The question is whether we can exploit that. We only need the link to the artifact.

Is this good enough?

Yes, that was what I was thinking about. Don't know if it works. That's why I was inquiring ;-)

The problem was that the htmldocs run did not produce any artifacts, so we had no chance of testing in any direction.

Otherwise what I found so far: https://gist.github.com/umohi/bfc7ad9a845fc10289c03d532e3d2c2f I can try that and give you a command that should work. But you will need an access token.

That looks like a way. Generating a limited access token, just to get the file, may be a possibility.

[hdiethelm]

And, yes, a soft-fail on Debian:sid may be appropriate. It is a rare occasion that sid breaks, but it is a blocker.

With c323d8e, the other packages are built, even if sid fails. But sid is shown red.
I can move the continue-on-error: to all steps, then it will show green. But I think red is fine, it failed and needs investigation.

[hdiethelm]

Otherwise what I found so far: https://gist.github.com/umohi/bfc7ad9a845fc10289c03d532e3d2c2f I can try that and give you a command that should work. But you will need an access token.

That looks like a way. Generating a limited access token, just to get the file, may be a possibility.

I found something that works:
https://docs.github.com/en/rest/actions/artifacts?apiVersion=2026-03-10

Token:

Get url's of all artifacts that match the branch and are named linuxcnc-doc:

TOKEN="YourToken"
BRANCH="ci_doc_build"
NAME="linuxcnc-doc"
curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "X-GitHub-Api-Version: 2026-03-10" \
  "https://api.github.com/repos/linuxcnc/linuxcnc/actions/artifacts?per_page=100&name=${NAME}"  | \
  jq ".artifacts[] | select(.workflow_run.head_branch==\"${BRANCH}\") | .archive_download_url"

Download the zip:

curl -L \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "X-GitHub-Api-Version: 2026-03-10" \
  https://api.github.com/repos/LinuxCNC/linuxcnc/actions/artifacts/7537361103/zip -o linuxcnc-doc.zip

Combined:

TOKEN="YourToken"
BRANCH=ci_doc_build
NAME="linuxcnc-doc"
DL_URL=$(curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${TOKEN}" \
-H "X-GitHub-Api-Version: 2026-03-10" \
"https://api.github.com/repos/linuxcnc/linuxcnc/actions/artifacts?per_page=100&name=${NAME}" | \
jq ".artifacts[] | select(.workflow_run.head_branch==\"${BRANCH}\") | .archive_download_url" | head -n1)

#DL_URL has quotes, remove them
DL_URL=${DL_URL//\"/}

curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${TOKEN}" \
-H "X-GitHub-Api-Version: 2026-03-10" \
"$DL_URL" -o linuxcnc-doc.zip

Looks like the links are sorted by date, so I can just take the first one.

Does that work for you? You need curl and jq Took some time to get all together, all i found was to download releases but we need artifacts from CI build.

As soon as this branch is merged, you can change BRANCH=ci_doc_build to BRANCH=master

Edit: Using ?name= is more efficient than filtering for name. Also, there is pagination, i set it to 100. So after 100 other builds without any master build, it will fail.

[hdiethelm]

I removed the commit creating packages.
Should be fine to merge if you are ok with it.

[BsAtHome]

That looks very good that we might be almost there to try.

@andypugh This might be able to restore the devel docs on the webserver. Can you check on the webserver if it has jq installed (try run jq --version)? There should be curl by default, I think.
If jq is there, then we have a "simple" way to poll github for new devel html content directly built from CI (via a cron job). If jq is unavailable, then we need to look at what alternatives can parse json and are available on the webserver (python, php, lua, perl,...).

[hdiethelm]

There will shure be a way. Otherwhise just copy over the binary of jq or cobble together something in bash / awk to get the right url string.
BTW: Artifacs expire after some time, default 90 days, so it might sometimes fail. It's also still github.
The curl --fail options might help handle errors.

[grandixximo]

This also bites package-indep: today's adios2/sid breakage failed package-indep (debian:sid) and fail-fast cancelled the bookworm/trixie indep builds. Could you apply the same allow_fail handling there? With that I'd close my overlapping #4155.

[andypugh]

Can you check on the webserver if it has jq installed (try run jq --version)?

Sorry, I missed this last night, and don't have the relevant keys on my work laptop. I can check when I get home.