ci: add Testing Farm test via tmt for Konflux CI by Roshan-R · Pull Request #4515 · coreos/coreos-assembler

Roshan-R · 2026-03-31T12:13:53Z

Add tmt test infrastructure to hook into Konflux CI via Testing Farm

The new test pulls the newly built coreos-assembler container image and verifies that it can successfully build a Fedora CoreOS image from scratch using a fedora-coreos-config checkout.

gemini-code-assist

Code Review

This pull request introduces a TMT-based testing framework for coreos-assembler, including Tekton integration for Testing Farm, TMT plans, and a test script for containerized builds. The reviewer identified several improvement opportunities: replacing hardcoded Git references in the Tekton configuration with dynamic parameters, refining the list of required packages in the TMT plan to include podman, and enhancing the test script's reliability by ensuring directory existence and applying proper variable quoting.

gemini-code-assist · 2026-03-31T12:15:07Z

+    value: https://github.com/Roshan-R/coreos-assembler
+  - name: GIT_REF
+    value: tmt-testing


The GIT_URL and GIT_REF parameters are currently hardcoded to a personal fork and a specific branch (Roshan-R/coreos-assembler and tmt-testing). For integration into the main repository, these should be updated to point to the official repository or use dynamic Tekton parameters to ensure the tests are executed against the correct source code.

gemini-code-assist · 2026-03-31T12:15:07Z

+      - wget
+      - jq
+      - git


The packages wget, jq, and git are being installed in the test environment but do not appear to be used by the host-side test.sh script. The git command used in the test is executed inside the coreos-assembler container. Conversely, the script relies on podman, which should be explicitly listed if it is not guaranteed to be present in the base image.

- podman

gemini-code-assist · 2026-03-31T12:15:07Z

+
+# IMAGE_URL is the coreos-assembler image that is built by konflux for each run.
+export COREOS_ASSEMBLER_CONTAINER="$IMAGE_URL"
+export COSA_DIR=$HOME/workspace/build


It is recommended to ensure that the $COSA_DIR directory exists before it is mounted as a volume. If the directory is missing, podman might create it with root ownership, which can lead to permission issues for the builder user inside the container. Additionally, quoting the path is a best practice to handle potential spaces.

Suggested change

export COSA_DIR=$HOME/workspace/build

export COSA_DIR="$HOME/workspace/build"

mkdir -p "$COSA_DIR"

gemini-code-assist · 2026-03-31T12:15:07Z

+    podman run --rm --security-opt=label=disable --privileged \
+    -v=${COSA_DIR}:/srv/ --device=/dev/kvm \
+    --device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp --name=cosa ${COREOS_ASSEMBLER_CONTAINER} "$@";


The volume mount and image variable should be quoted to prevent word splitting or issues with special characters. Also, the trailing semicolon at the end of the command is unnecessary in a bash script.

Suggested change

podman run --rm --security-opt=label=disable --privileged \

-v=${COSA_DIR}:/srv/ --device=/dev/kvm \

--device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp --name=cosa ${COREOS_ASSEMBLER_CONTAINER} "$@";

podman run --rm --security-opt=label=disable --privileged \

-v "${COSA_DIR}:/srv/" --device=/dev/kvm \

--device=/dev/fuse --tmpfs=/tmp -v /var/tmp:/var/tmp --name=cosa "${COREOS_ASSEMBLER_CONTAINER}" "$@"

Roshan-R · 2026-04-01T09:19:22Z

/retest

Roshan-R · 2026-04-08T14:02:05Z

/retest

joelcapitao · 2026-04-14T08:14:55Z

/retest

Roshan-R · 2026-04-17T09:17:14Z

/retest

Roshan-R · 2026-04-20T09:44:45Z

/retest

Roshan-R · 2026-04-22T11:37:55Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a testing framework using tmt for coreos-assembler, including a new test plan, a build test script, and updates to the Tekton pipeline for memory resource management. Feedback suggests adding the fetch flag to the build command to ensure necessary metadata is downloaded, increasing the allocated disk space to prevent out-of-space errors during image generation, and removing a fixed container name in the podman command to avoid potential naming conflicts.

gemini-code-assist · 2026-04-22T11:40:17Z

+    set -x
+    podman run --rm --security-opt=label=disable --privileged \
+    -v="${COSA_DIR}":/srv/ --device=/dev/kvm \
+    --device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp --name=cosa "${COREOS_ASSEMBLER_CONTAINER}" "$@";


Using a fixed name --name=cosa for the podman container can lead to conflicts if the test environment is reused or if multiple tests are run in parallel on the same host. Since the container is run with --rm, the name is not strictly necessary for identification. It's safer to omit it.

Suggested change

--device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp --name=cosa "${COREOS_ASSEMBLER_CONTAINER}" "$@";

--device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp "${COREOS_ASSEMBLER_CONTAINER}" "$@";

jbtrystram · 2026-04-28T15:12:18Z

+mkdir -p "$COSA_DIR"
+cosa init --force https://github.com/coreos/fedora-coreos-config --branch testing-devel
+# Test if the newly built container can build the fcos image
+cosa build


let's also run a set of test.
See

coreos-assembler/.cci.jenkinsfile

Line 42 in 211dee6

kola(cosaDir: "/srv", addExtTests: ["${env.WORKSPACE}/ci/run-kola-self-tests"])

Roshan-R · 2026-04-30T07:12:22Z

/retest

Roshan-R · 2026-05-08T09:19:26Z

/retest

…e memory for prepare-sboms Bump the pipeline-docker-build-multi-platform-oci-ta bundle to the latest hash. The prepare-sboms step in the build-images task requires more memory with our current workload and was hitting OOM. Increase memory requests and limits to 2Gi to ensure reliable execution.

c4rt0

Just a comment based on today's Jenkins experience.

Rolv-Apneseth

Minor things about utils.sh but otherwise LGTM

Roshan-R · 2026-06-01T08:21:10Z

/retest

joelcapitao · 2026-06-01T11:23:56Z

/retest

joelcapitao · 2026-06-01T14:26:05Z

/retest

Roshan-R · 2026-06-01T15:02:12Z

/retest

Roshan-R · 2026-06-01T15:27:02Z

/retest

Roshan-R · 2026-06-02T07:36:13Z

/retest

c4rt0

Some comment, but other than that this /lgtm

joelcapitao

The code looks good but the TestingFarm jobs are failing because of :

[2026-06-02 08:11:28] [E] [test-schedule-tmt-connect] Failed to get list of plans:
---v---v---v---v---v---

No metadata found in the '.' directory. Use 'tmt init' to get started.

The exception was caused by 1 earlier exceptions

Cause number 1:

    Unable to find tree root for '/var/ARTIFACTS/git-main13u1u670'.

What's the difference between those jobs and your coreos-assembler-testing-x86-64-rosh job ?

Introduce a tmt-based test suite for Testing Farm integration in Konflux CI. The new workflow validates that the freshly built coreos-assembler container can initialize and build Fedora CoreOS artifacts from fedora-coreos-config, generate QEMU images, and run kola integration, upgrade, and self-tests.

Roshan-R · 2026-06-10T08:59:29Z

The code looks good but the TestingFarm jobs are failing because of :
[2026-06-02 08:11:28] [E] [test-schedule-tmt-connect] Failed to get list of plans:
---v---v---v---v---v---

No metadata found in the '.' directory. Use 'tmt init' to get started.

The exception was caused by 1 earlier exceptions

Cause number 1:

    Unable to find tree root for '/var/ARTIFACTS/git-main13u1u670'.
What's the difference between those jobs and your coreos-assembler-testing-x86-64-rosh job ?

@joelcapitao the jobs are failing since the tmt code is not yet in the coreos-assmebler main branch. the coreos-assembler-testing-x86-64-rosh job is setup to read the tmt code from my branch tmt-testing and hence can access the newly created tmt code. Other than that there is no difference between the two and the other jobs would start to pass once this PR gets merged and the code is available to main branch.

joelcapitao · 2026-06-10T10:42:16Z

@joelcapitao the jobs are failing since the tmt code is not yet in the coreos-assmebler main branch. the coreos-assembler-testing-x86-64-rosh job is setup to read the tmt code from my branch tmt-testing and hence can access the newly created tmt code. Other than that there is no difference between the two and the other jobs would start to pass once this PR gets merged and the code is available to main branch.

Thank you for the clarification. This means we're not testing the changes we're doing in tmt/ into the same PR. Those changes (once merged to main) will be run in the next PRs.
I think we need to fetch the right codebase (the one from the PR) from the metadata of the Konflux Snapshot being tested, and not rely on static GIT_URL and GIT_REF, what do you think ?

Roshan-R · 2026-06-10T10:59:17Z

@joelcapitao the jobs are failing since the tmt code is not yet in the coreos-assmebler main branch. the coreos-assembler-testing-x86-64-rosh job is setup to read the tmt code from my branch tmt-testing and hence can access the newly created tmt code. Other than that there is no difference between the two and the other jobs would start to pass once this PR gets merged and the code is available to main branch.

Thank you for the clarification. This means we're not testing the changes we're doing in tmt/ into the same PR. Those changes (once merged to main) will be run in the next PRs.
I think we need to fetch the right codebase (the one from the PR) from the metadata of the Konflux Snapshot being tested, and not rely on static GIT_URL and GIT_REF, what do you think ?

Yeah that would be the better solution to this, let me see how to get it working

joelcapitao · 2026-06-10T11:05:27Z

Yeah that would be the better solution to this, let me see how to get it working

We could use similar approach that we did for f-c-c c.f https://github.com/coreos/fedora-coreos-pipeline/blob/main/tmt/tests/init.sh. In that case we're getting the metadata directly from the OCI image itself.

Roshan-R · 2026-06-10T11:42:10Z

Yeah that would be the better solution to this, let me see how to get it working

We could use similar approach that we did for f-c-c c.f https://github.com/coreos/fedora-coreos-pipeline/blob/main/tmt/tests/init.sh. In that case we're getting the metadata directly from the OCI image itself.

Looks like there is no org.opencontainers.image.source label for the coreos-assembler image made using konflux. should we add it to the image as a label or try another way?

joelcapitao · 2026-06-10T12:08:46Z

Looks like there is no org.opencontainers.image.source label for the coreos-assembler image made using konflux. should we add it to the image as a label or try another way?

We don't explicitly add them but Konflux does it for us, see below:

skopeo inspect -n docker://quay.io/redhat-user-workloads/coreos-tenant/coreos-assembler-main:on-pr-32d209f69317a30ab9e14375c7250ea32a0b8cfa | jq .Labels
{
  "architecture": "x86_64",
  "build-date": "2026-06-10T08:57:15Z",
  "io.buildah.version": "1.42.2",
  "license": "MIT",
  "name": "fedora",
  "org.opencontainers.image.created": "2026-06-10T08:57:15Z",
  "org.opencontainers.image.license": "MIT",
  "org.opencontainers.image.licenses": "MIT",
  "org.opencontainers.image.name": "fedora",
  "org.opencontainers.image.revision": "32d209f69317a30ab9e14375c7250ea32a0b8cfa",
  "org.opencontainers.image.source": "https://github.com/Roshan-R/coreos-assembler",
  "org.opencontainers.image.title": "fedora",
  "org.opencontainers.image.url": "https://fedoraproject.org/",
  "org.opencontainers.image.vendor": "Fedora Project",
  "org.opencontainers.image.version": "43",
  "quay.expires-after": "5d",
  "vcs-ref": "32d209f69317a30ab9e14375c7250ea32a0b8cfa",
  "vcs-type": "git",
  "vendor": "Fedora Project",
  "version": "43"
}

Roshan-R · 2026-06-10T14:00:21Z

/retest

Roshan-R · 2026-06-10T18:43:06Z

/retest

Roshan-R · 2026-06-11T12:38:03Z

/retest

joelcapitao · 2026-06-11T12:52:01Z

@Roshan-R I did some research and found out the task already supports what we want :) Check out https://gitlab.com/testing-farm/integrations-konflux/-/tree/main/task/testing-farm-konflux-scheduler?ref_type=heads#optional-inputs-to-testing-farm-request

Overrides for auto-resolved source

GIT_URL → --git-url (falls back to SOURCE_GIT_URL or value derived from SNAPSHOT)
GIT_REF → --git-ref (falls back to PR/MR merge head or value derived from SNAPSHOT)

Roshan-R · 2026-06-11T12:54:54Z

@Roshan-R I did some research and found out the task already supports what we want :) Check out https://gitlab.com/testing-farm/integrations-konflux/-/tree/main/task/testing-farm-konflux-scheduler?ref_type=heads#optional-inputs-to-testing-farm-request
Overrides for auto-resolved source

GIT_URL → --git-url (falls back to SOURCE_GIT_URL or value derived from SNAPSHOT)
GIT_REF → --git-ref (falls back to PR/MR merge head or value derived from SNAPSHOT)

yepp, I just found out about this a while ago as well. I have edited the integration tests and have started a re-run. let's see if it will work :)

joelcapitao · 2026-06-11T12:54:58Z

@Roshan-R I did some research and found out the task already supports what we want :) Check out https://gitlab.com/testing-farm/integrations-konflux/-/tree/main/task/testing-farm-konflux-scheduler?ref_type=heads#optional-inputs-to-testing-farm-request

Oh I see you've already taken that path as I can't see GIT_URL and GIT_REF anymore in the main integration tests !

gemini-code-assist Bot reviewed Mar 31, 2026

View reviewed changes

Roshan-R force-pushed the tmt-testing branch from f2ff904 to 51baff3 Compare April 22, 2026 11:33

Roshan-R force-pushed the tmt-testing branch from 51baff3 to 814ea11 Compare April 22, 2026 11:38

gemini-code-assist Bot reviewed Apr 22, 2026

View reviewed changes

jbtrystram requested changes Apr 28, 2026

View reviewed changes

Roshan-R force-pushed the tmt-testing branch 2 times, most recently from 51dbef0 to fa6e4a4 Compare May 5, 2026 09:02

Roshan-R force-pushed the tmt-testing branch from 8ff4a94 to f3aa1b7 Compare May 13, 2026 07:57

Roshan-R force-pushed the tmt-testing branch 2 times, most recently from 3836375 to 3c5bfc2 Compare May 14, 2026 00:14

c4rt0 reviewed May 15, 2026

View reviewed changes

Comment thread tmt/tests/kola.fmf Outdated

Roshan-R force-pushed the tmt-testing branch 4 times, most recently from 48c0022 to d2f61ed Compare May 20, 2026 09:54

Rolv-Apneseth reviewed May 26, 2026

View reviewed changes

Comment thread tmt/tests/utils.sh Outdated

Comment thread tmt/tests/utils.sh Outdated

Roshan-R force-pushed the tmt-testing branch from d2f61ed to ea2b9da Compare June 1, 2026 07:29

Roshan-R requested review from jbtrystram and joelcapitao June 3, 2026 10:02

c4rt0 reviewed Jun 8, 2026

View reviewed changes

Comment thread tmt/tests/utils.sh

joelcapitao reviewed Jun 9, 2026

View reviewed changes

Roshan-R force-pushed the tmt-testing branch from ea2b9da to 32d209f Compare June 10, 2026 08:55

	export COSA_DIR=$HOME/workspace/build
	export COSA_DIR="$HOME/workspace/build"
	mkdir -p "$COSA_DIR"

	--device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp --name=cosa "${COREOS_ASSEMBLER_CONTAINER}" "$@";
	--device=/dev/fuse --tmpfs=/tmp -v=/var/tmp:/var/tmp "${COREOS_ASSEMBLER_CONTAINER}" "$@";

Conversation

Roshan-R commented Mar 31, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Roshan-R commented Apr 1, 2026

Uh oh!

Roshan-R commented Apr 8, 2026

Uh oh!

joelcapitao commented Apr 14, 2026

Uh oh!

Roshan-R commented Apr 17, 2026

Uh oh!

Roshan-R commented Apr 20, 2026

Uh oh!

Roshan-R commented Apr 22, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot Apr 22, 2026

Choose a reason for hiding this comment

Uh oh!

jbtrystram Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Roshan-R commented Apr 30, 2026

Uh oh!

Roshan-R commented May 8, 2026

Uh oh!

c4rt0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Rolv-Apneseth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Roshan-R commented Jun 1, 2026

Uh oh!

joelcapitao commented Jun 1, 2026

Uh oh!

joelcapitao commented Jun 1, 2026

Uh oh!

Roshan-R commented Jun 1, 2026

Uh oh!

Roshan-R commented Jun 1, 2026

Uh oh!

Roshan-R commented Jun 2, 2026

Uh oh!

c4rt0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joelcapitao left a comment

Choose a reason for hiding this comment

Uh oh!

Roshan-R commented Jun 10, 2026

Uh oh!

joelcapitao commented Jun 10, 2026

Uh oh!

Roshan-R commented Jun 10, 2026