hack/deploy.sh failed on fedora server 43

[andreiperepiolkin]

Ive tried deploy using hack/deploy.sh on fresh fedora 43 server with no ui.
First installation failed with no podman or qemu, so Ive installed podman and qemu.

Next it failed because of no-limit in /playbooks/install-minikube.yaml is not supported for qemu.

So Ive changed it to be like this:

 - name: Start minikube
      command: >-
          minikube start
          --kubernetes-version={{ kubernetes_version }}
          --embed-certs
          --container-runtime=cri-o
          --addons ingress,ingress-dns
          --cpus 8
          --memory 12g
          --delete-on-failure=true
          --disable-metrics=true
          {{ ramdisk_command }}

After that process hang up on setting up gerrit. It constantly printes:

...
2026-01-04T01:25:54+01:00       INFO    Waiting 10s for gerrit statefulset to be ready...
2026-01-04T01:26:04+01:00       INFO    Waiting 10s for gerrit statefulset to be ready...
2026-01-04T01:26:14+01:00       INFO    Waiting 10s for gerrit statefulset to be ready...
2026-01-04T01:26:24+01:00       INFO    Waiting 10s for gerrit statefulset to be ready...
2026-01-04T01:26:34+01:00       INFO    Waiting 10s for gerrit statefulset to be ready...
2026-01-04T01:26:44+01:00       INFO    Waiting 10s for gerrit statefulset to be ready...
2026-01-04T01:26:54+01:00       INFO    Waiting 10s for gerrit statefulset to be ready...
...

user@sf-test:~$ kubectl describe pods gerrit-0
Name:             gerrit-0
Namespace:        sf
Priority:         0
Service Account:  default
Node:             minikube/10.0.2.15
Start Time:       Sat, 03 Jan 2026 23:55:42 +0100
Labels:           app=sf
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=gerrit-996c47c8c
                  run=gerrit
                  statefulset.kubernetes.io/pod-name=gerrit-0
Annotations:      <none>
Status:           Running
SeccompProfile:   RuntimeDefault
IP:               10.244.0.7
IPs:
  IP:           10.244.0.7
Controlled By:  StatefulSet/gerrit
Init Containers:
  gerrit-init:
    Container ID:    cri-o://eec46afd4091a67c96d4caa56090e6d875c40593193554aaa1d183036f7a9695
    Image:           quay.io/software-factory/gerrit:3.6.4-8
    Image ID:        eafe9d99a8baee0c79edfd049c965b07ec0c0ab02855c199a2b750c061cf6ffe
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      sh
      -c
      #!/bin/bash

      set -ex

      export HOME=/gerrit
      # The /dev/./urandom is not a typo. https://stackoverflow.com/questions/58991966/what-java-security-egd-option-is-for
      JAVA_OPTIONS="-Djava.security.egd=file:/dev/./urandom -Xms${JVM_XMS} -Xmx${JVM_XMX}"

      echo "Initializing Gerrit site ..."
      java ${JAVA_OPTIONS} -jar /var/gerrit/bin/gerrit.war init -d ~/ --batch --no-auto-start --skip-plugins

      echo "Installing plugins ..."
      cp -u /var/gerrit-plugins/* ~/plugins

      cat << EOF > ~/.gitconfig
      [user]
          name = SF initial configurator
          email = admin@${FQDN}
      [gitreview]
          username = admin
      [push]
          default = simple
      EOF

      echo "Ensure admin user"
      # This command is noop if admin user already exists
      pynotedb create-admin-user --email "admin@${FQDN}" --pubkey "${GERRIT_ADMIN_SSH_PUB}" \
        --all-users ~/git/All-Users.git --scheme gerrit

      echo "Setting Gerrit config file ..."
      git config -f ~/etc/gerrit.config --replace-all gerrit.canonicalWebUrl "https://gerrit.${FQDN}"
      git config -f ~/etc/gerrit.config --replace-all auth.type "DEVELOPMENT_BECOME_ANY_ACCOUNT"
      git config -f ~/etc/gerrit.config --replace-all sshd.listenaddress "*:29418"
      git config -f ~/etc/gerrit.config --unset-all httpd.listenUrl
      git config -f ~/etc/gerrit.config --add httpd.listenUrl "proxy-https://*:8080/"
      git config -f ~/etc/gerrit.config --replace-all user.email "gerrit@${FQDN}"
      git config -f ~/etc/gerrit.config --replace-all sendemail.enable "false"

      echo "Install the ready.sh script"
      cat << EOF > ~/ready.sh
      echo "Waiting for httpd"
      curl --fail http://localhost:8080/config/server/version

      echo "Waiting for sshd"
      python3 -c 'import socket; socket.socket(socket.AF_INET, socket.SOCK_STREAM).connect(("localhost", 29418))'
      EOF
      chmod +x ~/ready.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 03 Jan 2026 23:58:20 +0100
      Finished:     Sun, 04 Jan 2026 00:01:51 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  768Mi
    Requests:
      cpu:     100m
      memory:  512Mi
    Environment:
      GERRIT_ADMIN_SSH_PUB:  <set to the key 'pub' in secret 'admin-ssh-key'>  Optional: false
      FQDN:                  sfop.me
      JVM_XMS:               256m
      JVM_XMX:               512m
    Mounts:
      /gerrit from gerrit (rw)
Containers:
  gerrit:
    Container ID:    cri-o://743b826dcf1521ef174f2d708baa8e769f0ed20af99723e87489ddd3c9bab347
    Image:           quay.io/software-factory/gerrit:3.6.4-8
    Image ID:        eafe9d99a8baee0c79edfd049c965b07ec0c0ab02855c199a2b750c061cf6ffe
    Ports:           8080/TCP (gerrit-httpd), 29418/TCP (gerrit-sshd)
    Host Ports:      0/TCP (gerrit-httpd), 0/TCP (gerrit-sshd)
    SeccompProfile:  RuntimeDefault
    Command:
      sh
      -c
      #!/bin/bash

      set -ex

      # The /dev/./urandom is not a typo. https://stackoverflow.com/questions/58991966/what-java-security-egd-option-is-for
      JAVA_OPTIONS="-Djava.security.egd=file:/dev/./urandom -Xms${JVM_XMS} -Xmx${JVM_XMX}"

      echo "Set local git config for gerrit admin"
      cat << EOF > ~/.gitconfig
      [user]
          name = SF initial configurator
          email = admin@${FQDN}
      [gitreview]
          username = admin
      [push]
          default = simple
      EOF

      echo "Setup .ssh/config to allow container exec of 'ssh gerrit'"
      mkdir -p ~/.ssh
      cat << EOF > ~/.ssh/config
      Host gerrit
      User admin
      Hostname ${HOSTNAME}
      Port 29418
      IdentityFile ~/.ssh/gerrit_admin
      EOF

      echo "Copy Gerrit Admin SSH keys on filesystem"
      echo "${GERRIT_ADMIN_SSH}" > ~/.ssh/gerrit_admin
      chmod 0600 ~/.ssh/gerrit_admin

      unset GERRIT_ADMIN_SSH

      echo "Running Gerrit ..."
      exec java ${JAVA_OPTIONS} -jar /var/gerrit/bin/gerrit.war daemon -d ~/
    State:          Running
      Started:      Sun, 04 Jan 2026 01:24:34 +0100
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Sun, 04 Jan 2026 01:16:11 +0100
      Finished:     Sun, 04 Jan 2026 01:19:28 +0100
    Ready:          False
    Restart Count:  16
    Limits:
      cpu:     1
      memory:  768Mi
    Requests:
      cpu:      100m
      memory:   512Mi
    Liveness:   exec [bash /gerrit/ready.sh] delay=5s timeout=5s period=20s #success=1 #failure=20
    Readiness:  exec [bash /gerrit/ready.sh] delay=0s timeout=5s period=10s #success=1 #failure=20
    Startup:    exec [bash /gerrit/ready.sh] delay=5s timeout=2s period=20s #success=1 #failure=10
    Environment:
      HOME:              /gerrit
      FQDN:              sfop.me
      JVM_XMS:           128m
      JVM_XMX:           512m
      GERRIT_ADMIN_SSH:  <set to the key 'priv' in secret 'admin-ssh-key'>  Optional: false
    Mounts:
      /gerrit from gerrit (rw)
  managesf-resources:
    Container ID:    cri-o://17cba8299b43e6a0665b791a97e4204d88bb80d62f221816fd54d4a017cedfe1
    Image:           quay.io/software-factory/sf-op-busybox:1.5-20250925-1
    Image ID:        b14b2038536f8bbab66df2d1043b14c0819ca930bc38e394e18295932e275e4e
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      sh
      -c
      #!/bin/sh

      if [ "${HOME}" == "/" ]; then
          echo "HOME can not be / dir!"
          exit 1
      fi

      mkdir -p ~/.ssh
      chmod 0700 ~/.ssh
      echo "${SF_ADMIN_SSH}" > ~/.ssh/id_rsa
      chmod 0400 ~/.ssh/id_rsa

      cat << EOF > ~/.ssh/config
      Host gerrit
      User admin
      Hostname ${GERRIT_SSHD_PORT_29418_TCP_ADDR}
      Port ${GERRIT_SSHD_SERVICE_PORT_GERRIT_SSHD}
      IdentityFile ~/.ssh/id_rsa
      StrictHostKeyChecking no
      EOF

      sleep inf

    State:          Running
      Started:      Sun, 04 Jan 2026 00:05:39 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  256Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      HOME:          /tmp
      FQDN:          sfop.me
      SF_ADMIN_SSH:  <set to the key 'priv' in secret 'admin-ssh-key'>  Optional: false
    Mounts:
      /etc/managesf from managesf-resources-config-vol (rw)
      /usr/share/managesf from managesf-resources-tooling-vol (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  gerrit:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  gerrit-gerrit-0
    ReadOnly:   false
  managesf-resources-config-vol:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      managesf-resources-config-map
    Optional:  false
  managesf-resources-tooling-vol:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        managesf-resources-tooling-config-map
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Normal   Killing    11m (x15 over 78m)     kubelet  spec.containers{gerrit}: Container gerrit failed startup probe, will be restarted
  Warning  BackOff    6m38s (x111 over 55m)  kubelet  spec.containers{gerrit}: Back-off restarting failed container gerrit in pod gerrit-0_sf(6bf302c7-91fd-4fec-935e-e64f4fb6e198)
  Normal   Pulled     3m4s (x17 over 85m)    kubelet  spec.containers{gerrit}: Container image "quay.io/software-factory/gerrit:3.6.4-8" already present on machine
  Warning  Unhealthy  90s (x122 over 81m)    kubelet  spec.containers{gerrit}: Startup probe failed: command timed out

Have to notice that before that, there was an error that some service is not able to connect/download from 8080 port...
Not sure this error might be attributed to, Ive disabled firewall on fedora through cockpit, yet it did not helped.

user@sf-test:~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:33259         0.0.0.0:*               LISTEN      25111/ssh
tcp        0      0 127.0.0.54:53           0.0.0.0:*               LISTEN      910/systemd-resolve
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      910/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1089/sshd: /usr/sbi
tcp        0      0 0.0.0.0:40857           0.0.0.0:*               LISTEN      25087/qemu-system-x
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      910/systemd-resolve
tcp        0      0 0.0.0.0:37611           0.0.0.0:*               LISTEN      25087/qemu-system-x
tcp6       0      0 :::9090                 :::*                    LISTEN      1/systemd
tcp6       0      0 :::22                   :::*                    LISTEN      1089/sshd: /usr/sbi
tcp6       0      0 ::1:33259               :::*                    LISTEN      25111/ssh
tcp6       0      0 :::5355                 :::*                    LISTEN      910/systemd-resolve
udp        0      0 127.0.0.54:53           0.0.0.0:*                           910/systemd-resolve
udp        0      0 127.0.0.53:53           0.0.0.0:*                           910/systemd-resolve
udp        0      0 127.0.0.1:323           0.0.0.0:*                           961/chronyd
udp        0      0 0.0.0.0:5355            0.0.0.0:*                           910/systemd-resolve
udp6       0      0 ::1:323                 :::*                                961/chronyd
udp6       0      0 :::5355                 :::*                                910/systemd-resolve

[mhuin]

Hello,

the deploy.sh is mostly used with centos 9 stream in our CI, an immediate workaround would be to use this OS instead.

[mhuin]

Ive tried deploy using hack/deploy.sh on fresh fedora 43 server with no ui. First installation failed with no podman or qemu, so Ive installed podman and qemu.

This is odd because podman should be installed by the second task in playbooks/install-minikube.yaml, which is called fairly early in deploy.sh - do you have more details about the issue you encountered here?

qemu also shouldn't be required, at least on centos 9 stream minikube uses the default container driver to run the cluster instead of a VM.

[andreiperepiolkin]

do you have more details about the issue you encountered here

Unfortunately not. May be I will be able to reproduce it.
I think once qemu was installed it was used as 'default' containerization tool, for some reason podman was ignored.
May be Ive clicked 'virtualisation host'(for initial package setup) during installation, but Im not sure.

[mhuin]

I have managed to run hack/deploy.sh successfully twice on a Fedora Cloud 43 Virtual Machine, the only pre-requisite was git (needed to clone sf-operator locally).

`
$ lscpu

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
`

`
$ free

total used free shared buff/cache available
Mem: 16360908 3493968 500296 23752 12721748 12866940
Swap: 8388604 140 8388464
`

`
$ cat /etc/os-release

NAME="Fedora Linux"
VERSION="43 (Cloud Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=43
VERSION_CODENAME=""
PRETTY_NAME="Fedora Linux 43 (Cloud Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:43"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f43/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=43
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=43
SUPPORT_END=2026-12-02
VARIANT="Cloud Edition"
VARIANT_ID=cloud
`

[andreiperepiolkin]

Ive re-deployed it and problem was solves by adding:
Permission to run passwordless sudo for podman

user ALL=(ALL) NOPASSWD: /usr/bin/podman

after

%wheel  ALL=(ALL)       ALL

In file /etc/sudoers

I guess my original idea to install podman and qemu manually was a mistake