exited 1: local erasure endpoints must use distinct physical disks

[Nightsee]

I have deployed a rustfs tenancy using the rustfs operator, But I am getting the error below, it is preventing the pods from starting.

exited 1: Initializing data directories: Initializing log directory: /logs Starting: /usr/bin/rustfs [FATAL] Server encountered an error and is shutting down: local erasure endpoints must use distinct physical disks; detected shared devices [sdg => /data/rustfs0, /data/rustfs1, /data/rustfs2, /data/rustfs3]. validation diagnostics: [path='/data/rustfs0', canonical='/data/rustfs0', st_dev='8:96', device_ids=[sdg]; path='/data/rustfs1', canonical='/data/rustfs1', st_dev='8:96', device_ids=[sdg]; path='/data/rustfs2', canonical='/data/rustfs2', st_dev='8:96', device_ids=[sdg]; path='/data/rustfs3', canonical='/data/rustfs3', st_dev='8:96', device_ids=[sdg]]. Set RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true only for local testing or CI to bypass this safety check

Note: I am using Directpv as a csi in my cluster to provision the necessary pvs on local disks. I have looked for a way to make directpv distribute the pvs accross disks reliably, but I couldn't find anything really usefull.

I see there in the logs I can skip the check by setting the RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true env variable. But I don't see exactly where on the config.
can you please provide more details on where I should set that ?

[GatewayJ]

You can set it on the Tenant custom resource under spec.env, for example:

spec:
  env:
    - name: RUSTFS_UNSAFE_BYPASS_DISK_CHECK
      value: "true"

For an existing Tenant, edit the Tenant resource, not the operator Deployment or Helm chart env:
kubectl edit tenant -n
That said, this should only be used for local testing or CI. The error means RustFS detected that /data/rustfs0, /data/rustfs1, /data/rustfs2, and /data/rustfs3 are all backed by the same physical device. For production, the safer fix is to make DirectPV place those PVCs on distinct physical disks, or use a 1 server x 1 volume layout if you only have one physical disk available.

[Nightsee]

You can set it on the Tenant custom resource under spec.env, for example:

spec:
  env:
    - name: RUSTFS_UNSAFE_BYPASS_DISK_CHECK
      value: "true"

For an existing Tenant, edit the Tenant resource, not the operator Deployment or Helm chart env: kubectl edit tenant -n That said, this should only be used for local testing or CI. The error means RustFS detected that /data/rustfs0, /data/rustfs1, /data/rustfs2, and /data/rustfs3 are all backed by the same physical device. For production, the safer fix is to make DirectPV place those PVCs on distinct physical disks, or use a 1 server x 1 volume layout if you only have one physical disk available.

hi,

I have tried this; but still the same issue, the pods won't start. here is my tenant yaml (I am using the rustfs operator):

`
apiVersion: rustfs.com/v1alpha1
kind: Tenant
metadata:
creationTimestamp: 2026-08-03T11:16:09Z
generation: 2
name: testenant
namespace: rustfs
resourceVersion: '80123319'
uid: 42abe47c-b920-4d21-bd08-678dc4710c28
spec:
pools:

• name: pool-0
  servers: 4
  persistence:
  volumesPerServer: 4
  volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
  requests:
  storage: 5Gi
  storageClassName: directpv-min-io
  image: rustfs/rustfs:latest
  mountPath: /data
  env:
• name: RUSTFS_UNSAFE_BYPASS_DISK_CHECK
  value: 'true'
  securityContext:
  runAsNonRoot: true
  `

[GatewayJ]

Thanks for the updated YAML. The placement of spec.env is correct.

We checked the current code:

• The Operator copies Tenant.spec.env into the RustFS container and detects environment changes when reconciling the StatefulSet.
• Current RustFS still supports RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true. When effective, it should log:

Skipping local physical disk independence validation due to explicit environment override

Therefore, this looks more like a reconciliation or rollout issue than a removed RustFS feature. Please check each layer below.

1. Check whether the Tenant update was observed

NS=rustfs
TENANT=testenant

kubectl -n "$NS" get tenant "$TENANT" \
  -o jsonpath='{.metadata.generation}{"\t"}{.status.observedGeneration}{"\n"}{range .spec.env[*]}{.name}={.value}{"\n"}{end}'

Please verify that:

• RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true is present.
• If status.observedGeneration is populated, it matches metadata.generation.

If the generations differ, the Operator has not completed reconciliation.

2. Check the generated StatefulSet

kubectl -n "$NS" get sts -l "rustfs.tenant=$TENANT" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\tenv="}{.spec.template.spec.containers[?(@.name=="rustfs")].env[?(@.name=="RUSTFS_UNSAFE_BYPASS_DISK_CHECK")].value}{"\trevision="}{.status.currentRevision}{"/"}{.status.updateRevision}{"\n"}{end}'

Expected result:

<statefulset>    env=true    revision=<same revision>/<same revision>

If the variable exists on the Tenant but not on the StatefulSet, please provide the Operator image version and reconciliation logs:

kubectl get deployment -A \
  -l app.kubernetes.io/name=rustfs-operator \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'

3. Check the actual Pods

kubectl -n "$NS" get pods -l "rustfs.tenant=$TENANT" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\tenv="}{.spec.containers[?(@.name=="rustfs")].env[?(@.name=="RUSTFS_UNSAFE_BYPASS_DISK_CHECK")].value}{"\timage="}{.spec.containers[?(@.name=="rustfs")].image}{"\timageID="}{.status.containerStatuses[?(@.name=="rustfs")].imageID}{"\n"}{end}'

If the StatefulSet contains env=true but the Pods do not, the rollout is incomplete. Check the update strategy:

kubectl -n "$NS" get sts -l "rustfs.tenant=$TENANT" \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.updateStrategy.type}{"\n"}{end}'

After confirming the StatefulSet template is correct, recreate one failing Pod and verify it before touching the others:

kubectl -n "$NS" delete pod <pod-name>

This deletes only the Pod object, not its PVCs.

4. Check the logs from the newly created Pod

kubectl -n "$NS" logs <pod-name> -c rustfs
kubectl -n "$NS" logs <pod-name> -c rustfs --previous

Please check whether the bypass warning appears. If the Pod specification shows env=true but a newly created Pod still reports the distinct-disk fatal error, please provide:

• The complete startup log
• The Pod imageID from step 3
• The Operator image version
• kubectl -n rustfs describe tenant testenant

Using rustfs/rustfs:latest can make exact-version diagnosis difficult, so a pinned RustFS release tag or image digest is recommended.

5. Confirm the DirectPV allocation

The diagnostics already show that all four local paths are backed by sdg. Please also provide:

kubectl directpv list drives --output wide
kubectl directpv list volumes --all
kubectl -n "$NS" get pvc -l "rustfs.tenant=$TENANT" -o wide

The bypass should only be used for testing or CI. With volumesPerServer: 4, each RustFS server is expected to have four independent local devices. If each node has only one physical disk, a new volumesPerServer: 1 layout is safer than presenting four directories from the same disk. Do not change an existing storage layout containing data without planning a migration.

Relevant RustFS bypass implementation: [endpoints.rs](https://github.com/rustfs/rustfs/blob/acce8b22536aa6ade42514b5f42a000cba24cf7a/crates/ecstore/src/layout/endpoints.rs#L1562-L1575).

---