Skip to content

feat: support of PodDisruptionBudgets#6864

Closed
SuperQ wants to merge 1 commit into
pingcap:mainfrom
SuperQ:superq/tikv_scale_subresource
Closed

feat: support of PodDisruptionBudgets#6864
SuperQ wants to merge 1 commit into
pingcap:mainfrom
SuperQ:superq/tikv_scale_subresource

Conversation

@SuperQ
Copy link
Copy Markdown
Contributor

@SuperQ SuperQ commented Apr 27, 2026

In order for PodDisruptionBudgets to work on various groups of Pods we need to implement the Scale subresource at the direct owner of the Pod.

Otherwise the PDB will return errors like:

status:
  conditions:
    - lastTransitionTime: "2026-04..."
      message: tikvs.core.pingcap.com does not implement the scale subresource
      reason: SyncFailed
      status: "False"
      type: DisruptionAllowed

Followup to #6213

Closes: #6901
Closes: #4221

@ti-chi-bot
Copy link
Copy Markdown
Contributor

ti-chi-bot Bot commented Apr 27, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign csuzhangxc for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot
Copy link
Copy Markdown
Contributor

ti-chi-bot Bot commented Apr 27, 2026

Hi @SuperQ. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@github-actions github-actions Bot added the v2 for operator v2 label Apr 27, 2026
@ti-chi-bot ti-chi-bot Bot added the size/M label Apr 27, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 37.97%. Comparing base (fe02609) to head (29edafd).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #6864   +/-   ##
=======================================
  Coverage   37.97%   37.97%           
=======================================
  Files         393      393           
  Lines       22605    22605           
=======================================
  Hits         8584     8584           
  Misses      14021    14021           
Flag Coverage Δ
unittest 37.97% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@liubog2008
Copy link
Copy Markdown
Member

It seems not reasonable to add /scale for instance crds. Could you comment your PDB spec?

@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented Apr 28, 2026

Sure, the PDB is pretty simple / standard.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: test-tikv
  namespace: test
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: tikv
      app.kubernetes.io/instance: test

My understanding is that the PodDisruptionBudget acts on the Pod resource and looks for the Pod's owner.

So, this example:

apiVersion: v1
kind: Pod
metadata:
  name: test-tikv-326vs
  namespace: test
  labels:
    app.kubernetes.io/component: tikv
    app.kubernetes.io/instance: test
    ...
  ownerReferences:
  - apiVersion: core.pingcap.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: TiKV
    name: test-tikv
    uid: ...

So when Kubernetes tries to gather all the Pods matching the label spec it looks at the owner references and finds that TiKV is the owner, no mention of TiKVGroup.

So the way things are now, the PDB can't work.

@liubog2008
Copy link
Copy Markdown
Member

https://kubernetes.io/docs/tasks/run-application/configure-pdb/#arbitrary-controllers-and-selectors

It seems that maxUnavailable is not supported for current implementation. I'll dig more about this issue.

Anyway, this PR cannot fix the issue.

@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 9, 2026

Oof, that's an unfortunate restriction.

So the only real option is that the TiDB Operator will need to create pods via an indirect method, for example each Pod needs to have an individual StatefulSet.

@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 9, 2026

If you would like, I can file a tracking issue detailing the need here.

@liubog2008
Copy link
Copy Markdown
Member

Now operator manages TIKVGroup -> TiKV -> Pod ownership. However, PDB cannot find TiKVGroup from pod's owner. It can be enhanced by kubernetes easily. But now, operator v2 cannot support PDB.

@liubog2008
Copy link
Copy Markdown
Member

What we can do now is only adding the leader eviction scheduler to one pod even if there are more than one pod is deleting.

@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 15, 2026

What needs to be done is have an intermediate Pod controller like this:

TIKVGroup -> TiKV -> StatefulSet -> Pod

Happy to help implement this.

@liubog2008
Copy link
Copy Markdown
Member

What needs to be done is have an intermediate Pod controller like this:

TIKVGroup -> TiKV -> StatefulSet -> Pod
Happy to help implement this.

It's useless. PDB needs to get replicas from pod's owner.

@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 15, 2026

It does work, I have filed #6901 with some details on how multiple StatefulSets can be combined into a single PDB.

In this new design, the Pod's owner is the StatefulSet, which supports maxUnavailable. This is how and why we implemented this in the valkey-operator.

@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 15, 2026

Now that we have the details we need in the issue, I am going to close this PR.

@SuperQ SuperQ closed this May 15, 2026
@liubog2008 liubog2008 reopened this May 20, 2026
Comment thread api/core/v1alpha1/pd_types.go Outdated
@liubog2008
Copy link
Copy Markdown
Member

liubog2008 commented May 20, 2026

I have verified the PDB in my own env.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"policy/v1","kind":"PodDisruptionBudget","metadata":{"annotations":{},"name":"tikv-pdb","namespace":"default"},"spec":{"maxUnavailable":1,"selector":{"matchLabels":{"pingcap.com/component":"tikv"}}}}
  creationTimestamp: "2026-05-20T01:47:48Z"
  generation: 1
  name: tikv-pdb
  namespace: default
  resourceVersion: "12004810"
  uid: 5d6d5d3a-ec36-4d9a-904b-1a8c89ec8cbd
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      pingcap.com/component: tikv
status:
  conditions:
  - lastTransitionTime: "2026-05-20T01:47:48Z"
    message: ""
    observedGeneration: 1
    reason: SufficientPods
    status: "True"
    type: DisruptionAllowed
  currentHealthy: 3
  desiredHealthy: 2
  disruptionsAllowed: 1
  expectedPods: 3
  observedGeneration: 1

There are still some issues of this PR:

  • Only spec.replicas path is needed. Both status and selector are useless. (But controller tools need status path to generate crd)
  • New field spec.replicas should be added.
  • spec.replicas should be optional, default to 1 and cannot be changed.

@SuperQ SuperQ force-pushed the superq/tikv_scale_subresource branch from 38939e1 to 70720b8 Compare May 20, 2026 07:05
@SuperQ SuperQ requested a review from liubog2008 May 20, 2026 07:05
@ti-chi-bot ti-chi-bot Bot added size/XL and removed size/M labels May 20, 2026
@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 20, 2026

Updated. I think we can probably remove the Scale subresource from the FooGroupSpec, since they don't directly create Pods.

Comment thread api/core/v1alpha1/pd_types.go Outdated
Comment thread api/core/v1alpha1/pd_types.go Outdated
Comment thread api/core/v1alpha1/pd_types.go Outdated
@liubog2008
Copy link
Copy Markdown
Member

@SuperQ I left some comments.

Updated. I think we can probably remove the Scale subresource from the FooGroupSpec, since they don't directly create Pods.

The scale of group(e.g. TiKVGroup) is for HPA and VPA. It has been used. The scale of instance(e.g. TiKV) is only for compatibility of PDB

In order for PodDisruptionBudgets to work on various groups of Pods
we need to implement the Scale subresource at the direct owner of
the Pod.

Otherwise the PDB will return errors like:

```yaml
status:
  conditions:
    - lastTransitionTime: "2026-04..."
      message: tikvs.core.pingcap.com does not implement the scale subresource
      reason: SyncFailed
      status: "False"
      type: DisruptionAllowed
```

Followup to pingcap#6213

Signed-off-by: SuperQ <superq@gmail.com>
@SuperQ SuperQ force-pushed the superq/tikv_scale_subresource branch from 70720b8 to 29edafd Compare May 20, 2026 08:36
@ti-chi-bot ti-chi-bot Bot added size/L and removed size/XL labels May 20, 2026
@SuperQ SuperQ requested a review from liubog2008 May 20, 2026 08:38
// +kubebuilder:validation:Maximum=1
// +default:value=1
// +optional
Replicas *int32 `json:"replicas"`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment about this field. "It's only designed for compatibility with PDB and cannot be changed"

// +kubebuilder:validation:Maximum=1
// +default:value=1
// +optional
Replicas *int32 `json:"replicas"`
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add omitempty in json

@liubog2008
Copy link
Copy Markdown
Member

Still two comments are left. Thanks @SuperQ

@liubog2008 liubog2008 changed the title Implement scale subresource on Pod owners feat: support of PodDisruptionBudgets May 22, 2026
@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 22, 2026

Should we also add a kubebuilder validation like:

  // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="replicas is immutable"

@SuperQ SuperQ closed this May 22, 2026
@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 22, 2026

Bah! I didn't mean to close this PR.

@SuperQ
Copy link
Copy Markdown
Contributor Author

SuperQ commented May 22, 2026

Opened #6912

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support PDB for k8s cluster version upgrade Support PodDisruptionBudget in v2

3 participants