Skip to content

implemented resource quota in default namespace#637

Open
phunga003 wants to merge 1 commit intomainfrom
resource-limiter
Open

implemented resource quota in default namespace#637
phunga003 wants to merge 1 commit intomainfrom
resource-limiter

Conversation

@phunga003
Copy link
Contributor

quota can be changed, and is specific to namespace. You can potentially create a privilidged namespace that allow for more resource access

@phunga003
Copy link
Contributor Author

Because all glados users are interacting with the k8s using a shared service account. This acts as a sort of shared resource pool for all users. This means that a user on glados can hold resources hostage by spawning nonsensical jobs.

This pr only address the probability of users submitting abnormally large jobs crippling the service by crashing the node.

More fine-grained user resource usage control must be done on the frontend (next.js) or backend (flask) to preemptively disallow users from starting new experiments if the quota is filled

Copy link
Contributor

@helena-donaldson helena-donaldson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Referring to Teams discussion, issue with Backend and Frontend pods present that needs addressing in configs. Additionally, once that is fixed, I would like to test what happens if a user requests a number of workers higher than 4, and if we reach 50 pods, what happens on the user side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants