Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

also log the delta when lendingLimit is updated #256

Merged
merged 1 commit into from
Oct 16, 2024

Conversation

dgrove-oss
Copy link
Collaborator

No description provided.

@dgrove-oss
Copy link
Collaborator Author

dgrove-oss commented Oct 16, 2024

An example showing 2 cordons followed by an uncordon.

2024-10-16T20:54:59.04138093Z	INFO	logr@v1.4.2/logr.go:280	Updated NoSchedule information	{"controller": "NodeMonitor", "namespace": "", "name": "test-worker", "reconcileID": "90b1bc8c-4c0b-4011-9de7-21aa331b3726", "Number NoSchedule Nodes": 1, "NoSchedule Resource Details": {"test-worker":{"cpu":"4","ephemeral-storage":"102625208Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","hugepages-32Mi":"0","hugepages-64Ki":"0","memory":"12231868Ki","nvidia.com/gpu":"8"}}}
2024-10-16T20:54:59.05508718Z	INFO	logr@v1.4.2/logr.go:280	Updated lending limits	{"controller": "SlackClusterQueueMonitor", "namespace": "", "name": "slack-cluster-queue", "reconcileID": "39f686e1-6095-460c-abbe-e62f181f0ca5", "Changed by": {"cpu":"-4","memory":"-12231868Ki","nvidia.com/gpu":"-8"}, "Updated Resources": [{"name":"cpu","nominalQuota":"8","lendingLimit":"4"},{"name":"memory","nominalQuota":"128Gi","lendingLimit":"121985860Ki"},{"name":"nvidia.com/gpu","nominalQuota":"10","lendingLimit":"2"},{"name":"nvidia.com/roce_gdr","nominalQuota":"1"},{"name":"pods","nominalQuota":"100"}]}
2024-10-16T20:57:40.114535465Z	INFO	logr@v1.4.2/logr.go:280	Updated NoSchedule information	{"controller": "NodeMonitor", "namespace": "", "name": "test-worker2", "reconcileID": "7ba8173c-98b4-4e95-9e75-47a43007ac9f", "Number NoSchedule Nodes": 2, "NoSchedule Resource Details": {"test-worker":{"cpu":"4","ephemeral-storage":"102625208Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","hugepages-32Mi":"0","hugepages-64Ki":"0","memory":"12231868Ki","nvidia.com/gpu":"8"},"test-worker2":{"cpu":"4","ephemeral-storage":"102625208Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","hugepages-32Mi":"0","hugepages-64Ki":"0","memory":"12231868Ki","nvidia.com/gpu":"8"}}}
2024-10-16T20:57:40.125715965Z	INFO	logr@v1.4.2/logr.go:280	Updated lending limits	{"controller": "SlackClusterQueueMonitor", "namespace": "", "name": "slack-cluster-queue", "reconcileID": "ee7ae069-f9d4-4359-80cd-e0e60ed966df", "Changed by": {"cpu":"-4","memory":"-12231868Ki","nvidia.com/gpu":"-2"}, "Updated Resources": [{"name":"cpu","nominalQuota":"8","lendingLimit":"0"},{"name":"memory","nominalQuota":"128Gi","lendingLimit":"109753992Ki"},{"name":"nvidia.com/gpu","nominalQuota":"10","lendingLimit":"0"},{"name":"nvidia.com/roce_gdr","nominalQuota":"1"},{"name":"pods","nominalQuota":"100"}]}
2024-10-16T20:57:55.203261472Z	INFO	logr@v1.4.2/logr.go:280	Updated NoSchedule information	{"controller": "NodeMonitor", "namespace": "", "name": "test-worker", "reconcileID": "d2583b93-6456-4e0e-8693-013d3ba63788", "Number NoSchedule Nodes": 1, "NoSchedule Resource Details": {"test-worker2":{"cpu":"4","ephemeral-storage":"102625208Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","hugepages-32Mi":"0","hugepages-64Ki":"0","memory":"12231868Ki","nvidia.com/gpu":"8"}}}
2024-10-16T20:57:55.213178556Z	INFO	logr@v1.4.2/logr.go:280	Updated lending limits	{"controller": "SlackClusterQueueMonitor", "namespace": "", "name": "slack-cluster-queue", "reconcileID": "86a422c5-871d-4cd8-b911-1fd33678ad3a", "Changed by": {"cpu":"4","memory":"12231868Ki","nvidia.com/gpu":"2"}, "Updated Resources": [{"name":"cpu","nominalQuota":"8","lendingLimit":"4"},{"name":"memory","nominalQuota":"128Gi","lendingLimit":"121985860Ki"},{"name":"nvidia.com/gpu","nominalQuota":"10","lendingLimit":"2"},{"name":"nvidia.com/roce_gdr","nominalQuota":"1"},{"name":"pods","nominalQuota":"100"}]}

@dgrove-oss dgrove-oss merged commit b45a492 into project-codeflare:main Oct 16, 2024
3 checks passed
@dgrove-oss dgrove-oss deleted the lllog branch October 16, 2024 21:09
@dgrove-oss dgrove-oss added the enhancement New feature or request label Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant