Speed up creation/deletion of game servers in a set #542

jkowalski · 2019-01-31T22:46:52Z

parallelizing create/delete calls in GSS controller
creating dedicated worker queues for creation/deletion requests
replacing default rate limiters with ones that don't have global QPS limit

That, plus general cleanup of GSS controller for readability.

Also, introduced GS state cache, so that actions taken by GSS controller are more precise because they don't rely on stale informer data.

agones-bot · 2019-01-31T22:52:11Z

Build Failed 😱

Build Id: 1b95cdde-44fb-447d-9605-251262bd7128

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2019-02-01T17:03:02Z

Build Failed 😱

Build Id: 26be4f54-3be0-46fe-82a8-0d0c7a234151

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2019-02-01T18:13:03Z

Build Failed 😱

Build Id: c46f8ace-8882-482e-9a5a-9ab9da04b119

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2019-02-01T18:20:14Z

Build Failed 😱

Build Id: e484653c-5ac5-4572-ba50-9e08152341fb

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2019-02-01T18:27:34Z

Build Failed 😱

Build Id: 67b09fe2-2b8d-4f56-99b7-0cd54e2f27d7

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel

Will wait to do a proper review once you're ready - but was just reading through, found a few minor cleanup things.

pkg/gameserversets/gameserver_state_cache_test.go

pkg/gameserversets/gameserver_state_cache.go

pkg/gameserversets/controller.go

agones-bot · 2019-02-02T00:05:05Z

Build Succeeded 👏

Build Id: d523e7e5-41c5-4c6a-adc8-4436300b3934

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-86e6868
image: gcr.io/agones-images/agones-sdk:0.8.0-86e6868
image: gcr.io/agones-images/agones-ping:0.8.0-86e6868
Linux C++ SDK (build: dev): agonessdk-0.8.0-86e6868-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-86e6868-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-86e6868-src.zip
SDK Server: agonessdk-server-0.8.0-86e6868.zip

A preview of the website will exist for the next 30 days:

https://86e6868-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-86e6868

agones-bot · 2019-02-02T00:08:35Z

Build Succeeded 👏

Build Id: 3ce57273-0b61-490d-b2e1-085cbedd55fd

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-19f98b5
image: gcr.io/agones-images/agones-sdk:0.8.0-19f98b5
image: gcr.io/agones-images/agones-ping:0.8.0-19f98b5
Linux C++ SDK (build: dev): agonessdk-0.8.0-19f98b5-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-19f98b5-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-19f98b5-src.zip
SDK Server: agonessdk-server-0.8.0-19f98b5.zip

A preview of the website will exist for the next 30 days:

https://19f98b5-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-19f98b5

markmandel

I'll probably go through this again, I want to make sure I understand what's happening here.

Added some notes though for mostly minor cleanup things.

pkg/gameservers/controller.go

pkg/gameserversets/controller.go

agones-bot · 2019-02-04T19:54:24Z

Build Succeeded 👏

Build Id: cc8bc9f1-d5ab-40be-b56b-4817ef843b41

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-d173b4a
image: gcr.io/agones-images/agones-sdk:0.8.0-d173b4a
image: gcr.io/agones-images/agones-ping:0.8.0-d173b4a
Linux C++ SDK (build: dev): agonessdk-0.8.0-d173b4a-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-d173b4a-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-d173b4a-src.zip
SDK Server: agonessdk-server-0.8.0-d173b4a.zip

A preview of the website will exist for the next 30 days:

https://d173b4a-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-d173b4a

ilkercelikyilmaz · 2019-02-04T22:37:10Z

pkg/gameservers/controller.go

-	c.workerqueue = workerqueue.NewWorkerQueue(c.syncGameServer, c.logger, stable.GroupName+".GameServerController")
+	c.workerqueue = workerqueue.NewWorkerQueueWithRateLimiter(c.syncGameServer, c.logger, stable.GroupName+".GameServerController",
+		workqueue.NewItemFastSlowRateLimiter(20*time.Millisecond, 500*time.Millisecond, 5))
+	c.creationWorkerQueue = workerqueue.NewWorkerQueueWithRateLimiter(c.syncGameServer, c.logger, stable.GroupName+".GameServerControllerCreation",


Should we consider having separate method for each workerqueue instead of syncGameServer method? Each method can only focus on the possible events rather than trying to process all.

queue is just a "dirty" bit, it's totally possible that the state of a resource will change underneath between enqueue and processing, so we should not assume anything about the state of each GS.

I understand your concern about queue. However the new "enqueueGameServerBasedOnState" method should put the messages to the appropriate queue. I think except syncGameServerDeletionTimestamp handler, every other handler ignore unexpected events.

agones-bot · 2019-02-04T23:48:55Z

Build Succeeded 👏

Build Id: 9aa1bfe8-8b91-4048-a754-aea47973403b

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-a9d1ba4
image: gcr.io/agones-images/agones-sdk:0.8.0-a9d1ba4
image: gcr.io/agones-images/agones-ping:0.8.0-a9d1ba4
Linux C++ SDK (build: dev): agonessdk-0.8.0-a9d1ba4-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-a9d1ba4-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-a9d1ba4-src.zip
SDK Server: agonessdk-server-0.8.0-a9d1ba4.zip

A preview of the website will exist for the next 30 days:

https://a9d1ba4-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-a9d1ba4

agones-bot · 2019-02-05T03:11:48Z

Build Succeeded 👏

Build Id: bc1da740-7d27-4cd9-ba8b-1ce95f647413

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-caf5cda
image: gcr.io/agones-images/agones-sdk:0.8.0-caf5cda
image: gcr.io/agones-images/agones-ping:0.8.0-caf5cda
Linux C++ SDK (build: dev): agonessdk-0.8.0-caf5cda-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-caf5cda-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-caf5cda-src.zip
SDK Server: agonessdk-server-0.8.0-caf5cda.zip

A preview of the website will exist for the next 30 days:

https://caf5cda-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-caf5cda

agones-bot · 2019-02-05T03:14:06Z

Build Succeeded 👏

Build Id: 3ffcf1db-2549-4fef-8d70-cb5e066c24e0

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-32cab47
image: gcr.io/agones-images/agones-sdk:0.8.0-32cab47
image: gcr.io/agones-images/agones-ping:0.8.0-32cab47
Linux C++ SDK (build: dev): agonessdk-0.8.0-32cab47-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-32cab47-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-32cab47-src.zip
SDK Server: agonessdk-server-0.8.0-32cab47.zip

A preview of the website will exist for the next 30 days:

https://32cab47-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-32cab47

markmandel

A couple more questions/comments, but otherwise I think this is good to go. Just took me a while to run through it, and understand it completely.

pkg/gameservers/controller.go

pkg/gameserversets/controller.go

markmandel · 2019-02-05T03:51:53Z

pkg/gameserversets/controller.go

+
+		// GS being deleted counts towards target replica count.
+		if !gs.ObjectMeta.DeletionTimestamp.IsZero() {
+			//handleGameServerUp(gs)


Minor comment cleanup

pkg/gameserversets/controller.go

markmandel

Just saw the health check, and a blank line cleanup - otherwise, good to go.

pkg/gameservers/controller.go

- parallelizing create/delete calls in GSS controller - creating dedicated worker queues for creation/deletion requests - replacing default rate limiters with ones that don't have global QPS limit

agones-bot · 2019-02-06T02:24:37Z

Build Succeeded 👏

Build Id: fced92f7-08c5-41ec-b203-eb4ee3e9c60b

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-2c821ed
image: gcr.io/agones-images/agones-sdk:0.8.0-2c821ed
image: gcr.io/agones-images/agones-ping:0.8.0-2c821ed
Linux C++ SDK (build: dev): agonessdk-0.8.0-2c821ed-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-2c821ed-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-2c821ed-src.zip
SDK Server: agonessdk-server-0.8.0-2c821ed.zip

A preview of the website will exist for the next 30 days:

https://2c821ed-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-2c821ed

markmandel

LGTM!

agones-bot · 2019-02-06T02:34:08Z

Build Succeeded 👏

Build Id: 8561d6f5-a946-41e8-ba21-291f52fcd4b7

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:0.8.0-b8d0644
image: gcr.io/agones-images/agones-sdk:0.8.0-b8d0644
image: gcr.io/agones-images/agones-ping:0.8.0-b8d0644
Linux C++ SDK (build: dev): agonessdk-0.8.0-b8d0644-dev-linux-arch_64.tar.gz
Linux C++ SDK (build: runtime): agonessdk-0.8.0-b8d0644-runtime-linux-arch_64.tar.gz
Linux C++ SDK (source): agonessdk-0.8.0-b8d0644-src.zip
SDK Server: agonessdk-server-0.8.0-b8d0644.zip

A preview of the website will exist for the next 30 days:

https://b8d0644-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/542/head:pr_542 && git checkout pr_542
helm install install/helm/agones --namespace agones-system --name agones --set agones.image.tag=0.8.0-b8d0644

jkowalski force-pushed the gameserverset-improvements branch from e3c6ab4 to 54c077c Compare February 1, 2019 16:48

markmandel reviewed Feb 1, 2019

View reviewed changes

jkowalski force-pushed the gameserverset-improvements branch 2 times, most recently from 86e6868 to 19f98b5 Compare February 1, 2019 23:58

markmandel reviewed Feb 2, 2019

View reviewed changes

jkowalski force-pushed the gameserverset-improvements branch from 19f98b5 to d173b4a Compare February 4, 2019 19:44

ilkercelikyilmaz reviewed Feb 4, 2019

View reviewed changes

jkowalski force-pushed the gameserverset-improvements branch from d173b4a to a9d1ba4 Compare February 4, 2019 23:39

jkowalski force-pushed the gameserverset-improvements branch 2 times, most recently from caf5cda to 32cab47 Compare February 5, 2019 03:03

markmandel reviewed Feb 5, 2019

View reviewed changes

markmandel added the area/performance Anything to do with Agones being slow, or making it go faster. label Feb 5, 2019

markmandel added this to the 0.8.0 milestone Feb 5, 2019

jkowalski force-pushed the gameserverset-improvements branch from 32cab47 to 2c821ed Compare February 6, 2019 02:14

markmandel reviewed Feb 6, 2019

View reviewed changes

pkg/gameservers/controller.go Show resolved Hide resolved

pkg/gameservers/controller.go Show resolved Hide resolved

Speed up creation/deletion of game servers in a set by:

b8d0644

- parallelizing create/delete calls in GSS controller - creating dedicated worker queues for creation/deletion requests - replacing default rate limiters with ones that don't have global QPS limit

jkowalski force-pushed the gameserverset-improvements branch from 2c821ed to b8d0644 Compare February 6, 2019 02:23

markmandel approved these changes Feb 6, 2019

View reviewed changes

markmandel merged commit f9e7e81 into googleforgames:master Feb 6, 2019

jkowalski deleted the gameserverset-improvements branch February 6, 2019 02:45

WontonSam mentioned this pull request Dec 1, 2023

[Snyk] Fix for 1 vulnerabilities WontonSam/Phiahplay-#9

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed up creation/deletion of game servers in a set #542

Speed up creation/deletion of game servers in a set #542

jkowalski commented Jan 31, 2019 •

edited

Loading

agones-bot commented Jan 31, 2019

agones-bot commented Feb 1, 2019

agones-bot commented Feb 1, 2019

agones-bot commented Feb 1, 2019

agones-bot commented Feb 1, 2019

markmandel left a comment

agones-bot commented Feb 2, 2019

agones-bot commented Feb 2, 2019

markmandel left a comment

agones-bot commented Feb 4, 2019

ilkercelikyilmaz Feb 4, 2019

jkowalski Feb 5, 2019

ilkercelikyilmaz Feb 5, 2019

agones-bot commented Feb 4, 2019

agones-bot commented Feb 5, 2019

agones-bot commented Feb 5, 2019

markmandel left a comment

markmandel Feb 5, 2019

jkowalski Feb 6, 2019

markmandel left a comment

agones-bot commented Feb 6, 2019

markmandel left a comment

agones-bot commented Feb 6, 2019

Speed up creation/deletion of game servers in a set #542

Speed up creation/deletion of game servers in a set #542

Conversation

jkowalski commented Jan 31, 2019 • edited Loading

agones-bot commented Jan 31, 2019

agones-bot commented Feb 1, 2019

agones-bot commented Feb 1, 2019

agones-bot commented Feb 1, 2019

agones-bot commented Feb 1, 2019

markmandel left a comment

Choose a reason for hiding this comment

agones-bot commented Feb 2, 2019

agones-bot commented Feb 2, 2019

markmandel left a comment

Choose a reason for hiding this comment

agones-bot commented Feb 4, 2019

ilkercelikyilmaz Feb 4, 2019

Choose a reason for hiding this comment

jkowalski Feb 5, 2019

Choose a reason for hiding this comment

ilkercelikyilmaz Feb 5, 2019

Choose a reason for hiding this comment

agones-bot commented Feb 4, 2019

agones-bot commented Feb 5, 2019

agones-bot commented Feb 5, 2019

markmandel left a comment

Choose a reason for hiding this comment

markmandel Feb 5, 2019

Choose a reason for hiding this comment

jkowalski Feb 6, 2019

Choose a reason for hiding this comment

markmandel left a comment

Choose a reason for hiding this comment

agones-bot commented Feb 6, 2019

markmandel left a comment

Choose a reason for hiding this comment

agones-bot commented Feb 6, 2019

jkowalski commented Jan 31, 2019 •

edited

Loading