Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(build): idempotent builds #5291

Merged
merged 20 commits into from
Nov 22, 2022

Conversation

slonka
Copy link
Contributor

@slonka slonka commented Nov 9, 2022

Signed-off-by: slonka slonka@users.noreply.github.com

Closes #5270

EDIT:

To be more precise what I don't like about this PR:

  • build date - is it even needed? we seem to only pass it around and there is no computation based on it
    • we could remove it altogether, but that is passed around in KDS so it seems a bit too much for this PR
    • use static date (this is what is done now) - if someone ever needs to figure out the version by this date it would be impossible
    • use only Y/M/D without hours / minutes - less wrong than the above, not sure it matters
  • changing binary creation / modification time
    • there seems to be no workaround for this, docker just uses it for the cache so 😭, but is this acceptable

I had a chat with Bart and we decided that we can always iterate on this.

Old notes:

Not sure what happened (maybe docker update or something on my mac) but the image build times from the issue are a lot faster when I tested them today. Anyway with this PR I managed to cut down make images time from ~35 seconds to ~20 seconds on my machine and even a bigger improvement ~115 seconds to ~85 seconds when running an example e2e test ( gtime -q -f "%e" make -j test/e2e/debug-fast K3D=true E2E_PKG_LIST=./test/e2e/deploy/... ).

Overall I'm not super happy with this change, there are a couple of hacks here, with your help I hope we can make it better.

Checklist prior to review

  • Link to docs PR or issue -- build improvement
  • Link to UI issue or PR -- not a UI issue
  • Is the issue worked on linked? -- yes
  • The PR does not hardcode values that might break projects that depend on kuma (e.g. "kumahq" as a image registry) -- yes
  • The PR will work for both Linux and Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
  • Unit Tests --
  • E2E Tests --
  • Manual Universal Tests --
  • Manual Kubernetes Tests --
  • Do you need to update UPGRADE.md? --
  • Does it need to be backported according to the backporting policy? --
  • Do you need to explicitly set a > Changelog: entry here or add a ci/ label to run fewer/more tests?

Changelog: feat(build): idempotent build

Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
…dencies

Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
@slonka slonka marked this pull request as ready for review November 16, 2022 15:02
@slonka slonka requested a review from a team as a code owner November 16, 2022 15:02
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
@slonka slonka force-pushed the feat/make-local-builds-idempotent branch from c1d77f9 to e151dde Compare November 17, 2022 10:30
Copy link
Contributor

@bartsmykla bartsmykla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have few suggestions

@slonka
Copy link
Contributor Author

slonka commented Nov 18, 2022

Note to self: version.sh is called from other places so other call sites need to be adjusted.

@bartsmykla
Copy link
Contributor

I've run come commands on my mbp (intel), here are the results:

# master

gtime -q -f "%e" make images
# 192.04
# 70.12

gtime -q -f "%e" make -j test/e2e/debug-fast K3D=true E2E_PKG_LIST=./test/e2e/ownership/...
# 255.82
# 261.80

# feat/make-local-builds-idempotent

gtime -q -f "%e" make images
# 131.41
# 35.25

gtime -q -f "%e" make -j test/e2e/debug-fast K3D=true E2E_PKG_LIST=./test/e2e/ownership/...
# 268.30
# 207.96

@slonka
Copy link
Contributor Author

slonka commented Nov 18, 2022

☝️ one note to this, running make images "seeded" cache for the subsequent build. Nevertheless we see 50% improvement in make images and it's reflected in e2e test.

Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
@slonka slonka changed the title feat(kumacp): idempotent builds WIP feat(kumacp): idempotent builds Nov 21, 2022
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
@slonka slonka changed the title feat(kumacp): idempotent builds feat(build): idempotent builds Nov 22, 2022
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Copy link
Contributor

@bartsmykla bartsmykla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few nits, but lgtm overall

slonka and others added 2 commits November 22, 2022 10:14
Co-authored-by: Bart Smykla <bartek@smykla.com>
Signed-off-by: Krzysztof Słonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
Signed-off-by: slonka <slonka@users.noreply.github.com>
@slonka
Copy link
Contributor Author

slonka commented Nov 22, 2022

I had to remove the "touch -mt" hack to cache the context sent do docker because it was caching too much.

Signed-off-by: slonka <slonka@users.noreply.github.com>
@slonka slonka merged commit 97d3a51 into kumahq:master Nov 22, 2022
@lahabana lahabana mentioned this pull request Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Idempotent builds
3 participants