Recompute docker dependencies across dev loops. #5121
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tejal29
changed the title
gsquared94
reviewed
Dec 7, 2020
Codecov Report
@@ Coverage Diff @@
## master #5121 +/- ##
==========================================
+ Coverage 72.19% 72.24% +0.04%
==========================================
Files 380 380
Lines 13303 13326 +23
==========================================
+ Hits 9604 9627 +23
Misses 3006 3006
Partials 693 693
Continue to review full report at Codecov.
|
tejal29
changed the title
tejal29
force-pushed
the
fix_get_dependency_cache
branch
from
62b2d0b to
a380575
Compare
tejal29
changed the title
tejal29
force-pushed
the
fix_get_dependency_cache
branch
from
927b56f to
f96742f
Compare
|
Performance stats I ran
|
|
@gsquared94, @briandealwis and @nkubala please take a look. |
tejal29
force-pushed
the
fix_get_dependency_cache
branch
from
49867f8 to
35bff8e
Compare
gsquared94
reviewed
Dec 9, 2020
| @@ -53,3 +52,8 @@ func GetDependencies(ctx context.Context, workspace string, a *latest.CustomArti | |||
| return list.Files(workspace, a.Dependencies.Paths, a.Dependencies.Ignore) | |||
| } | |||
| } | |||
|
|
|||
| func getDockerBuildConfig(ws string, artifact string, a *latest.CustomArtifact) docker.BuildConfig { | |||
| dockerfile := a.Dependencies.Dockerfile | |||
There was a problem hiding this comment.
should we normalize the Dockerfile path here itself instead of later?
There was a problem hiding this comment.
That is a good suggestion. ThenormalizeDockerpath is used a lot many places. while going through the code, there is a huge scope of avoiding repetition across docker.SyncMap and docker.GetDependencies
I don't want to error out here in this stage of creating a BuildConfig
tejal29
added a commit
to tejal29/skaffold
that referenced
this pull request
Dec 9, 2020
…#5121) * only read chached values in createContext * remove sf.do * Move key from dockerfilePath to artifact name * Fix kaniko test
IsaacPD
pushed a commit
that referenced
this pull request
Dec 9, 2020
* Recompute docker dependencies across dev loops. (#5121) * only read chached values in createContext * remove sf.do * Move key from dockerfilePath to artifact name * Fix kaniko test * Release notes for v1.17.2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #5110 & #5115
In #4896, we cache results for
docker.GetDependenciessince there were getting computed atleast 2 times for docker builds.However, this PR ended up caching the results across dev loops.
In this PR,
GetCachedDependencieswhich will be used in subsequent calls.This fixes issue in Skaffold v1.17.1 unable to stream build output when copying from directory of larger context #5115 where same dockerfile was used in different artifacts.
Alternate approach considered,
dependencyCachefor each loop.docker.GetDependencyis a public method.dependencyCacheat end of each dev loopThis would have required adding
pkg/skaffold/dockerpackage dependency inpkg/skaffold/runner. If all dependencies were cached, we could have addedInvalidateDependencyCache()toBuilderinterface. However only docker dependencies are cached, so went with the approach to add a new methodGetCachedDependenciesand use it when creating tar contextTesting notes
skaffold devtouch node/src/new.js