How to separate 3rd party dependency layer from in-house jar dependencies ?

[mzagar]

Hi,

Is there a way to tell jib to put some jar dependencies into an 'extra' layer or to put all dependency jars into spearate layers: 1 jar - 1 layer ?

Here's my use case:

• my application depends on 3rd party jars (such as spring, gson, etc...) but also on other in-company jars.
• those in-company jar dependencies change much often than 3rd party jars and new releases are pushed on daily basis.
• when I push my app image and the pull it during deployment then whole 3rd party jars layer gets pulled even if it only had 1 jar dependency changed.

If I could somehow tell jib to put those 'fast changing' in-company jar dependencies into a layer separate from 3rd party jars then I would get much less overhead when pushing/pulling images in case some of in-company jar dependency version changes because 3rd party jars layer would not be touched and instead of pulling 70MB image change I would pull only a 5MB layer (with all in-company jars inside, those changed aswell as those not changed).

In my mind this layer would function the same way 'SNAPSHOT' layer does, only I could specify which jar releases go to this layer, something like:

• 3rd party dependencies layer
• extra dependencies layer
• project deps layer
• snapshot layer
• resources layer
• class layer
• files layer

Coming back to my original question - instead of this complicated extra layer jar selection maybe it would make sense to have an option to put every dependency jar to a separate layer - that way whichever jar changes, only those layers/jars would get pulled.

[mzagar]

I wrote a quick-and-dirty implementation (maven only) which works well for my usecase:
master...mzagar:project-dependencies-selector

With projectDependenciesSelector config option I can easily select exactly which dependencies I want to put in project dependency layer by matching them using a regex.

For example, I can force all dependencies starting with infobip- to end up in project dependency layer and get it separated from other 3rd party and snapshot dependencies:

<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <configuration>
        <projectDependenciesSelector>^infobip-.*</projectDependenciesSelector>
    </configuration>
</plugin>

I am ready to polish my pull request if you would consider accepting it.
If not, could you please suggest some alternatives?

Thanks

[chanseokoh]

Hi @mzagar

That's a very interesting use case that makes us think we should have a way to support. For a long time, we've had a lot discussions around layering various sorts of dependencies, and looks like we still need to think about this more.

maybe it would make sense to have an option to put every dependency jar to a separate layer

I think you should avoid this. There used to have a very small limit (of 42) as to how many layers you can have, which I believe has been mostly lifted on modern kernels, but I think many still may face another limit (of 127, for example): docker/docs#8230

[hendrikhalkow]

#1724 does exactly what you want.

[OneCricketeer]

Should this be closed?

[mzagar]

Hi @OneCricketeer, for me this is still an issue so I vote not close it.

Unfortunately #1724 does not solve my problem.

Here's a structure of infobip-hello maven project I am building:

• resulting docker image is built inside infobip-hello-service maven submodule
• 

After mvn jib:dockerBuild following jib layers are created in infobip-hello-service docker image :

• 

After diving into docker image built with jib 2.1.0 here's the content of jib layers :

1. dependencyJars layer

• all 3rd party dependencies are store at this layer (commons, grpc, hibernate, ...)

• in-house dependencies - which are same as 3rd party dependencies from perspective of this module are highlighted (infobip-*)

• 

2. snapshotDependencyJars layer

• infobip-hello-service module picked up infobip-hello-connector submodule dependency as expected
• 

3. projectDependencyJars

• not present since pnly project dependency is to infobip-hello-connector submodule which is in snapshotDependencyJars layer

4. app resources layer

• as expected contains src/main/resources content
• 

5. app class layer

• as expected
• 

6. jib extra directory

• as expected, contains files added explicitly
• 

My challenge is that in dependencyJars layer (1) in-house (infobip-*) dependencies are present alongside 3rd party dependencies and I would like to avoid it - what I would like to happen is:

• for those inhouse deps to be present in separate layer because they change more often than 3rd party deps - i.e. when infobip-mml dependency is changed then the whole 80MB of this layer needs to be pulled over the network but only single 2MB jar file changed
• to be able to control which dependencies I put in this 'separate layer' - since I as developer know which dependencies are changed more often than others and for which of them I could use this optimization.

If I am doing something wrong in using jib and I can accomplish this easily already please point me how to do it.

If not, I still am in favor of my suggestion to introduce this 'separate layer' where user could control with a regex which dependencies should be stored in this layer - sounds like a generic enough solution which can help to optimize docker images for such usecases as I described, but is not mandatory to be used.

Any suggestion how to go about this is greatly appreciated - I am ready to invest my time and create a PR for whichever approach would help me solve this problem.

Thanks,
Mario

[chanseokoh]

@mzagar we're not going to close this. For now, I don't see an immediate workaround in your situation to have this level of fine-grained control.

As we are getting numerous feature requests to Jib (many of which are often very specific to certain use-cases, don't align well with Jib's model or intentions, are too complex to support without polluting our codebase or configuration with special code, etc.), we decided to introduce an extension framework to Jib (#2401) rather than trying to implement all of them one way or another. With the extension system, anyone will be able to extend and/or modify Jib's behavior relatively easily and have a fine-grained control over most of the aspects of image building. I think that will cover most of such issues and FRs, including this one.

(FTR: #2450 for the list asking for the feature of fine-grained layer control.)

[mzagar]

sounds great, thanks!

any eta or a task I could follow the progress of the extension framework?

[chanseokoh]

We are working on it, but I think it's going to take a while. We aim to complete this at least in the second quarter.

Tracking: #2401

[chanseokoh]

@mzagar @hendrikhalkow @OneCricketeer

The Jib Extension Framework is now available with the latest Jib versions. You can easily extend and tailor the Jib plugins behavior to your liking.

We've written a general-purpose layer-filter extension that enables fine-grained layer control, including deleting files and moving files into new layers. The extension will cover your use-case to create a separate layer for in-hours jar dependencies (or any new layer for any files).

• Maven Layer-Filter Extension
• Gradle Layer-Filter Extension

For general information about using and writing extensions, take a look at the Jib Extensions repo.

[mzagar]

@chanseokoh just wanted to say the layer-filter extension works just the way we need, thanks!