[Build]: Fix docker images built multiple times issue #9253

xumia · 2021-11-14T03:19:08Z

Why I did it

The same docker image is built multiple times after upgrading to bullseye, the build time is increased to about 15 hours from 6 hours.
See log: https://dev.azure.com/mssonic/be1b070f-be15-4154-aade-b1d3bfb17054/_apis/build/builds/50390/logs/9
Line 1437: 2021-11-11T11:15:02.7094923Z [ building ] [ target/docker-sonic-telemetry.gz ]
Line 1446: 2021-11-11T11:37:41.1073304Z [ finished ] [ target/docker-sonic-telemetry.gz ]
Line 1459: 2021-11-11T11:38:20.6293007Z [ building ] [ target/docker-sonic-telemetry.gz-load ]
Line 1462: 2021-11-11T11:38:28.1250201Z [ finished ] [ target/docker-sonic-telemetry.gz-load ]
Line 2906: 2021-11-11T18:57:42.8207365Z [ building ] [ target/docker-sonic-telemetry.gz ]
Line 2917: 2021-11-11T19:43:47.1860961Z [ finished ] [ target/docker-sonic-telemetry.gz ]
Line 3997: 2021-11-11T22:49:35.0196252Z [ building ] [ target/docker-sonic-telemetry.gz ]
Line 4002: 2021-11-11T23:14:00.4127728Z [ finished ] [ target/docker-sonic-telemetry.gz ]

How I did it

Place the python wheels in another folder relative to the build distribution.

How to verify it

make sonic-broadcom.bin
make sonic-broadcom.raw

Make sure the docker build images, such as target/docker-sonic-telemetry.gz, not built multiple times.
https://dev.azure.com/mssonic/build/_build/results?buildId=51084&view=results

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

xumia · 2021-11-15T04:20:52Z

/azp run

azure-pipelines · 2021-11-15T04:20:57Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

xumia · 2021-11-15T04:21:14Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2021-11-15T04:21:26Z

Azure Pipelines successfully started running 1 pipeline(s).

saiarcot895 · 2021-11-15T17:24:50Z

For the Mellanox build failure, I think you'll need to remove the Python 2 dependencies from the base image build in slave.mk (lines 951-960). The reason it's not failing for the vs platform is because those Python 2 rules are not read for Bullseye (see rules/sonic-config.mk).

saiarcot895 · 2021-11-15T17:43:43Z

For the broadcom build failure, the python wheels caching isn't taking into account whether it's being build for Buster or Bullseye. Because of this, for sonic-yang-models, it's using a cached python wheel built for Bullseye (not sure how it got cached in the first place) for the Buster build.

To fix this, in Makefile.cache, $(eval $(call FLAGS_DEP_RULES, $(SONIC_PYTHON_WHEELS), $(PYTHON_WHEELS_PATH),flags)) on line 486 needs to be changed to $(eval $(call FLAGS_DEP_RULES, $(SONIC_PYTHON_WHEELS), $(PYTHON_WHEELS_PATH),flags,$(BLDENV))).

saiarcot895

Changes noted in earlier comments

xumia · 2021-11-16T23:39:16Z

@saiarcot895 , thanks for your comments. We can set the BLDENV as a common flag, applying it for all targets, right?

saiarcot895 · 2021-11-17T00:11:25Z

@saiarcot895 , thanks for your comments. We can set the BLDENV as a common flag, applying it for all targets, right?

Possibly, I'm not entirely sure on that. The docker container image files are currently in the target/ directory, and not in a BLDENV-specific directory, so I'm wondering if there are cases where more rebuilds are done than necessary. It would still result in a proper image, but there could maybe be some extra docker image builds.

xumia · 2021-11-17T02:09:29Z

@saiarcot895 , For some docker images should be relative to a BLDENV, such as docker-base-buster, docker-config-engine-buster. For the other dockers, such as docker-orchagent, it is buster currently, but it has no chance of buster or bullseye in undetermined status.
In short, for a docker image, the distribution should be fixed for the same commit. It should has not such issue.
Let's take a look for the build. Currently, it blocks by one of a test case failed in /sonic/src/sonic-config-engine/tests/test_cfggen.py, the nightly ci build also had such issue.
Triggered a new build in my fork merging branch: https://dev.azure.com/mssonic/build/_build/results?buildId=51787&view=results

xumia · 2021-11-17T03:34:01Z

/azp run

azure-pipelines · 2021-11-17T03:34:06Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

xumia · 2021-11-17T03:39:55Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2021-11-17T03:40:09Z

Azure Pipelines successfully started running 1 pipeline(s).

lguohan · 2021-11-17T18:26:16Z

from the log, sonic-config-engine py2 build succeeded, but not sonic-config-engine py3 version.
https://dev.azure.com/mssonic/be1b070f-be15-4154-aade-b1d3bfb17054/_apis/build/builds/51787/logs/9

lguohan · 2021-11-18T17:34:59Z

/azp run

azure-pipelines · 2021-11-18T17:35:05Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

lguohan · 2021-11-19T01:09:14Z

/azp run

azure-pipelines · 2021-11-19T01:09:19Z

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

lguohan · 2021-11-19T01:14:05Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2021-11-19T01:14:16Z

Azure Pipelines successfully started running 1 pipeline(s).

xumia · 2021-11-19T05:45:02Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2021-11-19T05:45:14Z

Azure Pipelines successfully started running 1 pipeline(s).

xumia · 2021-11-23T07:18:14Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2021-11-23T07:18:25Z

Azure Pipelines successfully started running 1 pipeline(s).

xumia · 2021-11-23T09:31:42Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2021-11-23T09:31:54Z

Azure Pipelines successfully started running 1 pipeline(s).

qiluo-msft · 2021-12-14T22:42:58Z

slave.mk

-PYTHON_DEBS_PATH = $(TARGET_PATH)/python-debs
-PYTHON_WHEELS_PATH = $(TARGET_PATH)/python-wheels
+PYTHON_DEBS_PATH = $(TARGET_PATH)/python-debs/$(BLDENV)
+PYTHON_WHEELS_PATH = $(TARGET_PATH)/python-wheels/$(BLDENV)


PYTHON_WHEELS_PATH

The path changes, and it will break other pipelines if copying artifacts from here. For example: https://github.com/Azure/sonic-platform-daemons/blob/2b0acfbdc48089acf9224113bbf9a61030314e87/azure-pipelines.yml#L117

Do you expect each repo owner to fix the paths?

@sujinmkang for awareness.

The same docker image is built multiple times after upgrading to bullseye, the build time is increased to about 15 hours from 6 hours. See log: https://dev.azure.com/mssonic/be1b070f-be15-4154-aade-b1d3bfb17054/_apis/build/builds/50390/logs/9 Line 1437: 2021-11-11T11:15:02.7094923Z [ building ] [ target/docker-sonic-telemetry.gz ] Line 1446: 2021-11-11T11:37:41.1073304Z [ finished ] [ target/docker-sonic-telemetry.gz ] Line 1459: 2021-11-11T11:38:20.6293007Z [ building ] [ target/docker-sonic-telemetry.gz-load ] Line 1462: 2021-11-11T11:38:28.1250201Z [ finished ] [ target/docker-sonic-telemetry.gz-load ] Line 2906: 2021-11-11T18:57:42.8207365Z [ building ] [ target/docker-sonic-telemetry.gz ] Line 2917: 2021-11-11T19:43:47.1860961Z [ finished ] [ target/docker-sonic-telemetry.gz ] Line 3997: 2021-11-11T22:49:35.0196252Z [ building ] [ target/docker-sonic-telemetry.gz ] Line 4002: 2021-11-11T23:14:00.4127728Z [ finished ] [ target/docker-sonic-telemetry.gz ] How I did it Place the python wheels in another folder relative to the build distribution. Co-authored-by: Ubuntu <xumia@xumia-vm1.jqzc3g5pdlluxln0vevsg3s20h.xx.internal.cloudapp.net>

xumia added the Build label Nov 14, 2021

xumia requested review from lguohan and saiarcot895 November 14, 2021 23:09

xumia marked this pull request as ready for review November 14, 2021 23:14

xumia requested a review from qiluo-msft as a code owner November 14, 2021 23:14

lguohan previously approved these changes Nov 15, 2021

View reviewed changes

saiarcot895 suggested changes Nov 15, 2021

View reviewed changes

xumia dismissed lguohan’s stale review via 592bac2 November 16, 2021 23:37

xumia force-pushed the fix-multiple-docker-build branch from 04949ca to 12d925e Compare November 23, 2021 05:55

xumia force-pushed the fix-multiple-docker-build branch from 12d925e to 67dce0c Compare December 2, 2021 05:38

Ubuntu added 3 commits December 8, 2021 03:02

Fix docker images built multiple times issue

a016460

Fix build mellanox failure

9b36e0a

Add the base image missing dependency SONIC_PLATFORM_API_PY3

3963693

xumia force-pushed the fix-multiple-docker-build branch from 67dce0c to 3963693 Compare December 8, 2021 03:05

xumia requested a review from liushilongbuaa December 9, 2021 06:52

saiarcot895 approved these changes Dec 9, 2021

View reviewed changes

lguohan merged commit 49dd5db into sonic-net:master Dec 9, 2021

lguohan mentioned this pull request Dec 9, 2021

[ci] Increase image timeout to 24 hours #9480

Closed

5 tasks

xumia added the Request for 202111 Branch For PRs being requested for 202111 branch label Dec 10, 2021

qiluo-msft reviewed Dec 14, 2021

View reviewed changes

judyjoseph added the Included in 202111 Branch label Dec 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Build]: Fix docker images built multiple times issue #9253

[Build]: Fix docker images built multiple times issue #9253

xumia commented Nov 14, 2021 •

edited

Loading

xumia commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

xumia commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

saiarcot895 commented Nov 15, 2021

saiarcot895 commented Nov 15, 2021 •

edited

Loading

saiarcot895 left a comment

xumia commented Nov 16, 2021

saiarcot895 commented Nov 17, 2021

xumia commented Nov 17, 2021 •

edited

Loading

xumia commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

xumia commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

lguohan commented Nov 17, 2021

lguohan commented Nov 18, 2021

azure-pipelines bot commented Nov 18, 2021

lguohan commented Nov 19, 2021

azure-pipelines bot commented Nov 19, 2021

lguohan commented Nov 19, 2021

azure-pipelines bot commented Nov 19, 2021

xumia commented Nov 19, 2021

azure-pipelines bot commented Nov 19, 2021

xumia commented Nov 23, 2021

azure-pipelines bot commented Nov 23, 2021

xumia commented Nov 23, 2021

azure-pipelines bot commented Nov 23, 2021

qiluo-msft Dec 14, 2021

qiluo-msft Dec 14, 2021

[Build]: Fix docker images built multiple times issue #9253

[Build]: Fix docker images built multiple times issue #9253

Conversation

xumia commented Nov 14, 2021 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

xumia commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

xumia commented Nov 15, 2021

azure-pipelines bot commented Nov 15, 2021

saiarcot895 commented Nov 15, 2021

saiarcot895 commented Nov 15, 2021 • edited Loading

saiarcot895 left a comment

Choose a reason for hiding this comment

xumia commented Nov 16, 2021

saiarcot895 commented Nov 17, 2021

xumia commented Nov 17, 2021 • edited Loading

xumia commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

xumia commented Nov 17, 2021

azure-pipelines bot commented Nov 17, 2021

lguohan commented Nov 17, 2021

lguohan commented Nov 18, 2021

azure-pipelines bot commented Nov 18, 2021

lguohan commented Nov 19, 2021

azure-pipelines bot commented Nov 19, 2021

lguohan commented Nov 19, 2021

azure-pipelines bot commented Nov 19, 2021

xumia commented Nov 19, 2021

azure-pipelines bot commented Nov 19, 2021

xumia commented Nov 23, 2021

azure-pipelines bot commented Nov 23, 2021

xumia commented Nov 23, 2021

azure-pipelines bot commented Nov 23, 2021

qiluo-msft Dec 14, 2021

Choose a reason for hiding this comment

qiluo-msft Dec 14, 2021

Choose a reason for hiding this comment

xumia commented Nov 14, 2021 •

edited

Loading

saiarcot895 commented Nov 15, 2021 •

edited

Loading

xumia commented Nov 17, 2021 •

edited

Loading