Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[build]: add the support of Apt-Cacher-NG proxy configuration to the … #3257

Closed
wants to merge 54 commits into from

Conversation

wangshengjun
Copy link
Contributor

[build]: add the support of Apt-Cacher-NG proxy configuration to the …
Signed-off-by: wangshengjun wangshengjun@asterfusion.com

What I did
Add the support of Apt-Cacher-NG proxy configuration to the slave docker build script.

Why I did it
The build process of basic slave docker always takes a long time to download the apt/debian package from the internet. We can introduce the apt cacher server to cut down the download time extremely.
So it makes sense to add the support of Apt-Cacher-NG proxy configuration.

How I verified it
I set up the apt cacher server in my local area network(IP :10.250.0.240, Port : 3142)
Build cmd as follows:
with apt_cacher_ng_proxy args
apt_cacher_ng_proxy=http://10.250.0.240:3142 make sonic-slave-build
without apt_cacher_ng_proxy args
make sonic-slave-build
Both works well.

@lguohan
Copy link
Collaborator

lguohan commented Aug 1, 2019

the proxy only works for the slave docker build, what about image build and docker build, can you add cache ng proxy for those as well?

@lguohan
Copy link
Collaborator

lguohan commented Aug 1, 2019

since this cache ng proxy only does apt cache, does it conflict with http_proxy option? Since http_proxy can be also used for pip package download.

@wangshengjun
Copy link
Contributor Author

since this cache ng proxy only does apt cache, does it conflict with http_proxy option? Since http_proxy can be also used for pip package download.

the cache ng proxy only does apt cache, it works fine with http_proxy option.

@wangshengjun
Copy link
Contributor Author

the proxy only works for the slave docker build, what about image build and docker build, can you add cache ng proxy for those as well?

Yes, the apt-cacher-ng proxy can add into the 'docker-base-strecth', so other dockers can inherit.

@wangshengjun wangshengjun force-pushed the master branch 2 times, most recently from fa22b3f to 9f41afa Compare August 1, 2019 11:53
…slave docker build script (sonic-net#3256)

Signed-off-by: wangshengjun <wangshengjun@asterfusion.com>
@@ -173,6 +175,7 @@ SONIC_BUILD_INSTRUCTION := make \
KERNEL_PROCURE_METHOD=$(KERNEL_PROCURE_METHOD) \
HTTP_PROXY=$(http_proxy) \
HTTPS_PROXY=$(https_proxy) \
APT_CACHER_NG_PROXY=$(apt_cacher_ng_proxy) \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

              [](start = 0, length = 13)

The indentation is not same as above lines

@@ -23,6 +23,10 @@ RUN rm -rf \
/var/cache/man/* \
/usr/share/locale/*

{% if APT_CACHER_NG_PROXY.strip() -%}
RUN echo 'Acquire::http::Proxy "{{APT_CACHER_NG_PROXY}}";' >> /etc/apt/apt.conf.d/02proxy
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/etc/apt/apt.conf.d/02proxy [](start = 70, length = 27)

Does it impact runtime proxy? If yes, production switches may have no access to build environment at all.

@@ -23,6 +23,10 @@ RUN rm -rf \
/var/cache/man/* \
/usr/share/locale/*

{% if APT_CACHER_NG_PROXY.strip() -%}
RUN echo 'Acquire::http::Proxy "{{APT_CACHER_NG_PROXY}}";' >> /etc/apt/apt.conf.d/02proxy
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

    [](start = 0, length = 8)

Suggest remove the leading spaces.

@@ -23,6 +23,10 @@ RUN rm -rf \
/var/cache/man/* \
/usr/share/locale/*

{% if APT_CACHER_NG_PROXY.strip() -%}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APT_CACHER_NG_PROXY [](start = 6, length = 19)

Suggest rename to APT_PROXY, Apt-Cacher-NG is a specific apt proxy implementation.

Copy link
Collaborator

@qiluo-msft qiluo-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As comments

renukamanavalan and others added 15 commits April 28, 2020 13:36
* Enable debug image build for kvm image.

* Fix a bug in cleaning up requests referring the netsmp_session being closed to avoid a crash, during netsnmp_session close/abort due to timeout.
RCA:
The netsnmp_agent_session requests use netsmp_subtree objects that matches the associated variable name.
The netsnmp_subtrees created through a netsnmp_session are tied to that session.
When a subagent connection is closed/dropped due to timeout, all associated netsnmp_subtree objects are fred.
Hence during a netsnmp_session close, all the delegated netsnmp_agent_sessions are scanned for requets that could be using netsnmp_subtree objects associated with this netsnmp_sesssion or its subsession. For each of the found request, they are explicitly marked to fail and a call is made to complete them.

But due to the bug in scanning, it leaves behind requests and hence later when the requests get processed, they refer the *now* freed netsnmp_subtree. As often these requests gets completed pretty soon, they escape crashing. But if it so happens that the freed memory happened to complete a memory unit, hence returned to kernel or it got reallocated & changed enough to crash, the snmpd crashes.

* Revert the changes

* Revert
…ce (sonic-net#3317)

Signed-off-by: wangshengjun <wangshengjun@asterfusion.com>
…n pam.d's config. (sonic-net#3322)

Present: Servers are listed in the same order as in redis-db
Fix: Save the sort o/p, hence use sorted list to write into pam.d's conf.
     As well convert priority to integer for use by sort.
…onic-net#3316', change the sample_output file of 'wait_for_intf.sh' (sonic-net#3325)

Signed-off-by: wangshengjun <wangshengjun@asterfusion.com>
Added sfputil.py for x86_64-accton_wedge100bf_32x-r0 and x86_64-accton_wedge100bf_65x-r0 devices.

Signed-off-by: Andriy Kokhan <akokhan@barefootnetworks.com>
* [DELL][Z9100,S6100,S6000] Platform 2.0 SFP Changes

Added support in sfp.py file which will be generic. Send the eeprom path and sfp_control path from chassis.py
…onic-net#3308)

Signed-off-by: brandon_chuang <brandon_chuang@CicadaBuildServer.accton.com.tw>
Signed-off-by: Ying Xie <ying.xie@microsoft.com>
[sairedis]:

ARM32 bit fixes, for 64bit printf format specifier (sonic-net#490)
fix wrong API type , add cmdline parse option , remove unuseful api call (sonic-net#499)
Use pointer to member function to reduce the footprint of collectCounters() (sonic-net#488)
[syncd] Add support for Innovium platform (sonic-net#496)
[saisdkdump]: Fix dump generation crash (sonic-net#493)

[swss]:

Add scope attribute to VNET table. (sonic-net#954)
[portsyncd]: Fix portsyncd restart case (sonic-net#1019)
[vnet]: Enable "vnet_orch_1" VS test case (sonic-net#1020)
[vlan] Add pytest cases to validate the behavior about add LAG member to vlan. (sonic-net#875)
[policerorch]: Add the capability to update policer rate/size (sonic-net#1017)
[aclorch]: Add MIRRORv6 support for NPS platform (sonic-net#1018)
[orchagent]: Avoid crash by setting g_syncMode flag before create_switch (sonic-net#1014)
[orchagent] Add support for Innovium platform (sonic-net#1005)
Provide broadcast IP while configuring interface ip address (sonic-net#1007)
[VLAN] Add pytest cases to validate different use-case of tagging_mode. (sonic-net#860)
[vstest]: Add test_port_config.py which include breakout port test. (sonic-net#866)
[vstests]: change 'test_acl_egress_table.py' to UNIX format (sonic-net#1010)
[vlan] Add pytest cases to verify data in app db and state db. (sonic-net#895)
[vstest]: Add the CRM tests (sonic-net#1013)
[warm-reboot]: add bgp eoiu support to speed up route reconcile (sonic-net#856)
[fdborch] support mac update (sonic-net#877)
[vxlanorch] Fix Logic of Vxlan tunnel removal (sonic-net#995)
Fix VLAN error introduced with new 4.9 kernel behavior (sonic-net#1001)
Config DB manual is being moved from Wiki to SWSS repo (sonic-net#1002)
Trap DHCPv6 packets for supporting ZTP over in-band interfaces using … (sonic-net#997)

Signed-off-by: Andriy Kokhan <akokhan@barefootnetworks.com>
[build] Add ipmitool
[dockers] Add innovium platform in orchagent + ipmitool in snmp
[platform/innovium] Add innovium platform
[device/celestica] Add x86_64-cel_midstone-r0 device for innovium
[device/delta] Add x86_64-delta_et-c032if-r0 device for innovium
[sonic-slave-stretch] Add texi2html

Signed-off-by: Tony Titus ttitus@innovium.com
…ic-net#3333)

[sonic-platform-common]

[sonic_sfp] Interpret sff 'int' element =0 as valid value (sonic-net/sonic-platform-common#51)
add more error code to get_transceiver_change_event ((sonic-net/sonic-platform-common#50)
[sonic_platform_base] support new-platform-api-based daemons ((sonic-net/sonic-platform-common#48)
sync change to sonic_platform_base/sonic_sfp and create symbol link ((sonic-net/sonic-platform-common#49)
Add parser support for Tx_RxLos,TxFault, PowerControl, ResetStatus in sff8436.py ((sonic-net/sonic-platform-common#45)
readd type_abbrv_name in sonic_sfp/sff8436.py ((sonic-net/sonic-platform-common#44)
[psu_base] get_status_led() returns current state of the status LED ((sonic-net/sonic-platform-common#39)
Fix abbrv name for OSFP ((sonic-net/sonic-platform-common#36)
[sff8436] support "Control Bytes" and "Options" ((sonic-net/sonic-platform-common#38)
sonic_sfp: avoid possible key error in get_physical_to_logical() ((sonic-net/sonic-platform-common#37)

[sonic-platform-daemons]

[xcvrd] Enhance xcvrd to handle new system level event/error (sonic-net/sonic-platform-daemons#39)
[xcvrd] Support both new platform API and old platform plugins (sonic-net/sonic-platform-daemons#38)
[psud] Support both new platform API and old platform plugins (sonic-net/sonic-platform-daemons#37)
[syseepromd] Support both new platform API and old platform plugins (sonic-net/sonic-platform-daemons#36)
Add missing import statemet (sonic-net/sonic-platform-daemons#32)
sonic_xcvrd: Support for DOM Threshold values for EEPROM dump (sonic-net/sonic-platform-daemons#29)
…onic-net#3290)

* [platform/cel]: update qsfp driver for dx010

* [device/celestica]: add sfp api

* [device/celestica]: add eeprom api
ndisc6 gathers a few diagnostic tools for IPv6 networks including:

 - ndisc6, which performs ICMPv6 Neighbor Discovery in userland,
 - rdisc6, which performs ICMPv6 Router Discovery in userland,
 - rltraceroute6, a UDP/ICMP IPv6 implementation of traceroute,
 - tcptraceroute6, a TCP/IPv6-based traceroute implementation,
 - tcpspray6, a TCP/IP Discard/Echo bandwidth meter,
 - addrinfo, easy script interface for hostname and address resolution,
 - dnssort, DNS sorting script.

Signed-off-by: Guohan Lu <gulv@microsoft.com>
* slave.mk
* .gitignore

The SONIC_PYTHON_STDEB_DEBS group is part of SONIC_CLEAN_DEBS
for cleaning with 'make clean', but this group is overlooked
since they reside in PYTHON_DEBS_PATH, not DEBS_PATH. This
fix creates a separate clean rule for SONIC_PYTHON_STDEB_DEBS.

Also noticed the .arch file showing up on 'git status' so
added it to .gitignore.

Signed-off-by: Greg Paussa <greg.paussa@broadcom.com>
shikenghua and others added 25 commits April 28, 2020 13:36
…net#3306)

* [docker-fpm-frr]: Generate separated staticd.conf for staticd

Generate staticd.conf by templates/staticd.conf.j2 with config DB data

* [docker-fpm-frr]: Remove default_route block from zebra.conf.j2

default_route block already moved to staticd.conf.j2

* [docker-fpm-frr]: Add test for staticd.conf.j2 template

* Add test for staticd.conf.j2 template

* Correct the sample output of zebra.conf.j2 template

* Fix a typo in test_zebra_frr

* [docker-fpm-frr]: Fix test_j2files test errors

* Fix test errors in test_j2files.py and test_j2files_t2_chassis_fe.py

* Fix typo in test_j2files_t2_chassis_fe.py
…#3359)

This commit also includes sonic-linux-kernel submodule update
Commits included:
f8b30b4 [Mellanox] Add hw-management driver patches (sonic-net#97)
feb786b Add psample and act_sample drivers (sonic-net#94)
15f8651 Update optoe driver to add CMIS (QSFP-DD, OSFP, ...) support (sonic-net#96)
Signed-off-by: johnson <JOHNSON.LU@deltaww.com>
…#3321)

Signed-off-by: Mykola Faryma <mykolaf@mellanox.com>
Signed-off-by: Andriy Kokhan <akokhan@barefootnetworks.com>
…onic-net#3352)

* Fast reboot mode should be enabled only 3 minutes after restart

* Advance sonic-quagga submodule
* [kernel] Upgrade kernel to version 4.9.168-1+deb9u5

- Picking up security patches.

Signed-off-by: Ying Xie <ying.xie@microsoft.com>

* advance linux-kernel sub module head
- increase timeout for power supply driver initialization
…ent v2.0.0191 (sonic-net#3371)

temp_xxxx_module{} => module{}_temp_xxxx
Updated z9264_platform.sh to change appropriate LED when HWSKU is changed.
mellanox simx platform don't have QSFP sysfs created, this script will cause timeout issue on simx platform
Signed-off-by: Kebo Liu <kebol@mellanox.com>
 Users can install docker with sudo, and it requires sudo to issue docker command. This will cause few errors in Makefile.work which need use docker without sudo.
…et#3323)

* Create the src archive in target/ when INSTALL_DEBUG_TOOLS=y

* Dropped -v (verbose flag) from tar command
…3366)

* [Makefile] make reset target changed to ONESHELL and multiarch docker
stop has been added in reset

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
…base_config.json (sonic-net#2182)

this is the first step to moving different databases tables into different database instances

in this PR, only handle multiple database instances creation based on user configuration at /etc/sonic/database_config.json

we keep current method to create single database instance if no extra/new DATABASE configuration exist in database_config.json file.

if user try to configure more db instances at database_config.json , we create those new db instances along with the original db instance existing today.

The configuration is as below, later we can add more db related information if needed:
{
...
"DATABASE": {
"redis-db-01" : {
"port" : "6380",
"database": ["APPL_DB", "STATE_DB"]
},
"redis-db-02" : {
"port" : "6381",
"database":["ASIC_DB"]
},
}
...
}

The detail description is at design doc at sonic-net/SONiC#271

The main idea is : when database.sh started, we check the configuration and generate corresponding scripts.

rc.local service handle old_config copy when loading new images, there is no dependency between rc.local and database service today, for safety and make sure the copy operation are done before database try to read it, we make database service run after rc.local

Then database docker started, we check the configuration and generate corresponding scripts/.conf in database docker as well.

based on those conf, we create databases instances as required.

at last, we ping_pong check database are up and continue


Signed-off-by: Dong Zhang d.zhang@alibaba-inc.com
…et#3379)

* [platform/cel]: Add new api installer

* [device/celestica]: Update APIs to support both pmon and base image

* [device/e1031]: update hwmon name
…et#3391)

Signed-off-by: chiourung_huang <chiourung_huang@edge-core.com>
optimize SFP module operations and fix issues.

- split initialization of variant categories of devices and initialize each category of devices only when needed, so that unnecessary dependencies can be avoided.
- update watchdog logic, only initializing watchdog when referenced.
- support platform.py and enable to initialize variant devices on a host/docker basis
- update init so that sonic_platform can be imported as a whole.
Signed-off-by: Zhenggen Xu <zxu@linkedin.com>
* Fixed initial state for eeprom.py and sfputil.py when thrift server is down
* Added transceiver plug-in/out event processing

Signed-off-by: Andriy Kokhan <akokhan@barefootnetworks.com>
Docker version above 18 has inconsistent behaviour with qemu.
Hence using the same version 18 used in sonic-slave

Signed-off-by: Antony Rheneus <arheneus@marvell.com>
… swsscommon (sonic-net#3264)

Update interfaces of bgpcfd from swsssdk to swsscommon to unify a suit of interface with other component. Meanwhile, we can listen multiple tables at one thread under swsscommon interface.

Signed-off-by: Ze Gan ganze718@gmail.com

- What I did
Move the interface of bgpcfgd from swsssdk to swsscommon. Because bgpcfgd need to listen more events in the future and we want to maintain one kind of APIs, swsscommon is more suitable than swsssdk.

- How I did it
Refactor the BGPConfigDaemon to two components, Daemon and BGPConfigManager. We can register new managers to the Daemon object if we want to listen more events.
@lguohan
Copy link
Collaborator

lguohan commented Nov 16, 2020

can you do the rebase and resolve the conflict? it is difficult to figure out what is the change now.

mssonicbld added a commit that referenced this pull request Apr 27, 2024
…atically (#18735)

#### Why I did it
src/sonic-utilities
```
* 9b463ca5 - (HEAD -> master, origin/master, origin/HEAD) [chassis][voq] Add fabric capacity monitoring cmds (#3255) (8 hours ago) [jfeng-arista]
* df94636b - Display target firmware version through CLI (#3274) (34 hours ago) [mihirpat1]
* cd5c0580 - Fix db_migrate.py show error and back trace while loading configuration on Linecard (#3257) (2 days ago) [Hua Liu]
* d48a8308 - Add Multi ASIC support for apply-patch (#3249) (3 days ago) [Xincun Li]
* b143ea6d - [chassis][voq]Add fabric monitoring commands. (#3239) (4 days ago) [jfeng-arista]
* 07d6d277 - [fast/warm-reboot] Retain TRANSCEIVER_INFO tables on fast/warm-reboot (#3240) (7 days ago) [Stepan Blyshchak]
* 8e5ff74f - Revert "Revert "route_check: Skip route checks if bgp feature is not enabled"" (#3270) (7 days ago) [anamehra]
* eb165f36 - Fix double hex to decimal conversion (#3267) (7 days ago) [Yuanzhe]
```
#### How I did it
#### How to verify it
#### Description for the changelog
xumia pushed a commit that referenced this pull request Aug 14, 2024
…19900)

Why I did it
The latest swss and dash-api commits are dependent on each other to build properly, if updated separately they won't build.

sonic-swss new commits:

* 29cea04a - (HEAD -> master, origin/master, origin/HEAD) [DASH] Remove deprecated 'action_type' refs (#3257) (3 hours ago) [Lawrence Lee]
sonic-dash-api new commits:

* dbb2d0f - (HEAD -> master, origin/master, origin/HEAD) Update for SAI API changes (3 days ago) [Lawrence Lee]
* 7cd1105 - deprecate pl_sip_encoding (6 days ago) [Lawrence Lee]
* 0df30ee - Merge branch 'master' of github.com:sonic-net/sonic-dash-api into route-group (6 days ago) [Lawrence Lee]
* 2721493 - deprecate privatelinknsg routing type (6 days ago) [Lawrence Lee]
* 3260f6a - update unit tests (6 days ago) [Lawrence Lee]
* 95fa234 - avoid breaking change (6 days ago) [Lawrence Lee]
* ab46204 - EOF newline (6 days ago) [Lawrence Lee]
* 2f58861 - add icmp redirection (6 days ago) [Lawrence Lee]
* 6e9ad22 - non-breaking route metering update (6 days ago) [Lawrence Lee]
* 722b6c2 - Rename action_type to routing type (6 days ago) [Lawrence Lee]
* 76ab87a - add newline to EOF (6 days ago) [Lawrence Lee]
* 747425a - Keep pl_sip_encoding (6 days ago) [Lawrence Lee]
*   597c4b8 - Merge branch 'route-group' of github.com:theasianpianist/sonic-dash-api into route-group (4 months ago) [Lawrence Lee]
|\  
| * ecec7fb - Merge branch 'master' into route-group (5 months ago) [Lawrence Lee]
* | 8b3de2d - update API to match new HLD schema (4 months ago) [Lawrence Lee]
|/  
* daa54b6 - add new message types to unittest (5 months ago) [Lawrence Lee]
* 16ca9f1 - add PA validation table (5 months ago) [Lawrence Lee]
* 8f7aaef - add route group table (5 months ago) [Lawrence Lee]
Work item tracking
Microsoft ADO (number only):
How I did it
How to verify it
matiAlfaro pushed a commit to Marvell-switching/sonic-buildimage that referenced this pull request Aug 21, 2024
…onic-net#19900)

Why I did it
The latest swss and dash-api commits are dependent on each other to build properly, if updated separately they won't build.

sonic-swss new commits:

* 29cea04a - (HEAD -> master, origin/master, origin/HEAD) [DASH] Remove deprecated 'action_type' refs (sonic-net#3257) (3 hours ago) [Lawrence Lee]
sonic-dash-api new commits:

* dbb2d0f - (HEAD -> master, origin/master, origin/HEAD) Update for SAI API changes (3 days ago) [Lawrence Lee]
* 7cd1105 - deprecate pl_sip_encoding (6 days ago) [Lawrence Lee]
* 0df30ee - Merge branch 'master' of github.com:sonic-net/sonic-dash-api into route-group (6 days ago) [Lawrence Lee]
* 2721493 - deprecate privatelinknsg routing type (6 days ago) [Lawrence Lee]
* 3260f6a - update unit tests (6 days ago) [Lawrence Lee]
* 95fa234 - avoid breaking change (6 days ago) [Lawrence Lee]
* ab46204 - EOF newline (6 days ago) [Lawrence Lee]
* 2f58861 - add icmp redirection (6 days ago) [Lawrence Lee]
* 6e9ad22 - non-breaking route metering update (6 days ago) [Lawrence Lee]
* 722b6c2 - Rename action_type to routing type (6 days ago) [Lawrence Lee]
* 76ab87a - add newline to EOF (6 days ago) [Lawrence Lee]
* 747425a - Keep pl_sip_encoding (6 days ago) [Lawrence Lee]
*   597c4b8 - Merge branch 'route-group' of github.com:theasianpianist/sonic-dash-api into route-group (4 months ago) [Lawrence Lee]
|\  
| * ecec7fb - Merge branch 'master' into route-group (5 months ago) [Lawrence Lee]
* | 8b3de2d - update API to match new HLD schema (4 months ago) [Lawrence Lee]
|/  
* daa54b6 - add new message types to unittest (5 months ago) [Lawrence Lee]
* 16ca9f1 - add PA validation table (5 months ago) [Lawrence Lee]
* 8f7aaef - add route group table (5 months ago) [Lawrence Lee]
Work item tracking
Microsoft ADO (number only):
How I did it
How to verify it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.