Permission issues with latest SONiC images

[BaluAlluru]

Description

Execute permissions of some of the files are revoked after 1st reboot. After the image is loaded,the box comes up and no issues observed.
But after the 1st reboot, the execute permissions of some of the files are revoked.

We found this issue when our platform service was not getting started after 1st reboot.
Permissions of the files in /usr/local/bin was getting changed after 1st reboot, as a result platform system service was not getting started.

We narrowed the issue to specific Jenkins image. This issue is observed in Jenkins 312 image and subsequent images.

Please have a look at this issue and suggest how to debug further.

Testing environment

Switch: Juniper-QFX5210-64C
ASIC: TH2
Branch: master

Here are the logs from problematic image (June 16th Jenkins image (312))
Platform specific util and monitor scripts are set with correct permissions in the code.

Below is the snippet from our code path:

ciju@sonic-server:~/sonic/sonic_mainline_june19/sonic-buildimage/platform/broadcom/sonic-platform-modules-juniper/qfx5210/utils$ ls -ltr
total 56
-rwxrwxr-x 1 ciju ciju 3964 Jun 19 03:25 README
-rwxrwxr-x 1 ciju ciju 3581 Jun 19 03:25 platform_poweroff
-rwxrwxr-x 1 ciju ciju 19534 Jun 19 03:25 juniper_qfx5210_util.py
-rwxrwxr-x 1 ciju ciju 27135 Jun 19 03:25 juniper_qfx5210_monitor.py

Logs from Box after loading the image:
Boot logs:

` Booting SONiC-OS-master.312-734b1c69

Loading SONiC-OS OS kernel ...Loading SONiC-OS OS kernel ...

Loading SONiC-OS OS initial ramdisk ...Loading SONiC-OS OS initial ramdisk ...

[ 3.712621] rc.local[530]: + sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version
[ OK ] Started System Logging Service.
[ OK ] Started containerd container runtime.
[ OK ] Started Initialize EDAC v3… Drivers For Machine Hardware.
[ OK ] Started Permit User Sessions.
[ OK ] Started OpenBSD Secure Shell server.
[ OK ] Started RAS daemon to log the RAS events.
[ OK ] Started Opennsl kernel modules init.
[ OK ] Found device /dev/ttyS0 `

Platform init service started:
`root@sonic:/home/admin# systemctl status qfx5210-platform-init.service
● qfx5210-platform-init.service - Juniper QFX5210 initialization service
Loaded: loaded (/etc/systemd/system/qfx5210-platform-init.service; enabled; v
Active: active (running) since Thu 2019-02-14 10:12:17 UTC; 28s ago
Process: 774 ExecStartPre=/usr/local/bin/juniper_qfx5210_util.py install (code
Main PID: 972 (python)
Tasks: 1 (limit: 4915)
Memory: 288.0M
CGroup: /system.slice/qfx5210-platform-init.service
└─972 python /usr/local/bin/juniper_qfx5210_monitor.py

root@sonic:/usr/local/bin# ls -ltr
total 161
-rwxr-xr-x 1 root root 3581 Jun 16 2020 platform_poweroff
-rwxr-xr-x 1 root root 19534 Jun 16 2020 juniper_qfx5210_util.py
-rwxr-xr-x 1 root root 27135 Jun 16 2020 juniper_qfx5210_monitor.py
-rwxr-xr-x 1 root root 3964 Jun 16 2020 README

As seen is the above log,all Platform specific files has execute permissions set.

Logs after 1st reboot:
[FAILED] Failed to start Juniper QFX5210 initialization service.
See 'systemctl status qfx5210-platform-init.service' for details.
[ OK ] Started netfilter persistent configuration.
[ OK ] Listening on Docker Socket for the API.
[ OK ] Found device /dev/ttyS0.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting LSB: service and resource monitoring daemon...
Starting containerd container runtime...
Starting Network Time Service...
Starting /etc/rc.local Compatibility...
[ 4.889911] rc.local[618]: + sonic-cfggen -y /etc/sonic/sonic_version.yml -v build_version
Starting Opennsl kernel modules init...
Starting Login Service...
Starting System Logging Service...
Starting RAS daemon to log the RAS events...
[ 5.328518] rc.local[618]: + SONIC_VERSION=master.312-734b1c69
[ OK ] Started Regular background program processing daemon.
[ 5.422019] rc.local[618]: + FIRST_BOOT_FILE=/host/image-master.312-734b1c69/platform/firsttime
Starting LSB: Execute the …-e command to reboot system...
[ 5.623030] rc.local[618]: + logger SONiC version master.312-734b1c69 starting up...
[ OK ] Started D-Bus System Message Bus.
[ 5.817964] rc.local[618]: + [ ! -e /host/machine.conf ]
Starting Permit User Sessions...
[ 5.967062] rc.local[618]: + . /host/machine.conf
Starting Kernel crash dump capture service...
[ 6.085966] rc.local[618]: + onie_arch=x86_64
Starting OpenBSD Secure Shell server...

`

`root@sonic:/home/admin# systemctl status qfx5210-platform-init.service
● qfx5210-platform-init.service - Juniper QFX5210 initialization service
Loaded: loaded (/etc/systemd/system/qfx5210-platform-init.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2019-02-14 10:11:58 UTC; 9min ago

Feb 14 10:11:58 sonic systemd[1]: Starting Juniper QFX5210 initialization service...
Feb 14 10:11:58 sonic systemd[527]: qfx5210-platform-init.service: Failed to execute command: Permission denied
Feb 14 10:11:58 sonic systemd[527]: qfx5210-platform-init.service: Failed at step EXEC spawning /usr/local/bin/juniper_qfx5210_util.py: Permission denied
Feb 14 10:11:58 sonic systemd[1]: qfx5210-platform-init.service: Control process exited, code=exited, status=203/EXEC
Feb 14 10:11:58 sonic systemd[1]: qfx5210-platform-init.service: Failed with result 'exit-code'.
Feb 14 10:11:58 sonic systemd[1]: Failed to start Juniper QFX5210 initialization service.
`

Issue observed because permissions of the platform specific files in /usr/local/bin are getting changed.
root@sonic:/usr/local/bin# ls -ltr
total 161
-rw-r--r-- 1 root root 3581 Jun 16 2020 platform_poweroff
-rw-r--r-- 1 root root 19534 Jun 16 2020 juniper_qfx5210_util.py
-rw-r--r-- 1 root root 27135 Jun 16 2020 juniper_qfx5210_monitor.py
-rw-r--r-- 1 root root 3964 Jun 16 2020 README

Steps to reproduce the issue:

1. Install Jenkins 312 image on the box (sonic-broadcom.bin)
2. SONIC boots and observed that Platform system service is started.
   3.Reboot the box. After reboot, Platform service is not started. As mentioned above, the issue is because permission of the platform specific files gets changed.

Here are the logs from working image (June 13th Jenkins image (310)).
The file permissions are not changed after 1st reboot.

root@sonic:/home/admin# systemctl status qfx5210-platform-init.service
● qfx5210-platform-init.service - Juniper QFX5210 initialization service
Loaded: loaded (/etc/systemd/system/qfx5210-platform-init.service; enabled; v
Active: active (running) since Thu 2019-02-14 10:12:17 UTC; 28s ago
Process: 774 ExecStartPre=/usr/local/bin/juniper_qfx5210_util.py install (code
Main PID: 972 (python)
Tasks: 1 (limit: 4915)
Memory: 288.0M
CGroup: /system.slice/qfx5210-platform-init.service
└─972 python /usr/local/bin/juniper_qfx5210_monitor.py

root@sonic:/usr/local/bin# ls -ltr
total 161
-rwxr-xr-x 1 root root 3581 Jun 13 2020 platform_poweroff
-rwxr-xr-x 1 root root 19534 Jun 13 2020 juniper_qfx5210_util.py
-rwxr-xr-x 1 root root 27135 Jun 13 2020 juniper_qfx5210_monitor.py
-rwxr-xr-x 1 root root 3964 Jun 13 2020 README
-rwxr-xr-x 1 root root 162 Jun 13 2020 host-ssh-keygen.sh
-rw-r--r-- 1 root root 8157 Jun 13 2020 wsdump.pyc
-rwxr-xr-x 1 root root 6412 Jun 13 2020 wsdump.py

From the above log, we can see that execute permissions of the files are set.

Additional Information:
We are suspecting below commit might be causing this issue.

commit 76a395c
Author: xumia 59720581+xumia@users.noreply.github.com
Date: Sat Jun 13 15:10:13 2020 +0800
[secure boot] Support rw files allowlist (#4585)
* Support rw files allowlist for Sonic Secure Boot
* Improve the performance
* fix bug
* Move the config description into a md file
* Change to use a simple way to remove the blank line
* Support chmod a-x in rw folder
* Change function name
* Change some unnecessary words
root@sonic:/home/admin# show version

SONiC Software Version: SONiC.master.312-734b1c69
Distribution: Debian 10.4
Kernel: 4.19.0-6-amd64
Build commit: 734b1c6
Build date: Tue Jun 16 14:22:15 UTC 2020
Built by: johnar@jenkins-worker-11
``

[BaluAlluru]

sonic_dump_sonic_20190214_101548.tar.gz

[xinliu-seattle]

Please retest with fix #4836.

[ciju-juniper]

Fix provided in #4836 rectified the issue

root@sonic:/usr/local/bin# ls -ltr
total 161
-rwxr-xr-x 1 root root  3581 Jun 23  2020 platform_poweroff
-rwxr-xr-x 1 root root 19534 Jun 23  2020 juniper_qfx5210_util.py
-rwxr-xr-x 1 root root 27135 Jun 23  2020 juniper_qfx5210_monitor.py

@Staphylo @qiluo-msft @lguohan Please go ahead and merge the patch.