bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

paulmon · 2019-05-20T23:09:16Z

When running the buildbot for Windows ARM32 test.pythoninfo must be run remotely.
Also seperating deploy step from both pythoninfo and test.bat

Requires buildmaster config change to work:
python/buildmaster-config#91

@zooba @zware

https://bugs.python.org/issue36511

paulmon · 2019-05-22T01:29:37Z

The debug part of the test pass should pass locally with these fixes, except for test_datetime.
See this PR for test_datetime: #13073
I am re-running test.bat now with these fixes, which takes more than an hour.
The last pass took 4 hours and 23 minutes on my secondary ARM test device, but on the buildbot ARM test device it's been taking about an hour and a half.
I still need to try the WindowsArm32ReleaseBuild steps, but expect the same results.

zooba

Would prefer a tighter skip on the tests, and deferring to @zware on the buildbot scripts (as with the build config change).

zooba · 2019-05-22T23:28:15Z

Lib/test/test_ssl.py

@@ -1345,6 +1346,7 @@ def test_load_verify_cadata(self):
            ctx.load_verify_locations(cadata=b"broken")


+    @unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")


This is certainly not a general issue - can we scope it down to the platform(s) where it crashes?

I will scope it down.

I think it's limited to UCRT. The problem is that SSL has FILE* from the ucrtbase.dll table and python has a FILE* from the ucrtbased.dll table.

I think we could limit it to win32+arm or platform.win32_is_iot( ). Unless there's a way to check if python is using UCRT? I couldn't find one.

Oh, it's because of mixing FILE* across debug/release builds? In that case we scope it down to Windows and enhance the skip message to say "Avoid mixing debug/release CRT on Windows"

paulmon · 2019-05-24T00:23:54Z

Enabling is_python_build( ) to return true on the test target uncovered more failing tests.
There is still one failure in test_pyexpat that I will be investigating next.

======================================================================
FAIL: test_exception (test.test_pyexpat.HandlerExceptionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\py\lib\test\test_pyexpat.py", line 453, in test_exception
    parser.Parse(b"<a><b><c/></b></a>", 1)
RuntimeError: a

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\py\lib\test\test_pyexpat.py", line 469, in test_exception
    self.assertIn('call_with_frame("StartElement"', entries[1][3])
AssertionError: 'call_with_frame("StartElement"' not found in ''

----------------------------------------------------------------------

paulmon · 2019-05-24T19:33:08Z

I simplified the deployment script to copy entire directories instead of picking files, except for PCBuild and Modules which have many files that aren't needed for the test run. This should make the script more resilient to test changes.

I think the Windows ARM32 test pass is green now when I run the scripts manually, if #13073 is included.

paulmon · 2019-05-24T19:33:47Z

Lib/test/test_pyexpat.py

@@ -465,7 +466,7 @@ def test_exception(self):
                                       "pyexpat.c", "StartElement")
            self.check_traceback_entry(entries[2],
                                       "test_pyexpat.py", "StartElementHandler")
-            if sysconfig.is_python_build():
+            if sysconfig.is_python_build() and not (sys.platform == 'win32' and platform.machine() == 'ARM'):


I'd rather fix this than skip it, but I haven't figured out where to start looking yet.

Lib/test/test_ssl.py

Lib/test/test_math.py

paulmon · 2019-06-13T17:23:24Z

@tiran can you please review?
Some of these changes are required to enable the Windows arm32 buildbot to start working.
Thanks!

zooba · 2019-06-19T17:29:29Z

@paulmon I'm happy to merge without Christian's review once the updates I suggested in my last one are done.

Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

paulmon · 2019-06-19T18:17:29Z

@zooba I merged your suggested changes. I think it's ready

zooba · 2019-06-19T18:22:05Z

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

paulmon · 2019-06-19T18:33:42Z

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

I'm not sure I'm following you. Are suggesting adding a check for arm/arm64 to Py_DEBUG_WIN32? That would be incorrect for Windows IoT Core x86/x64 which also use UCRT for everything.

zooba · 2019-06-19T18:44:25Z

@paulmon The Py_DEBUG_WIN32 change needs the references to that variable updated as well. I didn't go through and suggest a change on each one.

I'm not sure I'm following you. Are suggesting adding a check for arm/arm64 to Py_DEBUG_WIN32? That would be incorrect for Windows IoT Core x86/x64 which also use UCRT for everything.

No, I'm saying elsewhere you are referring to Py_DEBUG_Win32 which is now undefined. They need to be changed to Py_DEBUG_WIN32 as well

paulmon · 2019-06-19T18:56:18Z

No, I'm saying elsewhere you are referring to Py_DEBUG_Win32 which is now undefined. They need to be changed to Py_DEBUG_WIN32 as well

Thanks. I noticed the whitespace change but somehow didn't see the uppercase change.

paulmon · 2019-06-19T19:27:28Z

@zooba Thanks for the help
I fixed test_ssl and test_math so they pass on my dev machine.
All of the PR checks pass now.

miss-islington · 2019-06-19T20:09:57Z

Thanks @paulmon for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

…bot (pythonGH-13454) (cherry picked from commit f355069) Co-authored-by: Paul Monson <paulmon@users.noreply.github.com>

bedevere-bot · 2019-06-19T20:10:11Z

GH-14244 is a backport of this pull request to the 3.8 branch.

zooba · 2019-06-19T20:32:59Z

@paulmon Seems like it still fails: https://buildbot.python.org/all/#/builders/197/builds/338

paulmon · 2019-06-19T20:59:26Z

@paulmon Seems like it still fails: https://buildbot.python.org/all/#/builders/197/builds/338

The raspberry pi was in a bad state. I have restarted it

paulmon · 2019-06-19T21:09:45Z

Also, ssh.exe and scp.exe somehow don't exist when running under the buildbot, which is breaking the script.

zooba · 2019-06-19T21:17:44Z

@paulmon Different PATH setting? Or perhaps it's running with different file redirection settings (as a 32-bit app, maybe)?

paulmon · 2019-06-19T22:09:16Z

@paulmon Different PATH setting? Or perhaps it's running with different file redirection settings (as a 32-bit app, maybe)?

The PATH is echoed at the top of the log and includes the PATH to ssh.
I hadn't thought of WoW as a possible cause. I installed from the link on python.org so I got the 32-bit version of python. I'll update to the 64-bit python and see if that helps. There isn't a SysWOW64 version of ssh.exe, so that could be it.

* master: (599 commits) Docs: Improved phrasing (pythonGH-14069) Remove redundant if check from optional argument function in argparse. (pythonGH-8766) bpo-37289: Add a test for if with ifexpr in the peephole optimiser to detect regressions (pythonGH-14127) Update What's New in Python 3.9 (pythonGH-14253) bpo-36511: Improve ARM32 buildbot scripts (pythonGH-14251) bpo-37151: remove _PyCFunction_FastCallDict (pythonGH-14269) Fix typo, 'widger' -> 'widget', in idlelib/tree.py (pythonGH-14263) Fix bpo number in News file. (pythonGH-14260) bpo-37342: Fix the incorrect nb_index's type in typeobj documentation (pythonGH-14241) Update What's New in Python 3.8 (pythonGH-14239) bpo-36710: Use tstate in pylifecycle.c (pythonGH-14249) Add missing single quote in io.TextIOWrapper.reconfigure documentation (pythonGH-14246) bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 buildbot (pythonGH-13454) bpo-37333: Ensure IncludeTkinter has a value (pythonGH-14240) bpo-37331: Clarify format of socket handler messages in the documentation. (pythonGH-14234) bpo-37258: Not a bug, but added a unit test and updated documentation. (pythonGH-14229) bpo-36710: Remove PyImport_Cleanup() function (pythonGH-14221) Fix name of '\0'. (pythonGH-14222) bpo-36710: Add tstate parameter in import.c (pythonGH-14218) Document typing.ForwardRef (pythonGH-14216) ...

…bot (pythonGH-13454)

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh

1d33fe6

the-knights-who-say-ni added the CLA signed label May 20, 2019

bedevere-bot added the awaiting review label May 20, 2019

paulmon mentioned this pull request May 20, 2019

bpo-36511: Fix Windows arm32 buildbot, pythoninfo, scp, and ssh python/buildmaster-config#91

Merged

Paul Monson added 2 commits May 21, 2019 12:07

add +d to match test.bat

3aadf52

100% test pass when manually running test.bat

7f17d29

fix file destination

9aaa2c4

zooba reviewed May 22, 2019

View reviewed changes

Paul Monson added 6 commits May 23, 2019 11:22

only skip ssl tests on Windows IoT

3f38d18

Avoid mixing debug/release CRT on Windows

4e147de

test for win32

9819647

propagate exit code

365701c

fix 2 more exit 0 calls

69c40d7

fixes for test_regrtest

086a1b9

Paul Monson added 5 commits May 24, 2019 09:35

merge upstream/master

042e0e9

add tools/parser for test_tools

e230795

copy entire Tools directory

1a1f12d

skip check on Windows ARM32

29601d8

restore 64-bit check in test_regrtest

786194d

paulmon commented May 24, 2019

View reviewed changes

zooba added skip news needs backport to 3.8 labels Jun 11, 2019

merge upstream/master

cd2d5c5

paulmon requested a review from tiran as a code owner June 11, 2019 23:08

zooba reviewed Jun 12, 2019

View reviewed changes

Lib/test/test_ssl.py Outdated Show resolved Hide resolved

Lib/test/test_math.py Outdated Show resolved Hide resolved

paulmon and others added 2 commits June 19, 2019 11:16

Update Lib/test/test_math.py

f5c917a

Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

Update Lib/test/test_ssl.py

e2fb14b

Co-Authored-By: Steve Dower <steve.dower@microsoft.com>

Merge remote-tracking branch 'upstream/master' into arm32_pythoninfo

ce3dfbc

Paul Monson added 2 commits June 19, 2019 11:45

fix mismatched paren

5c355ab

fix Py_DEBUG_WIN32 spelling

f1ce1bf

zooba merged commit f355069 into python:master Jun 19, 2019

bedevere-bot removed the awaiting review label Jun 19, 2019

bedevere-bot removed the needs backport to 3.8 label Jun 19, 2019

paulmon mentioned this pull request Jun 19, 2019

bpo-36511 add arm32 buildbot diagnostic messages #14251

Merged

paulmon deleted the arm32_pythoninfo branch July 17, 2019 20:30

lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019

bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 build…

cd062ef

…bot (pythonGH-13454)

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020

bpo-36511: Add buildbot scripts and fix tests for Windows ARM32 build…

a739c78

…bot (pythonGH-13454)

paulmon mannequin mentioned this pull request May 20, 2022

Add Windows ARM32 buildbot #80692

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

paulmon commented May 20, 2019 •

edited by bedevere-bot

Loading

paulmon commented May 22, 2019

zooba left a comment

zooba May 22, 2019

paulmon May 22, 2019

zooba May 23, 2019

paulmon May 23, 2019

paulmon commented May 24, 2019

paulmon commented May 24, 2019

paulmon May 24, 2019

paulmon commented Jun 13, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

paulmon commented Jun 19, 2019

miss-islington commented Jun 19, 2019

bedevere-bot commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

paulmon commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019 •

edited

Loading

		@@ -1345,6 +1346,7 @@ def test_load_verify_cadata(self):
		ctx.load_verify_locations(cadata=b"broken")


		@unittest.skipIf(Py_DEBUG, "Crashes on debug python builds")

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

bpo-36511: Fix Windows arm32 buildbot pythoninfo, scp, and ssh #13454

Conversation

paulmon commented May 20, 2019 • edited by bedevere-bot Loading

paulmon commented May 22, 2019

zooba left a comment

Choose a reason for hiding this comment

zooba May 22, 2019

Choose a reason for hiding this comment

paulmon May 22, 2019

Choose a reason for hiding this comment

zooba May 23, 2019

Choose a reason for hiding this comment

paulmon May 23, 2019

Choose a reason for hiding this comment

paulmon commented May 24, 2019

paulmon commented May 24, 2019

paulmon May 24, 2019

Choose a reason for hiding this comment

paulmon commented Jun 13, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

paulmon commented Jun 19, 2019

miss-islington commented Jun 19, 2019

bedevere-bot commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019

paulmon commented Jun 19, 2019

zooba commented Jun 19, 2019

paulmon commented Jun 19, 2019 • edited Loading

paulmon commented May 20, 2019 •

edited by bedevere-bot

Loading

paulmon commented Jun 19, 2019 •

edited

Loading