Support runtest.py to work on Windows when no_pty=True #640

cy20lin · 2023-07-07T16:29:42Z

Tweak runtest.py script so that it could now partially run on Windows. The pty part is not implemented, and would raise if --no-pty is not specified.

For the implementation, the threading and queue modules are used to read the subprocess.stdout pipe with timeout, as the select module doesn't work on Windows.

OldLiu001 · 2023-07-08T03:59:45Z

I briefly tested it and it seemed to add some strange token at the end of each line, like \x16 or something.

This caused both my VBScript implementation and the repository's original PowerShell implementation by Joel Martin to fail the test (although it produced the expected results).

…indows

cy20lin · 2023-07-08T20:43:29Z

Thanks for your feedback. The special character \x16 was introduced at commit 18616b1 by @kanaka.

As I have no clear Idea about how the string '\x16\r' work at Runner.writeline on posix, I keep that part intact on posix platform and only modify the code to write the normal str with line_break to the stdin of the subprocess on Windows.

I've tested the powershell implementation by Joel Martin from step0 to step9, and the tests passed smoothly. Except for the step5, you might wants to increase --test-timeout to avoid timeout when testing (def! res2 (sum2 10000 0)).

Regarding stepA, there is an hanging issue when testing (readline "mal-user> "), the same issue is also found when testing on Windows WSL Ubuntu. I'm now fixing that issue and will update if I make further progress.

OldLiu001 · 2023-07-09T02:05:25Z

Thanks for update.

But when I test step0 with PowerShell impl, got this:

Testing basic string
] -> FAIL (line 3): -> ['',abcABC123
    Expected : 'abcABC123\\\r'
    Got      : 'abcABC123'
Testing string containing spaces
] -> FAIL (line 7):rld\r' -> ['',hello mal world
    Expected : 'hello\\ mal\\ world\\\r'
    Got      : 'hello mal world'
Testing string containing symbols
] -> FAIL (line 11):\r' -> ['',[]{}"'* ;:()
    Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
    Got      : '[]{}"\'* ;:()'
Test long string
TEST: 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)\r' -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'] -> FAIL (line 16):
    Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
Non alphanumeric characters
] -> FAIL (line 20):
    Expected : '!\\\r'
    Got      : '!'
] -> FAIL (line 22):
    Expected : '\\&\\\r'
    Got      : '&'
] -> FAIL (line 24):
    Expected : '\\+\\\r'
    Got      : '+'
] -> FAIL (line 26):
    Expected : ',\\\r'
    Got      : ','
] -> FAIL (line 28):
    Expected : '\\-\\\r'
    Got      : '-'
] -> FAIL (line 30):
    Expected : '/\\\r'
    Got      : '/'
] -> FAIL (line 32):
    Expected : '<\\\r'
    Got      : '<'
] -> FAIL (line 34):
    Expected : '=\\\r'
    Got      : '='
] -> FAIL (line 36):
    Expected : '>\\\r'
    Got      : '>'
] -> FAIL (line 38):
    Expected : '\\?\\\r'
    Got      : '?'
] -> FAIL (line 40):
    Expected : '@\\\r'
    Got      : '@'
] -> FAIL (line 43):
    Expected : '\\^\\\r'
    Got      : '^'
] -> FAIL (line 45):
    Expected : '_\\\r'
    Got      : '_'
] -> FAIL (line 47):
    Expected : '`\\\r'
    Got      : '`'
] -> FAIL (line 49):
    Expected : '\\~\\\r'
    Got      : '~'
------- Optional Functionality --------------
------- (Not needed for self-hosting) -------
Non alphanumeric characters
] -> SOFT FAIL (line 58):
    Expected : '\\#\\\r'
    Got      : '#'
] -> SOFT FAIL (line 60):
    Expected : '\\$\\\r'
    Got      : '$'
] -> SOFT FAIL (line 62):
    Expected : '%\\\r'
    Got      : '%'
] -> SOFT FAIL (line 64):
    Expected : '\\.\\\r'
    Got      : '.'
] -> SOFT FAIL (line 66):
    Expected : '\\|\\\r'
    Got      : '|'

FAILURES:
]:> ['',abcABC123 3): abcABC123
    Expected : 'abcABC123\\\r'
    Got      : 'abcABC123'
]:> ['',hello mal worldello mal world
    Expected : 'hello\\ mal\\ world\\\r'
    Got      : 'hello mal world'
]:> ['',[]{}"'* ;:()): []{}"'* ;:()
    Expected : '\\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\\r'
    Got      : '[]{}"\'* ;:()'
FAILED TEST (line 16): hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() [ -> ['',hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"'* ;:() []{}"'* ;:() []{]:'*)
    Expected : 'hello\\ world\\ abcdefghijklmnopqrstuvwxyz\\ ABCDEFGHIJKLMNOPQRSTUVWXYZ\\ 0123456789\\ \\(;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\ ;:\\(\\)\\ \\[\\]\\{\\}"\'\\*\\)\\\r'
    Got      : 'hello world abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ 0123456789 (;:() []{}"\'* ;:() []{}"\'* ;:() []{}"\'*)'
]:> ['',!ST (line 20): !
    Expected : '!\\\r'
    Got      : '!'
]:> ['',&ST (line 22): &
    Expected : '\\&\\\r'
    Got      : '&'
]:> ['',+ST (line 24): +
    Expected : '\\+\\\r'
    Got      : '+'
]:> ['',,ST (line 26): ,
    Expected : ',\\\r'
    Got      : ','
]:> ['',-ST (line 28): -
    Expected : '\\-\\\r'
    Got      : '-'
]:> ['',/ST (line 30): /
    Expected : '/\\\r'
    Got      : '/'
]:> ['',<ST (line 32): <
    Expected : '<\\\r'
    Got      : '<'
]:> ['',=ST (line 34): =
    Expected : '=\\\r'
    Got      : '='
]:> ['',>ST (line 36): >
    Expected : '>\\\r'
    Got      : '>'
]:> ['',?ST (line 38): ?
    Expected : '\\?\\\r'
    Got      : '?'
]:> ['',@ST (line 40): @
    Expected : '@\\\r'
    Got      : '@'
]:> ['',^ST (line 43): ^
    Expected : '\\^\\\r'
    Got      : '^'
]:> ['',_ST (line 45): _
    Expected : '_\\\r'
    Got      : '_'
]:> ['',`ST (line 47): `
    Expected : '`\\\r'
    Got      : '`'
]:> ['',~ST (line 49): ~
    Expected : '\\~\\\r'
    Got      : '~'
]:> ['',#ED TEST (line 58): #
    Expected : '\\#\\\r'
    Got      : '#'
]:> ['',$ED TEST (line 60): $
    Expected : '\\$\\\r'
    Got      : '$'
]:> ['',%ED TEST (line 62): %
    Expected : '%\\\r'
    Got      : '%'
]:> ['',.ED TEST (line 64): .
    Expected : '\\.\\\r'
    Got      : '.'
]:> ['',|ED TEST (line 66): |
    Expected : '\\|\\\r'
    Got      : '|'

TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step0_repl.mal):
    5: soft failing tests
   19: failing tests
    0: passing tests
   24: total tests

I fix it:

elif os.name == 'nt':
        expects = ["%s%s" % (t.out, re.escape(t.ret))]

with this:

elif os.name == 'nt':
        expects = ["%s%s" % (t.out, re.escape(t.ret)),
                   "%s%s" % (t.out, re.escape(t.ret.strip()))]

Then step0 of powershell & vbs can pass smoothly.

And when I test step1,

With powershell I got these 'FAILURES':

FAILURES:
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ')', got EOF"
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ']', got EOF"
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : 'Exception: expected \'"\', got EOF'
 -> ['.*(EOF|end of input|unbalanced).*\r',]:
    Expected : '.*(EOF|end of input|unbalanced).*\r'
    Got      : "Exception: expected ')', got EOF"
 -> ['{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r',]:
    Expected : '{"a([1-3])" \\1 "a(?!\\1)([1-3])" \\2 "a(?!\\1)(?!\\2)([1-3])" \\3}\r'
    Got      : '{"a2" 2 "a1" 1 "a3" 3}'

TEST RESULTS (for D:\@Resources\编程\@Repositories\mal\impls\tests\step1_read_print.mal):
    0: soft failing tests
    9: failing tests
  108: passing tests
  117: total tests

With my vbs impl, It stuck in ;; whole line comment (not an exception)\r,

] -> SUCCESS  {:b   {  :cde     3   }  }}\r' -> ['',{:a {:b {:cde 3}}}
] -> SUCCESS1}\r' -> ['',{"1" 1}
] -> SUCCESSr' -> ['',({})
Testing read of comments
TEST: ' ;; whole line comment (not an exception)\r' -> ['',] -> TIMEOUT (line 226)

Exception: TestTimeout('TIMEOUT (line 226)')
Output before exception:
user>

So I test my vbs's step1 with pipe:

mal\impls\vbs> .\run_step1_read_print.cmd < ..\tests\step1_read_print.mal

It did not get stuck and generate the expected result.

user> user> 1
user> user> 7
user> user> 7
user> user> -123
user> user> user> user> user> +
user> user> abc
user> user> abc
user> user> abc5
user> user> abc-def
user> user> user> user> -
user> user> -abc
user> user> ->>
user> user> user> user> (+ 1 2)
user> user> ()
user> user> ()
user> user> (nil)
user> user> ((3 4))
user> user> (+ 1 (+ 2 3))
user> user> (+ 1 (+ 2 3))
user> user> (* 1 2)
user> user> (** 1 2)
user> user> (* -3 6)
user> user> (() ())
user> user> user> user> (1 2 3)
user> user> user> user> user> user> user> user> user> user> nil
user> user> true
user> user> false
user> user> user> user> "abc"
user> user> "abc"
user> user> "abc (with parens)"
user> user> "abc\"def"
user> user> ""
user> user> "\\"
user> user> "\\\\\\\\\\\\\\\\\\"
user> user> "&"
user> user> "'"
user> user> "("
user> user> ")"
user> user> "*"
user> user> "+"
user> user> ","
user> user> "-"
user> user> "/"
user> user> ":"
user> user> ";"
user> user> "<"
user> user> "="
user> user> ">"
user> user> "?"
user> user> "@"
user> user> "["
user> user> "]"
user> user> "^"
user> user> "_"
user> user> "`"
user> user> "{"
user> user> "}"
user> user> "~"
user> user> "!"
user> user> user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unterminated string, got EOF.
user> user> Exception: unbalanced parentheses.
user> user> Exception: unbalanced parentheses.
user> user> user> user> (quote 1)
user> user> (quote (1 2 3))
user> user> (quasiquote 1)
user> user> (quasiquote (1 2 3))
user> user> (unquote 1)
user> user> (unquote (1 2 3))
user> user> (quasiquote (1 (unquote a) 3))
user> user> (splice-unquote (1 2 3))
user> user> user> user> user> :kw
user> user> (:kw1 :kw2 :kw3)
user> user> user> user> [+ 1 2]
user> user> []
user> user> []
user> user> [[3 4]]
user> user> [+ 1 [+ 2 3]]
user> user> [+ 1 [+ 2 3]]
user> user> ([])
user> user> user> user> {}
user> user> {}
user> user> {"abc" 1}
user> user> {"a" {"b" 2}}
user> user> {"a" {"b" {"c" 3}}}
user> user> {"a" {"b" {"cde" 3}}}
user> user> user> user> {"a1" 1 "a2" 2 "a3" 3}
user> user> {:a {:b {:cde 3}}}
user> user> {"1" 1}
user> user> ({})
user> user> user> user> user> 1
user> user> 1
user> user> user> user> (deref a)
user> user> user> user> user> user> user> user> user> (with-meta [1 2 3] {"a" 1})
user> user> user> user> user> user> "\n"
user> user> "#"
user> user> "$"
user> user> "%"
user> user> "."
user> user> "\\"
user> user> "|"
user> user> user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> 1
user> user> user> 1
user> user>

And then I manually test all tests in step1_read_print.mal with my vbs impl, It also not get stuck.

Here is my test script (Batch File, placed in same folder with runtest.py):

@echo off
pushd "%~dp0"
goto p
for %%a in (
	step0_repl
	step1_read_print
) do (
	echo @pushd "%%~dp0" ^& @cscript -nologo %%a.vbs > .\impls\vbs\run_%%a.cmd
	python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\vbs\run_%%a.cmd"
	pause
)
exit
:p
for %%a in (
	step0_repl
	step1_read_print
) do (
	echo @pushd "%%~dp0" ^& @powershell ".\%%a.ps1" > .\impls\powershell\run_%%a.cmd
	python runtest.py --no-pty "%~dp0impls\tests\%%a.mal" "%~dp0impls\powershell\run_%%a.cmd"
	pause
)

May I ask what your PowerShell testing environment is? I think there may be slight differences in our environment, which may result in your PowerShell testing passing while mine cannot.

Fix the hanging issue when testing `(readline "mal-user> ")` on powershell stepA_mal.ps1 implementation. The issue is found on Windows and Windows WSL Ubuntu.

cy20lin · 2023-07-09T22:51:25Z

I fix the hanging issue when testing on stepA. I test from step0 to stepA again for all my environments and the runtest.py script runs smoomthly.

I have two testing environments, which are:

First
- Windows 11
- Python 3.11.4
- PSVersion 5.1.22621.1778
Second
- Ubuntu 22.04 WSL on Windows 11
- Python 3.10.6
- PSVersion 7.3.5

To test in my first environment, I use the following script.

@echo off
@REM test.cmd
@REM Put this script in the same directory as the runtest.py file

@REM # How to run
@REM # O. Start In powershell shell environment
@REM # 1. Change to Mal project directory 
@REM cd path\to\mal
@REM # 2. Run the test and log the result
@REM .\test.cmd | Tee-Object -FilePath test-nt.log

for %%a in (
    step0_repl
    step1_read_print
    step2_eval
    step3_env
    step4_if_fn_do
    step5_tco
    step6_file
    step7_quote
    step8_macros
    step9_try
    stepA_mal
) do (
	python runtest.py --rundir "impls\powershell" --test-timeout 60 --deferrable --optional --no-pty "..\tests\%%a.mal" powershell ".\%%a.ps1"
)

To test in my second environment, I use the following script.

#!/usr/bin/bash
# test.bash

# Put this script in the same directory as the runtest.py file

# How to run
# # 1. Change to Mal project directory 
# cd path/to/mal
# # 2. Run the test and log the result
# bash test.bash | tee test-posix.log

names=(
    step0_repl
    step1_read_print
    step2_eval
    step3_env
    step4_if_fn_do
    step5_tco
    step6_file
    step7_quote
    step8_macros
    step9_try
    stepA_mal
)

for name in "${names[@]}"
do
    python runtest.py --rundir "impls/powershell" --test-timeout 60 --deferrable --optional --no-pty "../tests/${name}.mal" pwsh "${name}.ps1"
done

One could refer to the attached files for the test log.

cy20lin · 2023-07-09T23:00:12Z

@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.

OldLiu001 · 2023-07-10T10:57:54Z

@OldLiu001 couldn't reproduce your issue on my machine though, hope the infomation above could help you located the problem.

I finally discovered the root cause of the problem.

In Windows, git defaults to changing the line feed encoding of the cloned text file to CRLF, while runtest.py only considers the case where the line feed encoding is LF (i.e. split('\n')).

Resulting in an additional \r at the end of the generated regular expression and leading to test failure.

OldLiu001 · 2024-08-08T10:37:13Z

小哥您好，好久不见，别来无恙？
我将您的代码一并合并到了我的分支中，若您仍有兴趣和有空闲，也烦请关注 #624 的相关讨论。

Hello, long time no see, how are you?
I have merged your code into my branch, if you are still interested and have spare time, please follow the discussion on #624.

kanaka · 2024-08-22T22:24:23Z

@cy20lin Thank you for your work on this. @OldLiu001 and I figured out a way to use WSL on Windows plus the runtest --no-pty option to enable testing implementations on Windows. It doesn't require significant modification to runtest.py to make it work (and my hesitation with anything that makes runtest.py is that it's already complicated and difficult for me to support as it is). I'm going to close this ticket since there is a workaround that I prefer. Again, thanks for your work on this and sorry for the very high latency in replying.

cy20lin added 3 commits July 7, 2023 21:16

Support runtest.py to work on Windows when no_pty=True

80f2141

Fix and make runtest.py compatible with python2 again

9e239ee

Fix runtest.py for unexpected TestTimeout exception

374f391

cy20lin mentioned this pull request Jul 7, 2023

First VBScript implementation #624

Merged

writeline without adding extra characters '\x16\r' in runtest.py on W…

ad3030e

…indows

Fix the hanging issue when testing stepA on Windows

dc8a9b4

Fix the hanging issue when testing `(readline "mal-user> ")` on powershell stepA_mal.ps1 implementation. The issue is found on Windows and Windows WSL Ubuntu.

This was referenced Jul 9, 2023

Tests don't work on native Windows #542

Open

runtest.py broken on Windows #446

Open

OldLiu001 added a commit to OldLiu001/mal that referenced this pull request Jul 10, 2023

fix CRLF support of runtest.py (kanaka#640)

01d46eb

OldLiu001 mentioned this pull request Jul 10, 2023

fix CRLF support of runtest.py (#640) cy20lin/mal#2

Closed

Respect platform dependent newline when reading test files

1b8c360

kanaka closed this Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support runtest.py to work on Windows when no_pty=True #640

Support runtest.py to work on Windows when no_pty=True #640

cy20lin commented Jul 7, 2023

OldLiu001 commented Jul 8, 2023 •

edited

Loading

cy20lin commented Jul 8, 2023

OldLiu001 commented Jul 9, 2023 •

edited

Loading

cy20lin commented Jul 9, 2023

cy20lin commented Jul 9, 2023

OldLiu001 commented Jul 10, 2023

OldLiu001 commented Aug 8, 2024

kanaka commented Aug 22, 2024

Support runtest.py to work on Windows when no_pty=True #640

Support runtest.py to work on Windows when no_pty=True #640

Conversation

cy20lin commented Jul 7, 2023

OldLiu001 commented Jul 8, 2023 • edited Loading

cy20lin commented Jul 8, 2023

OldLiu001 commented Jul 9, 2023 • edited Loading

cy20lin commented Jul 9, 2023

cy20lin commented Jul 9, 2023

OldLiu001 commented Jul 10, 2023

OldLiu001 commented Aug 8, 2024

kanaka commented Aug 22, 2024

OldLiu001 commented Jul 8, 2023 •

edited

Loading

OldLiu001 commented Jul 9, 2023 •

edited

Loading