Running Micropython in a CPython subprocess on Unix

[kdschlosser]

OK so this is the skinny on what I am wanting to do but I seem to be having an issue with it working.

I am building MicroPython with a user module and I want to be able to text if the user module is functioning properly. How I wanted to do this was by creating a CPython unittest script that would open a subprocess to MicroPython and I could send over the code to execute in the subprocess. For whatever reason I am not getting any output when I run Micropython. I have trued running a shell and passing the command to execute micropython and I have tried directly sending the micropython command to subprocess. It doesn't want to run. I am not sure if there is some kind of mechanics in MicroPython that mucks it up from being able to be used in this manner.

Any information would be helpful.

Thanks.
--Kevin

[andrewleech]

Hi @kdschlosser this certainly can work, in the past I used a custom cpython unit test runner script that would execute a unix build of micropython with subprocess.

These days I don't bother with cpython though, I just write my tests using the micropython-lib version of unittest and run everything on micropython.

I'm not sure I understand your full issue though, if you run your compiled copy of micropython from a terminal shell, do you get the repl and can you run code Magnitsky from there?
If you try to run a basic $ micropython hello _world.py script from a terminal shell does this work?
Or is it only when using cpython subprocess that it fails to output?

[shariltumin]

Using subprocess from CPython to call MicroPython works.

sharil@kaki5:~/micropython/scripts/tests
$ ls -l hello.py Hello.py
-rw-rw-r-- 1 sharil sharil  24 Oct 28 18:35 hello.py
-rw-rw-r-- 1 sharil sharil 119 Oct 28 18:35 Hello.py
sharil@kaki5:~/micropython/scripts/tests
$ cat hello.py
print('Hello from MP')

sharil@kaki5:~/micropython/scripts/tests
$ cat Hello.py
import subprocess

print('Hello from CPython')
subprocess.run(["micropython", "hello.py"] )
print('Bye from CPython')

sharil@kaki5:~/micropython/scripts/tests
$ python Hello.py
Hello from CPython
Hello from MP
Bye from CPython

Now you need to run 'Hello.py' in the same directory as 'hello.py', unless you have included the absolute path to the file.

$ cd /tmp
sharil@kaki5:/tmp
$ python ~/micropython/scripts/tests/Hello.py
Hello from CPython
micropython: can't open file 'hello.py': [Errno 2] No such file or directory
Bye from CPython

Send us any error messages to help us understand what went wrong.

[kdschlosser]

Now If I am not mistaken in those tests you are instructing MicroPython to run a script. code is not being passed to the repl using stdin of the subprocess.

The repl works fine if I launch MicroPython from the shell. I get a prompt and it prints as it should. I do not however get the repl when I launch it from a subprocess. This is what I am doing to run MicroPython from a subprocess

import subprocess
import sys
import os


ctrl_c = b'\x03'  # 2 times to exit any running code
ctrl_d = b'\x04'  # exit paste mode committing what has been pasted
ctrl_e = b'\x05'  # enter paste mode
paste_mode_prompt = b'==='
repl_prompt = b'>>>'

code = b'''\
for i in range(1000):
    print(i)
'''

p = subprocess.Popen(
    [os.path.expandvars('$HOME/lv_micropython/ports/unix/build-standard/micropython')],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    stdin=subprocess.PIPE,
)

entered_repl = False
entered_paste_mode = False

while p.poll() is None:
    for line in iter(p.stdout.readline, b''):
        line = line.strip()
        if line:
            sys.stdout.write(line.decode('utf-8') + '\n')

            if line == repl_prompt:
                if not entered_repl:
                    p.stdin.write(ctrl_e)
                    p.stdin.flush()
                    entered_repl = True
                else:
                    p.stdin.close()
            elif line == paste_mode_prompt:
                if not entered_paste_mode:
                    p.stdin.write(code)
                    p.stdin.flush()
                    entered_paste_mode = True
                else:
                    p.stdin.write(ctrl_d)
                    p.stdin.flush()

if not p.stdout.closed:
    p.stdout.close()

if not p.stderr.closed:
    p.stderr.close()

sys.stdout.flush()

I get zero output from the subprocess. I should be seeing

MicroPython v1.20.0-700-g841ece132 on 2023-10-28; linux [GCC 11.4.0] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>>

which I am not getting at all when I run it as a subprocess. I am not sure why.

[kdschlosser]

It seems like I am not able to interact with MicroPython due to it behaving in a different manner then when it is run from the shell.

[andrewleech]

Looks like Unix port checks that it's running with a tty before entering repl

micropython/ports/unix/main.c

Line 732 in 2fda94c

if (isatty(0) || inspect) {

Maybe if you run subprocess with shell=True it'll behave like you want.

[kdschlosser]

I tried using the shell argument and it still didn't behave.

[kdschlosser]

That really stinks that it is made like that. I wanted to be able to feed in code via the repl and realtime monitor things like memory while performing the unit tests. IDK if you have heard of LVGL or not. LVGL is a graphics framework and there is a binding that is made so it will run in MicroPython. I wanted to do unit tests in a manner that would kind of simulate an MCU. I was going to have it dump the drame buffer output to stdout so that it could be read using subprocess without having a large amountof additional overhead like adding some other means of transferring the information between the 2 processes.

I will make a feature request and see if that is something that could be turned off using a build argument this way it makes it a bit more flexible. It might get added, ya never know. There is an applicable use case for it.

[kdschlosser]

OH! it looks like there is a way to do it. Have to set MICROPYINSPECT to 1 in the environment.

[kdschlosser]

yessir that did the trick. Awesome! I am happy with that.

[picpic020960]

bonjour ,
same test :
MicroPython v1.19 .... Use Ctrl-D to exit, Ctrl-E for paste mode
ok but after , no prompt '>>>'
v1.19 bad version or i needs tricks
thank's for help

[shariltumin]

May I suggest two easier alternatives. Please set MICROPYINSPECT before running the script. Thanks Kevin! - for figuring this out.

$ export MICROPYINSPECT=1

The first of these uses subprocess.communicate().

$ python my_test_3.py
Output:-
MicroPython v1.20.0-555-g92717a95c-dirty on 2023-10-04; linux [GCC 9.4.0] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> x=2*3
>>> print(x)
6
>>> print(x**4)
1296
>>> 
>>> for i in range(10):
...     print(i)
... 
0
1
2
3
4
5
6
7
8
9
>>> 

The second is a modification based on an idea from Kevin's test script.

$ python my_test_1.py
MicroPython v1.20.0-555-g92717a95c-dirty on 2023-10-04; linux [GCC 9.4.0] version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> x=23
>>> print(x)
23
>>> print(x**24)
480250763996501976790165756943041
>>> 
>>> for i in range(10):
...   print(i)
... 
0
1
2
3
4
5
6
7
8
9
>>>DONE

The two scripts are as follows.

1. my_test_3.py

#
# my_test_3.py 

from subprocess import Popen, PIPE

p = Popen(['micropython'], stdin=PIPE, stdout=PIPE, stderr=PIPE, text=True)

# List of cmd input strings
inputs = [
"x=2*3\n",
"print(x)\n",
"print(x**4)\n",
'''
for i in range(10):
    print(i)

''',

]

# Sending each cmd input string one after the other
for input_str in inputs:
    p.stdin.write(input_str)
    p.stdin.flush()  # Flush the input buffer

# Read the output from the subprocess
res, err = p.communicate()
print("Output:-")
for r in res.split('\n'):
    print(r)

2. my_test_1.py

#
# my_test_1.py

from subprocess import Popen, PIPE
import sys

cmd = [
"x=23\n",
"print(x)\n",
"print(x**24)\n",
'''
for i in range(10):
  print(i)

''',
'\n',  # need extra blank line to terminate
]

with Popen(["micropython"], 
   stdout=PIPE,
   stderr=PIPE,
   stdin=PIPE,
   text=True) as proc:
   inx=0
   ww = ''
   while 1:
      w = proc.stdout.read(1)
      if w != '\n':
         ww+=w
         if ww.find(">>>") >= 0:
            print(ww, end='')
            ww = ''
            if inx < len(cmd):
               proc.stdin.write(cmd[inx])
               proc.stdin.flush()
               inx+=1
            else:
               print('DONE')
               break
      else:
         print(ww)
         ww = ''

I hope they are of some use to someone.

[kdschlosser]

no worries m8, Thanks goes to @andrewleech for providing the link to the code that was blocking the repl because of using isatty. I clicked the link and say there was another condition and scrolled up a little bit to locate what that condition exactly was. once I saw it was a check for an environment variable i knew it was going to be good to go from that point. So TY @andrewleech for tracking down the code that was blocking it from working.

[picpic020960]

thank's for reply.
same stuck with this examples so I thinks that V1.19 is bad.
How to were more sure ?

[kdschlosser]

Give me a few to locate the test code I used. I will pass it on over to you. I know it works 100% by just running the script from CPython. The only thing you will have to change in the script is the path to MicroPython

[kdschlosser]

found it.

no need to set any environment variables or anything like that. change the path and run the script. If your build of micropython is good then it should work.

import subprocess
import sys
import os

MICROPYTHON_PATH = '$HOME/lv_micropython/ports/unix/build-standard/micropython'

ctrl_c = b'\x03'  # 2 times to exit any running code
ctrl_d = b'\x04'  # exit paste mode committing and running what has been pasted
ctrl_e = b'\x05'  # enter paste mode

paste_mode_prompt = b'==='
repl_prompt = b'>>>'

code = b'''\
for i in range(1000):
    print(i)
'''

os.environ['MICROPYINSPECT'] = '1'

p = subprocess.Popen(
    [os.path.expandvars(MICROPYTHON_PATH)],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    stdin=subprocess.PIPE,
    env=os.environ
)

entered_repl = False
entered_paste_mode = False

output = b''
while p.poll() is None:
    char = p.stdout.read(1)
    if char != b'':
        output += char

        if b'999' in output:
            sys.stdout.write(output.decode('utf-8'))
            sys.stdout.flush()
            p.stdin.close()

        elif repl_prompt in output:
            sys.stdout.write(output.decode('utf-8'))
            sys.stdout.flush()
            output = output.split(repl_prompt, 1)[-1]

            if not entered_repl:
                p.stdin.write(ctrl_e)
                p.stdin.flush()
                entered_repl = True

        elif paste_mode_prompt in output:
            sys.stdout.write(output.decode('utf-8'))
            sys.stdout.flush()
            output = output.split(paste_mode_prompt, 1)[-1]
            if not entered_paste_mode:
                p.stdin.write(code)
                p.stdin.write(ctrl_d)
                p.stdin.flush()
                entered_paste_mode = True

if not p.stdout.closed:
    p.stdout.close()

if not p.stderr.closed:
    p.stderr.close()

sys.stdout.flush()

[picpic020960]

thank's
this script version is OK with micropython V1.19
the trick is the read character by character , not readline.
(build the last version of micropython = KO . mpthreadport.c issue , not clear and no workaround , I continue with 1.19)

[kdschlosser]

Yeah that's because the prompts don't end in a new line. They actually end with a space. It's just best to dump the output to a buffer 1 character at a time and then check the buffer for what you need and split the buffer a single time shaving off the front of it before the data that you looked for.

To be able to tell where the code I pasted in ends and any output begins I made it so that # end is added to the end of the code that I past in. once the ctrl code get sent and the code runs there is not going to be an easy way to know what is what unless you put "=== " at the beginning of each line of the code you pasted in and then you can tell where the codes end in the output. so by adding the # end on the last line of code that is sent when you read the output from MicroPython that is what you look for. when it is gotten then you can send the ctrl code to exit paste mode which will run the code you just pasted in.. that way you know any output that happens from that point until you get another repl code prompt is going to be from the code you pasted in.

[kdschlosser]

I have already hammered out a rough framework to run unit tests using CPython and passing data between MicroPython and CPython for the tests.

The reason why I wanted to go this route is it is an easy way to send GUI frame buffer data and be able to save the invalid data to an output png file using PIL. This way the error image can be a build artifact in a CI and visually inspected for the issues with it. using the stdout and stdin built into MicroPython is already set up and how it is written makes it fast and not too memory intensive. It simplifies the entire process which makes things easier from a maintenance standpoint.

I also already have a plethora of examples that are written that can now be used to run the tests with and they don't have to be modified in any way to be able to do that. I run each example as a test case and a subprocess is launched for each test case, this ensure a clean sandbox environment for each example to run in. because the subprocess is persistent between each test in a test case I am able to write the code needed to check input and output from the different objects. 1/2 the work is now already done.

[kdschlosser]

I am going to build a more complete framework for using CPython to run unit tests on code that is written for MicroPython. This can be a handy thing to use. I am also going to build a framework specifically made for passing data back and forth this way as well. MicroPython can be packaged with other software if wanting to provide realtime scripting support for whatever reason.