[Feature Request] Run maestro on multiple devices at the same time on a single computer

[jimingzhi]

Is your feature request related to a problem? Please describe.
A computer can only run one maestro even if there are multiple devices connected, and if i want to run multiple maestro, i need multiple computers, which is a waste of resources.

Describe the solution you'd like
Run maestro on multiple devices at the same time on a single computer. If multiple devices are connected to the PC, run the ‘-device’ command to select a specific device to run the maestro on.

[holzgeist]

⚠️ This is just a hack and I have no idea about other implications, but:

• setting connectToExistingSession = false here, and
• running the two instances with different values for --port (i.e. maestro --device <deviceA> --port 7001 test ... and maestro --device <deviceB> --port 7002 test ...)

allows me to run two sessions in parallel.

My use case is a complete e2e test of a chat app, where two instances of the app exchange chat messages

[btrautmann]

I'm trying to understand if this is the issue I'm running into.

I'm running maestro tests on Android and iOS at the same time on a self-hosted github actions runner.

I see the following on Android when doing so:

io.grpc.StatusRuntimeException: INTERNAL: http2 exception
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
	at maestro_android.MaestroDriverGrpc$MaestroDriverBlockingStub.deviceInfo(MaestroDriverGrpc.java:634)
	at maestro.drivers.AndroidDriver$deviceInfo$1.invoke(AndroidDriver.kt:180)
	at maestro.drivers.AndroidDriver$deviceInfo$1.invoke(AndroidDriver.kt:179)
	at maestro.drivers.AndroidDriver.runDeviceCall(AndroidDriver.kt:1062)
	at maestro.drivers.AndroidDriver.deviceInfo(AndroidDriver.kt:179)
	at maestro.Maestro.fetchDeviceInfo(Maestro.kt:63)
	at maestro.Maestro.access$fetchDeviceInfo(Maestro.kt:39)
	at maestro.Maestro$cachedDeviceInfo$2.invoke(Maestro.kt:47)
	at maestro.Maestro$cachedDeviceInfo$2.invoke(Maestro.kt:46)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at maestro.Maestro.getCachedDeviceInfo(Maestro.kt:46)
	at maestro.Maestro.deviceInfo(Maestro.kt:57)
	at maestro.orchestra.Orchestra.initJsEngine(Orchestra.kt:217)
	at maestro.orchestra.Orchestra.runFlow(Orchestra.kt:107)
	at maestro.cli.runner.TestSuiteInteractor.runFlow(TestSuiteInteractor.kt:250)
	at maestro.cli.runner.TestSuiteInteractor.runTestSuite(TestSuiteInteractor.kt:82)
	at maestro.cli.command.TestCommand$handleSessions$1$1$results$1$1$1.invoke(TestCommand.kt:281)
	at maestro.cli.command.TestCommand$handleSessions$1$1$results$1$1$1.invoke(TestCommand.kt:258)
	at maestro.cli.session.MaestroSessionManager.newSession(MaestroSessionManager.kt:102)
	at maestro.cli.session.MaestroSessionManager.newSession$default(MaestroSessionManager.kt:54)
	at maestro.cli.command.TestCommand$handleSessions$1$1$results$1$1.invokeSuspend(TestCommand.kt:258)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)
Caused by: io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:353)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

On iOS, I see something like:

Error: Exception in thread "main" java.nio.file.NoSuchFileException: /Users/runnersmac/Library/Logs/maestro/2024-09-25_123706
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
	at java.base/java.nio.file.Files.readAttributes(Files.java:18[51](https://github.com/Betterment/mobile/actions/runs/11037192389/job/30657462405#step:11:52))
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:220)
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277)
	at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323)
	at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71)
	at java.base/java.nio.file.Files.walk(Files.java:3918)
	at java.base/java.nio.file.Files.walk(Files.java:3973)
	at maestro.utils.FileUtils.zipDir(FileUtils.kt:26)
	at maestro.debuglog.DebugLogStore.finalizeRun(DebugLogStore.kt:91)
	at maestro.cli.AppKt.main(App.kt:139)

Both platforms pass when run alone.

Is this the same symptom you are seeing @jimingzhi @holzgeist (or something like it) or does Maestro actually output an error saying something like "cannot run more than once test at a time"?

I'll give @holzgeist's suggestion a try and report back--it would seem that you should just be able to specify the port and that should be enough 🤔

[holzgeist]

Hi @btrautmann
I actually worked around this issue by not testing live chats on two phones, but logging out and logging in as the second person on the same phone/emulator..

As far as I remember, the error message looked different. There was something about binding ports (but I don't remember which ports on which host..)

[btrautmann]

Hi @btrautmann I actually worked around this issue by not testing live chats on two phones, but logging out and logging in as the second person on the same phone/emulator..

As far as I remember, the error message looked different. There was something about binding ports (but I don't remember which ports on which host..)

Interesting, thanks for confirming. I see a noticeable improvement in reliability when only running one at a time, but we definitely need to run these in parallel as our suite is ~20 minutes long, so having it be 40+ minutes is no good!

[ice-cap0]

i'm able to run both on my machine, and obtain the results.

however >80% of the time I get this error which leaves maestro process hanging

maestro --port 7001 --device e-1 test test1.yaml
# other terminal
maestro --port 7002 --device e-2 test test2.yaml

Exception in thread "main" java.nio.file.NoSuchFileException: /Users/me/Library/Logs/maestro/2024-12-04_125846
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
        at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
        at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:220)
        at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277)
        at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323)
        at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71)
        at java.base/java.nio.file.Files.walk(Files.java:3918)
        at java.base/java.nio.file.Files.walk(Files.java:3973)
        at maestro.utils.FileUtils.zipDir(FileUtils.kt:26)
        at maestro.debuglog.DebugLogStore.finalizeRun(DebugLogStore.kt:91)
        at maestro.cli.AppKt.main(App.kt:145)

i tried setting debug-output and flatten-debug-output, same error

any ideas?

[Del-S]

I am also able to run multiple tests on my machine but the problem comes when I move to gitlab CI/CD. For now we have one machine (single runner) which should be able to run 2 emulators and allow us to run two different tests (jobs) in parallel. The thing is that it fails (only one of the jobs not both) when the tests run in parallel because it cannot find an element which is on the screen and I am not sure why.

15:32:05.393 [ INFO] maestro.cli.runner.TestSuiteInteractor.invoke: Assert that "Dev tools" is visible FAILED

Even the commands json file contains the child which contain the "Dev tools" text.

"children" : [ {
  "attributes" : {
    "text" : "Dev tools",
    "ignoreBoundsFiltering" : "false",
    "clickable" : "false",
    "bounds" : "[44,111][317,197]",
    "enabled" : "true",
    "focused" : "false",
    "checked" : "false",
    "scrollable" : "false",
    "selected" : "false",
    "class" : "android.widget.TextView"
  },

If the tests do not run in parallel it works fine so I am not really sure what is going on. Both emulators and maestro commands use different ports but there is probably still some connection between them which breaks it which is unfortunate.

We would love to have the option to run multiple tests in parallel in our own environment.