Domino Workflow with Selenium Remote Web Driver

[tdari]

Hey there! I'm experimenting with some browser automation using Domino. I can run my Domino piece and perform tasks within a Docker container with Selenium Remote WebDriver installed. Suppose I create a workflow using these components. Is it feasible to pull this Docker image from Docker Hub, execute all the components within this container, and only remove the container with Web Driver installed when the workflow is complete? What would be a better way to achieve this, if any?

[vinicvaz]

hey @tdari nice you are trying browser automation, good use case!
Not sure if I got what you need so corret me if I'm wrong.
What you need is to make the chrome driver a piece that will start at the begging, stay running until the last piece to finish its process and than shut down the chrome driver piece, is it right?

[tdari]

Yes, exactly. Normally people run their scripts and close the driver at the end. However, I can't close the driver because I want to perform various tasks using different pieces within this container. I want to separate the responsibilities of these pieces and include a piece to quit the driver at the end.

[vinicvaz]

Got it. Ok, we have two situations here:

The first one is that your Chrome Driver piece would need to have a public endpoint to be accessible from the other pieces. Locally it might work but for sure running in k8s it will not since the pieces are not be sharing the same network.

The second is that we don't have a direct way to "finish/kill/stop" a task in Domino.
What I would suggest is to do the following, but if this work will work only running locally because of the problem I said before.

1. Create your Chrome Driver piece as a http server but add to it a route called /shutdown.
2. Use the Domino open source HTTP request piece to send a request to your piece, in your case you can use http://localhost:4444 as url.
3. The http piece will depend from all tasks of your workflow.

Your workflow would look something like:

---

ISSUES
The issue of being able to finish loop tasks at the end of any process is important and should be addressable, I think it should not be so difficult to do an interface for that.
The issue of being able to open http server pieces that can be discovered in the network by other pieces in the same workflow is more complicated and I'll have to think if we can do it now since it may implicate into security issues.

@tdari thanks for bringing this and sorry for not being able to give you a complete answer.

@nathan-vm @luiztauffer I'd would love to hear your thoughts about these two points.

[tdari]

@vinicvaz thank you for pointing out the other issues that may happen. I will also check the docs first because there may not be an option to perform operations without closing the driver. Thank you for your time again.

[nathan-vm]

@tdari I understand your idea, but the "network event" implementation between pieces can be a little tricky in k8s and airflow, one way around is create a chrome instance piece who recive an array of instructions and evaluate then in order, something like:

Piece A - export instruction "go to url"
Piece B - export instruction "take screenshot"
Piece C - export instruction "foo bar"

and then connect this three pieces to "Chrome piece"

Piece A \
Piece B - - -> ChromePiece -> export array of outputs
Piece C /

ChromePiece will be responsible for have the instance of selenium or pupperteer and evaluate the instruction in the order you want, and then export results (if apply) in the same order.

Thats make sense for your solution ?

[tdari]

That might be much simpler. I can parse inputs into different script parts and run the parsed script in a single piece. Thanks, @nathan-vm

[vinicvaz]

@tdari I'll close this discussion but feel free to contact us again if need assistance with this.
Also let us know if you were able to do what nathan suggested :)

[nathan-vm]

@tdari I don't know if you implemented the solution I brought or not, but it ended up that I myself needed to use Selenium with Domino and I ended up creating a piece exactly the way I said. I made a very simple version, but maybe it's a start for your needs. Additionally, if you need any specific support with Domino, do not hesitate to contact us.

Here is the code of my Selenium implementation: https://github.com/nathan-vm/web-browser-scrapper-piece

[tdari]

@nathan-vm Yes, I’ve implemented it in a different way than you do. Thanks, I'll check that out too.