RFC - Creating an openxla/dev-tools repo and building automation

[stellaraccident]

I would like to contribute a variant of my in-progress development tools repository to the openxla organization: https://github.com/stellaraccident/openxla-dev-tools. I suggest simply shortening it to dev-tools once brought into the organization-proper.

The above represents a few hours of coding to build the boilerplate tooling that we can use for managing repositories, dependencies, and corresponding automation in the OpenXLA projects, and is a first step to helping us build workflows that bump dependencies and enable release automation.

In consultation with various folks, I concluded that a dedicated tools project was in order, and I chose to make that a cohesive python package that can be installed by anyone with a one liner. In addition, we wanted to make it straight-forward for folks (and CI) to check out individual projects and have a single script to run that would update their deps (without requiring installation of more advanced dev tooling packages like this). That way, we should be able to have a flow where advanced, cross-repo users can use this tools package as their daily driver, while people focused on a single project get exactly what they need (by being able to run sync_deps.py locally and always be up to date).

As an example (not yet completely refactored but to show the idea), here is a draft of the openxla-pjrt-plugin project using this tools package to manage its deps: https://github.com/openxla/openxla-pjrt-plugin/pull/29

Regardless of where we take such dev-tools and automation in the future, it seems that we definitely need a centrally accessible dev-tools project, and I therefore propose that we bootstrap it with the code I have started writing.

[jpienaar]

Being able to easily setup and use SGTM This reminds me of the "recent" https://github.com/google/git-mmrepo repo.

Nit: I'd suggest openxla-dev-tools still, org is elided when forking and checking out, that makes origin difficult to see.

[stellaraccident]

Being able to easily setup and use SGTM This reminds me of the "recent" https://github.com/google/git-mmrepo repo.

Yeah, let's say I've learned a few things since then about not trying to do too much or create something too general. This is quite specific to our projects.

I think I've basically gotten this to a point of good utility now. I've overhauled the openxla-pjrt-plugin repo to be managed via this mechanism, and I implemented dep rolling of both the source and binary (python deps).

• This workflow now runs at 12:30PM each day to roll all source and binary pinned deps and creates/approves a PR for submission (if tests pass). I need to touch base with Geoffrey next week for some permission and de-hacking stuff but it basically works.
• Here is an example of an auto-roll PR that submitted itself.
• pjrt-plugin instructions are a lot nicer now that everything is properly pinned.
• I implemented what is needed for a continuous bump for main deps but at that frequency, we probably want a centralized dep roller bot to coordinate that vs just having a bunch of random GitHub actions trying to not step on each other. One step at a time.
• It shouldn't be too hard to create such a bot since it is mostly just choreographing a few basic commands.

See the current README for the state of what is implemented. I think we can roughly call this ready to promote to a real repo.

[aaron-schneider]

Submitting request to include @jpienaar's suggestion (openxla/openxla-dev-tools).