Merge docs and source repositories

[daquinteroflex]

We'd like to merge the documentation with the source code to have a more maintainable documentation environment and configuration. However, we don't want to give rtd organization access. One way to solve this is to implement rtd web hooks as described on these instructions The strategy to perform the actual repository merge could follow something like this blog post

This issue is to track progress integrating the docs and source code, and whatever arises from doing that.

[daquinteroflex]

Some notes on this based on this blog

1. Re-arrange each repo so they can merge without conflicts
2. Add each repo as a remote source for the monorepo
3. Merge (rebase?) the repos with -allow-unrelated-histories

Basically, this works as long as there are no existing related files that can clash between documentation and source code.

From docs:

• Remove pyproject.toml
• Remove docs/requirements.txt as ported to new tidy3d/ pyproject.toml
• Maintain .readthedocs.yaml to integrate
• Maintain .binder/ requirements
• .github/workflows should not clash between the projects
• We need to update out all those sys.path.insert to the new paths.
• Remove .pytest.ini docs
• Move docs README.md to the docs source folder pending refactor
• Remove tidy3d_docs.py
• Move docs version management back to __version__.py on the source code repo
• Decide on the future of the docs .git_scripts
• Remove docs tests/requirements.txt
• Remove .gitmodules as submodule no longer required
• Delete .gitignore
• Delete .gitattributes
• Maintain test_notebooks.py configuration

Source code:

• Upgrade and merge the two .gitignore files
• Upgrade the two .gitattributes files

Decisions:

• Are we happy with this merge approach in order to maintain git history?
• Are we happy to stick to the source code tidy3d license?
• Are we happy to restructure the docs/ folder so that it only contains documentation inside there?

[daquinteroflex]

A few things to note, each of those pull requests contains the non-conflicting repository status of the docs and source code, in my own personal forks. The idea is that by changing the remote now of the docs repo (which again the remotes are my own personal forks), it should be possible to rebase all the git history of the docs into the main repository.

It is important to mention we should be able to support progressive integration from the docs to the source until whenever we want, as long as all the work is contained within the docs/ directory.

Whenever we want to integrate with the develop branch, then we just should need to merge #1234 At that point it becomes a pain to keep updating the other documentation repo. Until then we can follow the update flow above, in order to keep the status of both repositories synced.

So I run:

daquintero@Lisas-MacBook-Pro tidy3d-docs % git remote add -f source https://github.com/daquinteroflex/tidy3d.git
Updating source
remote: Enumerating objects: 14332, done.
remote: Counting objects: 100% (902/902), done.
remote: Compressing objects: 100% (419/419), done.
remote: Total 14332 (delta 615), reused 615 (delta 483), pack-reused 13430
Receiving objects: 100% (14332/14332), 59.00 MiB | 10.51 MiB/s, done.
Resolving deltas: 100% (10898/10898), done.
From https://github.com/daquinteroflex/tidy3d
 * [new branch]        develop            -> source/develop
 * [new branch]        merge_docs         -> source/merge_docs
 * [new branch]        prepare_repo_merge -> source/prepare_repo_merge
 * [new branch]        repo_merge         -> source/repo_merge

I think I'll follow the set of instructions in the guide rather than rebasing, as it only makes sense to rebase after this branch is fully complete and full of all the revenant commits and documentation from the other repo.

daquintero@Lisas-MacBook-Pro tidy3d-docs % git merge source/repo_merge --no-commit --allow-unrelated-histories
Automatic merge went well; stopped before committing as requested
daquintero@Lisas-MacBook-Pro tidy3d-docs % git commit -am "FEAT: Successful merge"
[repo_merge ea2f7e07] FEAT: Successful merge
daquintero@Lisas-MacBook-Pro tidy3d-docs % git push source repo_merge
Enumerating objects: 6436, done.
Counting objects: 100% (6436/6436), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2007/2007), done.
Writing objects: 100% (6426/6426), 486.16 MiB | 35.32 MiB/s, done.
Total 6426 (delta 4362), reused 6407 (delta 4351), pack-reused 0
remote: Resolving deltas: 100% (4362/4362), completed with 3 local objects.
To https://github.com/daquinteroflex/tidy3d.git
   bb496e22..ea2f7e07  repo_merge -> repo_merge

[tylerflex]

Are we happy to restructure the docs/ folder so that it only contains documentation inside there?

Yea I think that should be fine.

Are we happy with this merge approach in order to maintain git history?
Are we happy to stick to the source code tidy3d license?

Maybe @momchil-flex has some thoughts on license and git history.

[momchil-flex]

Are we happy with this merge approach in order to maintain git history?
Are we happy to stick to the source code tidy3d license?

Maybe @momchil-flex has some thoughts on license and git history.

License sounds fine. Re the merge approach, is that explained in detail somewhere? It is not clear to me from just the messages here.

[daquinteroflex]

Sounds good! So the merge approach is in development still. The idea currently is to move all example notebooks to a separate repo, and merge the docs/source code to improve shared packaging and development. This should be nearly tiny impact on the current web examples deployment approach, and should streamline the documentation builds.

…

On Tue, 7 Nov 2023 at 21:55, momchil-flex ***@***.***> wrote: Are we happy with this merge approach in order to maintain git history? Are we happy to stick to the source code tidy3d license? Maybe @momchil-flex <https://github.com/momchil-flex> has some thoughts on license and git history. License sounds fine. Re the merge approach, is that explained in detail somewhere? It is not clear to me from just the messages here. — Reply to this email directly, view it on GitHub <#1231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDV5U6VOQ3SWQXHYBII6IZLYDKN3TAVCNFSM6AAAAAA67OGHEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBQGA4TSOJVHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[momchil-flex]

The part I'm still not sure about is that you seem to be saying this can preserve the docs history? Would that mean all of the docs commits applied on top of the current tidy3d frontend? But somehow with notebooks removed?

[daquinteroflex]

There’s a few ways to do this. Check out this pull request #1234 . We can first merge them in with all the history and then remove them when we add the examples repo submodule. We can also just start from a clean slate if we don’t want to preserve history. We can move things around to convert it just in a notebook repo as well and then plug that into the submodule later. Or just make a new notebook repo from the state it’s in now. There’s a few options on what we could do. Also open to ideas on how to preserve things best. What do you prefer?

…

On Tue, 7 Nov 2023 at 22:24, momchil-flex ***@***.***> wrote: The part I'm still not sure about is that you seem to be saying this can preserve the docs history? Would that mean all of the docs commits applied on top of the current tidy3d frontend? But somehow with notebooks removed? — Reply to this email directly, view it on GitHub <#1231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDV5U6TYR65OUGPFQZJFURDYDKRKJAVCNFSM6AAAAAA67OGHEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBQGE3TAOBSGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[daquinteroflex]

Worth adding we could rebase the docs history into the frontend if that’s how we wanted to track the history if we were to go with that option

…

On Tue, 7 Nov 2023 at 22:30, Dario Quintero ***@***.***> wrote: There’s a few ways to do this. Check out this pull request #1234 . We can first merge them in with all the history and then remove them when we add the examples repo submodule. We can also just start from a clean slate if we don’t want to preserve history. We can move things around to convert it just in a notebook repo as well and then plug that into the submodule later. Or just make a new notebook repo from the state it’s in now. There’s a few options on what we could do. Also open to ideas on how to preserve things best. What do you prefer? On Tue, 7 Nov 2023 at 22:24, momchil-flex ***@***.***> wrote: > The part I'm still not sure about is that you seem to be saying this can > preserve the docs history? Would that mean all of the docs commits applied > on top of the current tidy3d frontend? But somehow with notebooks removed? > > — > Reply to this email directly, view it on GitHub > <#1231 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BDV5U6TYR65OUGPFQZJFURDYDKRKJAVCNFSM6AAAAAA67OGHEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBQGE3TAOBSGM> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[momchil-flex]

I'd say probably no need to preserve the docs history in the frontend. We have it in the docs repo if ever needed.

[daquinteroflex]

Sounds good!

TLDR: I'll just copy things manually as a first pass, and then do it again the day we decide to transform the docs repo into a notebooks repo.

So I've been thinking how to implement this at the moment. Before, when merging the whole history, it kind of created two remote links between the two repositories, which meant that they could be both updated progressively until the day we decided to modify the docs one into just notebooks. In the sense that all commits could be propagated along to the new combined repository from both tidy3d and tidy3d-docs repositories.

Without maintaining git history, I could just copy the files over, but then this also removes a "live" commit link between the two repos, as I'd have to do that manually every time until we merge. Maybe that's fine actually.

I'm still investigating now how to implement this without necessarily keeping development history, but letting it progressively be integrate-able. Just because it might still take a few more days to sort out the rest of the documentation porting and writing the docs for the rest of the team on how to set up.

Another approach is that we maintain the history in order to maintain the progressive integration features, but we remove the notebooks on the new combined repo, and then we just modify the docs to notebook repo. The benefit of this is that the docs files that have nothing to do with the notebooks retain their history, and the docs repo maintains all its notebook history too. The real motivation though is the progressive integration aspect of this though.

I'm thinking of how to sort this and will update this message whenever I've found a potential solution for progressive integration without commit history mapping.

So I've been writing how the docs repo should work in order to have a clear set of boundaries of where the projects interact, and where dependencies are managed and scripts included flexcompute-readthedocs/tidy3d-docs#323 (comment)

[daquinteroflex]

I'm closing this as this has been decided and implemented in the main issue in any case #1148