Deactivate Unnecessary CIs for Docs

[justusschock]

🚀 Feature

Run only Doctest and Docbuilding CIs if changes are only for docs

Motivation

With #10176 we saw a lot of commits changing the documentation only.
For such commits, it is not necessary to run the full CI. After discussing with @tchaton , we agreed that we cannot rely only on the github label but also have to inspect the git diff to verify only docs were changed.

Pitch

Only run necessary CI pipelines

Alternatives

keep as is

Additional context

Could maybe do the other way around and only run CIs if changes on tests or package. That would then automatically also exclude readme etc.

---

If you enjoy Lightning, check out our other projects! ⚡

• Metrics: Machine learning metrics for distributed, scalable PyTorch applications.

• Flash: The fastest way to get a Lightning baseline! A collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning

• Bolts: Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

• Lightning Transformers: Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.

cc @Borda @rohitgr7 @carmocca @akihironitta @Borda @daniellepintz

[Borda]

we cannot rely only on the github label but also have to inspect the git diff to verify only docs were changed

this can e simply configures by path in GHA
https://github.com/PyTorchLightning/pytorch-lightning/blob/9237106451f97393b17009a0ca571b6ff5ba5484/.github/workflows/ci_dockers.yml#L10

[carmocca]

I'm not sure that's right. The docstrings in the code are used to autofill the docs. So any change to a source file can impact the generated docs.

Additionally, docs code examples have a few imports which could break accidentally if left untested. For example:

https://github.com/PyTorchLightning/pytorch-lightning/blob/9237106451f97393b17009a0ca571b6ff5ba5484/docs/source/conf.py#L364-L379

[justusschock]

@carmocca that‘s true and there‘s probably nothing we can do about that, but when directly editing rst-files for example, it should be possible.

And for the doctests and examples: running those takes significantly less time than running our whole pipeline :)

[carmocca]

Sorry! I misunderstood the proposal. You are saying:

• Any source file was edited: test everything
• Only docs files were edited: test only docs

In that case, I agree. And it's quite easy as Jirka mentioned. Here's the syntax docs: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#patterns-to-match-file-paths

[carmocca]

There's a big limitation with this proposal which is required jobs. They still need to run even if the workflow has signaled to skip them: actions/runner-images#1281

[justusschock]

Can't we just skip all steps then? So that the overall workflow still passes?

[carmocca]

I guess that would work but it's

1. ugly to write. we'd have to add skips everywhere
2. misleading, as the PR will appear to be actually running the jobs for authors unaware of this.

[mhadole-tc]

Potential solution:

1. Disable status checks for workflows that are not supposed to trigger on every PR and use GitHub status API to update status to fail if workflow fails.
2. Enable status checks for actions that do not have paths conditional triggers.

[Borda]

1. Disable status checks for workflows that are not supposed to trigger on every PR and use GitHub status API to update status to fail if workflow fails.

do you have some examples? just be aware the docs are testing alignment with codebase so we can drop pytest if edits are only in docs but not vise-versa...

2. Enable status checks for actions that do not have paths conditional triggers.

what do you mean, badges?

[stale]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!