[SPARK-49495][DOCS] Document and Feature Preview on the master branch via Live GitHub Pages Updates

[yaooqinn]

What changes were proposed in this pull request?

This pull request introduces functionalities that enable 'Document and Feature Preview on the master branch via Live GitHub Pages Updates'.

Why are the changes needed?

• Instead of limited 72-hour voting phases, it provides the developer community with more opportunities to preview and verify the documentation contents.
• Instead of waiting for the final announcement of an official spark feature release, users can now preview some of the ongoing documented features, increasing the willingness to upgrade, sensing breaking changes in advance, and reducing the burden during the final upgrades.

Does this PR introduce any user-facing change?

No

How was this patch tested?

An example has been established by this repo - https://github.com/yaooqinn/spark-gh-pages

• Broken logo links are fixed at [SPARK-49494][DOCS] Use spark logos from spark-website in docs #47966

Was this patch authored or co-authored using generative AI tooling?

no

[HyukjinKwon]

While it's great to have the documentation for the latest development branch, I wonder if it's better to have the actual full nightly build and/or frequent release instead via setting up the release automation process.

My concern is that the Google result might show this page first, and people might get confused of, e.g., the new features in development.

[HyukjinKwon]

Actually one of my colleagues is working on setting up the release automation, cc @bozhang2820 WDYT?

[yaooqinn]

Thank you for the feedback @HyukjinKwon.

I wonder if it's better to have the actual full nightly build

This action can be scheduled w/ cron or via push, I tended to submit this PR using the push-and-cancelation trigger for a more up-to-date doc view.

We can benefit from this for a post-review on docs-related PRs

frequent release

While I'm also a supporter of the 'Release Early, release often' principle, after being the RM for 3.5.2, I think this could cost much on the infrastructures, such as 'pypi', 'ASF svn', 'mvn central', 'docker registry', e.t.c.

My concern is that the Google result might show this page first

This site will be marked as 'in-progress' or something else to reduce the ambiguity. Also, some directives of reStructuredText or markdown can be used in the doc build phase to enhance the warnings within a page in follow-up PRs.

[dongjoon-hyun]

Thank you, @yaooqinn . I guess we need a vote for this because GitHub Pages is a new channel from Apache Spark perspective.

If we use the official nightly channel, we may avoid the vote. However, IIUC, you want to use GitHub Pages specifically here, right, @yaooqinn ?

• https://nightlies.apache.org/spark/

[yaooqinn]

Thank you @dongjoon-hyun, I will raise a discussion thread in dev soon

[dongjoon-hyun]

Thank you!

[HyukjinKwon]

+1 for going for a vote. I think that'd be easier to decide.

[yaooqinn]

FYI, I have sent a mail to the dev https://lists.apache.org/thread/xojcdlw77pht9bs4mt4087ynq6k9sbqq

Stale

[dongjoon-hyun]

I saw the vote thread a few minutes ago. I voted. Thank you! :)

https://lists.apache.org/thread/lw6rqkvvth44ykncpjy52qjctmbdyrzq

[dongjoon-hyun]

+1, LGTM. I believe we can merge this after the vote passes.

[HyukjinKwon]

This makes the build dependent on each other. E.g., if you have commit 1,2,3 at the same time, build 1 has to be finished before starting build 2. Can we maybe just do this as a scheduled job to save resources?

[yaooqinn]

if you have commit 1,2,3 at the same time, build 1 has to be finished before starting build 2

1 and 2 will be canceled by 3.

[yaooqinn]

Thank you @dongjoon-hyun and @HyukjinKwon