Doctest overhaul

[staticfloat]

I want to minimize the number of times we actually build Julia on CI.
This PR re-uses the output of build_x86_64-linux-gnu, and also adds a
second, signed step to deploy the docs using an encrypted
DOCUMENTER_KEY.

[staticfloat]

@fredrikekre @mortenpi I would like to have the docs we build on this repository be uploaded to a different place than the main docs that get built on JuliaLang/julia#master. What's the easiest way for us to deploy to a similar, but different, location? It could be a different branch of docs.julialang.org, or even a different git repository. I leave it to your discretion. :)

[fredrikekre]

I suppose those docs could serve as previews, right? We can make a new repo, staging-docs.julialang.org or something for that then.

[mortenpi]

Sorry, I haven't really kept myself in the loop with the CI changes.. what's going on here? What builds will these be? I guess it's related to JuliaCI/julia-buildbot#273? I see that we're duplicating the docs manifest here, which doesn't seem ideal. You mentioned #master, but what about release builds?

[DilumAluthge]

I agree that it is suboptimal to store a duplicate copy of the docs manifest in this repo.

Can we instead use the docs manifest that already exists in the JuliaLang/julia repo?

[staticfloat]

I'm trying to separate the "building" of docs (which is quite Julia and Documenter version-specific) with the "uploading" of docs (which should not be julia or Documenter specific, I hope!)

Additionally, I am trying to minimize attack surface. The upload step has access to our DOCUMENTER_KEY. I'd like to ensure that someone doesn't do something nasty like insert a new dependency into the documenter manifest in a PR, run that PR, then get access to our DOCUMENTER_KEY. The way we do that is by signing the manifest and project TOML files, but that's a bit of a bother to maintain if we need to bump Documenter for building the docs so often. So this project is specifically only for uploading, since that should be fine to use an older Documenter version.

In fact, I'd be fine with not using documenter at all if the upload step is easily recreated without importing the entire package.

[DilumAluthge]

Oh I see. I'm 100% on board with that.

Actually, yes, we shouldn't need any manifest at all for the (signed) upload step. For the (signed) upload step, we should download the built docs from a previous (unsigned) docbuild step, and then just do a git push or whatever we need to do. We definitely shouldn't need to install or run Documenter in the (signed) upload step.

[staticfloat]

Yeah, I just don't really know what this line does

[fredrikekre]

Basically git clone + commit + push, but Documenter does some postprocessing of the built docs too, so not sure it is trivial to skip.

[DilumAluthge]

but Documenter does some postprocessing of the built docs too

We would just need to find a way to do that postprocessing during the (unsigned) docbuild step.

[DilumAluthge]

Could you rename the utilities/docs folder to utilities/docs_upload? That would make it more clear (even from a quick glance) that the manifest and project in that folder are only for the purposes of the upload step.

[mortenpi]

I also agree with the goal, I think that makes a lot of sense. So this would deploy everything, i.e. master, pre-release docs, and tags? And the signing is just there to make sure that the input environment hasn't been modified?

Replicating deploydocs here would mean duplicating quite a bit of fragile code, and would also duplicate the maintenance effort of that code. deploydocs generates a few files, and makes sure that the correct directory structure and symlinks get generated in the Git repo. Using an older Documenter version to push would work most of the time, but e.g. this PR is one that required a simultaneous change to deploydocs and makedocs to work correctly. With the way the makedocs-deploydocs interaction is right now, I don't see any really good answers here.

[staticfloat]

So this would deploy everything, i.e. master, pre-release docs, and tags? And the signing is just there to make sure that the input environment hasn't been modified?

Correct, and correct.

With the way the makedocs-deploydocs interaction is right now, I don't see any really good answers here.

Could we "deploy" the docs to a local folder, then do only the git push steps in the signed steps? E.g. can we have documenter do everything except pushing to the actual target in the unsigned builder?

[DilumAluthge]

Could we "deploy" the docs to a local folder, then do only the git push steps in the signed steps? E.g. can we have documenter do everything except pushing to the actual target in the unsigned builder?

Yeah, this is what we need.

[mortenpi]

It shouldn't be too hard to modify Documenter such that it wouldn't git push. However, the post-processing should happen on a cloned Git repository (i.e. the full Pages branch). How would you want to handle passing that to the signed step? I am not sure you can really trust a Git repo since you can set up hooks etc.

[staticfloat]

Can't we run the full post-processing step in the unsigned builder, then all the signed builder does is copy some files somewhere? I wouldn't copy a git repository in its entirety, I would tarball up the content from the unsigned builder, then on the signed builder i would clone the relevant git repository, delete all contents within the folder, then unpack the tarball copied from the unsigned builder, and push it up.

[mortenpi]

The postprocessing modifies the whole Pages branch (updates the symlinks, and needs to know what is on the branch to generate the version selector). I suppose we could tarball the whole working directory up and then unpack that in the signed step. Another potential option would be to generate a .patch file for the whole commit and then apply that patch in the signed step?

[staticfloat]

I'm happy with either (I'd probably go with the .tar approach, myself).

How can I get Documenter to do all that work but not push it at the end?

[mortenpi]

So, I think we could patch Documenter like this: JuliaDocs/Documenter.jl@bd1e344, and set deploydocs(archive=...) in the make.jl script. Then you should end up with a tarball, rather than git push getting called.

It seems to work fine locally, but could we test it here with the mp/deploy-archive before merging this into Documenter? It should work if we just update Documenter to that branch after cloning the Julia repository (and patch the make.jl script).

[mortenpi]

Suggested change

	make -C doc html doctest=true
	make -C doc deploy doctest=true

I think?

[mortenpi]

We also need to have DOCUMENTER_KEY set because of https://github.com/JuliaLang/julia/blob/b11ccae719c5826cc5c1629356c0b057e25b1b45/doc/make.jl#L340. I am not sure if e.g. an empty string would work though..

[mortenpi]

Alright, if DOCUMENTER_KEY is set to an empty string, Documenter should be able to unpack it fine into an empty file. However, I am not sure what ssh will do if it tries to read an empty key file (with this .ssh/config).

So if that doesn't work, what might work is patching make.jl further with

struct BuildBotConfig <: Documenter.DeployConfig end
# Add these two lines:
Documenter.authentication_method(::Documenter.DeployConfig) = Documenter.HTTPS
Documenter.authenticated_repo_url(::Documenter.DeployConfig) = "git@github.com:JuliaLang/docs.julialang.org.git"

to use the HTTPS authentication path.

[staticfloat]

Alright, it's all looking good to me! I think I'm confident enough to push this through now. Will clean this up, and merge the changes to JuliaLang/julia. Thanks guys!

[staticfloat]

@mortenpi can you merge your Documenter branch first? I figure it's best if we don't use a branch on base Julia

[mortenpi]

Yep, will have it in 0.27.20 ASAP.

[mortenpi]

JuliaRegistries/General#63939

@staticfloat you have a branch with the necessary changes to the Julia master branch, right?

[staticfloat]

Yes: https://github.com/JuliaLang/julia/tree/sf/documenter_overhaul

[staticfloat]

Alright, this is all looking really good!

Final question: is there some way we can "deploy" these changes somewhere that's not the actual deployment target? Should we push to a non-gh-pages branch on the docs.julialang.org.git repository? Should we push to S3 somewhere? Should we push to a different repository if we're not on master?

[mortenpi]

We don't have anything set up for that, but pushing to a different docs.julialang.org branch should be straightforward. But why do you want to deploy those builds?

[staticfloat]

So that people can see the changes they just made, live somewhere.

[mortenpi]

Hmm. Would that be for PRs only then? And only for PRs from JuliaLang/julia or also forks?

In principle, we have the PR previews feature, so we could push to the same gh-pages, but that would only be safe for non-fork PRs. Pushing to a different branch wouldn't be too useful, since you can't access that as a web page. But one could always create a separate docs-previews repo that serves its own gh-pages under a different subdomain.

[staticfloat]

Why would that only be safe for non-fork PRs? (Also, I’m going to merge this as-is, and we’ll continue discussion of improvements regardless)

[mortenpi]

Why would that only be safe for non-fork PRs?

The way it's set up, the make.jl script still has full control over the contents of the gh-pages branch, and you could easily e.g. nuke all the docs by making it produce an empty tarball (we can probably restore by reverting on the docs.julialang.org repo, but still).

If we set up another repo or S3 for forks, then we could mitigate this though. For what its worth, with GHA, I have recently been leaning towards using build artifacts for previews, rather than trying to deploy them somewhere custom. Not sure if Buildkite has anything like that, or would that require setting up a custom S3?