Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy option to enable continuous deployment #1745

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

Conversation

tophtucker
Copy link
Contributor

@tophtucker tophtucker commented Oct 11, 2024

Running notes

  • Whether you build locally or in cloud, the deploy method ends with pollForProcessingCompletion. In the local case, that just means uploading files; in the cloud case, it means doing a whole cloud build. Should we have the same timeout in both cases?
  • Currently this prompts you to enable continuous deployment even if your project directory is ineligible, so you know it would be an option if the criteria were met, and so we can provide informative errors about why it's not available. Sound good?
  • Once your continuous deployment setting is saved, do we ever prompt you about it again? Is there a command to turn it on, or do we instruct you to edit deploy.json?
  • The continuous deployment setting is a property of the source code, but the GitHub connection and build environment are properties of the data app in our database, i.e. the “deploy target”. Those can change independently: e.g. you might unlink GitHub without touching the code, and your next deploy command will fail.
    • State like “continuous deployment is enabled but you don’t have GitHub linked”
    • Can’t tell when it changed though

src/deploy.ts Outdated Show resolved Hide resolved
src/deploy.ts Outdated Show resolved Hide resolved
Comment on lines +543 to +544
// Disables continuous deployment if there’s no env/source & we can’t link GitHub
if (continuousDeployment) continuousDeployment = await this.maybeLinkGitHub(deployTarget);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fil says: if you enable continuous deployment, it should stay on, and if we can't connect to github for whatever reason (you're not in a repo, or no git remote, or no link in our database), the deploy should just fail.

workspaceLogin: deployTarget.workspace.login,
projectSlug: deployTarget.project.slug
});
if (latestCreatedDeployId !== deployTarget.project.latestCreatedDeployId) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chatting with fil: there's no guarantee that the new deploy ID is the one you just kicked off; maybe your colleague click the deploy button around the same time. currently, the postProjectBuild method can't return the new deploy ID because it just dispatches a message to the job-manager, which at some point will get around to making a new deploy. two options:

  • create deploy id earlier, in api, and pass to job manager (which feels like it might mess with our messaging protocol? i don't know all the reasons it was designed like it is today)
  • pass some unique string to the api, which will pass it to the job manager, which will eventually respond with it, which would let us identify that the deploy was our own

but fil thinks this is probably not a blocking issue with this PR

Comment on lines -194 to -196
const deployConfig = await this.getUpdatedDeployConfig();
const deployTarget = await this.getDeployTarget(deployConfig);
const buildFilePaths = await this.getBuildFilePaths();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fil notes that this could be parallelized: getBuildFilePaths doesn't depend on the previous two; it's filesystem i/o, as opposed to talking to the api. might speed up local deploys a bit

deployId = await this.createNewDeploy(deployTarget);
await this.uploadFiles(deployId, buildFilePaths);
await this.markDeployUploaded(deployId);
}
return await this.pollForProcessingCompletion(deployId);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this pollForProcessingCompletion pattern, in the continuous deployment case, keeps the user's terminal busy even though at that point all the work is being done on our servers. i mean, they can always ctrl-c, and we could tell them it's safe to do that. we could return early and say "when your deploy is done, if it succeeds, it will be visible at this url". but maybe it's better to wait so that we can return an error code if it fails?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fil thinks it's correct as-is; if someone has scripted this, they can run yarn deploy to kick off a cloud build, and they will receive a CliError code if the cloud deploy fails.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given a magic wand, would you want to stream the build logs here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mythmon Eventually, yeah!

src/deploy.ts Outdated
let deployTarget: DeployTargetInfo;
if (deployConfig.workspaceLogin && deployConfig.projectSlug) {
try {
const project = await this.apiClient.getProject({
workspaceLogin: deployConfig.workspaceLogin,
projectSlug: deployConfig.projectSlug
});
deployTarget = {create: false, workspace: project.owner, project};
const environment = await this.apiClient.getProjectEnvironment({id: project.id});
Copy link
Contributor Author

@tophtucker tophtucker Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at some point (maybe not in this PR) i want to combine getProjectEnvironment into getProject. it could at least be part of that api response; ideally we'd also fold the project_config table into new columns on the projects table.

(currently, getProjectEnvironment has slightly different permissions: it's only accessible by members of the project's workspace who can view the project. for a public project, we should not return those fields. but that would still be doable if it were one table and one endpoint; we'd just serve a redacted version in the public response.)

edit: fil points out that if we don't do it now, we'll have old framework projects in the wild and we'd have to continue supporting the getProjectEnvironment for a long time. so i guess i should do it now!

Copy link
Contributor Author

@tophtucker tophtucker Oct 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API side done in https://github.com/observablehq/observablehq/pull/18390; still gotta adopt it here

await this.markDeployUploaded(deployId);
private async maybeLinkGitHub(deployTarget: DeployTargetInfo): Promise<boolean> {
if (deployTarget.create) {
throw new Error("Incorrect deployTarget state");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fil says: update to something more user-readable

Suggested change
throw new Error("Incorrect deployTarget state");
throw new Error("Inconsistent deploy target state");

they still wouldn't know what to do with that but at least they could send it to us…

if (deployTarget.create) {
throw new Error("Incorrect deployTarget state");
}
if (!this.effects.isTty) return false;
Copy link
Contributor Author

@tophtucker tophtucker Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should move down to the "else" block below. we only need to check tty when we're depending on user interaction. if the build_environment_id and source are already set up just fine, then we should return true even if it's not an interactive terminal.

(fil notes: there's also process.stdout.isTTY, which says if we can write color codes and such.)

src/deploy.ts Outdated
Comment on lines 244 to 246
if (deployTarget.environment.build_environment_id && deployTarget.environment.source) {
// can do cloud build
return true;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fil notes: if we broke our local git configuration, shouldn't we check it? do we want to check if everything is correct? there's a good chance everything will work, but…

if you have erased your .git folder, or you have changed your remotes, or if you're not on the right branch…

should we check that local and remote git refs match?? the user is expecting a cloud deploy of the same application state they are looking at locally!! if they have local unstaged changes, or are on a different branch, they could get surprising results from doing a cloud build.

if observable cloud is configured to pull from the main branch, and you're on toph/onramp locally, what are you expecting to happen when you run yarn deploy?

and it'll get more confusing with branch previews. how does it work on vercel? on vercel, there's no way to trigger a deploy except to push, so there's no question here. maybe that should be our norm? except you still wanna be able to re-run data loaders, which depend on changing external state not captured by commits.

Copy link
Contributor Author

@tophtucker tophtucker Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe at a minimum we should just check that the local git repo still exists, still has that remote, and is on the same branch. (i.e. move all the tests in the "else" block up above the if/else.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also verify that the repo hasnt been unlinked on the web side

return true;
} else {
// We only support cloud builds from the root directory so this ignores this.deployOptions.config.root
const isGit = existsSync(".git");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: log cwd when running yarn deploy. you can run yarn deploy from a child directory like src, but i think it still runs in the context of the root directory, in which case this is correct.

// We only support cloud builds from the root directory so this ignores this.deployOptions.config.root
const isGit = existsSync(".git");
if (isGit) {
const remotes = (await promisify(exec)("git remote -v", {cwd: this.deployOptions.config.root})).stdout
Copy link
Contributor Author

@tophtucker tophtucker Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that in loader.ts we make our promises "by hand" (and with spawn instead of exec), but in create.ts we already use promisify, so… seems fine!

.split("\n")
.filter((d) => d)
.map((d) => d.split(/\s/g));
const gitHub = remotes.find(([, url]) => url.startsWith("https://github.com/"));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fil instead has, i think, the SSH case, which we should also check for:

❯ git remote -v
observable	git@github.com:observablehq/pangea.git (fetch)
observable	git@github.com:observablehq/pangea.git (push)
origin	git@github.com:Fil/pangea.git (fetch)
origin	git@github.com:Fil/pangea.git (push)

src/deploy.ts Outdated
.map((d) => d.split(/\s/g));
const gitHub = remotes.find(([, url]) => url.startsWith("https://github.com/"));
if (gitHub) {
const repoName = formatGitUrl(gitHub[1]);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatGitUrl will also be different in the SSH case

src/deploy.ts Outdated Show resolved Hide resolved
src/deploy.ts Outdated
if (authedRepo) {
// Set branch to current branch
const branch = (
await promisify(exec)("git rev-parse --abbrev-ref HEAD", {cwd: this.deployOptions.config.root})
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think cwd may be wrong here, it should be running in the normal cwd of the command, at the root, since we don't support non-root repos/projects

}
}
} else {
this.effects.clack.log.error("No GitHub remote found; cannot enable continuous deployment.");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should exit with a CliError rather than just logging a message and continuing on with a local deploy

Comment on lines +531 to +532
`Do you want to enable continuous deployment? ${faint(
"This builds in the cloud and redeploys whenever you push to this repository."
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dunstan says this could note upfront that continuous deployment requires a GitHub repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants