RFC: Specification for the version management

[hansl]

Requesting comments on this proposal. This is coming from a week of brainstorming and whiteboarding with Paul, Diego and myself.

Essentially this would allow us to manage versions independently from project to project, create a project with always the latest version, allow users to start new projects without an internet connection, let user upgrade at their own pace (even if it means their test net is versions behind the main net), and let us run integration tests on any version necessary (even custom SHAs). This cover all use cases I could think of, so if you think of something else let me know so I can integrate it.

This does not require a significant amount of code from our part. As a matter of fact the mocked CLI implement some of this proposal already (bundling and checking versions).

The complex part will be the migration scripts, but I would like to defer that work to after the 1.0 release.

[nomeata]

Not sure if we are at the right time to discuss this, but secure software upgrade is like crypto: Dangerous if you just roll your own solution. Just too many corner cases to worry about.

Do you know about https://theupdateframework.github.io/?

[hansl]

Thanks for the feedback @nomeata. I might not have made it clear here, but no binary is actually updated. The global dfx is essentially a version manager in this case; being able to switch and download different versions. In that sense, it’s closer to (e.g.) rustup. Does that make it clearer? I don’t think we can use the link you posted considering this.

I’ll try to update my document to clarify this stance.

[ghost]

A few questions:

• Why would developers want more than one version of dfx installed?
• What happens when there are breaking changes in the DFINITY client? How does dfx know about them?
• What is a user still allowed to do locally on a deprecated/unsupported version of dfx versus what (e.g. deploy) do we prevent them from doing and how?

These all orbit the general problem of the DFINITY client constantly changing and individual nodes potentially running different versions of it.

[ghost]

Why would developers want more than one version of dfx installed?

To be clear, I can think of a couple possibilities. I'm just wondering how common this will actually be vs having a single version of dfx that supports all currently stable versions of the DFINITY client.

[nomeata]

An upgrade command can be made available to ping our servers, download the latest release,
install it in the user-level cache, and update the project's dfx_version field (if in a project)
to the new version.

Unless I am completely mistaken. This sounds very much like what you propose has strong overlaps with the problem space that that TUF is solving: Querying some repository for available versions, and downloading them. This makes it a package manager (even if there is only one package, “the SDK” initially), and if we consider this infrastructure security critical (e.g. are worried about people getting security upgrades this way, and are worried about attackers trying to lure them to use a “wrong” repository, or trick them into not seeing an upgrade), then the problems that the TUF researchers have identified and solved apply.

(Maybe this particular PR is the wrong place to discuss how to implement the design securely, and all I want to achieve right now is to get this project onto the radar, so that we don’t needlessly reinvent the wheel (or worse, ignore certain treats, such as an attacker presenting a properly signed but stale version of the repository to the user).

[knl]

Shouldn't we let our clients decide when they want to upgrade? That is, provide dfx upgrade?

[hansl]

This only happens when the version isn't specified by the user, e.g. when in a project (the configuration file has the version).

[knl]

This might be a dangerous assumption, as we might need to change something in order to get things right. That is, "never needs to be updated" might fly, however, it would be prudent to want the ability for this global CLI to be "re-installed".

[hansl]

I'll loosen up the wording a bit. This is a case where "hopefully we won't need to".

[hansl]

@nomeata

Maybe this particular PR is the wrong place to discuss how to implement the design securely

I agree, unless those security concerns clash with this high-level proposal, I feel they are an implementation detail. This proposal is mainly to solve the version management issue from a high level design.

[hansl]

@stanleygjones

Why would developers want more than one version of dfx installed?

I think the most relevant case is that they aren't ready to test against the latest. One other case would be if projects/devs share a common test net that is still running on an older version.

For the DFINITY client this is most definitely not an issue; we have to provide backward compatibility for probably ever (or close to). For the SDK itself though it might mean a lot of little things like command/flag changes and the developer isn't ready to upgrade its CI/CD.

What happens when there are breaking changes in the DFINITY client? How does dfx know about them?

I think if there's a breaking change in the client we have bigger problems, but this basically ensure that a version of DFINITY you are using for a project will always have the same API. When you're ready to upgrade, run dfx upgrade and we can either try to migrate the user's code, or tell the user what to do in the cases we can't.

What is a user still allowed to do locally on a deprecated/unsupported version of dfx versus what (e.g. deploy) do we prevent them from doing and how?

Anything that affects a remote client (even if it's not mainnet) should query the version and make sure it's compatible somehow (this is handwaving as I don't know how this should be validated). I'll update the document for this special test.

[paulyoung]

I would like to give this the attention it deserves which means I probably won't take a proper look until tomorrow.

[hansl]

@paulyoung There's no rush. Most of the points here we discussed offline so it should feel familiar.

[ghost]

My concerns have been satisfied. Thanks.

[paulyoung]

Thanks for putting so much thought into this.

I left some comments and suggestions. I think some are more important than others.

[hansl]

@paulyoung Please take another look when you have time.

[paulyoung]

I made a couple of suggestions that I think are worth considering.

https://github.com/dfinity-lab/sdk/pull/30/files#diff-759bf5c10560dc8fe61d6924fbc361ccR9 uses "global binary cache" instead of "user-level cache". Should we do the same in this document?

[hansl]

It's in the home folder, so it won't be global. We could discuss whether it's worth having a global cache (and where that would be; /usr/share/dfinity, and what permissions to give?), if you want to discuss the idea. My opinion is that it's simpler to do it at the user level (home folder) and give 700 permissions, so I would rather change the code/comments to be technically correct.