Docs for v5

[countvajhula]

Summary of Changes

A few more documentation updates, including:

• crosslink in relevant places to the qi/list module
• document our release practices for users, with guidance on getting different kinds of behavior that we might want

Public Domain Dedication

• In contributing, I relinquish any copyright claims on my contribution and freely release it into the public domain in the simple hope that it will provide value.

(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)

[benknoble]

Just registering a few questions/opinions.

The bits about CD and "versioning backends" are higher priority to me, but I would not consider any of this blocking.

[benknoble]

True—but how often do we push to main? If I think back to major releases, we have normally used long-lived integration branches (which is typically contra-CD). In other words, we have both some CD (any time we push/merge to main) and some non-CD (long-lived integration branches).

Maybe this is selective memory: I don't recall very many merges to main outside of release branches in recent times.

So perhaps we should call this aspirational? Any chance we can break up the long-lived integration into something that can be merged right now without breaking anything is a win in my book.

[countvajhula]

That's a good point. I guess the difference is one of intention. In cases where our intention wasn't clear but we needed to iterate on a set of changes (which would help reveal the right design later), we opted to use an integration branch. But in cases where we have a good idea of what we're trying to do (even if it may later turn out that this was a misstep), we would merge on the main branch. That is, in principle we would merge changes on the main branch where the intention is clear even if it's already known to be backwards-incompatible. We don't want to use the main branch for experimentation where we aren't already sure of what we're trying to do. We would use an integration branch for that.

So perhaps we should call this aspirational? Any chance we can break up the long-lived integration into something that can be merged right now without breaking anything is a win in my book.

Yes, you're right. re: aspirational, see my other comment below.

[benknoble]

Arguably we should be able to do some experiments on the main branch: if the experiment doesn't break things (or can be hidden away in a module that isn't "public", or can be turned off via feature flag, or some such), it can be merged to ease the development burden (smaller PRs, smaller commits, etc.). That's more typical of true CI/CD in my experience.

But it's also not worth arguing about ;) I'll leave this unresolved for future readers, but I'm satisfied for now.

[benknoble]

Obviously this is a bit controversial within the Racket community—I wonder if it's worth summarizing these points early, something like:

Qi breaks with traditional Racket package development and occasionally ships breaking changes in a SemVer-style major release. For that reason, we recommend pinning to a Qi tag using a Git package source¹ rather than following the main "qi" Racket package source. Keep reading for details and examples.

Footnotes

1. Don't forget to @tech{} ↩

[countvajhula]

That's a great way of putting it. I'll adopt your suggestion.

FTR, I think what we are really trying to do is something different from semver -- and I suspect we are trying to live in a Denxi world where multiple versions of the "same" library can coexist on the same system, and where solving "backwards compatibility" is not a problem at all, and "dependency hell" is just precluded entirely. Why even bother trying to reconcile dependency "versions" across packages when we can just treat them as what they really are: distinct collections of code (yet, without duplicating what's common, like Git's blobs)? It seems that dependency versioning problems all stem from using one name to refer to two different things. Thinking of you, @zyrolasting.

But while we live in Raco land, it does feel as though following semantic versioning (I like "semver-style," to give some hint that we adopt it reluctantly and hope to find better alternatives!) is a reasonable compromise. We haven't strictly been doing that thus far, but we can aim to do that more assiduously going forward.

[benknoble]

That's a great way of putting it. I'll adopt your suggestion.

Glad you liked it :)

FTR, I think what we are really trying to do is something different from semver -- and I suspect we are trying to live in a Denxi world where multiple versions of the "same" library can coexist on the same system, and where solving "backwards compatibility" is not a problem at all, and "dependency hell" is just precluded entirely. Why even bother trying to reconcile dependency "versions" across packages when we can just treat them as what they really are: distinct collections of code (yet, without duplicating what's common, like Git's blobs)? It seems that dependency versioning problems all stem from using one name to refer to two different things. Thinking of you, @zyrolasting.

Denxi seemed to be concerned with how to constrain/execute/interpret the meaning of/etc. those multiple copies. Here I think we are just thinking about how to have Qi, Qi2, and Qi3 live together—there's a reason Racket punts on that in the package manager and says "have separate packages" 😅 getting it right as a package manager is hard. Just look at how cargo, npm, and yarn do things for some examples.

I don't think having multiple Git versions available really makes it feasible to install 2 separate versions of Qi because of the immediate module conflicts.

But I may have also completely misunderstood—I now recall Sage saying something about layering or meshing code together, so maybe there's some "there" there.

But while we live in Raco land, it does feel as though following semantic versioning (I like "semver-style," to give some hint that we adopt it reluctantly and hope to find better alternatives!) is a reasonable compromise. We haven't strictly been doing that thus far, but we can aim to do that more assiduously going forward.

True enough. It is neat that Qi wants to support both raco's model and a SemVer-ish model.

[benknoble]

After all, any introduced bug is technically backwards-incompatible, as, indeed, is the fix for the bug!

This is at least somewhat controversial, and typically depends on the bug and its blast radius. I think different groups handle this differently, so it might be worth calling out that this is Qi's stance rather than a broad point.

In particular: at $DAYJOB we often rely on patch (or minor) versions to get bugfixes regularly in our dependencies.

[countvajhula]

Do you mean that you depend on something like x.y and then updating would automatically pull in any versions like x.y.z1 and x.y.z2 that may happen to have been recently posted? Or do you explicitly bump the dependency version from x.y.z1 to x.y.z2?

[benknoble]

The former rather than the latter (I personally try to avoid pinning to x.y.z, because then every time a CVE is announced we have to go update those pins; instead, we just seamless update to the newest z, possibly by re-locking dependencies in systems that do that).

Anyway, the text has moved and I didn't look to see if it still considers bug fixes backwards-incompatible changes.

[benknoble]

On the other hand, the branch strategy above supports these semantics to a sensible extent -- that of receiving necessary bug fixes, but not gratuitous "improvements" that may unwittingly break your application, even if they aren't intended to be backwards-incompatible.

I think that should be extend---that (em-dash, no spaces; my use of no spaces is traditional but has apparently become less normal).

Also, traditional SemVer is supposed to help you not get broken by such "gratuitous 'improvements'," so maybe the point there is that sometimes mistakes happen?

[countvajhula]

Yes, I feel that semver is trying to solve a problem (and the solution is leaky because of partial knowledge, mistakes, etc. -- and, e.g., I agree with @greghendershott when he points out some issues with it), but it seems to be much better to transcend the problem by weakening the scope of names (like "qi," as referring to a nebulous, changing collection of code, vs referring to something immutable which need not be the same as what another package calls "qi"). IIUC this is what Sage has been talking about for a long time.

But it's very possible that following something like semver may be the best compromise for the moment (while hopefully, at the same time, encouraging adoption of better alternatives -- it would be great to include a section here on using Qi with Denxi, Sage, if that's possible today).

P.S. I'll review -- vs --- use and I confess I don't know when to use which. Thanks for the reminder!

[benknoble]

Yes, I feel that semver is trying to solve a problem (and the solution is leaky because of partial knowledge, mistakes, etc. -- and, e.g., I agree with @greghendershott when he points out some issues with it),

Sure—SemVer has its issues, but it's better than not knowing what any version number change meant in the pre-SemVer days. (I still read release notes, so it's never pure speculation; the kind of bump just tells me what to pay extra attention to.)

Some kind of "manifest" that advertises what a version provides could be neat—something I thought of while reading Greg's post. But there's a lot to build to support stuff like that even within a single language ecosystem (see "ambitious" below).

but it seems to be much better to transcend the problem by weakening the scope of names (like "qi," as referring to a nebulous, changing collection of code, vs referring to something immutable which need not be the same as what another package calls "qi"). IIUC this is what Sage has been talking about for a long time.

I only vaguely follow "cutting the ties" here from what I recall of Denxi, but it does seem ambitious for Qi to take these steps alone—especially in that we want Qi to remain relevant to Racketeers, or we'll have lost our (small!) audience.

But it's very possible that following something like semver may be the best compromise for the moment (while hopefully, at the same time, encouraging adoption of better alternatives -- it would be great to include a section here on using Qi with Denxi, Sage, if that's possible today).

+1

P.S. I'll review -- vs --- use and I confess I don't know when to use which. Thanks for the reminder!

https://practicaltypography.com/hyphens-and-dashes.html ;)

[countvajhula]

I've incorporated your suggestions @benknoble , and also made the wording less controversial. It's still not perfect but definitely an improvement.

[greghendershott]

I think Jay McCarthy is the first person I heard talk about versioning as primarily a social issue best solved socially not technologically. That idea really surprised me at first.

The biggest issue is, do you want other people/software to use/depend on you? If you don't, because it's still alpha/beta WIP, that's simple. Just make sure people know that, and they can proceed accordingly. Probably they can't/won't use your stuff in packages that they want to offer as stable and dependable. Transitivity.

If someday/maybe you do want something to be used by other people, in software they in turn offer to others as stable... then you're kind of entering into a bargain. The exact nature of the bargain, is up to you and them. There are different ways to handle it.

Anyway I think there are two main kinds of changes to consider:

• Backward compatible or not ("breaking" changes). You probably can't tell people to use your stuff, then keep breaking their use of it. You probably need to keep it available indefinitely. You can say that some subset won't improve, anymore, but you can't say it will disappear. It's fine to advise people to depend on a git commit/tag, if that's how you want to do it.

• New things. In a lang like Racket, I'd suggest just bumping the plain old version number for people's deps and raco pkg update to refresh. But just for that. Beyond that, they can do things like try to dynamic-require your new function, and proceed accordingly depending on whether it's available. (I do this all the time in Racket Mode, to optionally use things in newer versions of Racket.)

Personally, I feel that SemVer (or not) is beside the point. In chaotic package ecosystems with lots of bad actors, SemVer is probably a way for package users to feel some control or stability. Horse left barn long ago; cope as best you can. But of course it's trying to define rules for what should be basic human courtesy (the social dimension), which is something we programmers tend to do. 😄

[benknoble]

s/outside of a production deployment/outside of a major version release

?

[benknoble]

I think Jay McCarthy is the first person I heard talk about versioning as primarily a social issue best solved socially not technologically. That idea really surprised me at first.

It might have been Greg and Sid that first exposed me to one of my new mantras¹: technology doesn't solve social problems.

Which is a really shorthand way of saying what Greg spelled out, for lots of different cases, and has a lot of nuance—some people think I mean that better food distribution systems wouldn't alleviate food insecurity, for example, when I really mean that food insecurity should be tackled primarily on the social side, possible using technology as a tool in the solution. The curse of shorthands is nuance lost, as anyone who reads or writes about "best practices" should know ;)

Footnotes

1. This was several years ago, and I have a vague memory of seeing it on the Racket Discourse, but I don't recall and I didn't dig for it. ↩

[zyrolasting]

It seems that dependency versioning problems all stem from using one name to refer to two different things. Thinking of you, @zyrolasting. -- @countvajhula

I now recall Sage saying something about layering or meshing code together, so maybe there's some "there" there. -- @benknoble

Denxi's input overriding example seems relevant here. The words "input", "output", "package", and "transaction" use Denxi's definitions, but it is true that Denxi resolves name conflicts.

it would be great to include a section here on using Qi with Denxi, Sage, if that's possible today -- @countvajhula

You know how Ruby has rvm and Node.js has nvm? These programs protect the user from versioning details. Say Qi gets a qvm. You have to ask the user to install a thing to install a thing, but they might prefer to do that than understand the underlying versioning rules.

A version manager must be stable, which works nicely with Racket package catalogs. So you'd say raco pkg install qvm , then (maybe) raco qvm install <version> <collection> to bind a set of collection paths to a set of modules known to work together. <version> can be whatever you want, since your source does not have to be Racket's package catalog at that point (EDIT: <collection> likely defaults to qi or qi2 depending if your <version> scheme cares to make that distinction). If you use Denxi to do this, then qvm might be a Denxi launcher. One would use more of Denxi when you want to use the same <collection> for different values of <version>.

Does this fit what's being discussed?