Publish .deb packages to Cloudsmith #49

dketterer · 2020-10-30T19:57:11Z

Hi!

so the issue is with the Debian ppa on gemfury.

It provides no InRelease or Release file. This file must contain a origin field (and maybe some others) so that unattended-upgrades can reference this as a "source" to which the auto updates are applied.

In Detail:

The Debian Repository Format specifies that there should be a InRelease file and in this file there are among others the optional fields:

Origin
Label
Suite
Codename

Unattended-upgrades is configured in the /etc/apt/apt.conf.d/50unattended-upgrades file. This file has two relevant sections like this:

Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
};

Unattended-Upgrade::Origins-Pattern {
        "origin=Debian,codename=${distro_codename},label=Debian";
        "origin=Debian,codename=${distro_codename},label=Debian-Security";
};

In the first one its of the form "<origin>:<archive>" and in the second one origin, codename, label and archive can be used. I am not exactly sure but I think archive is the same as Suite in the InRelease file.

The info about unattended-upgrades is from this blog post since the original wiki entry is not so detailed.

So initially this is just the issue with the missing InRelease/Release file but I can see how this turns out to be a bigger thing that might require a move away from gemfury. I don't know how you handle this stuff, I am not a Debian packaging pro, I just stumbled over this stuff while trying to setup and understand unattended-upgrades.

I am curious hearing what you think about that.

Best
Daniel

The text was updated successfully, but these errors were encountered:

francislavoie · 2020-11-01T07:39:49Z

Thanks!

We're evaluating whether https://cloudsmith.io can serve us better, how exactly could I find out whether it supports what you need? I'm not sure what to look for.

dketterer · 2020-11-01T09:19:06Z

I looked at this cloudsmith doc page but I am not able to tell from this. I also tried to find a working example from cloudsmith but was unsuccessful.

Maybe have a look at the structure of the debian repo of Jitsi: and their installation instructions. This is how it should look like.

Especially this Release or InRelease (the signed version) files are what I am looking for. Bonus if you start signing your packages and distribute the gpg-key over a channel where a service like cloudsmith can not interfer, for example directly from caddyserver.com.

francislavoie · 2020-11-01T14:22:48Z

Alright, well we have our latest version uploaded to cloudsmith if you'd like to give it a shot.

https://cloudsmith.io/~caddy/repos/caddy/setup/#formats-deb

I can't figure out how to download the InRelease file from a browser, I'm getting 404s for all the URLs I've tried (many of them) so 🤷‍♂️, maybe you could figure out how to construct the URL properly.

I do see this when I apt update though, so I think it should have what you want:

Get:9 https://dl.cloudsmith.io/public/caddy/caddy/deb/ubuntu focal InRelease [6,441 B]

Re signing, see #42.

Cloudsmith automatically signs releases we upload, you can find the GPG key in the link above. They do give us the option to upload our own GPG key (@mholt does have one he typically uses for signing commits) but uploading the private key seems like a bad idea. We could make Github sign them maybe, but we had trouble with that in the past when we tried to enable it for https://github.com/goreleaser/goreleaser which is the tool we use to build everything (including the .deb that we upload) from configuration. Hopefully you're satisfied with trusting Cloudsmith, considering their whole business model hinges on being trustable for distribution 😅

lskillen · 2020-11-01T14:44:28Z

Folks:
https://dl.cloudsmith.io/public/caddy/caddy/deb/ubuntu/dists/focal/InRelease

Happy to help with questions over here! ;) You can contact support via Intercom at anytime too.

francislavoie · 2020-11-01T14:47:34Z

Thanks @lskillen I have no idea how I didn't hit that one, I tried all kinds of similar combinations 🙈

lskillen · 2020-11-01T15:17:35Z

Ha! It can be inscrutable unless you know what you're looking for. We have (short-term) plans to add a simple file listing index view for things like the debian (and other) indexes, but documenting helps. To break it down:

https://dl.cloudsmith.io/public/caddy/caddy/deb/ubuntu/dists/focal/InRelease

Parts defined by Cloudsmith:

https://dl.cloudsmith.io/ - Our download domain, of course, and always https-only; this can be customised.
public/ - The access token; this is a public repository, so no tokens are needed.
caddy/caddy/ - The account identifier caddy plus the repository name caddy; these disappear for custom domains.
deb/ - The packaging format; if this was RedHat, you'd see rpm here.

Parts defined by Debian:

ubuntu/ - The codename for the Debian-based distribution; for Ubuntu in this case, for Debian it is just debian.
dists/ - Always just "dists". Defined by Debian.
focal/ - The code for the distribution release; Ubuntu Focal in this case; for Debian Buster it is just buster.
InRelease - The Release file with a signature in it (thus "In" and "Release" -> InRelease).

From the Release file, you'll also see the metadata links that Debian uses to find the package indices. These are relative to the Release file. For example, getting the amd64 packages index, referenced as main/binary-amd64/Packages.

https://dl.cloudsmith.io/public/caddy/caddy/deb/ubuntu/dists/focal/main/binary-amd64/Packages

Finally, and a bit weirdly, the Filename referred in there, for actually downloading the package, is relative to the root just after the distribution; e.g.

https://dl.cloudsmith.io/public/caddy/caddy/deb/ubuntu/pool/any-distro/main/c/ca/caddy_2.2.1_linux_amd64.deb

We'll see about adding this additional detail to the documentation, since it's useful to know for exploring!

Reference: https://wiki.debian.org/DebianRepository/Format

Aside: If you'd like to control the keys, uploading your own private key is the way to go. It's what other companies distributing at-scale are doing, who either sell their products, or at least want to stamp their brand/name onto it. Typically they'll use both custom domains plus custom keys for distribution. It doesn't have to be the master key though, you can create subkeys for signing that are separately revokable from the master key: https://wiki.debian.org/Subkeys

dketterer · 2020-11-01T15:19:11Z

So those are the Properties right now:

Origin: cloudsmith/caddy/caddy
Codename: focal
Date: Sat, 31 Oct 2020 20:53:09 UTC
Architectures: amd64 arm64 armel armhf i386 ppc64el s390x
Components: main
Suite: focal
Label: source=none
Acquire-By-Hash: yes

Maybe you can set Origin to caddy and Label to something more meaningful like stable for the case that there might be a unstable release in the future.

For the installation instructions I would recommend using the manual instructions on the Caddy installation guide as it is bad practice to download some script and pipe it to sh. And also maybe combine it for ubuntu with a bash part that extracts the codename from /etc/os-release: codename=$(. /etc/os-release;echo $VERSION_CODENAME).

lskillen · 2020-11-01T15:26:18Z

Changing the Origin and Label would be a feature request, but do-able. ;-) At the moment these are auto-generated based on where the repository comes from (the origin), and how the repository is accessed in terms of authentication (label). So they're more-or-less static, but I can understand wanting to customise them. Just need to be aware that changing them after-the-fact breaks clients because the apt client stores them upon first access / retrieval of the source.

The way that most people deal with separate release channels is to use multiple repositories; e,g. instead of caddy/caddy, there would be caddy/stable and caddy/unstable (or caddy/release and caddy/beta, whatever makes the most sense semantically for you and your users). That way, you've got different channels, and they're automatically annotated into the Origin field.

You certainly can just the built-in key too. The private key is stored encrypted on our side, and it doesn't change unless you tell Cloudsmith to rotate the key (or upload your own one). Although we always keep the old keys for reference, so that we can continue to service packages signed with an older key. The latest key is always the one used for signing metadata.

francislavoie · 2020-11-01T15:42:19Z

That's great @lskillen, thanks for the rundown.

@lskillen We'll see about adding this additional detail to the documentation, since it's useful to know for exploring!

Yeah, I think having it in https://help.cloudsmith.io/docs/debian-repository would be great. It's where I looked first to try and figure out the structure.

@dketterer For the installation instructions I would recommend using the manual instructions on the Caddy installation guide as it is bad practice to download some script and pipe it to sh.

Yeah that's the plan, once we've got everything in place.

@lskillen The way that most people deal with separate release channels is to use multiple repositories; e,g. instead of caddy/caddy, there would be caddy/stable and caddy/unstable

Interesting, yeah we might do that. Right now, we skip uploading to gemfury on releases with a special tag (see https://github.com/caddyserver/caddy/blob/937ec342010894f7a6239883b10f6a107ff82c9f/.github/workflows/release.yml#L76) but we could upload to a separate repo for those pretty easily instead.

@lskillen It doesn't have to be the master key though, you can create subkeys for signing that are separately revokable from the master key: wiki.debian.org/Subkeys

Right, okay. FYI @mholt ☝️ I'll let you decide how you want to handle that.

francislavoie · 2020-11-02T04:12:56Z

Alright I just made these repos:

https://dl.cloudsmith.io/public/caddy/stable/deb/ubuntu/dists/focal/InRelease
~~https://dl.cloudsmith.io/public/caddy/unstable/deb/ubuntu/dists/focal/InRelease~~ see below

And I deleted the caddy one (because have those two makes more sense).

Might rename unstable to testing before we start advertising it. TBD.

dketterer · 2020-12-02T12:03:16Z

Sorry closed to fast, I keep this open until the install instructions are updated.

francislavoie · 2020-12-28T21:58:42Z

Okay I just renamed the other one to testing

https://dl.cloudsmith.io/public/caddy/testing/deb/ubuntu/dists/focal/InRelease

I hope to get everything sorted with @mholt's help before v2.3 is released (probably next week I think?)

francislavoie · 2020-12-28T22:40:16Z

@lskillen caddyserver/caddy#3941 FYI if you'd like to take a peek?

I opted to use the CLI tool directly rather than using the cloudsmith github action, because we have a bunch of files to upload and it seemed easier to do that way.

I've been setting any-distro/any-version on the cloudsmith web UI when uploading packages. Should I be adding --tags "any-distro/any-version" to the commands as well, or will it default to that?

Edit: Nevermind, I read https://help.cloudsmith.io/docs/supported-formats and I see that I need to do cloudsmith push deb caddy/xcaddy/any-distro/any-version $filename 👍

francislavoie · 2020-12-29T21:25:35Z

@lskillen it seems that uploading a custom GPG key requires a paid plan? We planned on using the free tier, as Caddy is an open source project.

Looking at https://cloudsmith.io/~caddy/repos/stable/settings/, I see:

Custom GPG Signing Key
Note: Custom signing keys require an upgraded plan that supports it.

lskillen · 2020-12-29T23:44:01Z

@francislavoie @mholt

Checked out caddyserver/caddy#3941 and it looks good to me. 👍 No issues with not using the GitHub action. We're (in the new year) going to release a "Setup CLI" action that preps an environment for you, but you're certainly free to roll your own.

For the custom GPG key, no sweat. It works because, by default, we offer Team-based plan features and free storage/bandwidth to all OSS projects. When a project wants/needs a feature from our Velocity+ range of features, we do that via (slightly more formal) sponsorship.

This is nothing crazy, and we don't make you sign anything; it just means we'd appreciate your help for promoting Cloudsmith in return for the free goodness. An example is a blog post, a more prominent linking back / endorsement, or occasional social media tweets; once you're happy! We're pretty flexible, and our marketing team is ultra-nice. :-)

You can read more about it here:
https://help.cloudsmith.io/docs/open-source-hosting-policy

No matter what, I've enabled the custom GPG key support for the Caddy project already, for you. We also treat everyone as if they're a paying customer too, regardless of how they use Cloudsmith.

mholt · 2020-12-30T03:15:47Z

@lskillen That's very generous -- thank you. We'd be happy to promote Cloudsmith as our preferred distributor for our Debian packages.

We'll get a badge added to the README as soon as we get a chance (kind of swamped catching up on things over the holidays, and prepping the 2.3 release which I hope will use Cloudsmith.)

francislavoie · 2020-12-30T03:34:59Z

Added the attribution in the README to my PR: https://github.com/caddyserver/caddy/pull/3941/files?short_path=b335630#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5

francislavoie · 2020-12-30T18:37:38Z

As a test run, we released the .deb for xcaddy v0.1.7, pushed to Cloudsmith via CI. Worked! (Took a couple re-runs cause we forgot a couple things but I digress)

https://github.com/caddyserver/xcaddy/runs/1627308051

https://cloudsmith.io/~caddy/repos/xcaddy/packages/

francislavoie · 2020-12-30T21:22:48Z

Alright, we got signed builds of Caddy https://cloudsmith.io/~caddy/repos/testing/packages/ (that's the testing repo, stable release coming in the next few days)

Only loose end to tie up is updating the docs to recommend using Cloudsmith instead of Gemfury.

francislavoie · 2020-12-30T21:39:56Z

@lskillen the install instructions we think we'll go with for https://caddyserver.com/docs/download

$ sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/cfg/gpg/gpg.155B6D79CA56EA34.key' | sudo apt-key add -
$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/cfg/setup/config.deb.txt?distro=ubuntu&codename=xenial' | sudo tee -a /etc/apt/sources.list.d/caddy-stable.list
$ sudo apt update
$ sudo apt install caddy

My concern is with the ?distro=ubuntu&codename=xenial bit. Since we're shipping a Go binary, we don't care what distro/codename it's installed on.

So I have a feature request; would it be possible to make https://dl.cloudsmith.io/public/caddy/stable/cfg/setup/config.deb.txt (with no query args) serve a list file with like any/any as the distro/codename (instead of a 404 as it currently does), and make that a functional repo?

Like I'd expect to see this in the list file:

# Source: Cloudsmith (support@cloudsmith.io)
# Repository: Caddy / stable
# Description: Fast, multi-platform web server with automatic HTTPS


deb https://dl.cloudsmith.io/public/caddy/stable/deb/any any main

deb-src https://dl.cloudsmith.io/public/caddy/stable/deb/any any main

It would be better for UX; users might be confused seeing ubuntu when they're installing on like raspbian or debian, etc.

lskillen · 2020-12-30T23:36:19Z

🚀

Ha! We were talking about that just yesterday with someone else. It partially works now, in that you can specify:

deb https://dl.cloudsmith.io/public/caddy/testing/deb/debian any-version main

Just that the distribution part, or debian in the example above, is still required; but maybe this ^^^ example is a bit less confusing because it mentions debian instead of ubuntu? So it'd instead be:

$ curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/cfg/setup/config.deb.txt?distro=debian&version=any-version' | sudo tee -a /etc/apt/sources.list.d/caddy-stable.list

We'll be looking to see if we can support using any-distro for client-side configuration shortly in the new year. Traditionally the any-distro and any-version support have been for simplifying server-side only, but I can see where it'd be useful to support it client-side too (i.e. this use-case). :-)

francislavoie · 2020-12-30T23:39:05Z

Cool, yeah that helps. Would be great if omitting the query would have that same behaviour 👍

lskillen · 2020-12-30T23:41:50Z

Not a bad idea! I'm sure there are caveats that I haven't thought through, so I'd need to check with the team. You'll probably not be too surprised to know that package management is filled with caveats and odd edge-case behaviour that makes handling things universally, a lot of "fun". 😂 Our job is smooth over those, though!

mholt · 2020-12-31T00:24:13Z

@lskillen Thanks for being so great to work with so far -- appreciate your help. Getting set up was a breeze.

lskillen · 2020-12-31T12:44:29Z

Glad to hear it, and you're very welcome! The team is very excited about having you on-board. Looking forward to helping you with package management for 2021 and beyond. 😁 If there's anything we can do to help, just shout here or Intercom (latter probably easier since you'll get access to the full team). Other than that, hope y'all have a Happy New Year!

francislavoie · 2021-01-01T02:30:38Z

Alright, this is done! Closing.

https://github.com/caddyserver/caddy/releases/tag/v2.3.0

https://caddyserver.com/docs/install#debian-ubuntu-raspbian

To switch from Gemfury, you just need to delete your old list file, then use the steps above:

$ sudo rm /etc/apt/sources.list.d/caddy-fury.list

dketterer · 2021-01-05T15:59:17Z

For anyone coming across this and searching for the correct setup in unattended-upgrades in Ubuntu 20.04:

/etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
        // ... other settings for the distro
        "cloudsmith/caddy/stable:any-version";
};

As Debian 10 has the Origins-Pattern section in this file per default, the solution there is:

Unattended-Upgrade::Origins-Pattern {
        "origin=cloudsmith/caddy/stable,archive=any-version";
};

francislavoie added the enhancement New feature or request label Nov 1, 2020

dketterer closed this as completed Dec 2, 2020

dketterer reopened this Dec 2, 2020

francislavoie mentioned this issue Dec 28, 2020

ci: Add pushing to cloudsmith caddyserver/caddy#3941

Merged

francislavoie changed the title ~~Fix Debian ppa to supports Unattended-upgrades~~ Publish .deb packages to Cloudsmith Dec 30, 2020

francislavoie mentioned this issue Dec 30, 2020

GPG signed releases - reconsider gemfury? #42

Closed

francislavoie closed this as completed Jan 1, 2021

francislavoie mentioned this issue Feb 16, 2022

Sign our own apks, deb and rpms goreleaser/goreleaser#2903

Closed

2 tasks

tucksaun mentioned this issue Apr 22, 2022

Sign (RPM) packages in the (YUM) repos symfony-cli/symfony-cli#63

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish .deb packages to Cloudsmith #49

Publish .deb packages to Cloudsmith #49

dketterer commented Oct 30, 2020

francislavoie commented Nov 1, 2020

dketterer commented Nov 1, 2020

francislavoie commented Nov 1, 2020 •

edited

Loading

lskillen commented Nov 1, 2020

francislavoie commented Nov 1, 2020

lskillen commented Nov 1, 2020 •

edited

Loading

dketterer commented Nov 1, 2020 •

edited

Loading

lskillen commented Nov 1, 2020 •

edited

Loading

francislavoie commented Nov 1, 2020

francislavoie commented Nov 2, 2020 •

edited

Loading

dketterer commented Dec 2, 2020

francislavoie commented Dec 28, 2020 •

edited

Loading

francislavoie commented Dec 28, 2020 •

edited

Loading

francislavoie commented Dec 29, 2020 •

edited

Loading

lskillen commented Dec 29, 2020 •

edited

Loading

mholt commented Dec 30, 2020

francislavoie commented Dec 30, 2020

francislavoie commented Dec 30, 2020

francislavoie commented Dec 30, 2020

francislavoie commented Dec 30, 2020 •

edited

Loading

lskillen commented Dec 30, 2020 •

edited

Loading

francislavoie commented Dec 30, 2020

lskillen commented Dec 30, 2020

mholt commented Dec 31, 2020

lskillen commented Dec 31, 2020

francislavoie commented Jan 1, 2021

dketterer commented Jan 5, 2021

Publish .deb packages to Cloudsmith #49

Publish .deb packages to Cloudsmith #49

Comments

dketterer commented Oct 30, 2020

francislavoie commented Nov 1, 2020

dketterer commented Nov 1, 2020

francislavoie commented Nov 1, 2020 • edited Loading

lskillen commented Nov 1, 2020

francislavoie commented Nov 1, 2020

lskillen commented Nov 1, 2020 • edited Loading

dketterer commented Nov 1, 2020 • edited Loading

lskillen commented Nov 1, 2020 • edited Loading

francislavoie commented Nov 1, 2020

francislavoie commented Nov 2, 2020 • edited Loading

dketterer commented Dec 2, 2020

francislavoie commented Dec 28, 2020 • edited Loading

francislavoie commented Dec 28, 2020 • edited Loading

francislavoie commented Dec 29, 2020 • edited Loading

lskillen commented Dec 29, 2020 • edited Loading

mholt commented Dec 30, 2020

francislavoie commented Dec 30, 2020

francislavoie commented Dec 30, 2020

francislavoie commented Dec 30, 2020

francislavoie commented Dec 30, 2020 • edited Loading

lskillen commented Dec 30, 2020 • edited Loading

francislavoie commented Dec 30, 2020

lskillen commented Dec 30, 2020

mholt commented Dec 31, 2020

lskillen commented Dec 31, 2020

francislavoie commented Jan 1, 2021

dketterer commented Jan 5, 2021

francislavoie commented Nov 1, 2020 •

edited

Loading

lskillen commented Nov 1, 2020 •

edited

Loading

dketterer commented Nov 1, 2020 •

edited

Loading

lskillen commented Nov 1, 2020 •

edited

Loading

francislavoie commented Nov 2, 2020 •

edited

Loading

francislavoie commented Dec 28, 2020 •

edited

Loading

francislavoie commented Dec 28, 2020 •

edited

Loading

francislavoie commented Dec 29, 2020 •

edited

Loading

lskillen commented Dec 29, 2020 •

edited

Loading

francislavoie commented Dec 30, 2020 •

edited

Loading

lskillen commented Dec 30, 2020 •

edited

Loading