Recent Gitea 1.16 (stable) breaks build on Fedora 35 because the go.mod was changed to require Go1.17

[fhuberts]

Description

I can no longer build the stable branch 1.16 on my Fedora 35 system.

go mod tidy fails, see below.

+ make 'TAGS=bindata sqlite sqlite_unlock_notify pam' 'LDFLAGS=-X "code.gitea.io/gitea/modules/setting.AppWorkPath=/var/lib/gitea" -X "code.gitea.io/gitea/modules/setting.CustomPath=/var/lib/gitea/custom" -X "code.gitea.io/gitea/modules/setting.PIDFile=/run/gitea/gitea.pid" -X "code.gitea.io/gitea/modules/setting.CustomConf=/etc/gitea/app.ini"' vendor build
go mod tidy && go mod vendor
go mod tidy: go.mod file indicates go 1.17, but maximum supported version is 1.16
make: *** [Makefile:374: vendor] Error 1

IMHO The stable branch should not break existing working setups.

Gitea Version

1.16

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

f9ea4ab

Operating System

Fedora 35 x64

How are you running Gitea?

not relevant

Database

PostgreSQL

[fhuberts]

Commit f9ea4ab breaks it

[nazunalika]

I'm also seeing this issue. I build gitea RPM's in copr and I attempted to patch it for Fedora 35 and RHEL 8 as both fail but without success. CentOS Stream 8 and Fedora 36 are unaffected.

I believe raising to minimum golang 1.17 wasn't the right move on the stable branch.

[wxiaoguang]

Golang doesn't think so, you see: https://go.dev/dl/ , you can only find go 1.17 and 1.18 now.

[fhuberts]

That is all fine and dandy, but you're breaking builds on the stable branch.
Please please revert back to go 1.16.

[wxiaoguang]

That is all fine and dandy, but you're breaking builds on the stable branch. Please please revert back to go 1.16.

I just tried to answer why, but I am not the decision maker, it needs the team to decide ......

[nazunalika]

Golang doesn't think so ... you can only find go 1.17 and 1.18 now

Right yes, I understand and I agree. I understand the importance of following upstream as close as possible. As a question to the rest of the gitea development team I have to ask though, 1.16 for gitea is the current stable branch, is it not? Seems odd to me to raise the minimum golang version in a stable branch rather than planning for it in a development capacity (eg for 1.17), which unfortunately breaks a lot of builds.

Reason I say this is a good chunk of stable distributions provide golang packages to their users and do not upgrade them right away (on top of using binaries from go.dev is a non-starter). For example, RHEL does large rebases every 6 months (RHEL 8.6 won't be released till May, which will have 1.17). Fedora, unless the maintainers have a reason, do not generally rebase versions in their current release. 1.18 for example is being built for Fedora 36 beta, and it is unlikely that it will come down to Fedora 35. There's no indication of this happening, at least from their maintainers.

I understand the development styles and plans are different from project to project in open source. It's these decisions though that just makes it difficult for users who are confined to what they have from their distribution if they wish to build gitea in a certain way (like having PAM support). And for some of us users, downloading directly from go.dev is typically a non-starter in some capacities.

Hopefully we can get some input and some information for myself and others who may be running into this issue.

[wxiaoguang]

Personally, I am neutral to such break (neither support nor objection). I understood how it matters and understand your points.

Why I don't treat it as not right because Gitea isn't a framework, and it doesn't promise to follow Linux distribution lifecycles. And Gitea can always be complied to binaries for all distributions (eg: use docker to compile), and you can always build it and run it for Fedora 35, in the end, there is no breaking.

And I agree that we can not say that it is not wrong because it indeed changes something for build, and it would cause problems for legacy system which uses shared build (I am not sure whether there is really such build, if yes, I believe the Gitea team should pay more attentions)

Take a real case for example, as a Debian user, Debian 11 only provides Golang1.15, does it mean that recent Gitea doesn't work for Debian? So my personal feeling is that since Gitea is an app, as long it as can be compiled and run correctly, then it could not be a serious break.

ps: to make the issue more clean, I edited the title to reflect the real topic, I think it needs to be decided by the maintainers and owners.

ps: ps: There is a drawback if the Gitea stable branch always uses the old Golang: it would never receive security patches from Golang, it might be a problem: https://go.dev/doc/devel/release#policy : "Each major Go release is supported until there are two newer major releases.", no more security fixes for old releases.

[42wim]

@fhuberts If you really want to build with 1.16, you can manually change the go.mod or run go mod tidy -go=1.16 (it works for now until some dependent library start requiring go 1.17 or go 1.18

But I agree with @wxiaoguang as Go security updates only supports the 2 most recent versions Gitea should always be on one of the latest major releases.

Fedora and Go have both 6 month release cycles and this seems to cause this sort of issues. It looks like F36 will have go 1.18 though :)

[fhuberts]

I understand all of the arguments.
However, you're making it much harder for gitea to be adopted by distributions when you do it like this.

You have only 1 stable branch and you're breaking things on it.
For distributions that is a huge NO NO, blocking adoption.

If you want to break stuff, you should create a new stable branch and break stuff there.
In this case, a 1.17 branch should have been created right before the breaking commit.

I really like gitea and am using it heavily, but these kinds of things just make it harder.
It also means that my build server has to run the bleeding edge all the time.
Worse even: I have to run Fedora rawhide in this case if I want to pick up the bugfixes on the stable branch!
And I'm running Fedora, let's not talk about if I'd run Debian...

I do hope you see the 'strangeness' of the situation...

[Gusted]

I agree with @fhuberts, the decision to backport the et al changes to update to go1.18 on the 1.16 branch is weird because it's a breaking change. So far I know we only backport bug fixes, but we don't backport breaking changes. Yet this was somehow a exception 🤷🏽.

The argument to only support two latest Go Release shouldn't matter to this branch, because anyone can build it with go1.16-go1.18 we should only put up that restriction for the master branch(v1.17), it's their "choice" to still have a outdated golang version. But restricting it on the stable branch is quite stupid, 1.16.5 has five security fixes and now some gitea Instances cannot update to include these security fixes because they have to figure out some way to bypass the distribution's golang version and get a newer one. There's no harm if we revert the change.

[wxiaoguang]

Then what if Go add security fixes in Golang 1.17 (but not Golang1.16)? Since Gitea 1.16 is the Gitea stable, should it take these Golang security fixes in or not?

(ps: I am not saying whether it is right or not, I am trying to get the question clear about how to follow the Golang security fixes in future, then we can have a guideline)

[fhuberts]

@wxiaoguang
Then you create a new stable branch and bump the required go version.

[wxiaoguang]

@wxiaoguang Then you create a new stable branch and bump the required go version.

How to? Now Gitea has two branches, one is 1.16.x, one is main(1.17-dev), what's the new stable branch do you mean?

[Gusted]

Then what if Go add security fixes in Golang 1.17 (but not Golang1.16)? Since Gitea 1.16 is the Gitea stable, should it take these Golang security fixes in or not?

See:

it's their "choice" to still have a outdated golang version

Like I said anyone with a newer go version can build the stable branch just fine. It's their risk to still have it(wherever they are forced to or not). But currently now they have to cherry-pick the security fixes, because they don't want to have a security risk of having a gitea instance public.

[fhuberts]

Yes, same situation if there is a new gcc version.

[wxiaoguang]

it's their "choice" to still have a outdated golang version

Like I said anyone with a newer go version can build the stable branch just fine. It's their risk to still have it(wherever they are forced to or not). But currently now they have to cherry-pick the security fixes, because they don't want to have a security risk of having a gitea instance public.

But do you want to release an official Gitea 1.16 with the (possible) unsecure Golang 1.16 if the Gitea branch 1.16 doesn't get updated to the active Golang?

Yes, same situation if there is a new gcc version.

Any reference? GCC is not the same as Golang, GCC has a lot of backports for old versions, it makes these old GCC run without security problems. But Golang ONLY supports recent 2 releases, that's the key problem.

[Gusted]

But do you want to release an official Gitea 1.16 with the (possible) unsecure Golang 1.16 if the Gitea branch 1.16 doesn't get updated?

What do you mean with the unsecure Golang 1.16? Do you mean the static builds that we provide or the ability to compile Gitea with 1.16?

Because for the former, I'm pretty sure the build server is running the latest go version fine. For the latter, We shouldn't be the one to be in control "Hey! You have a outdated version, which can be a security risk!!! Please update before we allow you to compile Gitea" at the maximum we should provide a warning for it at runtime, but not disallow the the compilation for it. Own risk and do such breaking changes on major version updates(e.g. now we have done with 1.17).

[wxiaoguang]

Let's take a look at the problem behind the issue:

1. Golang only supports recent 2 releases, now is Go1.17 and Go1.18. Then Go1.16 won't receive any security fixes.
2. If there is a serious security problem fixed in Go1.17 and Go1.18, the problem is still in Go1.16.
3. If stable Gitea1.16 keeps using Go1.16, how can the security problem get fixed? Or can we accept that leave the security problem in Gitea1.16 until next release?

If this answer can be answered, I think we can make an agreement about how to handle such problems in future.

[Gusted]

1. Golang only supports recent 2 releases, now is Go1.17 and Go1.18. Then Go1.16 won't receive any security fixes.

Yes! So it's indeed weird from distributions to not update go, especially because go is backwards-compatible.

2. If there is a serious security problem fixed in Go1.17 and Go1.18, the problem is still in Go1.16.

Yup, the official go team won't provide patches. Not sure if certain distributions provide security patches for their binaries to still include these.

3. If stable Gitea1.16 keeps using Go1.16, how can the security problem get fixed?

Well, if a user is compiling Gitea with go1.16(not the other version) we cannot do much, our option is to disallow compilation(current behavior) or to warn them at runtime(Not sure since when we're security notification server).

3. Or can we accept that leave the security problem in Gitea1.16 until next release?

I think at this point we're taking the security too serious, the user has decided to still use go1.16 so that means that every other binary is also vulnerable to open security issues. We wouldn't be the only project that is vulnerable(if we leave out the attack vector of such security issue). If we really want to do something we should instead opt-in for such warning at runtime... So users can take their time until Gitea 1.17 is released to have their own go version updated to a newer one but still experience the Gitea bug and security fixes.

[wxiaoguang]

That statement is about users, not about the official Gitea. If I understand correctly, your propose is that at the moment, even if there are some security problems which would never be fixed in Go1.16 (because of EOL), then still make official Gitea 1.16 (stable) use Go1.16 (EOL)?

Personally, my answer is:

• My preference is to make Gitea stable follow the active Golang releases
• If most agree not to follow and keep using the EOL Golang, I am fine, too.

Need more people to vote @go-gitea/owners @go-gitea/maintainers

[Gusted]

then still make official Gitea 1.16 (stable) use Go1.16 (EOL)?

Are you talking about static builds or users building Gitea?

Because Like I said, if user's go version is 1.17 or 1.18 Go simply build with those versions, they don't magically go to some compatibility mode to build Gitea. We're only targetting users that still have go1.16 installed any other users with 1.17 or 1.18 won't notice anything.

• My preference is to make Gitea stable follow the active Golang releases

My preference still would be to not break User experience on Gitea stable and let them miss out on Gitea security updates.

[guillep2k]

It's been a long time since I've participated in this project, so please take my words with a grain of salt.

As I understand, the version bump was done in order to use new syntax for the tooling, but the actual build could still be compatible with go1.16 (except for the fact that go.mod asks specifically for go1.17).

If my assumption is correct (I'm not sure), then perhaps we could enable the go version configuration/override via environment variables or some Makefile syntax, so:

1. Older distributions could override the version to go1.16 and be happy.
2. We could use the latest or most convenient go version for our distribution binaries.
3. We could use the latest or most convenient go version for our CI/CD chain.

So, it could all be reduced to whether my assumption is correct.

[lunny]

I think for official Gitea binary, we should keep it security ASAP.
For those want(have to) compile Gitea themselves, we should allow them to do that. But we can detect go version and have a warning when users compile Gitea with a lower version of Go. The change could be in Makefile.

[tboerger]

Raising the required go version is a breaking change, this requires at least a new minor version, something like that should NEVER be done within a patch release.

[lunny]

OK. If nobody against that, let's change go.mod to go 1.16 in Gitea v1.16.6 but add a warning in Makefile if go version less than 1.17.

[zeripath]

Honestly I think @fhuberts is right here. I should have stopped the change to use go 1.17 in Gitea 1.16.5. However, I think Gitea 1.17 should be released with a minimum go 1.18.

[6543]

Gitea should alwasy use the latest stable golang for there build chain (CI, ...)
But we should allow older golang builds

For the specific backport, I agreed on merging since it fixes bugs

[tboerger]

Maybe add a simple build check to the ci system for older go versions, that would avoid such issues in the future

[zeripath]

We already have this:

gitea/.drone.yml

Line 112 in be9ef15

image: golang:1.17 # this step is kept as the lowest version of golang that we support

[wxiaoguang]

If I understand correctly, the conclusion is:

1. Change the go version in current Gitea stable (1.16) go.mod to go 1.16
2. For Gitea stable branch, the go version in go.mod should never be changed, let users take their own security risk to build with a unsupported Golang, even if they are building their own official Gitea packages for their end users.
3. Gitea official releases are always built with active/supported Golang.
4. If a Gitea stable is using an unsupported Golang, and a dependency package need a supported Golang to fix a security problem, then Gitea won't update the required Golang version. For example:
   • Gitea stable uses a unsupported Golang, and depends on PackageA
   • PackageA has a security problem, and the new version of PakcageA with security fix only supports a active/supported Golang
   • Gitea stable won't use the new version of PackageA because Gitea stable must support the old Golang to compile old PackageA
   • This assumption is that such case should seldom occur because hopefully all vendor packages should be safe and won't break build.
5. For all Gitea dev->stable release, make Gitea use the latest Golang, to make sure the used Golang has the longest support time.

[silverwind]

As I understand, the version bump was done in order to use new syntax for the tooling, but the actual build could still be compatible with go1.16 (except for the fact that go.mod asks specifically for go1.17).

Correct, the only part dictating 1.17 is the go run syntax in the Makefile, everything else, including the build should work on 1.16.

[techknowlogick]

Hey, sorry for all the trouble this has caused. This is entirely on me, I was rush in fixing go1.18 issues and trying to release a security release at the same time that I removed support for go1.16. When precedence states, as with all other releases we don't bump minimum go version for a release branch (although we may start building it with newer versions, but still leaving that older versions of go for people to build themselves). Hopefully the linked PR can be applied as a patch to the fedora release until we can cut another 1.16.x release.

[fhuberts]

Thanks for listening and fixing this issue!