cmd/go: add *_unix.go recognition (and presumably unix build tag)

[robpike]

It does understand *_windows.go, and I was surprised to (re)learn that it does not support *_unix.go. Would it do so, things would be clearer and all the *_unix.go files that already exist wouldn't need to maintain their

// +build darwin dragonfly freebsd linux netbsd openbsd solaris

tags as the list grows.

[bradfitz]

More concretely, does this mean "unix" would mean anything in a pre-defined list of GOOS values? Would we proactively add unsupported GOOS values that might be supported later, like we did with GOARCH values, or would "unix" mean anything that's not windows or plan9 or nacl? Wait, is nacl unix?

Also, should we define "posix"? Currently we have some _posix.go files which include windows.

I don't object to this, but I'm just curious which definition we'd use.

[robpike]

All good questions for which I don't know the answer.

My main concern is maintaining the long list of os names in the build tags. Making things easier would be an advantage too.

Discussion needed.

[j7b]

I'd suggest the focus should be on user code and the standard library, and a new build tag is required (perhaps "unix"). The standard library has clear manifestations of a notion of "unix" in packages intended for general use (most obviously in net), and I'd suggest any GOOS that implements these should satisfy the build constraint word "unix." For example, if nacl's net package supports network types "unix", "unixgram" and "unixpacket" it would satisfy some of the predicates the "unix" build constraint should require.

The *_unix.go convention could implicitly define +build unix.

[robpike]

That's pretty much what I mean.

[dominikh]

Related: #6325

[jimmyfrasche]

Looking at the list of OSes, the BSD's may be the most homogeneous and largest sublist. But do you count Solaris, Darwin? Even if not, just defining bsd as dragonfly freebsd netbsd openbsd seems handy regardless of any other tags that might be included. If Darwin and Solaris are included, the build tag in the OP shrinks down to // +build linux bsd

Looking at the feature sets, posix may be the simplest target, but which subset of posix? Is it just the shared subset of all the unix-like OSes currently supported? When a new posix-y OS is added does that mean it can't be added to posix if it's missing one thing the other posix OSes support or do the set of guaranteed features lose a member? Since posix is so vast and as far as I know in practice little agreed upon, that seems the most problematic.

Looking at the stdlib, there seems to be very little agreement in practice among which OSes to include, even in *_unix.go and *_posix.go files. Of course, that doesn't preclude having build tags like // +build unix,!freebsd when needed.

Looking to the future, if the list is not proactive, as Brad put it, that means a new OS could suddenly match with a new version of Go when it should not. That may be worrisome but then again currently we have the situation where a new GOOS doesn't match anything without manual intervention. Of course, even a future proofed list is only as good as targets currently known and can't contain as yet uninvented OSes, and then the same issues arise as with posix.

[j7b]

From a user code perspective it shouldn't matter if the "unix" is bsd, lunix, solaris or other. I'd consider mandating cgo for test purposes for "unix" candidates and having a cgo-centric test suite to make sure all the unix-type-things for a given Go toolchain aren't at odds with the C toolchain as known to the Go toolchain for a given GOOS.

[rsc]

This is basically a straight duplicate of #6325, as @dominikh points out. The problem here, as there, is that "unix" changes based on various details - sometimes it includes Linux, sometimes not, for example. It's not clear we want to be in the business of defining it exactly one way. Also, this should not affect users terribly much, given that we try to provide OS-independent APIs to the largest extent possible.

[robpike]

You're skirting the point here, which is just to avoid maintaining the huge list of Unix variants Go supports. It's to discriminate the basic difference between Windows and Unix. Windows has many versions, but there is only one predefined "windows" tag, in the sense the foo_windows.go works. I'm asking for the same courtesy for Unix.

[jimmyfrasche]

Ignoring ios and android for the moment, as they're essentially defined to be "variants" of darwin and linux, and only looking at the GOOS build tags, the complete list, as of 1.8, is:

windows plan9 nacl linux solaris darwin dragonfly openbsd netbsd freebsd

The goal, as I understand it, is to be able to use *_GOOS.go as much as possible, by defining one or more "super-GOOS" tags that match a large subset of the above tags.

I'm going to ignore the implications of defining unix or posix and just look at what's in the stdlib so there's some data. (Apologies if I made any mistakes).

Looking at the *_posix.go files in the $GOROOT/src† the definition of posix uses seems to be almost equivalent to !plan9, with occasion to additionally specify !nacl (3/15 of the files) and once adding !solaris.

(Windows is not what most people think of when they thing posix. An alternative would be to allow something like x_not-T.go to be equivalent to a // +build !T directive and rename most of those as _not-plan9.go.)

† using

find -type f -name "*_posix.go" -exec grep "^// [+]build " {} \; | cut -d" " -f3- | sort

For the *_unix.go files,I took the list of tags, removed two that, despite the name, only had to do with cgo, took out all arches, android build tags, etc., manually expanded the negations, reordered them so all the tags were in the same order and then counted the occurrences of each unique subset of GOOSes:

18: darwin dragonfly freebsd linux nacl netbsd openbsd solaris
12: darwin dragonfly freebsd linux netbsd openbsd solaris
3: darwin dragonfly freebsd linux nacl netbsd openbsd plan9 solaris
2: darwin dragonfly freebsd linux netbsd openbsd
2: darwin dragonfly freebsd linux nacl netbsd openbsd solaris windows
1: dragonfly freebsd linux nacl netbsd openbsd solaris
1: dragonfly freebsd linux netbsd openbsd
1: dragonfly linux netbsd openbsd solaris
1: freebsd linux netbsd

Looking at that it seems !windows,!plan9 would get the most existing *_unix.go files, fit most people's expected definition, but often require an additional // +build !nacl directive.

In practice how build tags are written seems to be somewhat ad hoc. Maybe gofmt could sort them in a standard way or golint warn if they're not alphabetical? Perhaps a new tool that deals solely with build constraints?

[jimmyfrasche]

Here's similar for every GOOS build tag :
https://gist.github.com/jimmyfrasche/24f45e37a35f02eab8dcd1fb2c1c6af2

[rsc]

It sounds like the real interesting use would be outside the standard library. Can someone survey what usages look like outside, like in the go corpus files?

[j7b]

Maybe the best expression of "unix" is !windows,!android,!plan9,!nacl as it's much easier to express why a GOOS is !unix and the build tag waters aren't muddied when user code has to pander to particular variants of things that aren't !unix.

[rsc]

On hold for someone to present evidence about the need for this outside the standard library.

[robpike]

Any code that needs to support variant operating systems would need this. For example, Upspin has

errors_unix.go:5:// +build darwin linux freebsd netbsd

Is that the right list? No. What is? I don't know, and the code will go out of date as new variants arise. The code in that file is Unix-generic, but what is Unix from Go's point of view?

It seems to me that if we can define golang.org/x/sys/unix then we can define build tag unix and _unix.go.

[vmarkovtsev]

I have gathered some stats on the GitHub dataset in BigQuery.

First I extracted all the "// +build" lines:

SELECT C.id as id, MIN(C.content) as content, MIN(F.path) as path, MIN(C.size) as size, MIN(F.repo_name) as repo
FROM `bigquery-public-data.github_repos.contents` as C
JOIN `bigquery-public-data.github_repos.files` as F
ON C.id = F.id AND F.path LIKE '%.go'
GROUP BY id

Query complete (251.7s elapsed, 2.26 TB processed)


SELECT line, COUNT(*) as n
FROM (
  SELECT SPLIT(content, '\n') as line
  FROM [eminent-airport-191509:github.go_content]
  HAVING LEFT(line, 9) = '// +build'
)
GROUP BY line
ORDER BY n DESC

Query complete (10.0s elapsed, 11.2 GB processed)

The raw result is attached as lines.csv.gz

Then I calculated the cooccurrences using a Neugram script
bake.txt and the result is attached as
result.csv.

Finally, I normalized the co-occurrence matrix by column (sum=1) and plotted it using
a Python script:

How to interpret: OS Y -> OS X cell is whiter if X occurs with Y more often than with the rest of the column. E.g. Plan9 is mentioned together with Windows more often than with anything else.
We can see that BSDs, Linux and Solaris tend to go together.

[trex58]

There is a new GOOS value to take into account now: aix .
We have ported GCC Go on AIX 7 and we are working now on porting golang.
Nearly all changes dealing with AIX and GCC Go have been submitted and merged.
We are now working on golang.org/x and it appears that it contains a lot of OS cases. Moreover, colleagues who are porting Go applications on AIX are facing many cases where they have to manage the OS (adding aix) though the code is far from the Operating System.

About golang.org/x, here are some recent (2018/05/30) data about how it depends on "linux", goos, and goarch . That provides an idea about the amount of work required for adding a new operating system or a new architecture.

Number of times the expression: '// +build ...linux...' appears in the sources of each package:
arch: 0
benchmarks: 3
blog: 0
build: 3
crypto: 8
exp: 14
image: 0
mobile: 34
net: 54
oauth2: 0
perf: 0
sync: 0
sys: 94
talks: 0
text: 0
tools: 8
vgo: 5

Number of times the key-word: 'linux' appears in the sources of each package:
arch: 1
benchmarks: 3
blog: 5
build: 488
crypto: 9
exp: 18
image: 0
mobile: 57
net: 153
oauth2: 0
perf: 12
sync: 0
sys: 283
talks: 15
text: 0
tools: 46
vgo: 70

Number of times the key-word: 'goos' appears in the sources of each package:
arch: 0
benchmarks: 1
blog: 1
build: 35
crypto: 0
exp: 0
image: 0
mobile: 19
net: 0
oauth2: 0
perf: 10
sync: 0
sys: 8
talks: 0
text: 0
tools: 6
vgo: 27

Number of times the key-word: 'goarch' appears in the sources of each package:
arch: 0
benchmarks: 1
blog: 0
build: 37
crypto: 0
exp: 0
image: 0
mobile: 8
net: 0
oauth2: 0
perf: 10
sync: 0
sys: 5
talks: 0
text: 0
tools: 6
vgo: 28

[mortdeus]

dang I thought this was going to be a proposal to port Go to run on the old PDP unix made at bell labs. :(

Also, I explored thinking about this awhile back but how hard would it be to port Go to run on top of a virtual OS layer like inferno?

[rsc]

On Jun 5 2017 I wrote:

On hold for someone to present evidence about the need for this outside the standard library.

Issue #38364 and #37503 may be evidence that the standard library is need enough, because it would make it easier to maintain Unix-like ports outside the main Go tree.

Perhaps it would be enough to define unix = all the Unixes and then systems that want to use a unix tag can filter with things like

// +build unix
// +build !linux

(There's a proposal to make these better. Still not ready to post that yet.)

[rsc]

Let's move discussion to #51572.

[gopherbot]

Change https://go.dev/cl/426296 mentions this issue: cmd/go/internal/imports: recognize "unix" build tag

[gopherbot]

Change https://go.dev/cl/426814 mentions this issue: [release-branch.go1.19] cmd/go/internal/imports: recognize "unix" build tag

[gopherbot]

Change https://go.dev/cl/437235 mentions this issue: all: use "unix" build tag where appropriate