Master fails to build with undefined: grpc.ClientConnInterface

[lazka]

On Ubuntu 22.04 and 24.04:

GO111MODULE=on go build -ldflags "-X github.com/etix/mirrorbits/core.VERSION=v0.5.1-82-gdb6fec7 -X github.com/etix/mirrorbits/core.BUILD=db6fec7-HEAD -X github.com/etix/mirrorbits/config.TEMPLATES_PATH=templates/" -o bin/mirrorbits .
# github.com/etix/mirrorbits/rpc
rpc/rpc.pb.go:1411:12: undefined: grpc.ClientConnInterface
rpc/rpc.pb.go:1415:16: undefined: grpc.SupportPackageIsVersion6
rpc/rpc.pb.go:1442:10: undefined: grpc.ClientConnInterface
rpc/rpc.pb.go:1445:27: undefined: grpc.ClientConnInterface
make: *** [Makefile:42: build] Error 1

commit db6fec71fa8bdfdb44007b61f355b4e5d8e2a564
Author: Arnaud Rebillout <arnaudr@kali.org>
Date:   Wed Sep 27 10:56:24 2023 +0700

    Add a CLI command to update the geolocation of a mirror (fix #96)
    
    Also provide a contrib script to make it easy to update the geolocation
    of all the mirrors in one go.

 cli/commands.go                  |  74 ++++++++++
 contrib/mirrorbits-geoupdate-all |  99 +++++++++++++
 rpc/rpc.go                       |  75 ++++++++++
 rpc/rpc.pb.go                    | 306 +++++++++++++++++++++++++--------------
 rpc/rpc.proto                    |   9 +-
 5 files changed, 457 insertions(+), 106 deletions(-)
 create mode 100755 contrib/mirrorbits-geoupdate-all

db6fec7

@elboulangero do you have any ideas?

[elboulangero]

Probably an issue with the generated file rpc/rpc.pb.go. I'd try to rebuild it:

rm rpc/rpc.pb.go
make rpc/rpc.pb.go

[elboulangero]

In my opinion generated files shouldn't be committed to the repo. But I'm not a Go developer, I don't know what are common practice in the Go community. But clearly it would avoid this kind of issue.

There are different versions of the tool to generate the code, and if I'm not mistaken, 1.3 and 1.5 are not compatible.

$ apt search protoc-gen-go

I believe I generated the code with the old 1.3, but I'm not 100% sure.

[lazka]

The rebuild didn't work here and leads to more errors, likely because I have an incompatible version.

Also not a go-developer, but given that the generation tools have to match the library versions suggests to me that committing the generated files is preferred.

[elboulangero]

How I build the package for a Debian bookworm system can be seen under https://salsa.debian.org/debian/mirrorbits/-/tree/debian/latest/debian?ref_type=heads.

Two things that really matter are:

• the list of build depends in the control file, that includes protoc-gen-go-1-3
• the line to rebuild the generated code in the rules file:

protoc -I rpc --go_out=plugins=grpc,Mgoogle/protobuf/empty.proto=github.com/golang/protobuf/ptypes/empty,Mgoogle/protobuf/timestamp.proto=github.com/golang/protobuf/ptypes/timestamp:rpc rpc/rpc.proto

This line is rather convoluted, but it was used as such to fix build failures with libprotobuf-dev 3.21.9 (details at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026674).

Can you try to rebuild the rpc.pb.go with the same command (and making sure it's protoc-gen-go-1-3 that's installed on your system and not protoc-gen-go-1-5?

[lazka]

Your command generates the same file as is currently in git here.

[elboulangero]

Alternatively, if you find yourself trying to build with protoc-gen-go-1-5, I have this patch floating around, from the time I tried to do that:

From: Arnaud Rebillout <arnaudr@kali.org>
Date: Mon, 11 Jul 2022 14:59:38 +0200
Subject: Add Go import path to .proto file
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This is required for golang-goprotobuf-dev 1.5.2-1 (currently in Debian
experimental), otherwise protoc fails with:

    protoc -I rpc --go_out=plugins=grpc:rpc rpc/rpc.proto
    protoc-gen-go: unable to determine Go import path for "rpc.proto"

    Please specify either:
    . a "go_package" option in the .proto source file, or
    . a "M" argument on the command line.

    See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information.

    --go_out: protoc-gen-go: Plugin failed with status code 1.

The fancy value '.;rpc' comes from:
https://github.com/golang/protobuf/issues/1102#issuecomment-619240905

Actually the option should specify the full Go path, however if I do that,
protoc generates the file rpc.pb.go in a completely different location, and the
value of 'package' in rpc.pb.go becomes a full path, rather than just "rpc".

I'm aiming at minimal changes here (because of my poor Go skills), so I prefer
to use this trick in order to keep the generated file unchanged, as much as
possible.

Forwarded: not-needed, Debian-specific
---
 rpc/rpc.proto | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/rpc/rpc.proto b/rpc/rpc.proto
index fed9eb3..b5b9eae 100644
--- a/rpc/rpc.proto
+++ b/rpc/rpc.proto
@@ -3,6 +3,8 @@ syntax = "proto3";
 import "google/protobuf/empty.proto";
 import "google/protobuf/timestamp.proto";
 
+option go_package = ".;rpc";
+
 service CLI {
     rpc GetVersion (google.protobuf.Empty) returns (VersionReply) {}
     rpc Upgrade (google.protobuf.Empty) returns (google.protobuf.Empty) {}
--
2.36.1

As you can see, I'm not really good at Go either...

[lazka]

Just do clarify, current master builds for you on bookworm?

[elboulangero]

Let me try, but I expect it does, bookworm is stable

[elboulangero]

Builds in Debian bookworm at first try.

In Debian unstable, error:

github.com/etix/mirrorbits/vendor/github.com/youtube/vitess/go/cgzip: exec: "pkg-config": executable file not found in $PATH

After adding pkgconf in the build depends, it builds.

[lazka]

Hm, I'm doing something wrong then, this is what I get in a clean bookworm container:

FROM debian:bookworm

RUN apt-get update -y && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y golang make pkg-config zlib1g-dev git && \
    apt-get clean

RUN git clone https://github.com/etix/mirrorbits

RUN cd mirrorbits;make

Step 4/4 : RUN cd mirrorbits;make
 ---> Running in 24444e1e28d0
go get -u github.com/golang/protobuf/protoc-gen-go
go: downloading github.com/golang/protobuf v1.5.4
go: downloading google.golang.org/protobuf v1.33.0
go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.
go: upgraded github.com/golang/protobuf v1.3.2 => v1.5.4
GO111MODULE=on go build -ldflags "-X github.com/etix/mirrorbits/core.VERSION=v0.5.1-89-g2a4a34c -X github.com/etix/mirrorbits/core.BUILD=2a4a34c-master -X github.com/etix/mirrorbits/config.TEMPLATES_PATH=templates/" -o bin/mirrorbits .
go: downloading github.com/pkg/errors v0.8.1
go: downloading github.com/op/go-logging v0.0.0-20160315200505-970db520ece7
go: downloading github.com/howeyc/gopass v0.0.0-20190910152052-7cb4b85ec19c
go: downloading google.golang.org/grpc v1.23.1
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/gomodule/redigo v0.0.0-20181026001555-e8fc0692a7e2
go: downloading golang.org/x/net v0.0.0-20190912160710-24e19bdeb0f2
go: downloading github.com/rafaeljusto/redigomock v0.0.0-20190202135759-257e089e14a1
go: downloading github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f
go: downloading github.com/youtube/vitess v0.0.0-20181105031612-54855ec7b369
go: downloading gopkg.in/tylerb/graceful.v1 v1.2.15
go: downloading github.com/oschwald/maxminddb-golang v1.5.0
go: downloading github.com/etix/goftp v0.0.0-20170217140226-0c13163a1028
go: downloading golang.org/x/sys v0.0.0-20190913121621-c3b328c6e5a7
go: downloading golang.org/x/crypto v0.0.0-20190911031432-227b76d455e7
go: downloading google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51
go: downloading golang.org/x/text v0.3.2
# github.com/etix/mirrorbits/rpc
rpc/rpc.pb.go:1411:12: undefined: grpc.ClientConnInterface
rpc/rpc.pb.go:1415:16: undefined: grpc.SupportPackageIsVersion6
rpc/rpc.pb.go:1442:10: undefined: grpc.ClientConnInterface
rpc/rpc.pb.go:1445:27: undefined: grpc.ClientConnInterface
make: *** [Makefile:42: build] Error 2
The command '/bin/sh -c cd mirrorbits;make' returned a non-zero code: 2

[lazka]

go get google.golang.org/grpc@latest; go mod tidy seems to fix it

[elboulangero]

Maybe it's this one: golang/protobuf#1596. Or maybe not.

[elboulangero]

I think the lines that matter in your logs are:

go get -u github.com/golang/protobuf/protoc-gen-go
[...]
go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.
go: upgraded github.com/golang/protobuf v1.3.2 => v1.5.4

Despite the line github.com/golang/protobuf v1.3.2 in go.mod, Go decides it knows better and it upgrades to 1.5.4, thus breaking the build. Although apparently you managed to fix it.

In contrast, to build the Debian package, I build offline with protoc-gen-go-1-3 installed in the environment, meaning I use v1.3.x.

[lazka]

Despite the line github.com/golang/protobuf v1.3.2 in go.mod, Go decides it knows better and it upgrades to 1.5.4, thus breaking the build. Although apparently you managed to fix it.

Yeah, the makefile explicitely updates protoc-gen-go to the latest version which pulls this in. If I fix it I still get the same build error though.

[lazka]

ah, looking at the Debian build again I see it is building against different version to the ones specific in the go.mod file... that explains it I guess.

I'll create a PR

[elboulangero]

Thanks for looking into it

[lazka]

I've also created #175 to improve some things which confused me along the way.