Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add module generator option #1400

Merged
merged 2 commits into from
May 26, 2020

Conversation

Humpheh
Copy link
Contributor

@Humpheh Humpheh commented May 26, 2020

The go protobuf compiler recently added a module option define a module prefix which is trimmed from the name of go package to determine the output directory. Currently when using paths=import a protobuf file with the go_package being example.com/path/to/example would be output under ./example.com/path/to/example/myfile.pb.go. This would usually be fine with GOROOT but with go modules it's less useful as the output directory structure does not have to be example.com/path/to etc.

I have tried to mirror the behaviour from the protobuf generator. The commit for which is here: https://go-review.googlesource.com/c/protobuf/+/219298/

@googlebot
Copy link

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@Humpheh
Copy link
Contributor Author

Humpheh commented May 26, 2020

@googlebot I signed it!

@googlebot
Copy link

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

@codecov-commenter
Copy link

Codecov Report

Merging #1400 into master will increase coverage by 0.07%.
The diff coverage is 77.77%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1400      +/-   ##
==========================================
+ Coverage   54.14%   54.22%   +0.07%     
==========================================
  Files          42       42              
  Lines        4375     4389      +14     
==========================================
+ Hits         2369     2380      +11     
- Misses       1750     1752       +2     
- Partials      256      257       +1     
Impacted Files Coverage Δ
...-gen-grpc-gateway/internal/gengateway/generator.go 43.47% <77.77%> (+4.86%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3217094...38f0e09. Read the comment docs.

@johanbrandhorst
Copy link
Collaborator

Hi Humphrey, thanks for your interest in the project! Let me understand how this overlaps with paths=source_relative, which has been my go-to for the issue with the default output files. Do you have a use case which is not covered by paths=source_relative? I had a quick look at the CL linked and I'm confused about why they added it, unless they want to deprecate paths=source_relative. Do you know?

@Humpheh
Copy link
Contributor Author

Humpheh commented May 26, 2020

Hey Johan, thanks for looking at this so quickly!

Recently I've been trying to upgrade our protobuf dependencies to later versions and I hit some snags with the output directories due to our current directory structure (which needs changing anyway). The following is an example which uses the go_out option but it was the same behaviour for the grpc-gateway.

Currently we have a folder which contains quite a few of our apis called apis, which contain a number of protobuf files (e.g. apis/a.proto and apis/b.proto) with imports relative to the root (e.g. apis/a.proto has import "apis/b.proto". We were previously generating the files within this directory via something like:

protoc -I . -I ../ --go_out=paths=source_relative:../mymessages a.proto

...which included ../ as a include so that the apis/b.proto would resolve. We have had some issues though so I'm trying to move it so we generate files from the root instead using something like

protoc -I . --go_out=paths=source_relative:mymessages apis/a.proto

The issue I then found was that the source_relative option here would output the files in mymessages/apis/myfile.go which didn't match the go_package defined in the protobuf spec. In this instance as I was using go modules I wasn't able to use the default paths behaviour either (although if my directory structure was from the GOROOT then it probably would've been fine.

This change allows me to run:

protoc -I . --go_out=module=github.com/Humpheh/example:. apis/a.proto` 

which respects the go_package and outputs the file in the directory we expect.

We've been moving towards a better directory structure for the apis so that they're not all under apis and instead under the correct directory structure but at the moment this has been slow work. The change in the go protobuf generator fixed that issue so I've had a go at applying it here also.

Copy link
Collaborator

@johanbrandhorst johanbrandhorst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your thorough explanation, one small style comment.

Is it implied that you would normally use this when invoking protoc from the root of the module?

Comment on lines 129 to 141
if g.modulePath != "" {
if g.pathType != pathTypeImport {
return "", errors.New("cannot use module= with paths=source_relative")
}
trimPath, pkgPath := g.modulePath+"/", file.GoPkg.Path+"/"
if !strings.HasPrefix(pkgPath, trimPath) {
return "", fmt.Errorf("%v: file go path does not match module prefix: %v", file.GoPkg.Path, trimPath)
}
name = filepath.Join(strings.TrimPrefix(pkgPath, trimPath), filepath.Base(name))
} else if g.pathType == pathTypeImport && file.GoPkg.Path != "" {
// Only respect the path value if module is not supplied
name = fmt.Sprintf("%s/%s", file.GoPkg.Path, filepath.Base(name))
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whenever I see something like else if I think it can usually be done neater with a switch statement. What do you think about:

Suggested change
if g.modulePath != "" {
if g.pathType != pathTypeImport {
return "", errors.New("cannot use module= with paths=source_relative")
}
trimPath, pkgPath := g.modulePath+"/", file.GoPkg.Path+"/"
if !strings.HasPrefix(pkgPath, trimPath) {
return "", fmt.Errorf("%v: file go path does not match module prefix: %v", file.GoPkg.Path, trimPath)
}
name = filepath.Join(strings.TrimPrefix(pkgPath, trimPath), filepath.Base(name))
} else if g.pathType == pathTypeImport && file.GoPkg.Path != "" {
// Only respect the path value if module is not supplied
name = fmt.Sprintf("%s/%s", file.GoPkg.Path, filepath.Base(name))
}
switch {
case g.modulePath != "" && g.pathType != pathTypeImport:
return "", errors.New("cannot use module= with paths=source_relative")
case g.modulePath != "":
trimPath, pkgPath := g.modulePath+"/", file.GoPkg.Path+"/"
if !strings.HasPrefix(pkgPath, trimPath) {
return "", fmt.Errorf("%v: file go path does not match module prefix: %v", file.GoPkg.Path, trimPath)
}
name = filepath.Join(strings.TrimPrefix(pkgPath, trimPath), filepath.Base(name))
case g.pathType == pathTypeImport && file.GoPkg.Path != "":
// Only respect the path value if module is not supplied
name = fmt.Sprintf("%s/%s", file.GoPkg.Path, filepath.Base(name))
}

Copy link
Collaborator

@johanbrandhorst johanbrandhorst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great!

@johanbrandhorst johanbrandhorst merged commit 29efd26 into grpc-ecosystem:master May 26, 2020
@johanbrandhorst
Copy link
Collaborator

Thanks for your contribution! Could you please cherry pick this against the v2 branch?

Humpheh added a commit to Humpheh/grpc-gateway that referenced this pull request May 27, 2020
* Add module generator option

* Feedback from code review
@Humpheh Humpheh deleted the feature/module-support branch May 27, 2020 09:46
@Humpheh
Copy link
Contributor Author

Humpheh commented May 27, 2020

Awesome, thanks @johanbrandhorst! I've made a cherry pick on the v2 branch here: #1409

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants