proposal: c-shared library compatible with independent Go application

[pjebs]

Proposal Details

I built a Go library:

package main

import "C"

func main() {  }

//export Add
func Add(a, b int32) int32 {
	return a + b
}

go build -o lib.so -buildmode=c-shared

I then built a Go application to consume the library:

package main

import (
	"fmt"

	"github.com/ebitengine/purego"
)

func main() {
	lib, _ := purego.Dlopen("./lib.so", purego.RTLD_LAZY)
	var add func(int32, int32) int32
	purego.RegisterLibFunc(&add, lib, "Add")

	//prints 42
	fmt.Println(add(40, 2))
}

I used purego package to build it to avoid cgo for my application. But the error is the same using cgo in the application.

I get a runtime error:

# command-line-arguments
Undefined symbols for architecture x86_64:
  "_Add", referenced from:
      __cgo_a844f0d618a1_Cfunc_Add in _x002.o
     (maybe you meant: __cgo_a844f0d618a1_Cfunc_Add)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
# command-line-arguments
cgo-gcc-prolog:47:33: warning: unused variable '_cgo_a' [-Wunused-variable]

It would be great if c-shared libraries could be made to be consumable by other Go applications in some way.

I understand it may be something about have the Go runtime running twice causing conflicts.

In the unrelated https://github.com/pkujhd/goloader package (which works as a linker), it does say that it shares the same Go runtime. Maybe some technique used in that package can be borrowed.

Maybe a build option to instruct the Go runtime to use different symbols.

[gabyhelp]

Related Issues

• cmd/cgo: "function symbol table not sorted by program counter" crash when using multiple shared libraries linked with -buildmode=c-archive #30822 (closed)
• cmd/compile: go build -buildmode=c-shared -linkshared produces .so that isn't loadable by dlopen #60957 (closed)
• cgo: duplicate definition linking cgo program with c-archive cgo library #49256 (closed)
• cmd/cgo: error while running C application which imports methods from the DLL created using c-shared in go #32101 (closed)
• cmd/go,cmd/cgo: link-time symbol resolution for buildmode=c-shared #12216
• cgo: multiple definition of cgo functions error when linking go-compiled c-archive library #20639 (closed)
• cmd/cgo: Conflicting types for generated symbols prevent LTO-enabled compilation #43830 (closed)
• runtime,cmd/compile: -buildmode=c-shared and dlopen-ing a shared library #16805
• cmd/cgo: undefined reference for C function with -buildmode=c-archive #25832
• cgo linking issue with C++ dynamic libraries #31723 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[ianlancetaylor]

To use a Go shared library with a Go program, you need to use either -buildmode=shared or -buildmode=plugin. There is no support for using -buildmode=c-shared to build a Go shared library to use with a Go program.

[pjebs]

@ianlancetaylor Will you remove shared build mode soon: #47788

Is there an example of how I can build a shared library and consume it (for Win, Mac and Linux, independently of course)

[ianlancetaylor]

I think the consensus on #47788, which is closed, is that we would work toward supporting a single -buildmode=shared library. As far as I know nobody has taken any steps toward that.

The overall intent is that plugins are the way to build a Go shared library and use it in a Go program. That said, the plugin package unfortunately has a lot of problems.

So unfortunately there is not at present any good way to build a Go shared library and use it in a Go program. But it's never going to be the case that using -buildmode=c-shared will work for that. That build mode is specifically for building a Go shared library and using it in a non-Go program.

[ianlancetaylor]

What is your actual goal?

[pjebs]

My ultimate goal is to create closed-source package that are written in Go and consumable only by Go applications.
It needs to work with Mac, Linux and Windows.

I was exploring different options and c-shared does not seem viable due to issues sharing the Go runtime.

@ianlancetaylor
So currently shared mode DOES WORK is what I'm getting. (The Go application will only use 1 Go library in my usecase). Is their any guidance or documentation on how a shared library can be loaded by a Go application?

[pjebs]

Why is an optional build option that dedepulicates the Go runtime symbols, memory mappings etc be bad? It seems doable and can be used by people who are willing to accept the risks (which can be documented).

[ianlancetaylor]

To the best of my knowledge -buildmode=c-shared does work for a single shared library. We have tests for that in cmd/cgo/internal/testshared.

Why is an optional build option that depulicates the Go runtime symbols, memory mappings etc be bad? It seems doable and can be used by people who are willing to accept the risks (which can be documented).

That was our (really my) attitude about -buildmode=plugin. It turned out to be a serious mistake, because when it fails people legitimately complain, and we don't have the bandwidth required to support them, and as far as we can tell nobody else does either. It's very tempting to say "here it is, try it out, at your own risk." In fact what happens is that people try it out, it works for them, then two years later it fails, and they are unhappy. For good reason. We can't make everyone happy, but we can at least avoid following roads that we know for sure will make people unhappy. To put it another way: we should not add features that we can't plausibly maintain.

[pjebs]

@ianlancetaylor

To the best of my knowledge -buildmode=c-shared does work for a single shared library. We have tests for that in cmd/cgo/internal/testshared.

I'm assuming you mean -buildmode=shared because that's what is in the tests.

[ianlancetaylor]

My apologies, I did mean -buildmode=shared.

[pjebs]

@ianlancetaylor Is there a penchant within the Go team for some way to obfuscate Go packages so that IP can be protected and packages can be sold.
That will increase Go use-cases considerably.

Go was originally designed to replace C/C++ according to Rob Pike. C/C++ devs don't have this problem.

[ianlancetaylor]

We have actually moved in the other direction. We used to support binary-only packages, but we dropped support for them in the 1.12 release.

#28152
https://go.dev/doc/go1.12#binary-only

I know that people won't find this convincing but my personal experience is that once you ship object files your code is available. There are tools out there that will take the object file and recreate an approximation of the source code from which it was generated. This is more true for Go than for languages like C/C++, because Go stack traces mean that Go object files include comprehensive source file and line number information. The only actual aspect of code that can't be extracted from an object file is the comments. That is not nothing, but it is not a lot.

So if you want to somehow protect your code, you need to use licensing agreements, not technical obfuscation measures, simply because technical obfuscation measures do not work.

As I say, I know that people won't find that convincing.

I'll also add that I would certainly like to make plugins work. But to make them practical is going to take a complete rethink of the current approach, and that is hard.

[pjebs]

@ianlancetaylor Are you saying -buildmode=shared produces object files?

[ianlancetaylor]

As far as recreating the original source code goes, a shared library provides the same information as an object file.

[pjebs]

@ianlancetaylor Would that be the case of the official plugin?

[ianlancetaylor]

If you mean: would an official plugin also permit end users to decompile the plugin to get an approximation of the original source code (without comments), then the answer is yes.

[pjebs]

@ianlancetaylor Would it be as easier or harder to decompile an official plugin compared to a -buildmode=shared library or object file (to read the source code)?

[thepudds]

Hi @pjebs, FWIW, in some markets & language ecosystems, it's relatively common for commercially-distributed software to use an obfuscator and not really rely on a "binary" distribution being human unreadable (for example, see here or here). This is mainly for the reasons Ian outlined above -- people can reverse a binary without too much difficulty for almost any language, and some languages make it easier than others.

In the Go world, it seems at least some commercial libraries like unipdf are distributed in obfuscated source code form (e.g., see a sample obfuscated unipdf file here). From a brief look, it seems the people behind unipdf might have their own proprietary obfuscator.

An open source Go obfuscator that I am somewhat familiar with is https://github.com/burrowers/garble. (4k stars, and the primary maintainer is Daniel Martí, a well-known and prolific Go contributor). The main use case for garble I think is a complete Go binary, but they are also considering tweaking things to be more friendly to obfuscating a Go library whose source is private or purchased, which sounds close to your use case. See burrowers/garble#369 for details.

I think legal protections and respect for IP are more meaningful, but in my experience with commercial software, a technical measure can reduce accidental abuse.

Distributing a binary or obfuscated source code can both be defeated, but both are enough of an obstacle that both should be roughly equivalent at reducing accidental abuse (and with less plausible deniability for "Oops!", an individual at a company is more concerned about later manager disapproval, or concerned about lawyers within a large company, or at least trigger some consciousness of wrong doing for a lone actor).

Also FWIW, personally I think obfuscated source code is a greater obstacle for someone malicious than a normal Go binary (in part because the massive security industry has many, many free & commercial tools for reversing binaries).

Based on your questions, I'm assuming there are some commercial interests involved, so perhaps it could be worthwhile to sponsor a garble maintainer to add support for burrowers/garble#369, or you could wait to see if it is implemented. Or maybe there is another Go obfuscator project out there that might be close to what you want.

My best guess, though, is that an obfuscated-source code solution is likely much closer than the Go project rethinking how to approach plugins and shared Go libraries and whatnot.

Finally, whether or not reducing accidental use is important I think depends on the software, the audience, the marketing strategy, and so on. A more user-friendly license enforcement strategy can literally be a competitive differentiator, or help find users that later turn to paying customers, but that's really for your business to determine what makes most sense.

In any event, sorry for the long post, and best of luck.

[ianlancetaylor]

Would it be as easier or harder to decompile an official plugin compared to a -buildmode=shared library or object file (to read the source code)?

With the current implementation of plugins, decompiling a plugin is essentially the same as decompiling a -buildmode=shared library or an object file.