Make Powergate CGO free & use testnet/3 #210

jsign · 2020-03-24T21:41:29Z

To have some context and motivation see #186.

Most of the deleted code wasn't really unnecessary but moved to:
https://github.com/textileio/lotus-client
https://github.com/textileio/lotus-devnet

Both repos have some work on that I pushed indiscriminately. From now on I'll stick to PRs to have a clearer history. Updates to those repos will be triggered by updating to a newer version of Lotus.

Also, while doing lotus-client I realized that with some PR and some OK from Lotus-team, might be reasonable to make direct-importing of the API part from Lotus. Now isn't possible since some dependency graph ends in the CGO world. (Which is what I fixed by building that almost-similar repo). Eventually, it would be nice to do that PR to have one less repo to maintain.

The lotus-devnetseems more useful in the long-run, so I dropped some words in the readme on how to use it.

The idea now is that Powergate uses lotus-client, which is a CGO free Lotus client. And use the docker-containers auto-generated by lotus-devnet to:

Use in the docker-file of the embedded mode. The network isn't run by powd anymore, but just a new container in the docker-compose.
In integration tests.

The above means Powergate isn't tied to any CGO thing anymore, so the Makefile, docker image building, and cross-platform compiling are much simplified.

While doing the CGO stuff, we decided to switch to testnet/3 to be a step ahead of what will happen. That took some extra work to get things working since the API was quite different compared to master.

The only thing that isn't good is that it seems that the devnet doesn't allow making more than one deal per run. I had to set an skip to the Renewal test that was working on master since, of course, that test implies making a second deal to renew the first one.
I'll be tracking an issue I created here to see if that can be fixed.

Apart from that, the migration went fine with enough work.

Closes #186
Closes #211
Closes #212
Closes #213

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

… a problem, better this way Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign

Comments for help reviewing.

jsign · 2020-03-26T17:49:20Z

.dockerignore

@@ -1,4 +1,3 @@
 **/docker-compose.yaml
 **/docker-compose-embedded.yaml
-**/Dockerfile
-extern


Bye externaka git-submodules.

jsign · 2020-03-26T17:51:19Z

.github/workflows/test.yml

-      - name: Set up Rust
-        uses: actions-rs/toolchain@v1
-        with:
-          profile: minimal
-          toolchain: nightly
-      - name: Install OpenCL
-        run: sudo apt-get update && sudo apt-get install -y mesa-opencl-icd ocl-icd-opencl-dev jq make


No Rust nor OpenCL for building the git-submodule.
Still self-hosted since running test is still heavy. Instead of running the devnet code, is running the devnet container.

jsign · 2020-03-26T18:24:53Z

Makefile

@@ -1,32 +1,18 @@
-SUBMODULES=.update-modules


Better to see without diffs.
Now it's super simple. Also closes #212.

jsign · 2020-03-26T19:21:09Z

api/server/server.go

@@ -93,6 +92,7 @@ type Config struct {
 	IpfsApiAddr         ma.Multiaddr
 	LotusAddress        ma.Multiaddr
 	LotusAuthToken      string
+	LotusMasterAddr     string


jsign · 2020-03-26T19:27:10Z

api/server/server.go

-		masterAddr, err = c.WalletDefaultAddress(context.Background())
-		if err != nil {
-			return nil, fmt.Errorf("getting default address: %s", err)
+		if masterAddr, err = c.WalletDefaultAddress(context.Background()); err != nil {


In embedded mode we get the master address as the default, like in the old days.
This because that whole devnet is ephemeral.

jsign · 2020-03-26T19:53:29Z

lotus/client.go

@@ -5,12 +5,10 @@ import (
 	"net/http"
 	"time"

-	"github.com/filecoin-project/lotus/api/apistruct"
-	"github.com/filecoin-project/lotus/build"
+	"github.com/filecoin-project/lotus/lib/jsonrpc"


Notice we use directly the jsonrpc impl of Lotus and not having our own (based on them). So one less thing to mantain.

That, plus not having to maintain all those API structs is much cleaner!

jsign · 2020-03-26T19:54:48Z

lotus/client.go

@@ -35,7 +33,8 @@ func New(maddr ma.Multiaddr, authToken string) (*apistruct.FullNodeStruct, func(
 	closer, err := jsonrpc.NewMergeClient("ws://"+addr+"/rpc/v0", "Filecoin",
 		[]interface{}{
 			&api.Internal,
-		}, headers, jsonrpc.WithReconnect(true, time.Second*3, 0))


The reconnect thing is still enabled. I did a PR some months ago in Lotus with this feature.
filecoin-project/lotus#1155

So, it's baked in now.

jsign · 2020-03-26T19:55:15Z

lotus/client.go

@@ -54,16 +53,6 @@ func New(maddr ma.Multiaddr, authToken string) (*apistruct.FullNodeStruct, func(

 }

-func NewEmbedded() (*apistruct.FullNodeStruct, func(), error) {


jsign · 2020-03-26T19:56:43Z

tests/ldevnet.go

-
-func CreateLocalDevnet(t *testing.T, numMiners int) (*ldevnet.LocalDevnet, address.Address, []address.Address, func()) {
-	dnet, err := ldevnet.New(numMiners, ldevnet.DefaultDuration)
+func CreateLocalDevnet(t *testing.T, numMiners int) (*apistruct.FullNodeStruct, address.Address, []address.Address) {


So now CreateLocalDevnet spinup a docker container instead of running the devnet per-se.
The good part is that all code called this method, so changing this was indistinguishable for the clients.

jsign · 2020-03-26T19:58:10Z

wallet/wallet.go

+	ty := crypto.SigTypeUnknown
+	if "bls" == typ {
+		ty = crypto.SigTypeBLS
+	} else if "secp256k1" == typ {
+		ty = crypto.SigTypeSecp256k1
+	} else {
+		return "", fmt.Errorf("unkown wallet type %s", typ)
+	}


A quick fix for the testnet/3 change since now it's typed, not a string.
I won't imagine the types of wallet to change much, so isn't that bad... but don't fully like this kind of ifs.

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

asutula

Huge improvements! Lots of small changes, but looks good to me.

asutula · 2020-03-27T02:49:30Z

.gitmodules

@@ -1,3 +0,0 @@
-[submodule "extern/filecoin-ffi"]
-	path = extern/filecoin-ffi
-	url = https://github.com/filecoin-project/filecoin-ffi.git


asutula · 2020-03-27T02:57:12Z

deals/deals.go

@@ -183,15 +190,15 @@ func pushNewChanges(ctx context.Context, api API, currState map[cid.Cid]*api.Dea
 				Size:          dinfo.Size,
 				PricePerEpoch: dinfo.PricePerEpoch.Uint64(),
 				Duration:      dinfo.Duration,
-				DealID:        dinfo.DealID,
+				DealID:        uint64(dinfo.DealID),


asutula · 2020-03-27T03:05:54Z

lotus/client.go

@@ -5,12 +5,10 @@ import (
 	"net/http"
 	"time"

-	"github.com/filecoin-project/lotus/api/apistruct"
-	"github.com/filecoin-project/lotus/build"
+	"github.com/filecoin-project/lotus/lib/jsonrpc"


That, plus not having to maintain all those API structs is much cleaner!

asutula · 2020-03-27T03:06:58Z

lotus/client.go

@@ -54,16 +53,6 @@ func New(maddr ma.Multiaddr, authToken string) (*apistruct.FullNodeStruct, func(

 }

-func NewEmbedded() (*apistruct.FullNodeStruct, func(), error) {


sanderpick

LGTM! 👋 CGO

jsign added 7 commits March 24, 2020 17:50

progress pr

b468e3c

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix docker embed

8b452ae

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

adjust to lotus-client narrwoing scope

5c155f6

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

update lotus-devnet docker image tag

35bb61b

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

tons of progress towards testnet/3

dad7010

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

more changes, finally compiles

20b8831

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix tests due to testnet/3

063b0c6

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign changed the title ~~Make Powergate CGO free~~ Make Powergate CGO free &move to testnet/3 Mar 25, 2020

jsign added 4 commits March 25, 2020 18:31

more changes for testnet/3 test passing

483af12

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

change tests since testnet/3 devnet doesnt support more than one deal

0500e6b

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

use timestamp for storage-ask

7a0d9c5

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix ask test

acd4231

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign changed the title ~~Make Powergate CGO free &move to testnet/3~~ Make Powergate CGO free & use testnet/3 Mar 25, 2020

jsign added 2 commits March 26, 2020 11:14

improve watching deals for deals

99118a6

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

disallow automatic restart on embed since stop might be indication of…

5621c9f

… a problem, better this way Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign self-assigned this Mar 26, 2020

jsign added this to the Sprint 33 milestone Mar 26, 2020

prepare for pr

a4391be

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign mentioned this pull request Mar 26, 2020

add building CLI and server binaries to Makefile #212

Closed

jsign added 2 commits March 26, 2020 16:00

update deps

3a35405

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

remove API interface from ask index. go mod tidy

198844e

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign commented Mar 26, 2020

View reviewed changes

jsign marked this pull request as ready for review March 26, 2020 19:59

jsign requested review from asutula and sanderpick March 26, 2020 19:59

fix build commands in Readme

e6ef6ab

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

remove GetCidInfo in favor to Show

99ee6bb

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

asutula approved these changes Mar 27, 2020

View reviewed changes

sanderpick approved these changes Mar 27, 2020

View reviewed changes

jsign merged commit 9500445 into master Mar 27, 2020

jsign deleted the jsign/pg5 branch March 27, 2020 17:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Powergate CGO free & use testnet/3 #210

Make Powergate CGO free & use testnet/3 #210

jsign commented Mar 24, 2020 •

edited

Loading

jsign left a comment

jsign Mar 26, 2020

jsign Mar 26, 2020

jsign Mar 26, 2020

jsign Mar 26, 2020

jsign Mar 26, 2020

jsign Mar 26, 2020

asutula Mar 27, 2020

jsign Mar 26, 2020

jsign Mar 26, 2020

asutula Mar 27, 2020

jsign Mar 26, 2020

jsign Mar 26, 2020

asutula left a comment

asutula Mar 27, 2020

asutula Mar 27, 2020

asutula Mar 27, 2020

asutula Mar 27, 2020

sanderpick left a comment

		@@ -54,16 +53,6 @@ func New(maddr ma.Multiaddr, authToken string) (*apistruct.FullNodeStruct, func(

		}

		func NewEmbedded() (*apistruct.FullNodeStruct, func(), error) {

Make Powergate CGO free & use testnet/3 #210

Make Powergate CGO free & use testnet/3 #210

Conversation

jsign commented Mar 24, 2020 • edited Loading

jsign left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asutula left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderpick left a comment

Choose a reason for hiding this comment

jsign commented Mar 24, 2020 •

edited

Loading