Document rclone setup #159

michielbdejong · 2023-02-13T08:03:32Z

I don't see any documentation about setting up an rclone server in https://developer.sciencemesh.io/docs/technical-documentation/ - are sites expected to run one?

I found https://github.com/cs3org/reva/blob/396dd741bcb4c86ab6be8165451b6ccdfdfcba75/examples/datatx/datatx.toml#L13
but I haven't found any docs yet about which version of rclone should be run.

CC @redblom

michielbdejong · 2023-02-20T13:55:54Z

Einstein uses transfer-create and then Marie should see the incoming share. Testing this with nrro-testing:

>> ocm-share-list-received
error: code=CODE_INTERNAL msg="error listing received shares" support_trace="468b619f04e99ecfe9753b1f8f8c52ab"

Next thing to try: with the revad configs from https://github.com/cs3org/reva/tree/master/examples/datatx, or at least with the JSON share manager.

michielbdejong · 2023-02-21T16:04:54Z

Now working in a gitpod.io/empty env, trying out this script:

#!/bin/bash
set -e

docker run --restart=always -d --network=testnet --name=revad1.docker -e HOST=revad1 pondersource/dev-stock-revad
docker run --restart=always -d --network=testnet --name=rclone1.docker rclone/rclone rcd -vv --rc-user=rcloneuser --rc-pass=eilohtho9oTahsuongeeTh7reedahPo1Ohwi3aek --rc-addr=0.0.0.0:5572 --server-side-across-configs=true --log-file=/dev/stdout
docker run --restart=always -d --network=testnet --name=revad2.docker -e HOST=revad2 pondersource/dev-stock-revad
docker run --restart=always -d --network=testnet --name=rclone2.docker rclone/rclone rcd -vv --rc-user=rcloneuser --rc-pass=eilohtho9oTahsuongeeTh7reedahPo1Ohwi3aek --rc-addr=0.0.0.0:5572 --server-side-across-configs=true --log-file=/dev/stdout

still debugging some errors with that, will continue tomorrow.

michielbdejong · 2023-02-22T09:18:21Z

2023/02/22 09:17:41 open /root/.reva-token: no such file or directory

michielbdejong · 2023-02-22T09:28:54Z

2023-02-22 09:28:16.282 ERR ../../reva/internal/grpc/services/ocmproviderauthorizer/ocmproviderauthorizer.go:111 > error getting provider info error="error: not found: revad1.docker" pid=25 pkg=rgrpc traceid=118323692a6bc6e469593dd395b5f0a4

michielbdejong · 2023-02-22T09:35:35Z

>> ocm-invite-forward -idp revad1.docker -token 4c510ada-c86b-4815-8820-42cdf82c3d51
error: code=CODE_INTERNAL msg="error forwarding invite:json: error sending accept post request: 401 Unauthorized: {\n  \"code\": \"UNAUTHENTICATED\",\n  \"message\": \"provider not authorized\"\n}" support_trace="baee63edc4caa2d59a1838e245920903"

michielbdejong · 2023-02-22T10:52:53Z

Switched to the pondersource/sciencemesh branch of reva (a fork of cs3org/master)

2023-02-22 10:50:12.342 ERR ../../reva/internal/http/services/ocmd/reqres.go:62 > provider not authorized error="error verifying mesh provider" pid=25 pkg=rhttp traceid=f8e3d484dd32339b8aed3472df82cd27

michielbdejong · 2023-02-22T10:57:45Z

And before that:

2023-02-22 10:50:12.341 ERR ../../reva/internal/grpc/services/ocmproviderauthorizer/ocmproviderauthorizer.go:125 > error verifying mesh provider error="internal error: json: ocm host not specified for mesh provider" pid=25 pkg=rgrpc traceid=eab4efa5a91af2973caf6f82bec00307

michielbdejong · 2023-02-22T11:59:50Z

With latest master of reva, revad1.docker says:

2023-02-22 11:52:07.119 ERR ../../reva/internal/grpc/services/ocmproviderauthorizer/ocmproviderauthorizer.go:124 > error verifying mesh provider error="json: error looking up client IP: lookup https://revad2.docker: no such host" pid=1 pkg=rgrpc traceid=e669f06a379e9522e3a30dcd62b160b1

michielbdejong · 2023-02-22T12:03:07Z

This is a response from net.LookupIP(ocmHost) so let's remove https:// from [i].services[j].host in providers.testnet.json

michielbdejong · 2023-02-22T12:32:43Z

With that, new error is:

2023-02-22 12:31:48.877 WRN ../../workspace/empty/reva/internal/grpc/services/ocminvitemanager/ocminvitemanager.go:252 > token not found pid=1 pkg=rgrpc traceid=f1d3761d25229567a3111e9a3912ab86

michielbdejong · 2023-02-22T12:41:41Z

Ah wait, I was confusing token with user_id.opaque_id. Fixed that, now it works!

>> ocm-invite-forward -idp revad1.docker -token 4891e4a3-0fed-4bae-8085-4be7b7f38b84
status:<code:CODE_OK trace:"0cc2e8209e0b349300b819d35d210b06" > user_id:<idp:"revad1.docker" opaque_id:"4c510ada-c86b-4815-8820-42cdf82c3d51" type:USER_TYPE_PRIMARY > email:"einstein@revad1-email.com" display_name:"Albert Einstein"

And I see ocm-find-accepted-users lists the other user on both sides, awesome. :) Now I can continue trying to do a data transfer :)

michielbdejong · 2023-02-22T12:47:16Z

Hm, einstein can create a transfer to marie with transfer-create but it ends up as SHARE_TYPE_USER and nothing happens yet.

Now, I think the next step is for Marie to accept the share but that is not working yet:

>> ocm-share-update-received -state accepted 6945bf85-0ab3-4bcb-8b52-c4d1ce7b2177
rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference

michielbdejong · 2023-02-22T13:32:05Z

This is the full log of revad2.docker:

023-02-22 13:31:13.171 DBG ../../workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:101 > skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/ListReceivedOCMShares pid=1 pkg=rgrpc traceid=3f68b853b68a2027ca7adbace5858e06
2023-02-22 13:31:13.172 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=299490 traceid=b3ddf6b54737d7b7eaed06bfa2129773 uri=/cs3.sharing.ocm.v1beta1.OcmAPI/ListReceivedOCMShares user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.173 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://[::1]:38908 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=1159659 traceid=3f68b853b68a2027ca7adbace5858e06 uri=/cs3.gateway.v1beta1.GatewayAPI/ListReceivedOCMShares user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.974 DBG ../../workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:101 > skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/GetReceivedOCMShare pid=1 pkg=rgrpc traceid=dcbd7874d3828c930a510b8027074cbf
2023-02-22 13:31:13.975 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=340750 traceid=cfcf95b1213ab17cf2c8e822d1d439d0 uri=/cs3.sharing.ocm.v1beta1.OcmAPI/GetReceivedOCMShare user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.975 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://[::1]:38924 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=1187299 traceid=dcbd7874d3828c930a510b8027074cbf uri=/cs3.gateway.v1beta1.GatewayAPI/GetReceivedOCMShare user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.976 DBG ../../workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:101 > skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/UpdateReceivedOCMShare pid=1 pkg=rgrpc traceid=abfb134e5fef9302ec335ab11db7afbc
2023-02-22 13:31:13.978 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=1606440 traceid=57b4767f02e99a3bffa8e6580967acad uri=/cs3.sharing.ocm.v1beta1.OcmAPI/UpdateReceivedOCMShare user-agent=grpc-go/1.52.0
2023-02-22 13:31:13.978 DBG ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=OK end="22/Feb/2023:13:31:13 +0000" from=tcp://127.0.0.1:58194 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=239180 traceid=4ddfd1f389442a68dc57837bfdb93050 uri=/cs3.sharing.ocm.v1beta1.OcmAPI/GetReceivedOCMShare user-agent=grpc-go/1.52.0
goroutine 5073 [running]:
runtime/debug.Stack()
        /home/gitpod/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /home/gitpod/go/src/runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/internal/grpc/interceptors/recovery.recoveryFunc({0x2265738, 0xc0044d69c0}, {0x17e3b20, 0x2e2d360})
        /workspace/empty/reva/internal/grpc/interceptors/recovery/recovery.go:47 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x2265738?, 0xc0044d69c0?}, {0x17e3b20?, 0x2e2d360?}, 0x0?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:383 +0x2a
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc000850000, {0x0, 0x0, 0x2edce60?})
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:421 +0x942
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).createOCMReference(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc000872bb0)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:429 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).UpdateReceivedOCMShare(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc0008cb700)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:358 +0x790
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler.func1({0x2265738, 0xc0044d6d50}, {0x1976b80?, 0xc0008cb700})
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3677 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x2265738, 0xc0044d6c90}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc00084c750)
        /workspace/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.37.0/interceptor.go:349 +0x528
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6c90?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/auth.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2ec0)
        /workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:116 +0x305
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?}, 0xc000a671f8?, 0x4d3c57?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/log.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2f40)
        /workspace/empty/reva/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/useragent.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0x1?, 0xc0002c2f80)
        /workspace/empty/reva/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/token.NewUnary.func1({0x2265738, 0xc0044d6930}, {0x1976b80, 0xc0008cb700}, 0xc004687620?, 0xc0002c2fa0)
        /workspace/empty/reva/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6930?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/appctx.NewUnary.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0x7f040e6fd518?, 0xc0002c2fc0)
        /workspace/empty/reva/internal/grpc/interceptors/appctx/appctx.go:42 +0x5e6
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6840?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0xc004809a20?, 0x17e2860?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler({0x1a1ece0?, 0xc00092a180}, {0x2265738, 0xc0044d6840}, 0xc000793c00, 0xc0008c6510)
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3679 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0xc0008c6a80, 0x2e489e0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:963 +0x28a
2023-02-22 13:31:13.979 ERR ../../workspace/empty/reva/internal/grpc/interceptors/recovery/recovery.go:49 > runtime error: invalid memory address or nil pointer dereference; stack: goroutine 5073 [running]:
runtime/debug.Stack()
        /home/gitpod/go/src/runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/internal/grpc/interceptors/recovery.recoveryFunc({0x2265738, 0xc0044d69c0}, {0x17e3b20, 0x2e2d360})
        /workspace/empty/reva/internal/grpc/interceptors/recovery/recovery.go:49 +0x65
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x2265738?, 0xc0044d69c0?}, {0x17e3b20?, 0x2e2d360?}, 0x0?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End.func1()
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:383 +0x2a
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0xc000850000, {0x0, 0x0, 0x2edce60?})
        /workspace/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.11.2/trace/span.go:421 +0x942
panic({0x17e3b20, 0x2e2d360})
        /home/gitpod/go/src/runtime/panic.go:884 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).createOCMReference(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc000872bb0)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:429 +0x212
github.com/cs3org/reva/internal/grpc/services/gateway.(*svc).UpdateReceivedOCMShare(0xc00092a180, {0x2265738, 0xc0044d6d50}, 0xc0008cb700)
        /workspace/empty/reva/internal/grpc/services/gateway/ocmshareprovider.go:358 +0x790
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler.func1({0x2265738, 0xc0044d6d50}, {0x1976b80?, 0xc0008cb700})
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3677 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x2265738, 0xc0044d6c90}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc00084c750)
        /workspace/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.37.0/interceptor.go:349 +0x528
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6c90?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/auth.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2ec0)
        /workspace/empty/reva/internal/grpc/interceptors/auth/auth.go:116 +0x305
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?}, 0xc000a671f8?, 0x4d3c57?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/log.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0xc0002c2ea0, 0xc0002c2f40)
        /workspace/empty/reva/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/useragent.NewUnary.func1({0x2265738, 0xc0044d69c0}, {0x1976b80, 0xc0008cb700}, 0x1?, 0xc0002c2f80)
        /workspace/empty/reva/internal/grpc/interceptors/useragent/useragent.go:38 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d69c0?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/token.NewUnary.func1({0x2265738, 0xc0044d6930}, {0x1976b80, 0xc0008cb700}, 0xc004687620?, 0xc0002c2fa0)
        /workspace/empty/reva/internal/grpc/interceptors/token/token.go:44 +0x158
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6930?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/internal/grpc/interceptors/appctx.NewUnary.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0x7f040e6fd518?, 0xc0002c2fc0)
        /workspace/empty/reva/internal/grpc/interceptors/appctx/appctx.go:42 +0x5e6
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x2265738?, 0xc0044d6840?}, {0x1976b80?, 0xc0008cb700?})
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x2265738, 0xc0044d6840}, {0x1976b80, 0xc0008cb700}, 0xc004809a20?, 0x17e2860?)
        /workspace/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1._GatewayAPI_UpdateReceivedOCMShare_Handler({0x1a1ece0?, 0xc00092a180}, {0x2265738, 0xc0044d6840}, 0xc000793c00, 0xc0008c6510)
        /workspace/go/pkg/mod/github.com/cs3org/go-cs3apis@v0.0.0-20230221082129-bcf2b5cf8870/cs3/gateway/v1beta1/gateway_api.pb.go:3679 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0xc0008c6a80, 0x2e489e0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1336 +0xd23
google.golang.org/grpc.(*Server).handleStream(0xc000aa83c0, {0x226fda0, 0xc0042b2000}, 0xc0047118c0, 0x0)
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:1704 +0xa2f
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /workspace/go/pkg/mod/google.golang.org/grpc@v1.52.0/server.go:963 +0x28a
 pid=1 pkg=rgrpc traceid=abfb134e5fef9302ec335ab11db7afbc
2023-02-22 13:31:13.979 ERR ../../workspace/empty/reva/internal/grpc/interceptors/log/log.go:66 > unary code=Internal end="22/Feb/2023:13:31:13 +0000" from=tcp://[::1]:38924 pid=1 pkg=rgrpc start="22/Feb/2023:13:31:13 +0000" time_ns=3756660 traceid=abfb134e5fef9302ec335ab11db7afbc uri=/cs3.gateway.v1beta1.GatewayAPI/UpdateReceivedOCMShare user-agent=grpc-go/1.52.0

michielbdejong · 2023-02-22T13:33:51Z

Line 429 of internal/grpc/services/gateway/ocmshareprovider.go is:

targetURI = fmt.Sprintf("webdav://%s@%s?name=%s", d.SharedSecret, share.Creator.Idp, share.Name)

michielbdejong · 2023-02-22T13:45:23Z

The problem is d.SharedSecret. I'll replace it with "asdf" and continue testing.

redblom · 2023-02-22T13:50:40Z

Just to be clear, for a transfer type share this part is not touched.

michielbdejong · 2023-02-22T13:54:04Z

it was touched by @gmgigi96 last week: https://github.com/cs3org/reva/blame/94ced01208864b66b21d2c54977c6df548d37974/internal/grpc/services/gateway/ocmshareprovider.go#L429

michielbdejong · 2023-02-22T13:56:16Z

-> split out to cs3org/reva#3677

michielbdejong · 2023-02-22T13:58:30Z

Next issue:
Hm, ocm-share-update-received returns OK but the state stays pending:

>> ocm-share-list-received
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                               | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE               | SHARETYPE       |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| 5d20f59c-e161-4e55-bd9d-847e82360d20 | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | opaque_id:"123e4567-e89b-12d3-a456-426655440000:fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2023-02-22 13:37:27 +0000 UTC | 2023-02-22 13:37:27 +0000 UTC | SHARE_STATE_PENDING | SHARE_TYPE_USER |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
>> ocm-share-update-received -state accepted 5d20f59c-e161-4e55-bd9d-847e82360d20
OK
>> ocm-share-list-received
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                               | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE               | SHARETYPE       |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| 5d20f59c-e161-4e55-bd9d-847e82360d20 | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | opaque_id:"123e4567-e89b-12d3-a456-426655440000:fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2023-02-22 13:37:27 +0000 UTC | 2023-02-22 13:37:27 +0000 UTC | SHARE_STATE_PENDING | SHARE_TYPE_USER |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
>>

redblom · 2023-02-22T14:00:11Z

it was touched by @gmgigi96 last week: https://github.com/cs3org/reva/blame/94ced01208864b66b21d2c54977c6df548d37974/internal/grpc/services/gateway/ocmshareprovider.go#L429

I mean, for a transfer type share there is no OCM reference being created. When accepting a share this function (createOCMReference) is not called.
So there must be something else going on.

redblom · 2023-02-22T14:03:38Z

Next issue: Hm, ocm-share-update-received returns OK but the state stays pending:

That is correct. The status IS pending immediately after accepting. The transfer is offloaded to rclone.
Next command would be transfer-list to find the share/transfer combo. And use the transfer id for the transfer-get-status command to discover the up to date status.
Ah apparently you know that. So basically the transfer is offloaded to rclone and it may take some time for the status to change.
If it stays at status pending than rclone may not be responding as expected.

michielbdejong · 2023-02-22T14:08:58Z

Ah ok. But I don't see it show up in transfer-list.
What I do is, on revad1.docker (Einstein):

>> login basic
username: einstein
password: OK
>> ocm-invite-generate
status:<code:CODE_OK trace:"ff1619d4c157866ccb1a73ff3229ad60" > invite_token:<token:"7626b9d8-489a-44d8-bdf0-859031fb7f31" user_id:<idp:"revad1.docker" opaque_id:"4c510ada-c86b-4815-8820-42cdf82c3d51" type:USER_TYPE_PRIMARY > expiration:<seconds:1677161020 > >

Then on revad2.docker (Marie):

>> login basic
username: marie
password: OK
>> ocm-invite-forward -idp revad1.docker -token 7626b9d8-489a-44d8-bdf0-859031fb7f31
status:<code:CODE_OK trace:"b610c39d6da2b8c9cdb7cdd81083e0b5" > user_id:<idp:"revad1.docker" opaque_id:"4c510ada-c86b-4815-8820-42cdf82c3d51" type:USER_TYPE_PRIMARY > email:"einstein@revad1-email.com" display_name:"Albert Einstein"

Then on revad1.docker (Einstein):

>> ocm-find-accepted-users
+--------------------------------------+---------------+------------------------+-------------+
| OPAQUEID                             | IDP           | MAIL                   | DISPLAYNAME |
+--------------------------------------+---------------+------------------------+-------------+
| f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | revad2.docker | marie@revad2-email.com | Marie Curie |
+--------------------------------------+---------------+------------------------+-------------+
>> mkdir /home/asdf
>> transfer-create -idp revad2.docker -grantee f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c /home/asdf
+--------------------------------------+---------------+--------------------------------------+---------------------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-----------------+-------------------------------+-------------------------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                                            | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | SHARETYPE       | CREATED                       | UPDATED                       |
+--------------------------------------+---------------+--------------------------------------+---------------------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-----------------+-------------------------------+-------------------------------+
| 8f981117-7937-4b95-949b-1288caee547a | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | storage_id:"123e4567-e89b-12d3-a456-426655440000" opaque_id:"fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | SHARE_TYPE_USER | 2023-02-22 14:06:28 +0000 UTC | 2023-02-22 14:06:28 +0000 UTC |
+--------------------------------------+---------------+--------------------------------------+---------------------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-----------------+-------------------------------+-------------------------------+

Then on revad2.docker (Marie):

>> ocm-share-list-received
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| #                                    | OWNER.IDP     | OWNER.OPAQUEID                       | RESOURCEID                                                               | TYPE              | GRANTEE.IDP   | GRANTEE.OPAQUEID                     | CREATED                       | UPDATED                       | STATE               | SHARETYPE       |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
| 7e40795a-8987-4cf2-ac1b-979437a1fea4 | revad1.docker | 4c510ada-c86b-4815-8820-42cdf82c3d51 | opaque_id:"123e4567-e89b-12d3-a456-426655440000:fileid-einstein%2Fasdf"  | GRANTEE_TYPE_USER | revad2.docker | f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c | 2023-02-22 14:06:28 +0000 UTC | 2023-02-22 14:06:28 +0000 UTC | SHARE_STATE_PENDING | SHARE_TYPE_USER |
+--------------------------------------+---------------+--------------------------------------+--------------------------------------------------------------------------+-------------------+---------------+--------------------------------------+-------------------------------+-------------------------------+---------------------+-----------------+
>> ocm-share-update-received -state accepted 7e40795a-8987-4cf2-ac1b-979437a1fea4
OK
>> transfer-list
+------------------+-------------+
| SHAREID.OPAQUEID | ID.OPAQUEID |
+------------------+-------------+
+------------------+-------------+
>>

redblom · 2023-02-22T14:22:17Z

Ah, it may be that the new OCM spec has not been implemented fully yet for transfer type shares.
It looks like the share is not discovered as a transfer type share:
https://github.com/cs3org/reva/blob/2b0cc8a125f2ffe338e9847c6a20969d6b0d92ce/internal/grpc/services/gateway/ocmshareprovider.go#L208
Can you detect rclone being called?

michielbdejong · 2023-02-22T14:28:51Z

No, nothing is happening in the logs of the rclone server.

michielbdejong · 2023-02-22T14:30:05Z

It's also worth noting that I don't even get this far if I don't set a fake value for d.SharedSecret in line 429 of internal/grpc/services/gateway/ocmshareprovider.go, due to cs3org/reva#3677

redblom · 2023-02-22T14:34:54Z

It's also worth noting that I don't even get this far if I don't set a fake value for d.SharedSecret in line 429 of internal/grpc/services/gateway/ocmshareprovider.go, due to cs3org/reva#3677

Yeah, but that method shouldn't even be called if we're dealing with a transfer.

redblom · 2023-02-22T14:35:03Z

Ok, I will test this myself locally, see what needs to be done.

redblom · 2023-02-22T16:12:11Z

I see that the share protocol of the transfer request is not set:
https://github.com/cs3org/reva/blob/2b0cc8a125f2ffe338e9847c6a20969d6b0d92ce/internal/grpc/services/ocmshareprovider/ocmshareprovider.go#L192-205

func (s *service) getProtocols(ctx context.Context, info *providerpb.ResourceInfo, methods []*ocm.AccessMethod) ocmd.Protocols {
	var p ocmd.Protocols
	for _, m := range methods {
		switch t := m.Term.(type) {
		case *ocm.AccessMethod_WebdavOptions:
			p = append(p, s.getWebdavProtocol(ctx, info, t))
		case *ocm.AccessMethod_WebappOptions:
			// TODO
		case *ocm.AccessMethod_TransferOptions:
			// TODO
		}
	}
	return p
}

So that must be the issue.

redblom · 2023-02-22T16:55:34Z

So I can fix the transfers.
But even regular ocm shares are not working. In createOCMReference the protocol is retrieved here:

Line 414	d, _ := getTransferProtocol(share)

This needs an implementation like getWebDAVProtocol to retrieve the secret from. That's where your nil pointer came from at:

Line 429 	targetURI = fmt.Sprintf("webdav://%s@%s?name=%s", d.SharedSecret, share.Creator.Idp, share.Name)

@michielbdejong regular sharing also fails right?

michielbdejong · 2023-04-05T11:13:20Z

CC @MahdiBaghbani @parhamin2010

redblom · 2023-05-08T09:42:23Z

Can be closed. Dealt with in PR cs3org/reva#3847. Transfers tutorial will follow shortly.

michielbdejong moved this to In Progress in Sciencemesh: road to production Feb 13, 2023

michielbdejong added this to Sciencemesh: road to production Feb 13, 2023

michielbdejong mentioned this issue Feb 20, 2023

error listing incoming shares from reva-cli #163

Open

michielbdejong added a commit to michielbdejong/reva that referenced this issue Feb 22, 2023

Work around pondersource/sciencemesh-php#159 (comment)

a3b36d2

michielbdejong moved this from In Progress to Todo in Sciencemesh: road to production Feb 22, 2023

redblom mentioned this issue Feb 22, 2023

d.SharedSecret undefined cs3org/reva#3677

Closed

mrvahedi68 mentioned this issue May 17, 2023

Contract 4 #194

Open

21 tasks

redblom moved this from Todo to Done in Sciencemesh: road to production May 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document rclone setup #159

Document rclone setup #159

michielbdejong commented Feb 13, 2023

michielbdejong commented Feb 20, 2023

michielbdejong commented Feb 21, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023 •

edited

Loading

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023

redblom commented Feb 22, 2023 •

edited

Loading

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023 •

edited

Loading

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023

redblom commented Feb 22, 2023

redblom commented Feb 22, 2023 •

edited

Loading

redblom commented Feb 22, 2023

michielbdejong commented Apr 5, 2023

redblom commented May 8, 2023

Document rclone setup #159

Document rclone setup #159

Comments

michielbdejong commented Feb 13, 2023

michielbdejong commented Feb 20, 2023

michielbdejong commented Feb 21, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023 • edited Loading

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023

redblom commented Feb 22, 2023 • edited Loading

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023 • edited Loading

michielbdejong commented Feb 22, 2023

michielbdejong commented Feb 22, 2023

redblom commented Feb 22, 2023

redblom commented Feb 22, 2023

redblom commented Feb 22, 2023 • edited Loading

redblom commented Feb 22, 2023

michielbdejong commented Apr 5, 2023

redblom commented May 8, 2023

michielbdejong commented Feb 22, 2023 •

edited

Loading

redblom commented Feb 22, 2023 •

edited

Loading

redblom commented Feb 22, 2023 •

edited

Loading

redblom commented Feb 22, 2023 •

edited

Loading