Config refactor, add ocis storage, fix oc storage sharing #461

butonic · 2020-09-01T10:19:44Z

I combined the driver configuration as a preparation to rename some of the environment variables. Several config structs were refactored.

use common struct for root, user layout, home enabled, shares dir
add new REVA_DRIVER_* env var names => will be adressed in https://github.com/owncloud/ocis/issues/494
add changelog
overwrite storage registry rules: fixes [GW] Non configurable storage backends product#193
BUG: oc storage broken, see Config refactor, add ocis storage, fix oc storage sharing #461 (comment)

PVince81 · 2020-09-01T12:31:22Z

code seems fine so far, what about the remaining checkboxes ?

did you retest with EOS also with this PR ?

PVince81 · 2020-09-02T09:54:17Z

BUG: with this PR alone and OC storage I now see the home folder appearing directly in the ocis-web file list. Probably something wrong with default values for home storage ?

PVince81 · 2020-09-04T09:05:16Z

@butonic please apply this to fix OC storage:

diff --git a/pkg/flagset/driverowncloud.go b/pkg/flagset/driverowncloud.go
index 8d5f0b5..a9f0274 100644
--- a/pkg/flagset/driverowncloud.go
+++ b/pkg/flagset/driverowncloud.go
@@ -45,7 +45,7 @@ func DriverOwnCloudWithConfig(cfg *config.Config) []cli.Flag {
                },
                &cli.BoolFlag{
                        Name:        "storage-owncloud-enable-home",
-                       Value:       false,
+                       Value:       true,
                        Usage:       "enable the creation of home storages",
                        EnvVars:     []string{"REVA_STORAGE_OWNCLOUD_ENABLE_HOME"},
                        Destination: &cfg.Reva.Storages.OwnCloud.EnableHome,

I wonder if there are other potential value issues

PVince81 · 2020-09-04T11:55:39Z

still unclear what "storagehome.go" and "storagehomedata.go" are supposed to be about. There is no "storageeoshome" nor "storageochome". If these need to be on a different level because they are semantically different, then maybe name them differently ? @butonic

butonic · 2020-09-07T08:47:27Z

applied: Config refactor, add ocis storage, fix oc storage sharing #461 (comment)
storagehome is the configuration for the storage provider mounted at /home - responsible for metadata operations
storagehomedata is the configuration for the data provider used by the storage mounted at /home - responsible for up and downloads

PVince81 · 2020-09-08T11:47:50Z

the latest changes fix owncloud/product#205

folder creation, upload, download, rename, move now work in a received share

PVince81 · 2020-09-08T11:54:32Z

some weird stuff here:


3. put_get...............   3. put_get............... FAIL (GET of `/remote.php/webdav/litmus/res' failed: 500 Internal Server Error)
--

PVince81 · 2020-09-08T11:58:48Z

I ran this locally with docker run -e LITMUS_URL="https://172.17.0.1:9200/remote.php/dav/files/4c510ada-c86b-4815-8820-42cdf82c3d51" -e LITMUS_USERNAME="einstein" -e LITMUS_PASSWORD="relativity" -e TESTS="basic" owncloud/litmus:latest , the first 500 on the "res" resource gives the following logs:

2020-09-08T13:55:19+02:00 WRN core access token not set pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG credentials obtained from credential strategy: &{Type:basic ClientID:einstein ClientSecret:relativity} pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG AuthenticateRequest: type:"basic" client_id:"einstein" client_secret:"relativity"  against localhost:9142 pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/Authenticate pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG skipping auth method=/cs3.auth.registry.v1beta1.RegistryAPI/GetAuthProvider pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:56214 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=210102 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.auth.registry.v1beta1.RegistryAPI/GetAuthProvider user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG skipping auth method=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG authenticated user entry={"Attributes":[{"ByteValues":["ZWluc3RlaW4="],"Name":"cn","Values":["einstein"]},{"ByteValues":["NGM1MTBhZGEtYzg2Yi00ODE1LTg4MjAtNDJjZGY4MmMzZDUx"],"Name":"ownCloudUUID","Values":["4c510ada-c86b-4815-8820-42cdf82c3d51"]},{"ByteValues":["QWxiZXJ0IEVpbnN0ZWlu"],"Name":"displayName","Values":["Albert Einstein"]},{"ByteValues":["ZWluc3RlaW5AZXhhbXBsZS5vcmc="],"Name":"mail","Values":["einstein@example.org"]},{"ByteValues":["MjAwMDA="],"Name":"uidnumber","Values":["20000"]},{"ByteValues":["MzAwMDA="],"Name":"gidnumber","Values":["30000"]}],"DN":"cn=einstein,ou=users,dc=example,dc=org"} pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4 user={"display_name":"Albert Einstein","id":{"idp":"https://localhost:9200","opaque_id":"4c510ada-c86b-4815-8820-42cdf82c3d51"},"mail":"einstein@example.org","opaque":{"map":{"gid":{"decoder":"plain","value":"MzAwMDA="},"uid":{"decoder":"plain","value":"MjAwMDA="}}},"username":"einstein"}
2020-09-08T13:55:19+02:00 INF user id:<idp:"https://localhost:9200" opaque_id:"4c510ada-c86b-4815-8820-42cdf82c3d51" > username:"einstein" mail:"einstein@example.org" display_name:"Albert Einstein" opaque:<map:<key:"gid" value:<decoder:"plain" value:"30000" > > map:<key:"uid" value:<decoder:"plain" value:"20000" > > >  authenticated pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:44082 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=205909606 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.auth.provider.v1beta1.ProviderAPI/Authenticate user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG FindProvider by path fn=/home pkg=rgrpc rules={"":"","/":"localhost:9152","/eos":"localhost:9158","/home":"localhost:9154","/oc":"localhost:9162","/public/":"localhost:10054","1284d238-aa92-42ce-bdc4-0b0000009152":"localhost:9152","1284d238-aa92-42ce-bdc4-0b0000009154":"localhost:9154","1284d238-aa92-42ce-bdc4-0b0000009158":"localhost:9158","1284d238-aa92-42ce-bdc4-0b0000009162":"localhost:9162"} service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:50914 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=686140 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:46864 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=965138 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.provider.v1beta1.ProviderAPI/CreateHome user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:50954 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=212795471 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.gateway.v1beta1.GatewayAPI/Authenticate user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 INF core access token generated pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG http routing head=remote.php pkg=rhttp service=reva tail=/dav/files/4c510ada-c86b-4815-8820-42cdf82c3d51/litmus/res traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/Stat pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG storageprovider: stat pkg=rgrpc req={"ref":{"Spec":{"Path":"/home/litmus/res"}}} service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG FindProvider by path fn=/home/litmus/res pkg=rgrpc rules={"":"","/":"localhost:9152","/eos":"localhost:9158","/home":"localhost:9154","/oc":"localhost:9162","/public/":"localhost:10054","1284d238-aa92-42ce-bdc4-0b0000009152":"localhost:9152","1284d238-aa92-42ce-bdc4-0b0000009154":"localhost:9154","1284d238-aa92-42ce-bdc4-0b0000009158":"localhost:9158","1284d238-aa92-42ce-bdc4-0b0000009162":"localhost:9162"} service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:50914 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=479218 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG storageprovider: unwrap pkg=rgrpc ref="path:\"/home/litmus/res\" " service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG storageprovider: unwrap: after trimMountPrefix fn=/home/litmus/res fsfn=/litmus/res mountPrefix=/home pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: resolve pkg=rgrpc ref="path:\"/litmus/res\" " service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: wrap begin EnableHome=true internal= pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: wrap end fn=/litmus/res internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap EnableHome=true pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap home external=/litmus/res internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4 trim=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files user=4c510ada-c86b-4815-8820-42cdf82c3d51
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap: internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res external=/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap EnableHome=true pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap home external=/litmus/res internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4 trim=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files user=4c510ada-c86b-4815-8820-42cdf82c3d51
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap: internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res external=/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:46864 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=2952228 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.provider.v1beta1.ProviderAPI/Stat user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:50954 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=5715068 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.gateway.v1beta1.GatewayAPI/Stat user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG skipping auth method=/cs3.gateway.v1beta1.GatewayAPI/InitiateFileDownload pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG storageprovider: stat pkg=rgrpc req={"ref":{"Spec":{"Path":"/home/litmus/res"}}} service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG FindProvider by path fn=/home/litmus/res pkg=rgrpc rules={"":"","/":"localhost:9152","/eos":"localhost:9158","/home":"localhost:9154","/oc":"localhost:9162","/public/":"localhost:10054","1284d238-aa92-42ce-bdc4-0b0000009152":"localhost:9152","1284d238-aa92-42ce-bdc4-0b0000009154":"localhost:9154","1284d238-aa92-42ce-bdc4-0b0000009158":"localhost:9158","1284d238-aa92-42ce-bdc4-0b0000009162":"localhost:9162"} service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:50914 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=579521 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG storageprovider: unwrap pkg=rgrpc ref="path:\"/home/litmus/res\" " service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG storageprovider: unwrap: after trimMountPrefix fn=/home/litmus/res fsfn=/litmus/res mountPrefix=/home pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: resolve pkg=rgrpc ref="path:\"/litmus/res\" " service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: wrap begin EnableHome=true internal= pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: wrap end fn=/litmus/res internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap EnableHome=true pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap home external=/litmus/res internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4 trim=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files user=4c510ada-c86b-4815-8820-42cdf82c3d51
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap: internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res external=/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap EnableHome=true pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap home external=/litmus/res internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4 trim=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files user=4c510ada-c86b-4815-8820-42cdf82c3d51
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap: internal=/var/tmp/reva/data/4c510ada-c86b-4815-8820-42cdf82c3d51/files/litmus/res external=/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:46864 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=3126398 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.provider.v1beta1.ProviderAPI/Stat user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG FindProvider by path fn=/home/litmus/res pkg=rgrpc rules={"":"","/":"localhost:9152","/eos":"localhost:9158","/home":"localhost:9154","/oc":"localhost:9162","/public/":"localhost:10054","1284d238-aa92-42ce-bdc4-0b0000009152":"localhost:9152","1284d238-aa92-42ce-bdc4-0b0000009154":"localhost:9154","1284d238-aa92-42ce-bdc4-0b0000009158":"localhost:9158","1284d238-aa92-42ce-bdc4-0b0000009162":"localhost:9162"} service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:50914 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=533436 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.registry.v1beta1.RegistryAPI/GetStorageProvider user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG storageprovider: unwrap pkg=rgrpc ref="path:\"/home/litmus/res\" " service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG storageprovider: unwrap: after trimMountPrefix fn=/home/litmus/res fsfn=/litmus/res mountPrefix=/home pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF file download data-server=http://localhost:9156/data/litmus/res fn=/home/litmus/res pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:46864 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=730778 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.storage.provider.v1beta1.ProviderAPI/InitiateFileDownload user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:50954 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=10013851 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.gateway.v1beta1.GatewayAPI/InitiateFileDownload user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 INF access token is already provided pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG sending request to internal data server pkg=rhttp service=reva target=http://localhost:9156/data/litmus/res traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF access token is already provided pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF tusd routing: path=/litmus/res pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: resolve pkg=rhttp ref="path:\"/litmus/res\" " service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: wrap begin EnableHome=false internal= pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG skipping auth method=/cs3.identity.user.v1beta1.UserAPI/GetUser pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 ERR home/vincent/Private/Work/workspace/reva/internal/grpc/services/userprovider/userprovider.go:107 > error getting user error="userprovidersvc: error getting user: error: not found: litmus" pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 INF unary code=OK end="08/Sep/2020:13:55:19 +0200" from=tcp://127.0.0.1:40472 pkg=rgrpc service=reva start="08/Sep/2020:13:55:19 +0200" time_ns=144499576 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/cs3.identity.user.v1beta1.UserAPI/GetUser user-agent=grpc-go/1.26.0
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap EnableHome=false pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap non-home external=/ internal= parts=[""] pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 DBG ocfs: unwrap: internal= external=/ pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 ERR datasvc: file not found error="error: not found: /" pkg=rhttp service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4
2020-09-08T13:55:19+02:00 WRN http end="08/Sep/2020:13:55:19 +0200" host=127.0.0.1 method=GET pkg=rhttp proto=HTTP/1.1 service=reva size=0 start="08/Sep/2020:13:55:19 +0200" status=404 time_ns=152128659 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/data/litmus/res url=/data/litmus/res
2020-09-08T13:55:19+02:00 WRN http end="08/Sep/2020:13:55:19 +0200" host=127.0.0.1 method=GET pkg=rhttp proto=HTTP/1.1 service=reva size=0 start="08/Sep/2020:13:55:19 +0200" status=404 time_ns=154838269 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/data url=/data
2020-09-08T13:55:19+02:00 ERR http end="08/Sep/2020:13:55:19 +0200" host=127.0.0.1 method=GET pkg=rhttp proto=HTTP/1.1 service=reva size=0 start="08/Sep/2020:13:55:19 +0200" status=500 time_ns=409890574 traceid=eb1575ebf58b1fca71e59a192c9505b4 uri=/remote.php/dav/files/4c510ada-c86b-4815-8820-42cdf82c3d51/litmus/res url=/remote.php/dav/files/4c510ada-c86b-4815-8820-42cdf82c3d51/litmus/res

the test also fails when using "remote.php/webdav" with litmus

there seems to be path mishandling here where "litmus" gets extracted as user:

2020-09-08T13:55:19+02:00 ERR home/vincent/Private/Work/workspace/reva/internal/grpc/services/userprovider/userprovider.go:107 > error getting user error="userprovidersvc: error getting user: error: not found: litmus" pkg=rgrpc service=reva traceid=eb1575ebf58b1fca71e59a192c9505b4

phil-davis · 2020-09-08T12:09:02Z

Also, this needs a rebase so that it runs with the latest core commit. That will stop some "silly" acceptance test reports like:

runsh: Total unexpected passed scenarios throughout the test run:
apiShareManagement/acceptSharesToSharesFolder.feature:15 apiShareManagement/acceptSharesToSharesFolder.feature:22 apiShareManagement/acceptSharesToSharesFolder.feature:30 apiShareManagement/acceptSharesToSharesFolder.feature:52

Those tests are in the expected failures file, but in this PR branch they are not being run, so they are getting reported as "unexpected passed scenarios" (which really means "unexpected not-failed scenarios" - that reports everything that was expected to fail but did not fail - it can "not fail" because it ran and passed, or also because it did not run at all)

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

… providers Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

phil-davis · 2020-09-08T12:42:54Z

https://cloud.drone.io/owncloud/ocis-reva/1016/2/7

--- Failed scenarios:

    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink1-createPublicLinkShare.feature:10
    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink1-createPublicLinkShare.feature:22
    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink2-copyFromPublicLink.feature:11
    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink2-copyFromPublicLink.feature:27
    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink2-copyFromPublicLink.feature:53
    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink2-copyFromPublicLink.feature:54
    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink2-copyFromPublicLink.feature:58
    /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink2-uploadToPublicLinkShare.feature:10

For example:

  Scenario: Get the mtime of a file inside a folder shared by public link using new webDAV version (run on OCIS)                                 # /drone/src/tests/acceptance/features/apiOcisSpecific/apiSharePublicLink1-createPublicLinkShare.feature:10
    Given user "Alice" has created folder "testFolder"                                                                                           # FeatureContext::userHasCreatedFolder()
    And user "Alice" has created a public link share with settings                                                                               # FeatureContext::userHasCreatedAPublicLinkShareWithSettings()
      | path        | /testFolder               |
      | permissions | read,update,create,delete |
    When the public uploads file "file.txt" to the last shared folder with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the new public WebDAV API # PublicWebDavContext::thePublicUploadsFileToLastSharedFolderWithMtimeUsingTheWebdavApi()
    Then as "Alice" file "testFolder/file.txt" should exist                                                                                      # FeatureContext::asFileOrFolderShouldExist()
      WebDav::listFolderAndReturnResponseXml Received empty response where XML was expected (Exception)
    And as "Alice" the mtime of the file "testFolder/file.txt" should be "Thu, 08 Aug 2019 04:18:13 GMT"                                         # FeatureContext::theMtimeOfTheFileShouldBe()
    And the mtime of file "file.txt" in the last shared public link using the WebDAV API should be "Thu, 08 Aug 2019 04:18:13 GMT"               # PublicWebDavContext::theMtimeOfFileInTheLastSharedPublicLinkUsingTheWebdavApiShouldBe()

It looks like uploads to public links are not working. Other scenarios in that list fail when they check an HTTP status. Uploads seem to be getting 401 - unauthorized.

Oh - that might be fixed in the last commit that just came...

PVince81 · 2020-09-08T13:19:34Z

seems "apiShareManagement/acceptSharesToSharesFolder.feature:22" is passing now according to https://cloud.drone.io/owncloud/ocis-reva/1017/5/1

@phil-davis should this then be adjusted in the core repo ?

PVince81 · 2020-09-08T13:45:39Z

@butonic please remove the line https://github.com/butonic/ocis-reva/blob/config-cleanup/tests/acceptance/expected-failures-on-OC-storage.txt#L189 and that should fix CI

phil-davis · 2020-09-08T13:58:54Z

seems "apiShareManagement/acceptSharesToSharesFolder.feature:22" is passing now according to https://cloud.drone.io/owncloud/ocis-reva/1017/5/1

@phil-davis should this then be adjusted in the core repo ?

Remove it from https://github.com/owncloud/ocis-reva/blob/master/tests/acceptance/expected-failures-on-OC-storage.txt

It is a core scenario that was failing on OCIS. Now it is passing, so the code here somewhere has improved it. With this code we can share a folder, accept the share, and access a file in the share.

I am a bit surprised that acceptSharesToSharesFolder.feature:15 is not also passing - that is a similar scenario for accepting a share of a file and downloading it.

phil-davis · 2020-09-08T14:04:07Z

cs3org/reva#1148 bumps the core commit id there, and the 4 new apiShareManagement/acceptSharesToSharesFolder.feature scenarios fail there, just like they failed here in ocis-reva before this PR.

It worries me a bit that one of the scenarios is fixed here - because some sort of fix for all 4 of those will be needed in cs3org/reva and I would have guessed that the fix from cs3org/reva would "bubble up" to here and be the fix here.

PVince81 · 2020-09-08T14:26:54Z

hmm, but this config cleanup PR doesn't update ocis-reva, it's currently independent.

so it means that something in the config changes is affected one of the tests, independently

PVince81 · 2020-09-08T14:30:27Z

@butonic do we need a PR to reva also to make CI use a similar configuration like here for OC storage overlapping ? (and potentially EOS as well)

PVince81 · 2020-09-09T08:20:47Z

feels like we're missing more bits for OCIS storage:

ocis help command is missing "reva-storage-ocis" and "reva-storage-ocis-data" services => moved to Prototyping for oCIS Storage Driver product#119 (comment)
update docs to specify how to configure and test with ocis storage (important for developers!) => moved to Prototyping for oCIS Storage Driver product#119 (comment)

and also the env variables already listed above

butonic · 2020-09-09T10:07:49Z

to run

REVA_STORAGE_HOME_DRIVER=ocis REVA_STORAGE_HOME_DATA_DRIVER=ocis REVA_STORAGE_OC_DRIVER=ocis REVA_STORAGE_OC_DATA_DRIVER=ocis OCIS_LOG_LEVEL=debug go run cmd/ocis/main.go server

butonic · 2020-09-09T10:11:33Z

instead of using 7 slots for storages use only two:

/home with the home storage
/spaces/users with a non home enabled config

when more are needed they need to be configured differently anyway

this would reduce the config confusion...

Also added slight hint about them being mounts with specific drivers.

PVince81 · 2020-09-10T08:08:26Z

I've pushed an update of the drone commit for core, let's see how the tests behave now

PVince81

👍

PVince81 · 2020-09-10T08:23:38Z

still the same output despite being up to date with master commit, I don't get it:

runsh: Total unexpected passed scenarios throughout the test run:
apiShareManagement/acceptSharesToSharesFolder.feature:22

I'll remove this for now so we can move forward

PVince81 · 2020-09-10T08:24:56Z

ah! I think this is legit: "owncloud/product#208 After accepting a share data in the received file cannot be downloaded"

we did fix downloading for received files.

but this only fixed line ":22", not ":15" of the file.

will still remove now

phil-davis · 2020-09-10T08:28:35Z

Yes, apiShareManagement/acceptSharesToSharesFolder.feature:22 is passing. It is always "a god thing" that more scenarios pass.

apiShareManagement/acceptSharesToSharesFolder.feature:15 still fails - so it would be nice to understand why this "file" scenario still fails, when "22" "folder" scenario pass.

But this PR can get merged...

ownclouders · 2020-09-10T08:31:27Z

Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- pkg/flagset/drivereos.go  1
- pkg/flagset/driverlocal.go  1
- pkg/command/storagehome.go  1
- pkg/command/drivers.go  1
- pkg/flagset/driverocis.go  1
- pkg/flagset/debug.go  1
- pkg/flagset/secret.go  1
- pkg/flagset/tracing.go  1
- pkg/command/storagehomedata.go  1
- pkg/flagset/driverowncloud.go  1
- pkg/flagset/ldap.go  1
         

Clones added
============
- pkg/flagset/authbearer.go  1
- pkg/command/drivers.go  13
- pkg/command/gateway.go  1
         

Clones removed
==============
+ pkg/flagset/storageocdata.go  -11
+ pkg/flagset/storagehomedata.go  -11
+ pkg/command/storageeos.go  -3
+ pkg/flagset/storageeos.go  -11
+ pkg/command/storageoc.go  -7
+ pkg/command/storageeosdata.go  -3
+ pkg/flagset/storageroot.go  -11
+ pkg/command/storagehome.go  -6
+ pkg/command/storageocdata.go  -8
+ pkg/flagset/storagehome.go  -11
+ pkg/flagset/users.go  -1
+ pkg/flagset/sharing.go  -1
+ pkg/flagset/storagepubliclink.go  -1
+ pkg/flagset/storageeosdata.go  -9
+ pkg/command/storageroot.go  -6
+ pkg/command/storagehomedata.go  -8
+ pkg/flagset/storageoc.go  -13

See the complete overview on Codacy

Merge: 52414c9 40f24f2 Author: Vincent Petry <pvince81@owncloud.com> Date: Thu Sep 10 10:56:48 2020 +0200 Merge pull request #461 from butonic/config-cleanup Config refactor, add ocis storage, fix oc storage sharing

butonic marked this pull request as draft September 1, 2020 10:19

PVince81 mentioned this pull request Sep 2, 2020

Prototyping for oCIS Storage Driver owncloud/product#119

Closed

2 tasks

PVince81 mentioned this pull request Sep 4, 2020

[OC storage] file operations not working as share recipient owncloud/product#205

Closed

butonic added 8 commits September 8, 2020 14:14

config cleanup

c998611

add ocis driver, arbitrary storage driver rules

1cd35ce

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

reordering to align with reva, add missing owncloud options

bb278dd

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix flag name

5156ac5

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

default oc driver to enable home

6054650

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

home creation is a config option for storage drivers, not for storage…

259e564

… providers Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

make only home storage default to enable home

6d4935c

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

make home data use same enable home override config

e3c081b

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic force-pushed the config-cleanup branch from 18713f6 to e3c081b Compare September 8, 2020 12:14

butonic added 2 commits September 8, 2020 14:29

add changelog

0159257

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix public sharing port, update generated docs

87b9469

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic self-assigned this Sep 8, 2020

butonic added the bug Something isn't working label Sep 8, 2020

butonic marked this pull request as ready for review September 8, 2020 12:38

butonic mentioned this pull request Sep 8, 2020

[GW] Non configurable storage backends owncloud/product#193

Closed

PVince81 changed the title ~~config cleanup~~ Config refactor, add ocis storage, fix oc storage sharing Sep 9, 2020

Merge branch 'master' into config-cleanup

67f927e

Add ocis to driver list in help

d462e32

Also added slight hint about them being mounts with specific drivers.

PVince81 self-requested a review September 9, 2020 10:48

owncloud deleted a comment from update-docs bot Sep 9, 2020

Update core commit

aa41066

PVince81 approved these changes Sep 10, 2020

View reviewed changes

Remove expected failure for download on received share

40f24f2

PVince81 mentioned this pull request Sep 10, 2020

Reva ocis storage #466

Merged

3 tasks

PVince81 merged commit 26465bb into owncloud:master Sep 10, 2020

butonic deleted the config-cleanup branch September 10, 2020 14:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Config refactor, add ocis storage, fix oc storage sharing #461

Config refactor, add ocis storage, fix oc storage sharing #461

butonic commented Sep 1, 2020 •

edited by PVince81

Loading

PVince81 commented Sep 1, 2020

PVince81 commented Sep 2, 2020

PVince81 commented Sep 4, 2020

PVince81 commented Sep 4, 2020

butonic commented Sep 7, 2020

PVince81 commented Sep 8, 2020 •

edited

Loading

PVince81 commented Sep 8, 2020

PVince81 commented Sep 8, 2020

phil-davis commented Sep 8, 2020 •

edited

Loading

phil-davis commented Sep 8, 2020 •

edited

Loading

PVince81 commented Sep 8, 2020

PVince81 commented Sep 8, 2020

phil-davis commented Sep 8, 2020 •

edited

Loading

phil-davis commented Sep 8, 2020

PVince81 commented Sep 8, 2020

PVince81 commented Sep 8, 2020

PVince81 commented Sep 9, 2020 •

edited

Loading

butonic commented Sep 9, 2020

butonic commented Sep 9, 2020

PVince81 commented Sep 10, 2020

PVince81 left a comment

PVince81 commented Sep 10, 2020

PVince81 commented Sep 10, 2020

phil-davis commented Sep 10, 2020

ownclouders commented Sep 10, 2020

Config refactor, add ocis storage, fix oc storage sharing #461

Config refactor, add ocis storage, fix oc storage sharing #461

Conversation

butonic commented Sep 1, 2020 • edited by PVince81 Loading

PVince81 commented Sep 1, 2020

PVince81 commented Sep 2, 2020

PVince81 commented Sep 4, 2020

PVince81 commented Sep 4, 2020

butonic commented Sep 7, 2020

PVince81 commented Sep 8, 2020 • edited Loading

PVince81 commented Sep 8, 2020

PVince81 commented Sep 8, 2020

phil-davis commented Sep 8, 2020 • edited Loading

phil-davis commented Sep 8, 2020 • edited Loading

PVince81 commented Sep 8, 2020

PVince81 commented Sep 8, 2020

phil-davis commented Sep 8, 2020 • edited Loading

phil-davis commented Sep 8, 2020

PVince81 commented Sep 8, 2020

PVince81 commented Sep 8, 2020

PVince81 commented Sep 9, 2020 • edited Loading

butonic commented Sep 9, 2020

butonic commented Sep 9, 2020

PVince81 commented Sep 10, 2020

PVince81 left a comment

Choose a reason for hiding this comment

PVince81 commented Sep 10, 2020

PVince81 commented Sep 10, 2020

phil-davis commented Sep 10, 2020

ownclouders commented Sep 10, 2020

butonic commented Sep 1, 2020 •

edited by PVince81

Loading

PVince81 commented Sep 8, 2020 •

edited

Loading

phil-davis commented Sep 8, 2020 •

edited

Loading

phil-davis commented Sep 8, 2020 •

edited

Loading

phil-davis commented Sep 8, 2020 •

edited

Loading

PVince81 commented Sep 9, 2020 •

edited

Loading