Skip to content
This repository has been archived by the owner on May 2, 2021. It is now read-only.

Docker test is flaky #187

Closed
xzfc opened this issue Oct 15, 2020 · 5 comments
Closed

Docker test is flaky #187

xzfc opened this issue Oct 15, 2020 · 5 comments
Labels
bug Something isn't working

Comments

@xzfc
Copy link

xzfc commented Oct 15, 2020

Failing CI jobs:

On my machine, the docker test failing with a ~30% chance, and there are at least three unique error messages. Logs from different failing runs on my machine:

Cannot support any of the requested Mechanisms 
--- FAIL: TestForwarderTestSuite (43.10s)
    --- PASS: TestForwarderTestSuite/TestHealthCheck (0.00s)
    --- PASS: TestForwarderTestSuite/TestKernelToKernel (1.39s)
    --- FAIL: TestForwarderTestSuite/TestKernelToMemif (20.19s)
        suite_util_test.go:144: 
            	Error Trace:	suite_util_test.go:144
            	            				suite_util_test.go:131
            	            				suite_util_test.go:137
            	            				suite_kernel_to_memif_test.go:100
            	Error:      	
            	Test:       	TestForwarderTestSuite/TestKernelToMemif
            	Messages:   	unable to ping 10.0.0.0/32 before context done: context deadline exceeded
    --- FAIL: TestForwarderTestSuite/TestMemifToKernel (0.87s)
        suite_memif_to_kernel_test.go:98: 
            	Error Trace:	suite_memif_to_kernel_test.go:98
            	Error:      	Received unexpected error:
            	            	rpc error: code = Unknown desc = Cannot support any of the requested Mechanisms: [cls:"LOCAL"  type:"MEMIF"  parameters:{key:"inodeURL"  value:"file:///tmp/forwarder-913438596/memif/0e14ef43-ea96-48a3-a850-db184fc622cf.memif.socket"}]
            	            	Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:58
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:71
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken.(*updateTokenClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/updatetoken/client.go:51
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer.(*injectPeerClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/utils/inject/injectpeer/client.go:61
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer.(*injectPeerClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/utils/inject/injectpeer/client.go:66
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	            	github.com/networkservicemesh/cmd-forwarder-vppagent/internal/ns.(*nsClient).Request
            	            		/build/internal/ns/client.go:53
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/mechanisms/sendfd/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/recvfd.(*recvFDClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/mechanisms/recvfd/client.go:52
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:71
            	            	github.com/networkservicemesh/sdk-vppagent/pkg/networkservice/mechanisms/memif.(*memifClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk-vppagent@v0.0.0-20201015042645-a705c0612ad9/pkg/networkservice/mechanisms/memif/client.go:59
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	            	github.com/networkservicemesh/sdk-vppagent/pkg/networkservice/connectioncontext/ipcontext/routes.(*setVppRoutesClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk-vppagent@v0.0.0-20201015042645-a705c0612ad9/pkg/networkservice/connectioncontext/ipcontext/routes/client.go:60
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:71
            	            	github.com/networkservicemesh/sdk-vppagent/pkg/networkservice/connectioncontext/ethernetcontext/macaddress.(*setMacVppClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk-vppagent@v0.0.0-20201015042645-a705c0612ad9/pkg/networkservice/connectioncontext/ethernetcontext/macaddress/client.go:62
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	            	github.com/networkservicemesh/sdk-vppagent/pkg/networkservice/connectioncontext/ipcontext/ipaddress.(*setVppIPClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk-vppagent@v0.0.0-20201015042645-a705c0612ad9/pkg/networkservice/connectioncontext/ipcontext/ipaddress/client.go:62
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:54
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:69
            	Test:       	TestForwarderTestSuite/TestMemifToKernel
    --- FAIL: TestForwarderTestSuite/TestMemifToMemif (20.04s)
        suite_util_test.go:166: 
            	Error Trace:	suite_util_test.go:166
            	            				suite_memif_to_memif_test.go:105
            	Error:      	
            	Test:       	TestForwarderTestSuite/TestMemifToMemif
            	Messages:   	unable to ping 10.0.0.0/32 before context done: context deadline exceeded
FAIL

Unable to find interface with prefix "k2kns" in netns "k2k-client" 
--- FAIL: TestForwarderTestSuite (19.98s)
    --- PASS: TestForwarderTestSuite/TestHealthCheck (0.00s)
    --- FAIL: TestForwarderTestSuite/TestKernelToKernel (1.39s)
        suite_util_test.go:202: 
            	Error Trace:	suite_util_test.go:202
            	            				suite_util_test.go:131
            	            				suite_util_test.go:186
            	            				suite_kernel_to_kernel_test.go:90
            	Error:      	
            	Test:       	TestForwarderTestSuite/TestKernelToKernel
            	Messages:   	Unable to find interface with prefix "k2kns" in netns "k2k-client"
    --- PASS: TestForwarderTestSuite/TestKernelToMemif (10.64s)
    --- PASS: TestForwarderTestSuite/TestMemifToKernel (3.66s)
    --- PASS: TestForwarderTestSuite/TestMemifToMemif (3.73s)
FAIL

(DELETE): failed to switch namespace: failed to get or create namespace 
--- FAIL: TestForwarderTestSuite (19.86s)
    --- PASS: TestForwarderTestSuite/TestHealthCheck (0.00s)
    --- FAIL: TestForwarderTestSuite/TestKernelToKernel (1.18s)
        suite_kernel_to_kernel_test.go:97: 
            	Error Trace:	suite_kernel_to_kernel_test.go:97
            	Error:      	Received unexpected error:
            	            	rpc error: code = Unknown desc = Error returned from github.com/networkservicemesh/sdk-vppagent/pkg/networkservice/commit/commitServer.Close: rpc error: code = FailedPrecondition desc = KeyErrors: [config/linux/l3/v2/route/10.0.0.1/32/client-9998e1df-1e80-4869-87aa-a46dff054cc0 (DELETE): failed to switch namespace: failed to get or create namespace (type:FD reference:"/proc/3530/fd/13" ): failed to get file /proc/3530/fd/13 from path: no such file or directory, linux/interface/client-9998e1df-1e80-4869-87aa-a46dff054cc0/address/static/10.0.0.0/32 (DELETE): failed to get or create namespace (type:FD reference:"/proc/3530/fd/13" ): failed to get file /proc/3530/fd/13 from path: no such file or directory]
            	            	Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Close
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:84
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/updatetoken.(*updateTokenClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/updatetoken/client.go:65
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/utils/inject/injectpeer.(*injectPeerClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/utils/inject/injectpeer/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/cmd-forwarder-vppagent/internal/ns.(*nsClient).Close
            	            		/build/internal/ns/client.go:73
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/mechanisms/sendfd/client.go:81
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/kernel.(*kernelMechanismClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/mechanisms/kernel/client.go:61
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/cmd-forwarder-vppagent/internal/ns.(*nsClient).Close
            	            		/build/internal/ns/client.go:73
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/refresh.(*refreshClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/refresh/client.go:124
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/heal.(*healClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/heal/client.go:88
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/updatepath.(*updatePathClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/updatepath/client.go:59
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/common/authorize/client.go:58
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/trace/client.go:78
            	            	github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Close
            	            		/root/go/pkg/mod/github.com/networkservicemesh/sdk@v0.0.0-20201015042109-4a2e1b7772e1/pkg/networkservice/core/next/client.go:82
            	Test:       	TestForwarderTestSuite/TestKernelToKernel
    --- PASS: TestForwarderTestSuite/TestKernelToMemif (10.58s)
    --- PASS: TestForwarderTestSuite/TestMemifToKernel (3.75s)
    --- PASS: TestForwarderTestSuite/TestMemifToMemif (3.76s)
FAIL

@denis-tingaikin denis-tingaikin added the bug Something isn't working label Oct 15, 2020
edwarnicke added a commit to edwarnicke/sdk that referenced this issue Oct 20, 2020
@edwarnicke
Explicitely close files for recvfd client *after* they don't matter.
587f0a1 
Previous we were allowing the recvfd files to be closed by their
finalizer and the GC.

This sometimes led to premature closing of files leading to errors like:

networkservicemesh/cmd-forwarder-vppagent#187

'Delete' case.  This fixes that by closing the files only after the
ctx has is Done.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>
@edwarnicke edwarnicke mentioned this issue Oct 20, 2020
Merged
@edwarnicke
Copy link
Member

edwarnicke commented Oct 20, 2020 •
edited
Loading

I think this may fix the 'Delete' case (but need to test it further):

networkservicemesh/sdk#548

@xzfc
Copy link
Author

xzfc commented Oct 20, 2020

Just did 150 test runs. The mentioned PR fixes the (DELETE) case. The other two remain.

  • 14 x Unable to find
  • 16 x cannot support
  • 0 x (DELETE)

@edwarnicke
Copy link
Member

  • 'cannot support' is quasi bug... it is a bug in the sense that sdk/networkservicemesh/common/mechanisms/ chain element is effectively suppressing the underlying errors at cause. That's definitely a bug. However, the real underlying error is not that a mechanism can't be found per se, but that some other bug has prevented successfully finding a mechanism. @xzfc could you look a little further up the logs and see what the root is behind the 'Unable to find' you are seeing?

  • 'Unable to find' is related to a race condition between vpp-agent (which returns the call to Update before all configuration is configured) and the check. vpp-agent was designed initially for purposes in which eventual consistency was desirable. For our case it is not. To fix this there is a feature to add a WaitDone flag to the vppagent Update. I was debugging that last night, and have filed some issues against vppagent on that, the one's relevant here are:

It is my belief that when fully and properly implemented WaitDone will resolve the 'Unable to find' issue.

@edwarnicke
Copy link
Member

This should help disambiguate the 'cannot support' issue:

networkservicemesh/sdk#557

fkautz pushed a commit to networkservicemesh/sdk that referenced this issue Oct 20, 2020
@edwarnicke
Explicitely close files for recvfd client *after* they don't matter. (#…
8b0bd57 
…548)

Previous we were allowing the recvfd files to be closed by their
finalizer and the GC.

This sometimes led to premature closing of files leading to errors like:

networkservicemesh/cmd-forwarder-vppagent#187

'Delete' case.  This fixes that by closing the files only after the
ctx has is Done.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>
nsmbot pushed a commit to networkservicemesh/cmd-registry-proxy-dns that referenced this issue Oct 20, 2020
Update go.mod and go.sum to latest version from networkservicemesh/sd…
a80d7bd 
…k@master networkservicemesh/sdk#548

networkservicemesh/sdk PR link: networkservicemesh/sdk#548

networkservicemesh/sdk commit message:
commit 8b0bd573228d24958ad9e74f643d083709e81844
Author: Ed Warnicke <hagbard@gmail.com>
Date:   Tue Oct 20 16:03:21 2020 -0500

    Explicitely close files for recvfd client *after* they don't matter. (#548)

    Previous we were allowing the recvfd files to be closed by their
    finalizer and the GC.

    This sometimes led to premature closing of files leading to errors like:

    networkservicemesh/cmd-forwarder-vppagent#187

    'Delete' case.  This fixes that by closing the files only after the
    ctx has is Done.

    Signed-off-by: Ed Warnicke <hagbard@gmail.com>

Signed-off-by: NSMBot <nsmbot@networkservicmesh.io>
nsmbot pushed a commit to networkservicemesh/sdk-kernel that referenced this issue Oct 20, 2020
Update go.mod and go.sum to latest version from networkservicemesh/sd…
2d56cbd 
…k@master networkservicemesh/sdk#548

networkservicemesh/sdk PR link: networkservicemesh/sdk#548

networkservicemesh/sdk commit message:
commit 8b0bd573228d24958ad9e74f643d083709e81844
Author: Ed Warnicke <hagbard@gmail.com>
Date:   Tue Oct 20 16:03:21 2020 -0500

    Explicitely close files for recvfd client *after* they don't matter. (#548)

    Previous we were allowing the recvfd files to be closed by their
    finalizer and the GC.

    This sometimes led to premature closing of files leading to errors like:

    networkservicemesh/cmd-forwarder-vppagent#187

    'Delete' case.  This fixes that by closing the files only after the
    ctx has is Done.

    Signed-off-by: Ed Warnicke <hagbard@gmail.com>

Signed-off-by: NSMBot <nsmbot@networkservicmesh.io>
nsmbot pushed a commit to networkservicemesh/cmd-registry-memory that referenced this issue Oct 20, 2020
Update go.mod and go.sum to latest version from networkservicemesh/sd…
78e8984 
…k@master networkservicemesh/sdk#548

networkservicemesh/sdk PR link: networkservicemesh/sdk#548

networkservicemesh/sdk commit message:
commit 8b0bd573228d24958ad9e74f643d083709e81844
Author: Ed Warnicke <hagbard@gmail.com>
Date:   Tue Oct 20 16:03:21 2020 -0500

    Explicitely close files for recvfd client *after* they don't matter. (#548)

    Previous we were allowing the recvfd files to be closed by their
    finalizer and the GC.

    This sometimes led to premature closing of files leading to errors like:

    networkservicemesh/cmd-forwarder-vppagent#187

    'Delete' case.  This fixes that by closing the files only after the
    ctx has is Done.

    Signed-off-by: Ed Warnicke <hagbard@gmail.com>

Signed-off-by: NSMBot <nsmbot@networkservicmesh.io>
nsmbot pushed a commit to networkservicemesh/cmd-nsmgr-proxy that referenced this issue Oct 20, 2020
Update go.mod and go.sum to latest version from networkservicemesh/sd…
5a40ce0 
…k@master networkservicemesh/sdk#548

networkservicemesh/sdk PR link: networkservicemesh/sdk#548

networkservicemesh/sdk commit message:
commit 8b0bd573228d24958ad9e74f643d083709e81844
Author: Ed Warnicke <hagbard@gmail.com>
Date:   Tue Oct 20 16:03:21 2020 -0500

    Explicitely close files for recvfd client *after* they don't matter. (#548)

    Previous we were allowing the recvfd files to be closed by their
    finalizer and the GC.

    This sometimes led to premature closing of files leading to errors like:

    networkservicemesh/cmd-forwarder-vppagent#187

    'Delete' case.  This fixes that by closing the files only after the
    ctx has is Done.

    Signed-off-by: Ed Warnicke <hagbard@gmail.com>

Signed-off-by: NSMBot <nsmbot@networkservicmesh.io>
nsmbot pushed a commit to networkservicemesh/sdk-vppagent that referenced this issue Oct 20, 2020
Update go.mod and go.sum to latest version from networkservicemesh/sd…
c9118b0 
…k@master networkservicemesh/sdk#548

networkservicemesh/sdk PR link: networkservicemesh/sdk#548

networkservicemesh/sdk commit message:
commit 8b0bd573228d24958ad9e74f643d083709e81844
Author: Ed Warnicke <hagbard@gmail.com>
Date:   Tue Oct 20 16:03:21 2020 -0500

    Explicitely close files for recvfd client *after* they don't matter. (#548)

    Previous we were allowing the recvfd files to be closed by their
    finalizer and the GC.

    This sometimes led to premature closing of files leading to errors like:

    networkservicemesh/cmd-forwarder-vppagent#187

    'Delete' case.  This fixes that by closing the files only after the
    ctx has is Done.

    Signed-off-by: Ed Warnicke <hagbard@gmail.com>

Signed-off-by: NSMBot <nsmbot@networkservicmesh.io>
edwarnicke pushed a commit to networkservicemesh/sdk-vppagent that referenced this issue Oct 21, 2020
@nsmbot
Update go.mod and go.sum to latest version from networkservicemesh/sd…
cd77a19 
…k@master networkservicemesh/sdk#548 (#407)

networkservicemesh/sdk PR link: networkservicemesh/sdk#548

networkservicemesh/sdk commit message:
commit 8b0bd573228d24958ad9e74f643d083709e81844
Author: Ed Warnicke <hagbard@gmail.com>
Date:   Tue Oct 20 16:03:21 2020 -0500

    Explicitely close files for recvfd client *after* they don't matter. (#548)

    Previous we were allowing the recvfd files to be closed by their
    finalizer and the GC.

    This sometimes led to premature closing of files leading to errors like:

    networkservicemesh/cmd-forwarder-vppagent#187

    'Delete' case.  This fixes that by closing the files only after the
    ctx has is Done.

    Signed-off-by: Ed Warnicke <hagbard@gmail.com>

Signed-off-by: NSMBot <nsmbot@networkservicmesh.io>

Co-authored-by: NSMBot <nsmbot@networkservicmesh.io>
@xzfc
Copy link
Author

xzfc commented Oct 21, 2020

The new combined error message:

    --- FAIL: TestForwarderTestSuite/TestMemifToKernel (0.87s)
        suite_memif_to_kernel_test.go:98: 
            	Error Trace:	suite_memif_to_kernel_test.go:98
            	Error:      	Received unexpected error:
            	            	rpc error: code = Unknown desc = Cannot support any of the requested Mechanisms: [cls:"LOCAL" type:"MEMIF" parameters:{key:"inodeURL" value:"file:///tmp/forwarder-580757190/memif/7f40781b-2dfd-448a-943a-1ec4fff0d9e8.memif.socket"}]: 2 errors occurred:
            	            		* Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Unsupported Mechanism: cls:"LOCAL"  type:"MEMIF"  parameters:{key:"inodeURL"  value:"file:///tmp/forwarder-580757190/memif/7f40781b-2dfd-448a-943a-1ec4fff0d9e8.memif.socket"}
            	            		* Error returned from github.com/networkservicemesh/api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Unsupported Mechanism: cls:"LOCAL"  type:"MEMIF"  parameters:{key:"inodeURL"  value:"file:///tmp/forwarder-580757190/memif/7f40781b-2dfd-448a-943a-1ec4fff0d9e8.memif.socket"}

Full logs: good run, bad run.

The error occurs in the nested mechanismServer. On both good and bad runs the request passes two mechanismServers: an outer and an inner. Here is the summary of how the request was going:

The outer mechanismServer supports "KERNEL_INTERFACE", "MEMIF", "SRV6" and "VXLAN".
request.Connection.GetMechanism == nil
request.MechanismPreferences() == [{"cls":"LOCAL","type":"MEMIF"}]

The request is passed to the server m.mechanisms["MEMIF"] here.

The inner mechanismServer supports "KERNEL_INTERFACE" only.

request.MechanismPreferences() == [{"cls":"LOCAL","type":"MEMIF"},{"cls":"LOCAL","type":"KERNEL_INTERFACE"},{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"\u003cnil\u003e"}},{"cls":"REMOTE","type":"SRV6"}]

Both on good and bad runs (on all requests except the one):

  • request.Connection.Mechanism == nil
  • The request is passed to server m.mechanisms["KERNEL_INTERFACE"].

And on bad run only (on one request):

  • request.Connection.Mechanism == {"cls":"LOCAL","type":"MEMIF","parameters":{"inodeURL":"file:///tmp/forwarder-580757190/memif/7f40781b-2dfd-448a-943a-1ec4fff0d9e8.memif.socket"}}
  • Because the Mechanism is not nil, and is not KERNEL_INTERFACE, the error is returned here.

illbegood pushed a commit to illbegood/sdk that referenced this issue Dec 20, 2020
@edwarnicke
Explicitely close files for recvfd client *after* they don't matter. (n…
8dff06c 
…etworkservicemesh#548)

Previous we were allowing the recvfd files to be closed by their
finalizer and the GC.

This sometimes led to premature closing of files leading to errors like:

networkservicemesh/cmd-forwarder-vppagent#187

'Delete' case.  This fixes that by closing the files only after the
ctx has is Done.

Signed-off-by: Ed Warnicke <hagbard@gmail.com>
Signed-off-by: Sergey Ershov <sergey.ershov@xored.com>
@haiodo haiodo closed this as completed Jan 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@haiodo @edwarnicke @xzfc @denis-tingaikin