Fixed recvfd leaks #1173

sol-0 · 2021-11-24T11:06:47Z

Signed-off-by: Oleg Solodkov oleg.solodkov@xored.com

Issue link

Fixes #1089

How Has This Been Tested?

Added unit testing to cover
Tested manually
Tested by integration testing
Have not tested

Types of changes

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

sol-0 · 2021-11-24T11:13:15Z

Hi @edwarnicke , this is an idea for #1089 fix. Could you please share your thoughts?

sol-0 · 2021-11-24T11:13:45Z

Local integration testing is in-progress.

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin · 2021-11-24T11:49:09Z

pkg/networkservice/chains/nsmgr/unix_test.go

@@ -14,7 +14,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-//+build !windows
+// +build !windows


Suggested change

// +build !windows

// +build linux

denis-tingaikin · 2021-11-24T11:49:19Z

pkg/networkservice/chains/nsmgr/unix_test.go

+const (
+	linuxOsName = "linux"
+)
+


Suggested change

const (

linuxOsName = "linux"

)

This change was required by golangci-lint

Just don't check on OS name. See at first comment.

Right. Thanks, Denis!

denis-tingaikin · 2021-11-24T11:50:20Z

pkg/networkservice/chains/nsmgr/unix_test.go

@@ -160,3 +164,49 @@ func Test_MultiForwarderSendfd(t *testing.T) {
 	require.NoError(t, err)
 	require.Equal(t, 1, counter.Closes())
 }
+
+func Test_TimeoutRecvfd(t *testing.T) {


Could you explain how this test checks the problem?

Client makes a request with an expired token thus triggering timeout (timeoutserver).

The test is incorrect.

Refresh was not disabled. So token can no be expired.

We need to verify that all files are removed.

Please add test for recvfd package.

Also, I think we do not need to use sandbox here. Because we simply need to check cleaning and sandbox is overkilled thing for it.

denis-tingaikin · 2021-11-24T11:50:43Z

pkg/networkservice/common/mechanisms/recvfd/server.go

@@ -107,3 +110,15 @@ func (r *recvFDServer) Close(ctx context.Context, conn *networkservice.Connectio
 	err = swapFileToInode(fileMap, conn.GetMechanism().GetParameters(), true)


Suggested change

err = swapFileToInode(fileMap, conn.GetMechanism().GetParameters(), true)

err = swapFileToInode(fileMap, conn.GetMechanism().GetParameters())

denis-tingaikin · 2021-11-24T11:50:55Z

pkg/networkservice/common/mechanisms/recvfd/server.go

 	// Get the grpcfd.FDRecver
 	recv, ok := grpcfd.FromContext(ctx)
 	if !ok {
+		r.closeFiles(conn)


Suggested change

r.closeFiles(conn)

denis-tingaikin · 2021-11-24T11:51:19Z

pkg/networkservice/common/mechanisms/recvfd/server.go

@@ -78,18 +78,21 @@ func (r *recvFDServer) Request(ctx context.Context, request *networkservice.Netw
 }

 func (r *recvFDServer) Close(ctx context.Context, conn *networkservice.Connection) (*empty.Empty, error) {
+	// Clean up the fileMap no matter what happens
+	defer r.fileMaps.Delete(conn.GetId())


Suggested change

defer r.fileMaps.Delete(conn.GetId())

defer r.closeFiles(conn)

pkg/networkservice/common/mechanisms/recvfd/server.go

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

edwarnicke · 2021-11-24T16:13:03Z

pkg/networkservice/chains/nsmgr/unix_test.go

@@ -14,13 +14,12 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

-// +build !windows
+// +build linux


why the switch to only linux? This precludes testing on the Mac OS... what was the driver?

Thanks, Ed!

This was a code change suggested during review and I somehow missed that it would remove tests for MacOS.
I'm going to revert this back to !windows.

@denis-tingaikin , please, share your thoughts in case you have any objections.

@edwarnicke This tag was incorrect. All tests actually check OS name on linux. So I suggested just changing the build tag instead of using skip on non-linux os names in tests...

@edwarnicke @denis-tingaikin , I'll check the tests with !windows on MacOS (with removed OS name check)

@sol-0 Please do not use !windows with removed OS name check. Note: these tests are testing grpcfd library. The library required linux...

FYI: Tests will work, but for MacOS they will not make sense because we are using null server/clients instead of server/clients based on grpcfd for non-linux systems...

@denis-tingaikin , ok, thanks, Denis! I'll revert the change.

See at an example of implementation grpcfd based chain element for non-linux systems

sdk/pkg/registry/common/recvfd/client_nolinux.go

Line 31 in c57ed95

return null.NewNetworkServiceEndpointRegistryClient()

edwarnicke · 2021-11-24T16:28:59Z

pkg/networkservice/common/mechanisms/recvfd/common.go

@@ -86,7 +86,7 @@ func recvFDAndSwapInodeToFile(ctx context.Context, fileMap *perConnectionFileMap
 	return err
 }

-func swapFileToInode(fileMap *perConnectionFileMap, parameters map[string]string, closeAllFiles bool) error {
+func swapFileToInode(fileMap *perConnectionFileMap, parameters map[string]string) error {


Have you tested this in integration? I ask... because I would expect this to cause issues. The purpose of the recvfd was to receive the file and hold it open for the life of the connection. This is because other things like for example memifproxy may need to be using it during the life of the connection.

Now please note, the and hold it open part may in fact not be necessary... but you will not be able to tell that from the unit testing in SDK. Testing in integration or with cmd-forwarder-vpp would give you a sense of whether you can get away with closing these files after the processing of a Request message or whether you need to hold them open for the life of the connection.

I'm hoping your are correct here... because I'd like to see this change work as submitted :)

@edwarnicke , thanks a lot for your comments!

As far as I understand, if there's a timeout, all files are also closed as closeAllFiles flag is true in Close method. The cleanup I added is for the case when there's no Peer in context (cancelContext passed from timeoutserver).

I'm definitely not going to commit this unless it's tested in integration (locally as well as on CI - will discuss the details with Denis).

@sol-0 Yeah... and please note: your change may be right. I hope its right. I was just wracking my brains as to why the closeAll had been there to begin with, and thought of possible (not certain) issues.

@edwarnicke You're correct. Currently, @sol-0 is verifying this change on integration tests.

@denis-tingaikin , @edwarnicke integration testing has been completed: https://github.com/networkservicemesh/integration-k8s-kind/actions/runs/1517061295

Local integration testing has also completed successfully.

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin

I very like that we can start testing recvfd mechanic.

Added a few comments on reducing the complexity of unit test.

Also please consider covering the registry recvfd server, because we changed pkg/registry/common/recvfd/server.go we should also cover changes by tests.

denis-tingaikin · 2021-11-30T20:20:52Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	"github.com/networkservicemesh/sdk/pkg/tools/sandbox"
+)
+
+type checkFileClosed struct {


Suggested change

type checkFileClosed struct {

type checkRecvfdServer struct {

denis-tingaikin · 2021-11-30T20:22:11Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+
+func (n *checkFileClosed) Request(ctx context.Context, request *networkservice.NetworkServiceRequest) (*networkservice.Connection, error) {
+	p, ok := peer.FromContext(ctx)
+	if !ok {


Suggested change

if !ok {

require.True(t, ok)

denis-tingaikin · 2021-11-30T20:22:38Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	}
+
+	transceiver, ok := p.Addr.(grpcfd.FDTransceiver)
+	assert.True(n.t, ok)


Suggested change

assert.True(n.t, ok)

require.True(n.t, ok)

denis-tingaikin · 2021-11-30T20:23:03Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	t *testing.T
+}
+
+type wrapTransceiver struct {


Suggested change

type wrapTransceiver struct {

type notifiableFDTransceiver struct {

denis-tingaikin · 2021-11-30T20:26:32Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//go:build linux


Why do we need this line?

This line was added by go fmt as I'm using 1.7. I've removed this line.

denis-tingaikin · 2021-11-30T20:30:37Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+func createFile(fileName string, t *testing.T) (inodeURLStr string, fileClosedContext context.Context, cancelFunc func()) {
+	f, err := os.Create(fileName)
+	require.NoError(t, err, "Failed to create and open a file: %v", err)
+
+	info, err := f.Stat()
+	assert.NoError(t, err)
+
+	stat, ok := info.Sys().(*syscall.Stat_t)
+	assert.True(t, ok)
+
+	err = f.Close()
+	require.NoError(t, err, "Failed to close file: %v", err)
+
+	fileClosedContext, cancelFunc = context.WithCancel(context.Background())
+	inodeURLStr = fmt.Sprintf("inode://%d/%d", stat.Dev, stat.Ino)
+
+	return
+}


I do not quite follow why do we need this.

This is required in case of multiple files to check that we're closing the same files that we created - I'm returning "inodeURLStr" to create a mapping inodeURLStr -> onFileClosedFunc which I'm later using in finalizer.

denis-tingaikin · 2021-11-30T20:33:17Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	dir, err := ioutil.TempDir(os.TempDir(), t.Name())
+	require.NoError(t, err)
+	defer func() {
+		_ = os.RemoveAll(dir)
+	}()


Please use t.TempDir()

denis-tingaikin · 2021-11-30T20:34:02Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	select {
+	case e := <-errCh:
+		assert.Failf(t, "Server failed to start: %v", e.Error())
+	default:
+	}


Suggested change

select {

case e := <-errCh:

assert.Failf(t, "Server failed to start: %v", e.Error())

default:

}

require.Len(t, errCh, 0)

denis-tingaikin · 2021-11-30T20:38:33Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+
+	inodeURLStr, fileClosedContext, cancelFunc := createFile(testFileName, t)
+
+	serveURL := &url.URL{Scheme: "unix", Path: s.Name(), Host: "0.0.0.0:5000"}


Could you explain why do wee need a Host for unix?

It's not required, I've removed it

denis-tingaikin · 2021-11-30T20:43:55Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	return
+}
+
+func createServerAndClient(ctx context.Context, t *testing.T, testServerChain *networkservice.NetworkServiceServer, serveURL *url.URL) (testClient networkservice.NetworkServiceClient) {


Suggested change

func createServerAndClient(ctx context.Context, t *testing.T, testServerChain *networkservice.NetworkServiceServer, serveURL *url.URL) (testClient networkservice.NetworkServiceClient) {

func createServerAndClient(ctx context.Context, t *testing.T, testServerChain *networkservice.NetworkServiceServer, serveURL *url.URL) networkservice.NetworkServiceClient {

Please do not use return without variable.

This looks complicated.

Split this function into two.

OR

Consider using endpoint package and client chain directly.

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin · 2021-12-03T10:51:08Z

pkg/tools/grpcfdutils/transceiver.go

+	var fileCh = make(chan *os.File)
+	go func() {
+		for f := range recv {
+			runtime.SetFinalizer(f, func(file *os.File) {


Please set finalizer in OnRecvFile implementation.

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin · 2021-12-05T15:00:48Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	var dir = t.TempDir()
+
+	s, err := os.Create(path.Join(dir, "test.sock"))
+	require.NoError(t, err)
+
+	var testFileName = path.Join(dir, t.Name()+".test")
+
+	inodeURLStr, fileClosedContext, cancelFunc := createFile(testFileName, t)
+
+	serveURL := &url.URL{Scheme: "unix", Path: s.Name()}
+
+	var testChain = chain.NewNetworkServiceServer(


Could you use one style for value assignment?

denis-tingaikin · 2021-12-05T15:01:02Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+func TestRecvfdClosesSingleFile(t *testing.T) {
+	t.Cleanup(func() { goleak.VerifyNone(t) })
+
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)


Time limit for unit test is 1s.

Suggested change

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)

ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)

denis-tingaikin

Looks better. Tests are still looking over complicated. Added a few comments to reduce complexity.

denis-tingaikin · 2021-12-05T15:04:01Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+
+	inodeURLStr = inodeURL.String()
+
+	return


Please don't use anonymous returns. I wonder why linter is not alerting on this line...

Apply this to other places.

denis-tingaikin · 2021-12-05T15:05:03Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+type checkRecvfdServer struct {
+	t                     *testing.T
+	onFileClosedCallbacks map[string]func()
+}
+
+func (n *checkRecvfdServer) Close(ctx context.Context, conn *networkservice.Connection) (*empty.Empty, error) {
+	return next.Server(ctx).Close(ctx, conn)
+}
+
+func (n *checkRecvfdServer) Request(ctx context.Context, request *networkservice.NetworkServiceRequest) (*networkservice.Connection, error) {
+	err := grpcfdutils.InjectOnFileReceivedCallback(ctx, n.onRecvFile)
+	require.NoError(n.t, err)
+
+	return next.Server(ctx).Request(ctx, request)
+}
+
+func (n *checkRecvfdServer) onRecvFile(fileName string, file *os.File) {
+	// checking a file is closed by recvfd by setting a finalizer
+	runtime.SetFinalizer(file, func(file *os.File) {
+		onFileClosedCallback, ok := n.onFileClosedCallbacks[fileName]
+		if ok {
+			onFileClosedCallback()
+		}
+	})
+}


Suggested change

type checkRecvfdServer struct {

t *testing.T

onFileClosedCallbacks map[string]func()

}

func (n *checkRecvfdServer) Close(ctx context.Context, conn *networkservice.Connection) (*empty.Empty, error) {

return next.Server(ctx).Close(ctx, conn)

}

func (n *checkRecvfdServer) Request(ctx context.Context, request *networkservice.NetworkServiceRequest) (*networkservice.Connection, error) {

err := grpcfdutils.InjectOnFileReceivedCallback(ctx, n.onRecvFile)

require.NoError(n.t, err)

return next.Server(ctx).Request(ctx, request)

}

func (n *checkRecvfdServer) onRecvFile(fileName string, file *os.File) {

// checking a file is closed by recvfd by setting a finalizer

runtime.SetFinalizer(file, func(file *os.File) {

onFileClosedCallback, ok := n.onFileClosedCallbacks[fileName]

if ok {

onFileClosedCallback()

}

})

}

denis-tingaikin · 2021-12-05T15:07:33Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+		&checkRecvfdServer{
+			t:                     t,
+			onFileClosedCallbacks: onFileClosedFuncs,
+		},


Suggested change

&checkRecvfdServer{

t: t,

onFileClosedCallbacks: onFileClosedFuncs,

},

checkcontext.NewServer(t, func(t *testing.T, c context.Context)

grpcfdutils.InjectOnFileReceivedCallback(ctx, func(...) {

//Do assertation here.

})

})

Please apply this for other recvfd tests

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin · 2021-12-06T10:27:21Z

pkg/networkservice/common/mechanisms/recvfd/server_test.go

+	t.Cleanup(func() { goleak.VerifyNone(t) })
+
+	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
+	defer cancel()
+
+	dir := t.TempDir()
+
+	s, err := os.Create(path.Join(dir, "test.sock"))
+	require.NoError(t, err)
+
+	testFileName := path.Join(dir, t.Name()+".test")
+
+	inodeURLStr, fileClosedContext, cancelFunc := createFile(testFileName, t)
+
+	onFileClosedCallbacks := map[string]func(){
+		inodeURLStr: cancelFunc,
+	}
+
+	serveURL := &url.URL{Scheme: "unix", Path: s.Name()}
+
+	testChain := chain.NewNetworkServiceServer(
+		checkcontext.NewServer(t, func(t *testing.T, c context.Context) {
+			injectErr := grpcfdutils.InjectOnFileReceivedCallback(c, func(fileName string, file *os.File) {
+				runtime.SetFinalizer(file, func(file *os.File) {
+					onFileClosedCallback, ok := onFileClosedCallbacks[fileName]
+					if ok {
+						onFileClosedCallback()
+					}
+				})
+			})
+
+			require.NoError(t, injectErr)
+		}),
+		recvfd.NewServer())
+
+	startServer(ctx, t, &testChain, serveURL)


This part is repeated in two tests.

Is it make sense to rework these tests on testify suite and move this setup logic into SetupSuite function?

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

…k@main PR link: networkservicemesh/sdk#1173 Commit: 23e4c36 Author: Sol-0 Date: 2021-12-07 21:10:50 +0700 Message: - Fixed recvfd leaks (#1173) * Fixed recvfd leaks Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Added const for OS name Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Fixed formatting Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Addressed review comments Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Added tests Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Addressed review comments Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Added registry recvfd server test Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Addressed review comments Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Addressed review comments Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Addressed review comments Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Addressed review comments Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com> * Addressed review comments, added test suite Signed-off-by: NSMBot <nsmbot@networkservicmesh.io>

Fixed recvfd leaks

c7b4986

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

sol-0 requested review from edwarnicke, denis-tingaikin, glazychev-art and Mixaster995 November 24, 2021 11:09

sol-0 added 2 commits November 24, 2021 18:16

Added const for OS name

5d58f01

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

Fixed formatting

1895c74

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin reviewed Nov 24, 2021

View reviewed changes

pkg/networkservice/common/mechanisms/recvfd/server.go Show resolved Hide resolved

Addressed review comments

9f4f9ab

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

edwarnicke reviewed Nov 24, 2021

View reviewed changes

sol-0 force-pushed the bugfix/1089 branch from c9b2eb6 to 9f4f9ab Compare November 24, 2021 19:39

This was referenced Nov 26, 2021

Testing fix for SDK issue 1089 networkservicemesh/deployments-k8s#3888

Closed

Testing SDK fix of 1089 networkservicemesh/integration-k8s-kind#494

Closed

sol-0 force-pushed the bugfix/1089 branch 2 times, most recently from 1d4c030 to dc5bd30 Compare November 29, 2021 17:44

Added tests

6f613aa

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

sol-0 force-pushed the bugfix/1089 branch from dc5bd30 to 6f613aa Compare November 29, 2021 17:52

sol-0 self-assigned this Nov 30, 2021

sol-0 marked this pull request as ready for review November 30, 2021 13:27

denis-tingaikin requested changes Nov 30, 2021

View reviewed changes

Addressed review comments

a21604d

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin reviewed Dec 3, 2021

View reviewed changes

sol-0 force-pushed the bugfix/1089 branch from a320cf5 to 1a1c059 Compare December 3, 2021 11:12

Addressed review comments

55d55c8

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

sol-0 force-pushed the bugfix/1089 branch from 1a1c059 to 55d55c8 Compare December 3, 2021 11:15

Addressed review comments

a178bfe

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

denis-tingaikin reviewed Dec 5, 2021

View reviewed changes

sol-0 force-pushed the bugfix/1089 branch from 5f17250 to 0a5d32d Compare December 6, 2021 09:04

Addressed review comments

007843e

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

sol-0 force-pushed the bugfix/1089 branch from 0a5d32d to 007843e Compare December 6, 2021 09:07

denis-tingaikin reviewed Dec 6, 2021

View reviewed changes

Addressed review comments, added test suite

b2faeee

Signed-off-by: Oleg Solodkov <oleg.solodkov@xored.com>

sol-0 force-pushed the bugfix/1089 branch from 2f3b247 to b2faeee Compare December 7, 2021 14:01

denis-tingaikin approved these changes Dec 7, 2021

View reviewed changes

denis-tingaikin merged commit 23e4c36 into networkservicemesh:main Dec 7, 2021

		@@ -107,3 +110,15 @@ func (r recvFDServer) Close(ctx context.Context, conn networkservice.Connectio
		err = swapFileToInode(fileMap, conn.GetMechanism().GetParameters(), true)

	defer r.fileMaps.Delete(conn.GetId())
	defer r.closeFiles(conn)

	type checkFileClosed struct {
	type checkRecvfdServer struct {

	type wrapTransceiver struct {
	type notifiableFDTransceiver struct {


		inodeURLStr, fileClosedContext, cancelFunc := createFile(testFileName, t)

		serveURL := &url.URL{Scheme: "unix", Path: s.Name(), Host: "0.0.0.0:5000"}

	func createServerAndClient(ctx context.Context, t testing.T, testServerChain networkservice.NetworkServiceServer, serveURL *url.URL) (testClient networkservice.NetworkServiceClient) {
	func createServerAndClient(ctx context.Context, t testing.T, testServerChain networkservice.NetworkServiceServer, serveURL *url.URL) networkservice.NetworkServiceClient {

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)

Fixed recvfd leaks #1173

Fixed recvfd leaks #1173

Conversation

sol-0 commented Nov 24, 2021 • edited Loading

Issue link

How Has This Been Tested?

Types of changes

sol-0 commented Nov 24, 2021

sol-0 commented Nov 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denis-tingaikin Nov 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sol-0 Nov 24, 2021 • edited Loading

Choose a reason for hiding this comment

denis-tingaikin Nov 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denis-tingaikin Nov 24, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denis-tingaikin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denis-tingaikin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sol-0 commented Nov 24, 2021 •

edited

Loading

denis-tingaikin Nov 24, 2021 •

edited

Loading

sol-0 Nov 24, 2021 •

edited

Loading

denis-tingaikin Nov 24, 2021 •

edited

Loading

denis-tingaikin Nov 24, 2021 •

edited

Loading