Initial pass on adding proxy peer server and configuration #10223

dboslee · 2022-02-08T19:27:26Z

This implements the proxy service for peer proxies to connect and dial nodes through.

Closes https://github.com/gravitational/cloud/issues/1305

russjones · 2022-02-09T19:43:49Z

@dboslee Ping us when this is ready to review.

lib/proxy/middleware.go

lib/proxy/server.go

…-server

NajiObeid · 2022-02-22T05:54:01Z

lib/proxy/server_test.go

+			expectErr: true,
+		},
+		{
+			desc: "certificates with correct role from different CAs",


I think this particular test case does not really represent what happens in the code.
The expected error is the result of the way the test is setup rather than actually be a legitimate error. I've removed it from my client branch as I was reworking the tests a bit after I implemented the equivalent of getConfigForClient on the client side.

https://github.com/gravitational/teleport/pull/10440/files#diff-8668137b10a4a8bf67506307e688582892039b3bd03ac2090b20ad0da9b9a3d1R49-R56

Not sure I understand the issue. This is specifically testing the server side cert validation. Could we leave this and have a separate test for the client side?

One change I might add is setting the tc.client.InsecureSkipVerify = true to ensure the failure happens server side.

Unless I am seeing this wrong, there's nothing in the code itself that prevents a connection between a client and a server with certs from two different CAs. It is just triggered by the way the test case is implemented.

The underlying call to ClientHandshake/ServerHandshake should return an error in this case.

…-server

russjones · 2022-03-04T23:08:50Z

@dboslee Is this PR ready for review?

dboslee · 2022-03-04T23:31:56Z

@dboslee Is this PR ready for review?

Yep, but I did open up this issue for discussion that could lead to the configuration changing slightly.

rosstimothy · 2022-03-07T14:11:38Z

api/types/networking_test.go

+		name    string
+		in      string
+		want    ProxyPeering
+		wantErr func(*testing.T, error)


Suggested change

wantErr func(*testing.T, error)

wantErr require.ErrorAssertionFunc

Will resolve this once we settle on a config format.

rosstimothy · 2022-03-07T14:26:45Z

api/types/types.proto

+    ProxyPeering ProxyPeering = 9 [ (gogoproto.jsontag) = "proxy_peering,omitempty" ];
+}
+
+// ProxyPeeringMode represents the cluster proxy peering mode.
+enum ProxyPeering {
+    // Disabled is set if proxy peering is disabled.
+    Disabled = 0;
+    // Enabled is set if proxy peering is enabled.
+    Enabled = 1;


Instead of the faux boolean could we do something that is both backward compatible and open to changes in the future?

Suggested change

ProxyPeering ProxyPeering = 9 [ (gogoproto.jsontag) = "proxy_peering,omitempty" ];

}

// ProxyPeeringMode represents the cluster proxy peering mode.

enum ProxyPeering {

// Disabled is set if proxy peering is disabled.

Disabled = 0;

// Enabled is set if proxy peering is enabled.

Enabled = 1;

TunnelDialingStrategy TunnelDialingStrategy = 9 [ (gogoproto.jsontag) = "tunnel_dialing_strategy,omitempty" ];

}

// TunnelDialingStrategy

enum TunnelDialingStrategy {

// Legacy/WarDialer/Whatever we are calling it

Legacy = 0;

// ProxyPeering

ProxyPeering = 1;

.....

// SomeNewMechanism in the future could go here

SomeNewMechanism = 2;

The naming in the suggestion could use some work - it was meant more to get the suggestion across than be anything concrete.

Makes sense. I would be curious to hear your thoughts on updating the configuration based on this discussion as well. Basically another configuration we want to add is reversetunnel_count which would be dependent on ProxyPeering being enabled.

It feels a bit wrong but maybe not a big deal for reversetunnel_count to be ignored when TunnelDialingStrategy is not equal to ProxyPeering.

I haven't fully looked into that discussion yet - but as far as allowing each type to have its own config parameters wouldn't that be similar to say how teleport storage configuration can have different parameters based on the type?

storage: type: etcd peers: [https://etcd-0.etcd:2379, https://etcd-1.etcd:2379, https://etcd-2.etcd:2379] tls_cert_file: /etc/etcd/certs/client-cert.pem tls_key_file: /etc/etcd/certs/client-key.pem tls_ca_file: /etc/etcd/certs/ca-cert.pem prefix: teleport

storage: type: dynamodb table_name: tablename region: us-east-2

lib/proxy/auth.go

lib/proxy/conn.go

lib/proxy/service_test.go

lib/proxy/conn.go

lib/proxy/service.go

rosstimothy · 2022-03-07T16:35:48Z

lib/proxy/service.go

+
+	sent, received := pipeConn(stream.Context(), streamConn, nodeConn)
+	log.Debugf("Closing dial request from peer. sent: %d reveived %d", sent, received)
+	return nil


Should this return any errors from streamConn/pipeConn instead of always returning nil?

Fixed comments on error handling here 39555e5

rosstimothy

Do you plan on adding any integration tests or extending the existing reverse tunnel tests to cover both agent mesh and proxy peering?

rosstimothy · 2022-03-21T13:19:28Z

lib/proxy/server_test.go

+			func() {
+				defer conn.Close()
+
+				client := proto.NewProxyServiceClient(conn)
+				_, err = client.DialNode(context.Background())
+				tc.assertErr(t, err)
+			}()


Does this really need to be wrapped in a function?

Suggested change

func() {

defer conn.Close()

client := proto.NewProxyServiceClient(conn)

_, err = client.DialNode(context.Background())

tc.assertErr(t, err)

}()

defer conn.Close()

client := proto.NewProxyServiceClient(conn)

_, err = client.DialNode(context.Background())

tc.assertErr(t, err)

@rosstimothy these server tests change a little bit on the client branch

It made sense when this wasn't wrapped in the t.Run(tc.desc, func(t *testing.T). But agreed not necessary now.

dboslee · 2022-03-21T16:06:21Z

Do you plan on adding any integration tests or extending the existing reverse tunnel tests to cover both agent mesh and proxy peering?

@rosstimothy I think that will make sense to add in the pr that has the changes for the agent/reverse tunneling.

…-server

pierrebeaucamp · 2022-04-06T18:12:05Z

@rosstimothy / @fspmarshall / @espadolini could we get another review please?

api/types/types.proto

api/types/tunnel_strategy.go

lib/proxy/conn.go

espadolini · 2022-04-07T10:26:08Z

lib/proxy/conn.go

+func (c *streamConn) Read(b []byte) (n int, err error) {
+	c.rLock.Lock()


Suggested change

func (c *streamConn) Read(b []byte) (n int, err error) {

c.rLock.Lock()

func (c *streamConn) Read(b []byte) (n int, err error) {

if len(b) == 0 {

return 0, nil

}

c.rLock.Lock()

I would rather let this through that way it will fail if the stream is closed.

A zero read would still be a noop if we had a non-empty buffer ready.

lib/proxy/conn.go

Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

rosstimothy · 2022-04-13T15:30:29Z

lib/proxy/conn.go

+}
+
+// streamConn wraps a grpc stream with a net.streamConn interface.
+type streamConn struct {


Can you add some test coverage for this and pipeConn? They seem like pretty fundamental components yet are only indirectly tested through DialNode. Some dedicated unit tests would ensure all the edge cases are taken care of

I've addressed these comments here a4053ec

rosstimothy · 2022-04-13T15:30:38Z

lib/proxy/conn.go

+	Recv() (*proto.Frame, error)
+}
+
+// streamConn wraps a grpc stream with a net.streamConn interface.


Suggested change

// streamConn wraps a grpc stream with a net.streamConn interface.

// streamConn wraps a grpc stream with a net.Conn interface.

rosstimothy · 2022-04-13T15:55:31Z

lib/proxy/conn.go

+	}
+}
+
+// Read reads data reveived over the grpc stream.


Suggested change

// Read reads data reveived over the grpc stream.

// Read reads data received over the grpc stream.

rosstimothy · 2022-04-13T15:58:27Z

lib/proxy/conn.go

+func (c *streamConn) Close() error {
+	var err error
+	if cstream, ok := c.stream.(grpc.ClientStream); ok {
+		err = cstream.CloseSend()


// CloseSend closes the send direction of the stream. It closes the stream
// when non-nil error is met. It is also not safe to call CloseSend
// concurrently with SendMsg.
https://pkg.go.dev/google.golang.org/grpc#ClientStream

Should we prevent Close and Write from being called concurrently?

rosstimothy · 2022-04-13T16:00:03Z

lib/proxy/service.go

+	"github.com/sirupsen/logrus"
+)
+
+// proxyService impelements the grpc ProxyService.


Suggested change

// proxyService impelements the grpc ProxyService.

// proxyService implements the grpc ProxyService.

rosstimothy · 2022-04-13T16:00:12Z

lib/proxy/service.go

+	log           logrus.FieldLogger
+}
+
+// DialNode opens a bidrectional stream to the requested node.


Suggested change

// DialNode opens a bidrectional stream to the requested node.

// DialNode opens a bidirectional stream to the requested node.

rosstimothy · 2022-04-13T16:01:27Z

api/types/tunnel_strategy.go

+	return nil
+}
+
+func (s *TunnelStrategyV1) CheckAndSetDefaults() error {


nit: missing godoc

rosstimothy · 2022-04-15T13:25:14Z

lib/proxy/conn_test.go

+		require.NoError(t, err)
+		require.Equal(t, len(data), n)


Assertions in spawned goroutines will cause problems due to semantics of t.FailNow. Here and in all other occurrences in this file the data needs to be sent over a channel and asserted on the main goroutine that is running the test

So I believe using assert fixes this since it calls testing.Errorf and testing.Log under the hood which allows the test to continue and fails at the end. Updated here 7c047ff.

If its preferred to use require I can move all the checks to the the main goroutine but using assert feels a little easier to read/reason about in this case.

dboslee force-pushed the david/proxy-peering-server branch from aae8f1c to 1bc2457 Compare February 8, 2022 19:36

Initial pass on adding proxy peer server and configuration

77ef008

dboslee force-pushed the david/proxy-peering-server branch from 1bc2457 to 77ef008 Compare February 8, 2022 19:56

Add license to test files

623947f

russjones added the cloud Cloud label Feb 9, 2022

Add dynamic client cert configuration

95cf727

dboslee marked this pull request as ready for review February 10, 2022 17:54

github-actions bot requested review from fspmarshall and jimbishopp February 10, 2022 17:54

dboslee requested review from knisbet, NajiObeid and russjones and removed request for jimbishopp February 10, 2022 17:54

NajiObeid reviewed Feb 10, 2022

View reviewed changes

lib/proxy/middleware.go Outdated Show resolved Hide resolved

NajiObeid reviewed Feb 10, 2022

View reviewed changes

lib/proxy/server.go Show resolved Hide resolved

NajiObeid reviewed Feb 10, 2022

View reviewed changes

lib/proxy/server.go Outdated Show resolved Hide resolved

NajiObeid reviewed Feb 10, 2022

View reviewed changes

lib/proxy/server.go Show resolved Hide resolved

dboslee added 2 commits February 11, 2022 12:22

Update server config defaults and address other feedback

705bf42

Merge remote-tracking branch 'origin/master' into david/proxy-peering…

5bd5492

…-server

NajiObeid reviewed Feb 22, 2022

View reviewed changes

dboslee added 2 commits February 24, 2022 13:32

Add peer address to proxy server

2997a92

Merge remote-tracking branch 'origin/master' into david/proxy-peering…

13752b5

…-server

pierrebeaucamp assigned dboslee Mar 1, 2022

russjones requested review from espadolini and rosstimothy and removed request for russjones March 3, 2022 19:25

rosstimothy reviewed Mar 7, 2022

View reviewed changes

dboslee requested a review from rosstimothy March 18, 2022 14:46

dboslee added 2 commits March 18, 2022 10:13

Merge branch 'master' into david/proxy-peering-server

4753bf9

Add license to new files

a817afa

rosstimothy reviewed Mar 21, 2022

View reviewed changes

dboslee added 2 commits March 23, 2022 08:55

Fix connection established message

142e64d

Clean up server test

c4d9ade

dboslee requested a review from rosstimothy March 31, 2022 14:29

dboslee added 2 commits April 5, 2022 12:30

Simplify stream conn to read and write directly to the stream

1d3e96f

Merge remote-tracking branch 'origin/master' into david/proxy-peering…

3e202d2

…-server

espadolini reviewed Apr 7, 2022

View reviewed changes

dboslee added 2 commits April 7, 2022 11:24

Simplify stream conn read and chunk large writes

c3dda48

Fix tunnel strategy and regenerate types.proto

be0a01b

dboslee requested review from espadolini and removed request for knisbet April 8, 2022 18:15

espadolini approved these changes Apr 11, 2022

View reviewed changes

lib/proxy/conn.go Outdated Show resolved Hide resolved

Update chunked writes

0d7bcb3

Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

rosstimothy reviewed Apr 13, 2022

View reviewed changes

Add streamConn and pipe tests and address other feedback

a4053ec

dboslee requested a review from rosstimothy April 14, 2022 17:39

Add license to conn_test.go

41a8e97

rosstimothy reviewed Apr 15, 2022

View reviewed changes

Fix concurrent test assertions

7c047ff

dboslee requested a review from rosstimothy April 18, 2022 16:37

rosstimothy approved these changes Apr 22, 2022

View reviewed changes

Merge branch 'david/proxy-peering' into david/proxy-peering-server

134364e

dboslee changed the base branch from master to david/proxy-peering April 29, 2022 20:53

dboslee merged commit 16a4c96 into david/proxy-peering Apr 29, 2022

dboslee deleted the david/proxy-peering-server branch April 24, 2023 17:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial pass on adding proxy peer server and configuration #10223

Initial pass on adding proxy peer server and configuration #10223

dboslee commented Feb 8, 2022 •

edited

Loading

russjones commented Feb 9, 2022

NajiObeid Feb 22, 2022

NajiObeid Feb 22, 2022

dboslee Feb 24, 2022

NajiObeid Feb 25, 2022

dboslee Mar 10, 2022

russjones commented Mar 4, 2022

dboslee commented Mar 4, 2022

rosstimothy Mar 7, 2022

dboslee Mar 10, 2022

rosstimothy Mar 7, 2022

dboslee Mar 7, 2022

rosstimothy Mar 7, 2022

rosstimothy Mar 7, 2022

dboslee Mar 10, 2022

rosstimothy left a comment

rosstimothy Mar 21, 2022

NajiObeid Mar 21, 2022

dboslee Mar 21, 2022

dboslee commented Mar 21, 2022

pierrebeaucamp commented Apr 6, 2022

espadolini Apr 7, 2022

dboslee Apr 7, 2022

espadolini Apr 11, 2022

rosstimothy Apr 13, 2022

dboslee Apr 14, 2022

rosstimothy Apr 13, 2022

rosstimothy Apr 13, 2022

rosstimothy Apr 13, 2022

rosstimothy Apr 13, 2022

rosstimothy Apr 13, 2022

rosstimothy Apr 13, 2022

rosstimothy Apr 15, 2022

dboslee Apr 18, 2022 •

edited

Loading

	wantErr func(*testing.T, error)
	wantErr require.ErrorAssertionFunc

		func (c *streamConn) Read(b []byte) (n int, err error) {
		c.rLock.Lock()

	// streamConn wraps a grpc stream with a net.streamConn interface.
	// streamConn wraps a grpc stream with a net.Conn interface.

	// Read reads data reveived over the grpc stream.
	// Read reads data received over the grpc stream.

	// proxyService impelements the grpc ProxyService.
	// proxyService implements the grpc ProxyService.

	// DialNode opens a bidrectional stream to the requested node.
	// DialNode opens a bidirectional stream to the requested node.

Initial pass on adding proxy peer server and configuration #10223

Initial pass on adding proxy peer server and configuration #10223

Conversation

dboslee commented Feb 8, 2022 • edited Loading

russjones commented Feb 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

russjones commented Mar 4, 2022

dboslee commented Mar 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosstimothy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dboslee commented Mar 21, 2022

pierrebeaucamp commented Apr 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dboslee Apr 18, 2022 • edited Loading

Choose a reason for hiding this comment

dboslee commented Feb 8, 2022 •

edited

Loading

dboslee Apr 18, 2022 •

edited

Loading