Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kuma-sidecar not starting when using Telepresence #853

Closed
xbauquet opened this issue Jun 25, 2020 · 4 comments · Fixed by #894 or #1394
Closed

Kuma-sidecar not starting when using Telepresence #853

xbauquet opened this issue Jun 25, 2020 · 4 comments · Fixed by #894 or #1394
Assignees
Labels
good first issue Good for newcomers

Comments

@xbauquet
Copy link
Contributor

xbauquet commented Jun 25, 2020

Summary

Telepresence is a really useful tool than allows you to substitute a pod in your kubernetes cluster by a docker image on your local computer. This image helps to understand how it works:

1_ceyHmoGefCl-EuWleiZFQw
Image modified from medium

I am using this tool during development to test my new app version directly in the kubernetes cluster (I don't need to have a local test cluster).

The problem is that when I use telepresence to replace one of my pods the kuma-sidecar is not starting alongside the pods set by telepresence in replacement of my normal pod.
Here are the logs of the kuma-sidecar:

2020-06-25T07:07:20.421Z	INFO	Skipping reading config from file
2020-06-25T07:07:20.421Z	INFO	kuma-dp.run	effective configuration	{"config": "controlPlane:\n  apiServer:\n    url: http://kuma-control-plane.kuma-system:5681\ndataplane:\n  mesh: default\n  name: app-dn-deployment-5ab90d93b67a4f72be34651114fd7de4-67679bbrtq74.datanostrum\n  drainTime: 30s\ndataplaneRuntime:\n  binaryPath: envoy\n  dataplaneTokenPath: /var/run/secrets/kubernetes.io/serviceaccount/token\n"}
2020-06-25T07:07:20.425Z	INFO	kuma-dp.run	picked a free port for Envoy Admin API to listen on	{"port": "9901"}
2020-06-25T07:07:20.425Z	INFO	kuma-dp.run	generated Envoy configuration will be stored in a temporary directory	{"dir": "/tmp/kuma-dp-553564296"}
2020-06-25T07:07:20.425Z	INFO	kuma-dp.run	starting Kuma DP	{"version": "0.5.1"}
2020-06-25T07:07:20.426Z	ERROR	kuma-dp.run	error while running Kuma DP	{"error": "listen unix /tmp/kuma-access-logs-app-dn-deployment-5ab90d93b67a4f72be34651114fd7de4-67679bbrtq74.datanostrum-default.sock: bind: invalid argument"}
Error: listen unix /tmp/kuma-access-logs-app-dn-deployment-5ab90d93b67a4f72be34651114fd7de4-67679bbrtq74.datanostrum-default.sock: bind: invalid argument

Steps To Reproduce

  1. With telepresence installed locally and un pod in your kubernetes cluster.
  2. Run a command similar to:
telepresence --swap-deployment app-dn-deployment --namespace myNameSpace --docker-run --rm -p 5005:5005 app-dn:latest
  1. Use kubectl to see that the kuma-sidecar is not starting and to see the logs
kubectl get pods -n myNameSpace
kubectl logs app-dn-deployment-5ab90d93b67a4f72be34651114fd7de4-67679bbrtq74 kuma-sidecar -n myNameSpace

Additional Details & Logs

  • Kubernetes version: 1.7.5-1
  • Kuma version: 0.5.1
  • Telepresence version: 0.105
@gdasson
Copy link
Contributor

gdasson commented Jul 3, 2020

I have the same issue for my application deployment. The error description isn't very helpful either. I am running version 0.6.0 of Kuma. I'll try to downgrade and see if this issue persists.

My configuration:
K8s version(EKS): 1.14
Kuma version: 0.6.0

@nickolaev
Copy link
Contributor

Looking into this, it seems we are hitting the Unix socket pathname limitation of 108 characters:
https://man7.org/linux/man-pages/man7/unix.7.html

A UNIX domain socket address is represented in the following
       structure:

           struct sockaddr_un {
               sa_family_t sun_family;               /* AF_UNIX */
               char        sun_path[108];            /* Pathname */
           };

       The sun_family field always contains AF_UNIX.  On Linux, sun_path is
       108 bytes in size; see also NOTES, below.

and:
/tmp/kuma-access-logs-app-dn-deployment-5ab90d93b67a4f72be34651114fd7de4-67679bbrtq74.datanostrum-default.sock
is 111 characters long. Maybe we should sanitize the value to ensure it fits the limit here -> https://github.com/Kong/kuma/blob/master/app/kuma-dp/pkg/dataplane/accesslogs/server.go#L39

@xbauquet
Copy link
Contributor Author

xbauquet commented Jul 6, 2020

I'm going to test that tomorrow. Thanks for looking into it.

@gdasson
Copy link
Contributor

gdasson commented Jul 8, 2020

Reducing the character length fixed my issue. Thanks!

@nickolaev nickolaev added bug good first issue Good for newcomers labels Jul 10, 2020
xbauquet added a commit to xbauquet/kuma that referenced this issue Jul 10, 2020
When the name of the dataplane is too long an error stop the side-car
from starting. This update reduces the size of the address and log an
explicit error if the address is still too long.

* reduce the size of access log address replacing 'kuma-access-logs' by
'kal'
* add explicit error log if the size of the address is still too long

Fix kumahq#853
xbauquet added a commit to xbauquet/kuma that referenced this issue Jul 10, 2020
When the name of the dataplane is too long an error stop the side-car
from starting. This update reduces the size of the address and log an
explicit error if the address is still too long.

* reduce the size of access log address replacing 'kuma-access-logs' by
'kal'
* add explicit error log if the size of the address is still too long

Fix kumahq#853
xbauquet added a commit to xbauquet/kuma that referenced this issue Jul 10, 2020
When the name of the dataplane is too long an error stop the side-car
from starting. This update reduces the size of the address and log an
explicit error if the address is still too long.

* reduce the size of access log address replacing 'kuma-access-logs' by
'kal'
* add explicit error log if the size of the address is still too long

Fix kumahq#853
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
3 participants