404 preview URL of the Eclipse Che Port extension

[Mbd06b]

Describe the bug

When I launch a default workspace, the Eclipse Che Port extension detects the running process and provides a Preview Link, but on visiting the preview link I'm getting a generic nginx 404.

I'm looking for pointers debugging this particular issue. I'm not sure where to start..

Che version

7.72@latest

Steps to reproduce

• Launch default spring-petclinic Workspace,
• Add maven vs code extensions, update/install dependencies.
• Maven Launch ... something like "mvn spring-boot:run"

Eclipse Che Port extension shows a Preview URL notification.

• Open link shows 404.

Expected behavior

The preview URL should show the landing page of the Spring Boot app launched.

Runtime

other (please specify in additional context)

Screenshots

After deploying, running the pet-clinic default workspace project, I get the preview URL popup.

Visiting the popup, I'm getting a 404 not found:

Preview Link:

Installation method

chectl/next

Environment

other (please specify in additional context)

Eclipse Che Logs

I've used chectl to save logs, there's a lot,  if there's something specific in a certain component that I should look for to share that would be helpful.

Additional context

I use Microk8s (ubuntu 22)
I use GCP for Google Cloud Domains and CloudDNS, but only to facilitate FQDN traffic and DNS settings to my cluster
and deploy chectl with a domain;.

In order to get valid https certificates working to reach the dashboard. I manually created a certificate resource using cert manager, and applied the suspected labels, in order to have cert-manager provied a secure certificate instead of the Eclipse-che Local certs.

Since it was the last thing I seem to get working before I was able to get into Eclipse Che, my recency-bias is telling me that there might be some configuration I missed that could be impacting how this workspace/preview routing handoff might be happening.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: che-tls
  namespace: eclipse-che
  labels:
    app.kubernetes.io/component: che-tls
    app.kubernetes.io/instance: che
    app.kubernetes.io/managed-by: che-operator
    app.kubernetes.io/name: che
    app.kubernetes.io/part-of: che.eclipse.org
spec:
  secretName: che-tls
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer
  dnsNames:
    - code.example.com
    - '*.code.example.com'`

[tolusha]

Could you have a look please
@amisevsk @azatsarynnyy

[amisevsk]

I tested on a more standard install and encountered no issues. Could you verify that the server is up and running (e.g. run curl localhost:8080 in a terminal after running the project)?

I suspect the issue is in the cluster configuration, specifically in the additional configuration

I use GCP for Google Cloud Domains and CloudDNS, but only to facilitate FQDN traffic and DNS settings to my cluster
and deploy chectl with a domain;.

In order to get valid https certificates working to reach the dashboard. I manually created a certificate resource using cert manager, and applied the suspected labels, in order to have cert-manager provied a secure certificate instead of the Eclipse-che Local certs.

The Che Operator is creating a separate ingress -- i.e. it's not serving the 8080 test-the-application endpoint through the main URL. Can you check that the ingress is configured correctly? It should be named something like <workspace-ID>-tools-8080-8080-tcp

For example, testing on OpenShift (which I have quickest access to), the workspace's URL is

https://<che-gateway-host>.<cluster-domain>.com/amisevsk/spring-petclinic/3100/

while the endpoint for 8080 is

http://amisevsk-spring-petclinic-8080-tcp.<cluster-domain>.com

(this is basically https://<username>-<workspace-name>-<port-num>-<protocol>.<cluster-domain>.com IIRC)

[Mbd06b]

Ok, verified the server is up and running. curl localhost:8080 returning as expected. The ingress and service inspected with kubectl look ok too. I'm still getting a 404 though. I'll keep digging.

[amisevsk]

Since it's an nginx error, I suspect there's something going wrong with the configuration of the ingress, though what that may be it's hard to say.

[Mbd06b]

$ kubectl get ingress workspace6dffd322a5e74667-tools-8080-8080-tcp -n mbd06b-che-udnte3 -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    che.routing.controller.devfile.io/component-name: tools
    che.routing.controller.devfile.io/endpoint-name: 8080-tcp
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  creationTimestamp: "2023-08-20T21:07:13Z"
  generation: 1
  labels:
    app.kubernetes.io/part-of: che.eclipse.org
    controller.devfile.io/devworkspace_id: workspace6dffd322a5e74667
  name: workspace6dffd322a5e74667-tools-8080-8080-tcp
  namespace: mbd06b-che-udnte3
  ownerReferences:
  - apiVersion: controller.devfile.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: DevWorkspaceRouting
    name: routing-workspace6dffd322a5e74667
    uid: 44aaac1e-0cb9-42ff-ae30-6de88bfa0e0c
  resourceVersion: "3581280"
  uid: 2a9fc3a5-b8bc-422c-b145-79e59dd74be0
spec:
  rules:
  - host: mbd06b-spring-petclinic-8080-tcp.code.example.com
    http:
      paths:
      - backend:
          service:
            name: workspace6dffd322a5e74667-service
            port:
              number: 8080
        path: /
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

$ kubectl describe svc workspace6dffd322a5e74667-service -n mbd06b-che-udnte3
Name:              workspace6dffd322a5e74667-service
Namespace:         mbd06b-che-udnte3
Labels:            app.kubernetes.io/component=exposure
                   app.kubernetes.io/name=eclipse-che
                   app.kubernetes.io/part-of=che.eclipse.org
                   controller.devfile.io/devworkspace_id=workspace6dffd322a5e74667
Annotations:       che.routing.controller.devfile.io/che-name: eclipse-che
                   che.routing.controller.devfile.io/che-namespace: eclipse-che
Selector:          controller.devfile.io/devworkspace_id=workspace6dffd322a5e74667
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.152.183.203
IPs:               10.152.183.203
Port:              ws-route  3030/TCP
TargetPort:        3030/TCP
Endpoints:         <none>
Port:              code-redirect-1  13131/TCP
TargetPort:        13131/TCP
Endpoints:         <none>
Port:              code-redirect-2  13132/TCP
TargetPort:        13132/TCP
Endpoints:         <none>
Port:              code-redirect-3  13133/TCP
TargetPort:        13133/TCP
Endpoints:         <none>
Port:              8080-tcp  8080/TCP
TargetPort:        8080/TCP
**Endpoints**:         <none>
Session Affinity:  None
Events:            <none>

From the ingress and service configuration, it's clear that the Ingress is set to forward traffic on the path / to workspace6dffd322a5e74667-service on port 8080.

However, is it a problem here that the service workspace6dffd322a5e74667-service has no Endpoints for any of its ports? including 8080. I'm currently suspecting this for my 404. Typically, no endpoints mean that the service doesn't have any pods to forward the traffic to, or the pods are not ready, but I can confirm that the app is running as expected by curling the port directly.

Thoughts? Does a working svc preview here have Endpoints defined?

I thought to try and edit endpoints: but those ports seem to be defined here just fine:

$ kubectl edit endpoints workspace6dffd322a5e74667-service -n mbd06b-che-udnte3

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2023-08-27T03:53:48Z"
  creationTimestamp: "2023-08-20T21:07:13Z"
  labels:
    app.kubernetes.io/component: exposure
    app.kubernetes.io/name: eclipse-che
    app.kubernetes.io/part-of: che.eclipse.org
    controller.devfile.io/devworkspace_id: workspace6dffd322a5e74667
  name: workspace6dffd322a5e74667-service
  namespace: mbd06b-che-udnte3
  resourceVersion: "5049029"
  uid: 212b844c-087a-4eb4-97ad-cd2525750a8b
subsets:
- addresses:
  - ip: 10.1.76.24
    nodeName: ethosengine
    targetRef:
      kind: Pod
      name: workspace6dffd322a5e74667-8559cffffd-jszhx
      namespace: mbd06b-che-udnte3
      uid: 248aa8e1-bcb1-424e-8312-922dd917aaee
  ports:
  - name: 8080-tcp
    port: 8080
    protocol: TCP
  - name: code-redirect-2
    port: 13132
    protocol: TCP
  - name: ws-route
    port: 3030
    protocol: TCP
  - name: code-redirect-1
    port: 13131
    protocol: TCP
  - name: code-redirect-3
    port: 13133
    protocol: TCP

Still trying to figure it out.

[amisevsk]

Is your workspace running here? The service won't have endpoints unless it can match its labelselector to a Pod.

It looks like your ingress isn't being picked up by nginx for some reason:

status:
  loadBalancer: {}

as for why... I still suspect it's related to SSL configuration. I'm not familiar with the full setup required, but its maybe relevant that your certificate is in the eclipse-che namespace, where the dashboard/main Che URL ingress lives, whereas the workspace-specific 8080 ingress is in the workspace's namespace.

[Mbd06b]

I'm going to dig into this tonight, particularly focusing on the secrets of the devworkspace-controller namespace.

One thing I noticed too is that if my authentication times-out, I get either a Fake or Self-Signed cert, but when my eclipse-che session is current, I get a fully secured TLS certificate for the workspace preview URL, which is picking up on my wildcard certificate. (both return 404), but I think the behavior is interesting.

Looking at the workspace certificates, Is eclipse che intentionally trying to only provide self-signed certs on the dev workspace previews?

Is there up-to-date documentation on using letsencrypt to setup certs either prior to or during deployment?

I manually created a "che-tls" certificate with all the labels I described originally, mainly because I'm struggling to find up-to-date documentation on deploying tls certificates for che, particularly when I have a clusterissuer with cert-manager that should be able to issue what I need.

[Mbd06b]

This has been a very persistent issue. 404 nginx ingress from from the Preview URL on spring-boot-pet clinic demo.

I've tried reinstalling a few times adjusting label, trying separate certificates, without any change on the preview url throwing the nginx 404.

Dns is fine
service is running ( I can curl from within the cluster and using in-cluster dns names).
Ingress points at port 8080 for the preview url host.
The ingress is not throwing any 404 log errors.

Is there a a working CheCluster resource, and Certificate examples that someone could share to follow that I might be missing here?
Is there additional custom configuration that needs to happen on the devworspace-controller somewhere?

[amisevsk]

If you can curl the service within the cluster, then the problem is the ingress itself not being picked up by nginx. Why that's happening, I'm not sure.

There are two things you could try:

• I'm not sure if Che is provisioning the ingress with TLS or not; if it's not you could try navigating to the non-secure version (http instead of https) and see if that works
• If it is trying to use TLS (which it seems to be, from annotations), you could try to configure the CheCluster's .spec.networking.annotations to set appropriate annotations to make the ingress use your custom certs. See kubectl explain checluster.spec.networking.annotations.

I'm not terribly experienced with configuring TLS on Kubernetes, so I'm not really sure what "good" settings are here, but maybe https://microk8s.io/docs/addon-cert-manager is a guide.

@tolusha does the above seem correct? This seems to be a certificates issue in Che Operator

[Mbd06b]

To try and narrow down the problem, I've cleared out all my namespaces, secrets, and certificates related to eclipse che, and did a fresh deploy. I'm still getting 404 on the running app preview served up with a Kubernetes Ingress Fake Cert. All the troubleshooting I've done so far is yielding similar results.

[Mbd06b]

Found it! Great suggestion focusing in on those networking annotations in the checluster.spec.networking.annotations
Here's my issue.

microk8s default ingress-controller uses a "public" Ingress Class name

And with that my dev preview is routing!

I'll leave it up to you fine folks if you want to document or update chectl to make it easier for those behind me. I know there's pretty limited support for -p microk8s flag.

[tolusha]

@Mbd06b
Awesome job, thank you!

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.