Query: 404s from api path when using web.route-prefix in 0.12.0

[gburek-fastly]

After an upgrade to Thanos 0.12.0, the query component only serves thanos UI, but any API requests to the /api/ path that uses the web.route-prefix, will return a 404 error.

Thanos, Prometheus and Golang version used:

$ ./thanos --version
thanos, version 0.12.0 (branch: HEAD, revision: 69f9c4e604d20b2e640f790957414cf3d19ecafa)
  build user:       root@aab79ed2e8d5
  build date:       20200415-09:36:49
  go version:       go1.13.6

What happened:

Thanos is configured with the following cli args:

$ ./thanos query --http-address=127.0.0.1:9090 --web.route-prefix=/thanos --web.external-prefix=/thanos

Current behavior against a 0.12.0 thanos binary:

$ curl "localhost:9090/thanos/"
<a href="/thanos/graph">Found</a>.
$ curl "localhost:9090/thanos/api/v1/query?query=time()&_=1587411669454"
404 page not found
$  curl "localhost:9090/thanos/api/v1/label/__name__/values?_=1587417389503"
404 page not found

Removing the web.route-prefix from the cli and curl path restores functionality for a modified path:

$ ./thanos query --http-address=127.0.0.1:9090 

$ curl "localhost:9090/"
<a href="/graph">Found</a>.
$ curl "localhost:9090/api/v1/query?query=time()&_=1587411669454"
{"status":"success","data":{"resultType":"scalar","result":[1587416528.202,"1587416528.202"]}}

In a browser, a red banner appears with the message: Error loading available metrics!

What you expected to happen:

Expected behavior would be to match the 0.11.0 thanos behavior:

$ ./thanos query --http-address=127.0.0.1:9090 --web.route-prefix=/thanos --web.external-prefix=/thanos

$ curl "localhost:9090/thanos/"
<a href="/thanos/graph">Found</a>.
$ curl "localhost:9090/thanos/api/v1/query?query=time()&_=1587411669454"
{"status":"success","data":{"resultType":"scalar","result":[1587414822.982,"1587414822.982"]}}
$  curl "localhost:9090/thanos/api/v1/label/__name__/values?_=1587417389503"
{"status":"success","data":null}

We use this functionality to host the thanos query endpoints behind an nginx reverse proxy. These curl urls appear to be sent by default on Thanos UI load and seem to be a basic test of the api path.

How to reproduce it (as minimally and precisely as possible):

Released 0.12.0 and 0.11.0 binaries, on linux or darwin, when run with --web.route-prefix=/thanos --web.external-prefix=/thanos will fail to resolve a simple time() query in the browser ui at or via curl, as above.

Anything else we need to know:

#2208 appears suspicious and may be related to this regression.

[yeya24]

I can reproduce this locally. Thanks for the report!

[yeya24]

From the code, seems we wrap the web route prefix twice, so the web endpoint becomes http://localhost:10904/thanos/thanos/api/v1/label/__name__/values

		if webRoutePrefix != "/" {
			router.Get("/", func(w http.ResponseWriter, r *http.Request) {
				http.Redirect(w, r, webRoutePrefix, http.StatusFound)
			})
			router = router.WithPrefix(webRoutePrefix)
		}

		ins := extpromhttp.NewInstrumentationMiddleware(reg)
		// TODO(bplotka in PR #513 review): pass all flags, not only the flags needed by prefix rewriting.
		ui.NewQueryUI(logger, reg, stores, webExternalPrefix, webPrefixHeaderName).Register(router, ins)

		api := v1.NewAPI(logger, reg, engine, queryableCreator, enableAutodownsampling, enablePartialResponse, replicaLabels, instantDefaultMaxSourceResolution)

		api.Register(router.WithPrefix(path.Join(webRoutePrefix, "/api/v1")), tracer, logger, ins)

[yeya24]

@gburek-fastly I opened a PR here #2484 It would be appreciated if you can test it in your environment

[gburek-fastly]

@yeya24 confirmed in the ui and with my curl tests

thanos-io/thanos ‹yeya24› » ./thanos query --http-address=127.0.0.1:9090 --web.route-prefix=/thanos --web.external-prefix=/thanos

» curl "localhost:9090/thanos/"
<a href="/thanos/graph">Found</a>.
» curl "localhost:9090/thanos/api/v1/query?query=time()&_=1587411669454"
{"status":"success","data":{"resultType":"scalar","result":[1587421890.119,"1587421890.119"]}}
 » curl "localhost:9090/thanos/api/v1/label/__name__/values?_=1587417389503"
{"status":"success","data":null}

[bwplotka]

Let's keep one issue open for this (:

Since @yeya24 is handling this, let's keep his open - let's continue the discussion there. Thanks for report! Glad we found it now, we are just cutting v0.12.1... cc @squat (:

Marking as a duplicate of #2486