Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(probeservices): httpx -> httpclientx #1576

Merged
merged 48 commits into from
Apr 30, 2024
Merged

refactor(probeservices): httpx -> httpclientx #1576

merged 48 commits into from
Apr 30, 2024

Conversation

bassosimone
Copy link
Contributor

@bassosimone bassosimone commented Apr 30, 2024

This issue replaces httpx with httpclientx for probeservices.

Part of ooni/probe#2723.


Here are some checks to make sure we' not changing API semantics.

GetTestHelpers

Before:

	err = c.APIClientTemplate.WithBodyLogging().Build().GetJSON(ctx, "/api/v1/test-helpers", &output)

After:

	// construct the URL to use
	URL, err := urlx.ResolveReference(c.BaseURL, "/api/v1/test-helpers", "")
	if err != nil {
		return nil, err
	}

	// get the response
	return httpclientx.GetJSON[map[string][]model.OOAPIService](ctx, URL, &httpclientx.Config{
		Client:    c.HTTPClient,
		Logger:    c.Logger,
		UserAgent: c.UserAgent,
	})

In both cases: uses the same client and user agent, uses the same URL path, uses logging.

OpenReport

Before:

	var cor model.OOAPICollectorOpenResponse
	if err := c.APIClientTemplate.WithBodyLogging().Build().PostJSON(ctx, "/report", rt, &cor); err != nil {

After:

	URL, err := urlx.ResolveReference(c.BaseURL, "/report", "")
	if err != nil {
		return nil, err
	}

	cor, err := httpclientx.PostJSON[model.OOAPIReportTemplate, *model.OOAPICollectorOpenResponse](
		ctx, URL, rt, &httpclientx.Config{
			Client:    c.HTTPClient,
			Logger:    c.Logger,
			UserAgent: c.UserAgent,
		},
	)

In both cases: uses the same client and user agent, uses the same URL path, uses logging.

SubmitMeasurement

Before:

	err := r.client.APIClientTemplate.WithBodyLogging().Build().PostJSON(
		ctx, fmt.Sprintf("/report/%s", r.ID), model.OOAPICollectorUpdateRequest{
			Format:  "json",
			Content: m,
		}, &updateResponse,

After:

	URL, err := urlx.ResolveReference(r.client.BaseURL, fmt.Sprintf("/report/%s", r.ID), "")
	if err != nil {
		return err
	}

	apiReq := model.OOAPICollectorUpdateRequest{
		Format:  "json",
		Content: m,
	}

	updateResponse, err := httpclientx.PostJSON[
		model.OOAPICollectorUpdateRequest, *model.OOAPICollectorUpdateResponse](
		ctx, URL, apiReq, &httpclientx.Config{
			Client:    r.client.HTTPClient,
			Logger:    r.client.Logger,
			UserAgent: r.client.UserAgent,
		},
	)

In both cases: uses the same client and user agent, uses the same URL path, uses logging.

Login

Before:

	var auth model.OOAPILoginAuth
	if err := c.APIClientTemplate.Build().PostJSON(
		ctx, "/api/v1/login", *creds, &auth); err != nil {

After:

	URL, err := urlx.ResolveReference(c.BaseURL, "/api/v1/login", "")
	if err != nil {
		return err
	}

	auth, err := httpclientx.PostJSON[*model.OOAPILoginCredentials, *model.OOAPILoginAuth](
		ctx, URL, creds, &httpclientx.Config{
			Client:    c.HTTPClient,
			Logger:    model.DiscardLogger,
			UserAgent: c.UserAgent,
		},
	)

In both cases: uses the same client and user agent, uses the same URL path, and we're not using logging.

MeasurementMeta

Before:

	var response model.OOAPIMeasurementMeta
	err := (&httpx.APIClientTemplate{
		BaseURL:    c.BaseURL,
		HTTPClient: c.HTTPClient,
		Logger:     c.Logger,
		UserAgent:  c.UserAgent,
	}).WithBodyLogging().Build().GetJSONWithQuery(ctx, "/api/v1/measurement_meta", query, &response)

After:

	// construct the URL to use
	URL, err := urlx.ResolveReference(c.BaseURL, "/api/v1/measurement_meta", query.Encode())
	if err != nil {
		return nil, err
	}
	return &response, nil

	// get the response
	return httpclientx.GetJSON[*model.OOAPIMeasurementMeta](ctx, URL, &httpclientx.Config{
		Client:    c.HTTPClient,
		Logger:    c.Logger,
		UserAgent: c.UserAgent,
	})

In both cases: uses the same client and user agent, uses the same URL path, and logging.

Psiphon

Before:

	client := c.APIClientTemplate.BuildWithAuthorization(s)
	return client.FetchResource(ctx, "/api/v1/test-list/psiphon-config")

After:

	// construct the URL to use
	URL, err := urlx.ResolveReference(c.BaseURL, "/api/v1/test-list/psiphon-config", "")
	if err != nil {
		return nil, err
	}

	// get response
	//
	// use a model.DiscardLogger to avoid logging config
	return httpclientx.GetRaw(ctx, URL, &httpclientx.Config{
		Authorization: s,
		Client:        c.HTTPClient,
		Logger:        model.DiscardLogger,
		UserAgent:     c.UserAgent,
	})

In both cases: uses the same client and user agent, uses the same URL path, and we're not logging.

Register

Before:

	var resp model.OOAPIRegisterResponse
	if err := c.APIClientTemplate.Build().PostJSON(
		ctx, "/api/v1/register", req, &resp); err != nil {

After:

	// construct the URL to use
	URL, err := urlx.ResolveReference(c.BaseURL, "/api/v1/register", "")
	if err != nil {
		return err
	}

	resp, err := httpclientx.PostJSON[*model.OOAPIRegisterRequest, *model.OOAPIRegisterResponse](
		ctx, URL, req, &httpclientx.Config{
			Client:    c.HTTPClient,
			Logger:    model.DiscardLogger,
			UserAgent: c.UserAgent,
		},
	)

In both cases: uses the same client and user agent, uses the same URL path, and we're not logging.

Tor

Before:

	client := c.APIClientTemplate.BuildWithAuthorization(s)
	err = client.GetJSONWithQuery(
		ctx, "/api/v1/test-list/tor-targets", query, &result)

After:

	// construct the URL to use
	URL, err := urlx.ResolveReference(c.BaseURL, "/api/v1/test-list/tor-targets", query.Encode())
	if err != nil {
		return nil, err
	}

	// get response
	//
	// use a model.DiscardLogger to avoid logging bridges
	return httpclientx.GetJSON[map[string]model.OOAPITorTarget](ctx, URL, &httpclientx.Config{
		Authorization: s,
		Client:        c.HTTPClient,
		Logger:        model.DiscardLogger,
		UserAgent:     c.UserAgent,
	})

In both cases: uses the same client and user agent, uses the same URL path, and we're not logging.

This diff includes a first attempt at consolidating the patterns with
which we invoke OONI and third-party API calls.

I have refactored the code of httpx and httpapi into a new package
called httpclientx, added some tests, started converting some parts
of the tree, and explained myself in a design document.

Part of ooni/probe#2700
Spotted thanks to a very paranoid check inside ./internal/oonirun.

Was not a problem before for `httpx` because of its usage pattern and may
or may not be a problem for the `httpapi` package (did not check since
this work is focused on replacing both `httpx` and `httpapi`).
Previously, we were gracefully handling this case, but honestly it is
not the best approach to pretend there's an empty structure if the server
breaks the API and returns `"null"` rather than an object.

That said, it was still awesome to have this test in place because it
helped us to figure out this extra condition of which httpclientx should
be aware and that this problem needs to be dealt with systematically
inside the httpclientx package.
Conflicts:
	internal/enginelocate/cloudflare.go
	internal/enginelocate/ubuntu.go
Conflicts:
	internal/oonirun/v2_test.go
As before, here I am going to ensure there's redundancy.
@bassosimone bassosimone marked this pull request as ready for review April 30, 2024 14:23
@bassosimone bassosimone requested a review from hellais as a code owner April 30, 2024 14:23
@bassosimone bassosimone merged commit ce62132 into master Apr 30, 2024
@bassosimone bassosimone deleted the issue/2723 branch April 30, 2024 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant