Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow max request size to be user-specified #4824

Merged
merged 4 commits into from
Jul 6, 2018
Merged

Conversation

jefferai
Copy link
Member

This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.

This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.

This turned out to be way more impactful than I'd expected because I
felt like the right granularity was per-listener, since an org may want
to treat external clients differently from internal clients. It's pretty
straightforward though.

This also introduces actually using request contexts for values, which
so far we have not done (using our own logical.Request struct instead),
but this allows non-logical methods to still get this benefit.
@jefferai jefferai added this to the 0.10.4 milestone Jun 24, 2018
@jefferai
Copy link
Member Author

Pretty trivially tested (the acutal limiting logic is the same, just the plumbing):

Limiting to smaller than default:

$ vault kv put -address=http://127.0.0.1:8200 secret/foo k=@CHANGELOG.md
Key              Value
---              -----
created_time     2018-06-24T16:27:25.632026852Z
deletion_time    n/a
destroyed        false
version          1

$ vault kv put -address=http://127.0.0.1:8202 secret/foo k=@CHANGELOG.md
Error writing data to secret/data/foo: Error making API request.

URL: PUT http://127.0.0.1:8202/v1/secret/data/foo
Code: 413. Errors:

* failed to parse JSON input: http: request body too large

$ truncate --size=21000 CHANGELOG.md 

$ vault kv put -address=http://127.0.0.1:8202 secret/foo k=@CHANGELOG.md
Key              Value
---              -----
created_time     2018-06-24T16:28:44.109742609Z
deletion_time    n/a
destroyed        false
version          2

Unlimited (ignore that I accidentally used vault write, doesn't matter):

$ vault write -address=http://127.0.0.1:8200 secret/foo k=@bin/vault
Error writing data to secret/foo: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/secret/foo
Code: 413. Errors:

* failed to parse JSON input: http: request body too large

$ vault write -address=http://127.0.0.1:8202 secret/foo k=@bin/vault
Error writing data to secret/foo: Error making API request.

URL: PUT http://127.0.0.1:8202/v1/secret/foo
Code: 404. Errors:


WARNING! The following warnings were returned from Vault:

  * Invalid path for a versioned K/V secrets engine. See the API docs for the
  appropriate API endpoints to use. If using the Vault CLI, use 'vault kv put'
  for this operation.

}
}

buf := bytes.NewBuffer(nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could use ioutil.ReadAll() here since you want []byte later.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair, I just kept that part of the logic that was there. Feel free to pull down the branch and change it yourself if you like, too!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@jefferai
Copy link
Member Author

Thanks!

@@ -792,7 +808,9 @@ CLUSTER_SYNTHESIS_COMPLETE:
// This needs to happen before we first unseal, so before we trigger dev
// mode if it's set
core.SetClusterListenerAddrs(clusterAddrs)
core.SetClusterHandler(vaulthttp.Handler(core))
core.SetClusterHandler(vaulthttp.Handler(&vault.HandlerProperties{
Core: core,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't we ever need a size limit on the cluster connection?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the time we get to the HTTP handler we've already handshaked the cluster connection so we already know if the connection is authorized -- if so we shouldn't limit it.

return errors.New("could not parse max_request_size from request context")
}
if max > 0 {
reader = http.MaxBytesReader(w, r.Body, max)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why do we use io.LimitReader above and http.MaxBytesReader here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Different function signatures, based on what's available.

@@ -26,6 +26,13 @@ const (
replTimeout = 10 * time.Second
)

// HanlderProperties is used to seed configuration into a vaulthttp.Handler.
// It's in this package to avoid a circular dependency
type HandlerProperties struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nitpick] Can we call this HandlerOptions instead, unless there was a reason to call this HandlerProperties?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I called it Properties since it's not just user-set options (e.g. Core).

@jefferai jefferai merged commit 5a2d80e into master Jul 6, 2018
@jefferai jefferai deleted the limit-request-size-user branch July 6, 2018 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants