Way to Read Request Body As String #459
Comments
|
Something like this should work: defmodule MyApp.Router
...
pipeline :browser do
plug :copy_req_body
plug :super
end
...
defp copy_req_body(conn, _) do
Plug.Conn.put_private(conn, :my_app_body, Plug.Con..read_body(conn))
end
end
defmodule MyAppController do
...
def show(conn, params) do
the_body = conn.private[:my_app_body]
end
endNote we don't update the |
|
@chrismccord thanks for the help. I had to modify your answer a bit (as my app is an API not a backend for a browser application). It still didn't work, let me know if my answer should work or if I messed something up: defmodule MyApp.Router
...
pipeline :api do
plug :copy_req_body
plug :super
end
pipe_through :api
...
end
# The rest is the sameInside of |
|
@rylev ah, please try this: defmodule MyApp.Router
...
pipeline :before do
plug :copy_req_body
plug :super
end
...
end |
|
@chrismccord that works :-) thanks! Just wondering: Is this the best way to handle this use case? I'm guessing the need for the raw body is seldom enough that this work around is fine, but could there be a nice API for optionally storing the raw body on the |
|
We should at least come up with a way to make it more performant since you don't need to read the request body on every single request. The easiest solution would be to match on the path_info. Let me get back to my desk and I'll have a better solution. |
|
Something like this would be a better approach outside of a mechanism in plug to keep the raw body around. This dance only reads the body twice for the API stack. Not beautiful, but I think it works for this less common use-case. Please reopen if you have issues. Thanks! defmodule MyApp.Router
...
@api_mount "/api"
pipeline :before do
plug MyApp.Plugs.CopyReqBody, mount: @api_mount
plug :super
end
scope @api_mount do
pipe_through :api
...
end
end
defmodule MyApp.Plugs.CopyReqBody do
import Plug.Conn
def init(mount: "/" <> mount), do: mount
defp call(%Plug.Conn{path_info: [mount | _rest]}, mount) do
put_private(conn, :my_app_body, read_body(conn))
end
defp call(conn, _), do: conn
end
defmodule MyAppController do
...
def show(conn, params) do
the_body = conn.private[:my_app_body]
end
end |
|
Is this still a valid approach? |
|
yup |
|
I'm a little confused about pipeline :before do
plug MyApp.Plugs.CopyReqBody, mount: @api_mount
plug :super
endAre these a special pipeline and plug that don't exist anymore? Or should I be including When I just add it to the existing pipelines, I get the body as an empty string, as we reported before you suggested |
|
Sorry, ignore |
|
Ahhh. That did the trick. For future reference by the community, a bare minimum solution (in the defp copy_req_body(conn, _) do
{:ok, body, _} = Plug.Conn.read_body(conn)
Plug.Conn.put_private(conn, :raw_body, body)
end
plug :copy_req_body
plug Plug.Parsers,
parsers: [:urlencoded, :multipart, :json],
pass: ["*/*"],
json_decoder: PoisonAs stated, that doesn't let you filter on only certain endpoints and can be pretty wasteful, but it's enough of a bare minimum example that anyone should be able to work with it. Thanks! |
|
New problem encountered: This only works with a body size of 1024 or less (it hangs with 1025). Not capturing the body works fine with > 1024, but if we capture the body we hang when we try to parse it later. |
|
Seems like you can configure the length and read length for Plug.Conn.read_body/2, but the defaults should be fine for this case. Is it possible that you are changing the defaults somewhere? http://hexdocs.pm/plug/Plug.Conn.html#read_body/2 Edit: "Is it possible that the defaults are being reset somewhere." |
|
My hypothesis is that read_body can only be called once. However, if the body is less than 1024, it's prefetched and then the read-once doesn't apply. However, once you have to fetch the next pit of the body, the first 1024 is thrown out. This causes the later read_body to timeout. Still digging through source code to confirm, but "Because the request body can be of any size, reading the body will only work once, as Plug will not cache the result of these operations." |
|
I tend to concur with @hamiltop. I was testing something very similar (reading & storing the entire body), and was trying to get it to support when |
|
I added the code suggested above and facing same :timeout issue as @hamiltop & @ksol. Stack trace More context available on http://stackoverflow.com/questions/33637418/elixir-phoenix-on-heroku-timeout-error-after-15-seconds and ninenines/cowboy#833 (comment) Source Code: https://github.com/endeepak/stub_on_web |
|
@endeepak To be clear, this only happens to you when you try to read/copy the request body, and does not happen if you use normal Plug.Parsers without the custom body copy? |
|
@chrismccord Yes. This issue started after adding code to copy request body. Other observations which might help
|
|
The "workaround" for now would be to not copy the request body. What is your usecase for needing the raw body? Also, I'm not positive this is a real cowboy issue without digging into how plug itself chunks the body reads. |
|
StubOnWeb is a stub http server which needs to record request and response for calls made to stubbed urls. Hence the workaround to not copy won't help. I'll dig into the code and try few stuff. Thanks for the help. If I find anything useful, I'll update here. |
|
My use case was signed requests from github webhooks. I needed the raw body On Tue, Nov 10, 2015 at 9:14 PM Deepak Narayana Rao <
|
|
My use case is actually the same as @hamiltop's. GH Webhooks are a blast! /s |
|
@hunterboerner https://github.com/hamiltop/ashes/blob/master/lib/ashes/github_webhook_plug.ex might be a good reference for you then. |
|
Perhaps the simplest short-term solution would be to have a custom parser(s) that copies the body? plug Plug.Parsers,
parsers: [MyApp.Parsers.URLENCODED, :multipart, :json],
pass: ["*/*"],
json_decoder: Poison
...
defmodule MyApp.Parsers.URLENCODED do
@moduledoc """
Parses urlencoded request body, and optionally copies the raw body
Copies raw body to `:raw_body` private assign when `:copy_raw_body`
private assign is true
"""
@behaviour Plug.Parsers
alias Plug.Conn
def parse(conn, "application", "x-www-form-urlencoded", _headers, opts) do
case Conn.read_body(conn, opts) do
{:ok, body, conn} ->
Plug.Conn.Utils.validate_utf8!(body, "urlencoded body")
decoded_body = Plug.Conn.Query.decode(body)
if conn.private[:copy_raw_body] do
{:ok, decoded_body, Plug.Conn.put_private(conn, :raw_body, body)}
else
{:ok, decoded_body, conn}
end
{:more, _data, conn} ->
{:error, :too_large, conn}
end
end
def parse(conn, _type, _subtype, _headers, _opts) do
{:next, conn}
end
end |
|
Thanks @chrismccord, your solution works for now. I'll keep an eye on this thread for the proper fix. For reference, commit with this fix endeepak/stub_on_web@4719255 |
|
I'm having trouble with this when implementing a stripe webhook and handing authentication. The body appears to be empty, which is needed for https://github.com/sikanhe/stripe-elixir Any ideas? |
|
@ch-andrewrhyne here's a super rough implementation of github webhooks that reads the body. it's just a Plug, no phoenix, but does the job: https://github.com/hamiltop/ashes/blob/master/lib/ashes/github_webhook_plug.ex |
|
Useful, but my issue is that I don't know how to access the raw body on the |
|
@ch-andrewrhyne here is the solution I use : https://gist.github.com/TechMagister/1a01b22c205309238617fb1fef1603a4 It's not perfect but it's the less intrusive way I found. |
|
@ch-andrewrhyne did you get this working? I am running into the same issue |
|
Sharing my solution for those working with stripe webhooks: https://gist.github.com/atomkirk/e05fbab86f34331ffe812a1b98f63851 |
|
Sharing my solution in case someone is looking for a lite-touch way to do this. My use case was similar to others here: verifying a Shopify request. I added a plug in In |
|
Thanks @curthasselschwert! Note you can use path.info instead of request path and that should be quite faster: Also note that Plug v1.6 will allow this to be done via an option to Plug.Parsers. |
|
I have encountered this problem as well. @curthasselschwert when I try |
|
@nitin21 the body can only be read once, and it is usually done in a Parser plug. Please see the new-ish https://hexdocs.pm/plug/Plug.Parsers.html#module-custom-body-reader |
|
My problem was interesting and solved using the above information. I will state it here incase something else is going on in the guts of handling a web request. These failed requests would not be logged by Phoenix - even adding onrequest and onresponse Cowboy hooks did not log the request. I cannot explain this but the fix was to create an explicate Another strange thing I noticed, (and only behind an AWS ALB / ECS) if you POST application/xml (with no explicate MIME handler / rejection), and never do a In local mode (curl to Docker/Phoenix) none of these errors could be replicated. |
|
For future searchers, Plug now has an elegant solution to this problem which was added by this PR: elixir-plug/plug#698 which allows you to do the following defmodule AuditorWeb.CacheBodyReader do
def read_body(conn, opts) do
{:ok, body, conn} = Plug.Conn.read_body(conn, opts)
body = [body | conn.private[:raw_body] || []]
conn = Plug.Conn.put_private(conn, :raw_body, body)
{:ok, body, conn}
end
def read_cached_body(conn) do
conn.private[:raw_body]
end
end
# endpoint.ex
plug Plug.Parsers,
parsers: [:urlencoded, :multipart, :json],
pass: ["*/*"],
body_reader: {AuditorWeb.CacheBodyReader, :read_body, []},
json_decoder: Phoenix.json_library() |
|
@minhajuddin this was a great end to this thread as I, too, was bit by the "verify request" problem. In your example above, what's this line doing: I ask because Thanks for posting this - was about to go down a very dark path! |
|
@coladarci So, for larger requests this function will be called multiple times as the request is read over multiple calls to |
|
Right, but won't |
|
hmm, you are right it would blow up, that code might need some rework :) |
If the body of a request is just JSON one could just decode the map within the controller instead of trapping it in a plug. This means it's simple and can be done in the controller. defmodule MyAppWeb.RawJsonController do
# ...
def get_raw(conn, params) do
raw_body = params |> Jason.encode!
end
endKind of a waste to backtrace but if the body expected is small then it's an option. |
|
@jochasinga - this is an entire thread around exactly that question and the answers right above your comment are still accurate. To provide an update on where we ended up: While the out of the box phoenix puts parsing inside the endpoint, we decided to move that parsing into our router so we can apply different parsing for different groups of routes via pipe_through. For example, a set of routes that all need to do some stupid verification of the raw body (note @jochasinga the raw body is NOT guaranteed to be the same as re-encoding the params), we can then add our custom plug to that pipeline to save the raw body for later use: And then: At which point, anywhere who has a conn (i.e for you, in your controller, or for us, in our auth plugs) can access the raw body by simply calling It's also worth noting that there are reasons Phoenix didn't opt to keep this raw body around, so if you only need it in certain places, it's probably wise to only add this special parsing to those places. Hope this helps! |
|
@coladarci can you please define
I understand this but in case of JSON they are the same. And this means you won't have to carry the raw body around because you only re-decode it once for that controller. I wanted to include it for completeness, but I did acknowledged the plug solution and even go with it for my project. |
|
What I meant was that the raw body has escaped spaces, newlines and the order of the keys is whatever the client sent, etc. The moment you parse it, you lose most, if not all, of that information. |
|
For those who end up here trying to understand why their body was missing some whitespace: For me it was because I tested with curl and its |
|
In our use case we need to verify an incoming webhook request by doing SHA256(app secret + raw request body). Since we just dump the webhooks and process them asynchronously, we need a way to store the raw body. Using the suggestions above, here's what we did:
|
@hamiltop Thank you. You saved me some hours. |
…ss-origin issues you cannot send application/json without chrome trying to send a preflight OPTIONS request, which blows up... so we need to either a) figure out what to do to allow cross-origin requests (not sure where phoenix sees the request as coming from at the moment, it happens inside of the chrome extension backgroud script) or b) figure out how to read the request body as plain text (phoenix doesnt make this that easy... heres an issue that talks about how to do it phoenixframework/phoenix#459)
|
I used this to drop some url parameters and works fine body = Map.drop(params, ["project", "environment"]) |> Jason.encode!You need to install https://github.com/michalmuskala/jason |
|
What would be a reliable way to do the same but for |
and others
I need to a way to access the body of a request as a raw string. Plug supports this functionality (https://github.com/elixir-lang/plug/blob/master/lib/plug/conn.ex#L418-L448) but as the docs say this can only be done once. Once the body is consumed, trying to read the body will result in an empty string.
I believe that if you use a Phoenix controller, reading the body as string is impossible. If a client sends a request with the content-type set to application/json Phoenix will automatically parse the body as params meaning any future attempts by client code to access the body will result in an empty string.
Is there any way for client code inside of a controller to access the body as a raw string?
The text was updated successfully, but these errors were encountered: