diff --git a/CHANGELOG.md b/CHANGELOG.md index 8be6e689..7f2a70f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,23 @@ +## Version 13.13.0 + +### Auditing max body length + +Payloads will be truncated in audit logs if they exceed the max supported (as configured by `http-verbs.auditing.maxBodyLength`). + +### WSProxyConfiguration + +`WSProxyConfiguration.apply` has been deprecated, use `WSProxyConfiguration.buildWsProxyServer` instead. + +There are some differences with `WSProxyConfiguration.buildWsProxyServer`: + * configPrefix is fixed to `proxy`. + * `proxy.proxyRequiredForThisEnvironment` has been replaced with `http-verbs.proxy.enabled`, but note, it defaults to false (rather than true). This is appropriate for development and tests, but will need explicitly enabling when deployed. + + +### Adds HttpClientV2 +This is in addition to `HttpClient` (for now), so can be optionally used instead. + +See [README](/README.md) for details. + ## Version 13.12.0 ### Supported Play Versions diff --git a/README.md b/README.md index a28024c4..61da2d69 100644 --- a/README.md +++ b/README.md @@ -33,12 +33,89 @@ Where `play-xx` is your version of Play (e.g. `play-28`). ## Usage +There are two HttpClients available. + +### uk.gov.hmrc.http.HttpClient + Examples can be found [here](https://github.com/hmrc/http-verbs/blob/master/http-verbs-test-common/src/test/scala/uk/gov/hmrc/http/examples/Examples.scala) -### URLs +URLs can be supplied as either `java.net.URL` or `String`. We recommend supplying `java.net.URL` and using the provided [URL interpolator](#url-interpolator) for correct escaping of query and path parameters. + + +### uk.gov.hmrc.http.client.HttpClientV2 + +This client follows the same patterns as `HttpClient` - that is, it also requires a `HeaderCarrier` to represent the context of the caller, and an `HttpReads` to process the http response. + +In addition, it: +- Supports streaming +- Exposes the underlying `play.api.libs.ws.WSRequest` with `transform`, making it easier to customise the request. +- Only accepts the URL as `java.net.URL`; you can make use of the provided [URL interpolator](#url-interpolator). + +Examples can be found in [here](/http-verbs-common/src/test/scala/uk/gov/hmrc/http/client/HttpClientV2Spec.scala) + +To migrate: + +```scala +httpClient.GET[ResponseType](url) +``` + +becomes + +```scala +httpClientV2.get(url"$url").execute[ResponseType] +``` + +and + +```scala +httpClient.POST[ResponseType](url, payload, headers) +``` + +becomes + +```scala +httpClientV2.post(url"$url").withBody(Json.toJson(payload)).addHeaders(headers).execute[ResponseType] +``` + -URLs can be supplied as either `java.net.URL` or `String`. We recommend supplying `java.net.URL` for correct escaping of query and path parameters. A [URL interpolator](https://sttp.softwaremill.com/en/latest/model/uri.html) has been provided for convenience. +#### Header manipulation +With `HttpClient`, replacing a header can require providing a customised client implementation (e.g. to replace the user-agent header), or updating the `HeaderCarrier` (e.g. to replace the authorisation header). This can now all be done with the `replaceHeader` on `HttpClientV2` per call. e.g. + +```scala +httpClientV2.get(url"$url").replaceHeader("User-Agent" -> userAgent).replaceHeader("Authorization" -> authorization).execute[ResponseType] +``` + +#### Using proxy + +With `HttpClient`, to use a proxy requires creating a new instance of HttpClient to mix in `WSProxy` and configure. With `HttpClientV2` this can be done with the same client, calling `withProxy` per call. e.g. + +```scala +httpClientV2.get(url"$url").withProxy.execute[ResponseType] +``` + +* It uses `WSProxyConfiguration.buildWsProxyServer` which needs enabling with `http-verbs.proxy.enabled` in configuration, which by default is `false`, for development. See [WSProxyConfiguration](CHANGELOG.md#wsproxyconfiguration) for configuration changes. + +#### Streaming + +Streaming is supported with `HttpClientV2`, and will be audited in the same way as `HttpClient`. Note that payloads will be truncated in audit logs if they exceed the max supported (as configured by `http-verbs.auditing.maxBodyLength`). + +Streamed requests can simply be passed to `withBody`: + +```scala +val reqStream: Source[ByteString, _] = ??? +httpClientV2.post(url"$url").withBody(reqStream).execute[ResponseType] +``` + +For streamed responses, use `stream` rather than `execute`: + +```scala +httpClientV2.get(url"$url").stream[Source[ByteString, _]] +``` + +### URL interpolator + +A [URL interpolator](https://sttp.softwaremill.com/en/latest/model/uri.html) has been provided to help with escaping query and parameters correctly. ```scala import uk.gov.hmrc.http.StringContextOps @@ -46,6 +123,7 @@ import uk.gov.hmrc.http.StringContextOps url"http://localhost:8080/users/${user.id}?email=${user.email}" ``` + ### Headers #### Creating HeaderCarrier @@ -120,7 +198,7 @@ class MyConnectorSpec extends WireMockSupport with GuiceOneAppPerSuite { } ``` -The `HttpClientSupport` trait can provide an instance of HttpClient as an alternative to instanciating the application: +The `HttpClientSupport` trait can provide an instance of `HttpClient` as an alternative to instanciating the application: ```scala class MyConnectorSpec extends WireMockSupport with HttpClientSupport { private val connector = new MyConnector( @@ -130,6 +208,8 @@ class MyConnectorSpec extends WireMockSupport with HttpClientSupport { } ``` +Similarly `HttpClientV2Support` can be used to provide an instance of `HttpClientV2`. + The `ExternalWireMockSupport` trait is an alternative to `WireMockSupport` which uses `127.0.0.1` instead of `localhost` for the hostname which is treated as an external host for header forwarding rules. This should be used for tests of connectors which call endpoints external to the platform. The variable `externalWireMockHost` (or `externalWireMockUrl`) should be used to provide the hostname in configuration. Both `WireMockSupport` and `ExternalWireMockSupport` can be used together for integration tests if required. diff --git a/build.sbt b/build.sbt index 7bacd961..d59dcf5e 100644 --- a/build.sbt +++ b/build.sbt @@ -59,6 +59,7 @@ lazy val sharedSources = shareSources("http-verbs-common") lazy val httpVerbsPlay28 = Project("http-verbs-play-28", file("http-verbs-play-28")) + .enablePlugins(BuildInfoPlugin) .settings( commonSettings, sharedSources, @@ -70,6 +71,10 @@ lazy val httpVerbsPlay28 = Project("http-verbs-play-28", file("http-verbs-play-2 AppDependencies.coreTestPlay28, Test / fork := true // akka is not unloaded properly, which can affect other tests ) + .settings( // https://github.com/sbt/sbt-buildinfo + buildInfoKeys := Seq[BuildInfoKey](version), + buildInfoPackage := "uk.gov.hmrc.http" + ) .dependsOn(httpVerbs) lazy val sharedTestSources = diff --git a/http-verbs-common/src/main/resources/reference.conf b/http-verbs-common/src/main/resources/reference.conf index b864a88d..09e41aa4 100644 --- a/http-verbs-common/src/main/resources/reference.conf +++ b/http-verbs-common/src/main/resources/reference.conf @@ -14,7 +14,13 @@ internalServiceHostPatterns = [ "^.*\\.service$", "^.*\\.mdtp$", "^localhost$" ] bootstrap.http.headersAllowlist = [] -http-verbs.retries { - intervals = [ "500.millis", "1.second", "2.seconds", "4.seconds", "8.seconds" ] - ssl-engine-closed-already.enabled = false +http-verbs { + retries { + intervals = [ "500.millis", "1.second", "2.seconds", "4.seconds", "8.seconds" ] + ssl-engine-closed-already.enabled = false + } + + auditing.maxBodyLength = 32665 + + proxy.enabled = false } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpDelete.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpDelete.scala index 7b7d3a88..955805f0 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpDelete.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpDelete.scala @@ -34,8 +34,8 @@ trait HttpDelete override def DELETE[O](url: String, headers: Seq[(String, String)] = Seq.empty)(implicit rds: HttpReads[O], hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(DELETE_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(DELETE_VERB, url)(doDelete(url, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(DELETE_VERB, url)(doDelete(url, allHeaders)) executeHooks(DELETE_VERB, url"$url", allHeaders, None, httpResponse) mapErrors(DELETE_VERB, url, httpResponse).map(rds.read(DELETE_VERB, url, _)) } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpErrorFunctions.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpErrorFunctions.scala index bc412463..995d3a05 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpErrorFunctions.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpErrorFunctions.scala @@ -16,6 +16,11 @@ package uk.gov.hmrc.http +import akka.stream.Materializer + +import scala.concurrent.{Await, TimeoutException} +import scala.concurrent.duration.{Duration, DurationInt} + trait HttpErrorFunctions { def notFoundMessage(verbName: String, url: String, responseBody: String): String = s"$verbName of '$url' returned 404 (Not Found). Response body: '$responseBody'" @@ -77,6 +82,51 @@ trait HttpErrorFunctions { // default followRedirect should mean we don't see 3xx... case status => Right(response) } + + /* Same as `handleResponseEither` but should be used when reading the `HttpResponse` as a stream. + * The error is returned as `Source[UpstreamErrorResponse, _]`. + */ + def handleResponseEitherStream( + httpMethod: String, + url : String + )( + response: HttpResponse + )(implicit + mat : Materializer, + errorTimeout: ErrorTimeout + ): Either[UpstreamErrorResponse, HttpResponse] = + response.status match { + case status if is4xx(status) || is5xx(status) => + Left { + val errorMessageF = + response.bodyAsSource.runFold("")(_ + _.utf8String) + val errorMessage = + // this await is unfortunate, but HttpReads doesn't support Future + try { + Await.result(errorMessageF, errorTimeout.toDuration) + } catch { + case e: TimeoutException => "" + } + UpstreamErrorResponse( + message = upstreamResponseMessage(httpMethod, url, status, errorMessage), + statusCode = status, + reportAs = if (is4xx(status)) HttpExceptions.INTERNAL_SERVER_ERROR else HttpExceptions.BAD_GATEWAY, + headers = response.headers + ) + } + // Note all cases not handled above (e.g. 1xx, 2xx and 3xx) will be returned as is + // default followRedirect should mean we don't see 3xx... + case status => Right(response) + } } object HttpErrorFunctions extends HttpErrorFunctions + +case class ErrorTimeout( + toDuration: Duration +) extends AnyVal + +object ErrorTimeout { + implicit val errorTimeout: ErrorTimeout = + ErrorTimeout(10.seconds) +} diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpGet.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpGet.scala index d54bdfdb..2b32713a 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpGet.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpGet.scala @@ -45,8 +45,8 @@ trait HttpGet extends CoreGet with GetHttpTransport with HttpVerb with Connectio val urlWithQuery = url + makeQueryString(queryParams) withTracing(GET_VERB, urlWithQuery) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(GET_VERB, urlWithQuery)(doGet(urlWithQuery, headers = allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(GET_VERB, urlWithQuery)(doGet(urlWithQuery, headers = allHeaders)) executeHooks(GET_VERB, url"$url", allHeaders, None, httpResponse) mapErrors(GET_VERB, urlWithQuery, httpResponse).map(response => rds.read(GET_VERB, urlWithQuery, response)) } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPatch.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPatch.scala index c9d0002b..34506a6c 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPatch.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPatch.scala @@ -42,8 +42,8 @@ trait HttpPatch hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(PATCH_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(PATCH_VERB, url)(doPatch(url, body, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(PATCH_VERB, url)(doPatch(url, body, allHeaders)) executeHooks(PATCH_VERB, url"$url", allHeaders, Option(HookData.FromString(Json.stringify(wts.writes(body)))), httpResponse) mapErrors(PATCH_VERB, url, httpResponse).map(response => rds.read(PATCH_VERB, url, response)) } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPost.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPost.scala index 1b542947..a734fb3e 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPost.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPost.scala @@ -42,8 +42,8 @@ trait HttpPost hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(POST_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(POST_VERB, url)(doPost(url, body, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(POST_VERB, url)(doPost(url, body, allHeaders)) executeHooks(POST_VERB, url"$url", allHeaders, Option(HookData.FromString(Json.stringify(wts.writes(body)))), httpResponse) mapErrors(POST_VERB, url, httpResponse).map(rds.read(POST_VERB, url, _)) } @@ -56,8 +56,8 @@ trait HttpPost hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(POST_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(POST_VERB, url)(doPostString(url, body, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(POST_VERB, url)(doPostString(url, body, allHeaders)) executeHooks(POST_VERB, url"$url", allHeaders, Option(HookData.FromString(body)), httpResponse) mapErrors(POST_VERB, url, httpResponse).map(rds.read(POST_VERB, url, _)) } @@ -70,8 +70,8 @@ trait HttpPost hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(POST_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(POST_VERB, url)(doFormPost(url, body, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(POST_VERB, url)(doFormPost(url, body, allHeaders)) executeHooks(POST_VERB, url"$url", allHeaders, Option(HookData.FromMap(body)), httpResponse) mapErrors(POST_VERB, url, httpResponse).map(rds.read(POST_VERB, url, _)) } @@ -83,8 +83,8 @@ trait HttpPost hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(POST_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(POST_VERB, url)(doEmptyPost(url, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(POST_VERB, url)(doEmptyPost(url, allHeaders)) executeHooks(POST_VERB, url"$url", allHeaders, None, httpResponse) mapErrors(POST_VERB, url, httpResponse).map(rds.read(POST_VERB, url, _)) } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPut.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPut.scala index 7ce60590..6711cf49 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPut.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpPut.scala @@ -36,8 +36,8 @@ trait HttpPut extends CorePut with PutHttpTransport with HttpVerb with Connectio hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(PUT_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(PUT_VERB, url)(doPut(url, body, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(PUT_VERB, url)(doPut(url, body, allHeaders)) executeHooks(PUT_VERB, url"$url", allHeaders, Option(HookData.FromString(Json.stringify(wts.writes(body)))), httpResponse) mapErrors(PUT_VERB, url, httpResponse).map(response => rds.read(PUT_VERB, url, response)) } @@ -50,8 +50,8 @@ trait HttpPut extends CorePut with PutHttpTransport with HttpVerb with Connectio hc: HeaderCarrier, ec: ExecutionContext): Future[O] = withTracing(PUT_VERB, url) { - val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) - val httpResponse = retry(PUT_VERB, url)(doPutString(url, body, allHeaders)) + val allHeaders = HeaderCarrier.headersForUrl(hcConfig, url, headers) :+ "Http-Client-Version" -> BuildInfo.version + val httpResponse = retryOnSslEngineClosed(PUT_VERB, url)(doPutString(url, body, allHeaders)) executeHooks(PUT_VERB, url"$url", allHeaders, Option(HookData.FromString(body)), httpResponse) mapErrors(PUT_VERB, url, httpResponse).map(rds.read(PUT_VERB, url, _)) } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpReadsLegacyInstances.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpReadsLegacyInstances.scala index cc4012fd..d1b8e5b7 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpReadsLegacyInstances.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpReadsLegacyInstances.scala @@ -16,50 +16,46 @@ package uk.gov.hmrc.http -import com.github.ghik.silencer.silent -import play.api.libs.json -import play.api.libs.json.{JsNull, JsValue} +import play.api.libs.json.{JsNull, JsValue, Reads} trait HttpReadsLegacyInstances extends HttpReadsLegacyOption with HttpReadsLegacyJson trait HttpReadsLegacyRawReads extends HttpErrorFunctions { @deprecated("Use uk.gov.hmrc.http.HttpReads.Implicits instead. See README for differences.", "11.0.0") - implicit val readRaw: HttpReads[HttpResponse] = new HttpReads[HttpResponse] { - @silent("deprecated") - def read(method: String, url: String, response: HttpResponse) = handleResponse(method, url)(response) - } + implicit val readRaw: HttpReads[HttpResponse] = + (method: String, url: String, response: HttpResponse) => + handleResponse(method, url)(response) } object HttpReadsLegacyRawReads extends HttpReadsLegacyRawReads trait HttpReadsLegacyOption extends HttpErrorFunctions { @deprecated("Use uk.gov.hmrc.http.HttpReads.Implicits instead. See README for differences.", "11.0.0") - implicit def readOptionOf[P](implicit rds: HttpReads[P]): HttpReads[Option[P]] = new HttpReads[Option[P]] { - def read(method: String, url: String, response: HttpResponse) = response.status match { - case 204 | 404 => None - case _ => Some(rds.read(method, url, response)) - } - } + implicit def readOptionOf[P](implicit rds: HttpReads[P]): HttpReads[Option[P]] = + (method: String, url: String, response: HttpResponse) => + response.status match { + case 204 | 404 => None + case _ => Some(rds.read(method, url, response)) + } } trait HttpReadsLegacyJson extends HttpErrorFunctions { @deprecated("Use uk.gov.hmrc.http.HttpReads.Implicits instead. See README for differences.", "11.0.0") - implicit def readFromJson[O](implicit rds: json.Reads[O], mf: Manifest[O]): HttpReads[O] = new HttpReads[O] { - def read(method: String, url: String, response: HttpResponse) = + implicit def readFromJson[O](implicit rds: Reads[O], mf: Manifest[O]): HttpReads[O] = + (method: String, url: String, response: HttpResponse) => readJson(method, url, handleResponse(method, url)(response).json) - } @deprecated("Use uk.gov.hmrc.http.HttpReads.Implicits instead. See README for differences.", "11.0.0") - def readSeqFromJsonProperty[O](name: String)(implicit rds: json.Reads[O], mf: Manifest[O]) = new HttpReads[Seq[O]] { - def read(method: String, url: String, response: HttpResponse) = response.status match { - case 204 | 404 => Seq.empty - case _ => - readJson[Seq[O]](method, url, (handleResponse(method, url)(response).json \ name).getOrElse(JsNull)) //Added JsNull here to force validate to fail - replicates existing behaviour - } - } + def readSeqFromJsonProperty[O](name: String)(implicit rds: Reads[O], mf: Manifest[O]): HttpReads[Seq[O]] = + (method: String, url: String, response: HttpResponse) => + response.status match { + case 204 | 404 => Seq.empty + case _ => + readJson[Seq[O]](method, url, (handleResponse(method, url)(response).json \ name).getOrElse(JsNull)) //Added JsNull here to force validate to fail - replicates existing behaviour + } - private def readJson[A](method: String, url: String, jsValue: JsValue)(implicit rds: json.Reads[A], mf: Manifest[A]) = + private def readJson[A](method: String, url: String, jsValue: JsValue)(implicit rds: Reads[A], mf: Manifest[A]): A = jsValue .validate[A] .fold( diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpResponse.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpResponse.scala index 60e4dd62..f8712c33 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpResponse.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/HttpResponse.scala @@ -16,6 +16,8 @@ package uk.gov.hmrc.http +import akka.stream.scaladsl.Source +import akka.util.ByteString import com.github.ghik.silencer.silent import play.api.libs.json.{JsValue, Json} @@ -44,6 +46,9 @@ trait HttpResponse { def json: JsValue = Json.parse(body) + def bodyAsSource: Source[ByteString, _] = + Source.single(ByteString(body)) + def header(key: String): Option[String] = headers.get(key).flatMap(_.headOption) @@ -106,6 +111,22 @@ object HttpResponse { } } + def apply( + status : Int, + bodyAsSource: Source[ByteString, _], + headers : Map[String, Seq[String]] + ): HttpResponse = { + val pStatus = status + val pBodyAsSource = bodyAsSource + val pHeaders = headers + new HttpResponse { + override def status : Int = pStatus + override def body : String = sys.error(s"This is a streamed response, please use `bodyAsSource`") + override def bodyAsSource: Source[ByteString, _] = pBodyAsSource + override def allHeaders : Map[String, Seq[String]] = pHeaders + } + } + def unapply(that: HttpResponse): Option[(Int, String, Map[String, Seq[String]])] = Some((that.status, that.body, that.headers)) } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/Retries.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/Retries.scala index e4761e58..36402e64 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/Retries.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/Retries.scala @@ -35,19 +35,35 @@ trait Retries { private val logger = LoggerFactory.getLogger("application") - def retry[A](verb: String, url: String)(block: => Future[A])(implicit ec: ExecutionContext): Future[A] = { + private lazy val sslRetryEnabled = + configuration.getBoolean("http-verbs.retries.ssl-engine-closed-already.enabled") + + def retryOnSslEngineClosed[A](verb: String, url: String)(block: => Future[A])(implicit ec: ExecutionContext): Future[A] = + retryFor(s"$verb $url") { case ex: SSLException if ex.getMessage == "SSLEngine closed already" => sslRetryEnabled }(block) + + @deprecated("Use retryOnSslEngineClosed instead", "14.0.0") + def retry[A](verb: String, url: String)(block: => Future[A])(implicit ec: ExecutionContext): Future[A] = + retryOnSslEngineClosed(verb, url)(block) + + def retryFor[A]( + label : String + )(condition: PartialFunction[Exception, Boolean] + )(block : => Future[A] + )(implicit + ec: ExecutionContext + ): Future[A] = { def loop(remainingIntervals: Seq[FiniteDuration]): Future[A] = { // scheduling will loose MDC data. Here we explicitly ensure it is available on block. block .recoverWith { - case ex @ `sslEngineClosedMatcher`() if remainingIntervals.nonEmpty => + case ex: Exception if condition.lift(ex).getOrElse(false) && remainingIntervals.nonEmpty => val delay = remainingIntervals.head - logger.warn(s"Retrying $verb $url in $delay due to '${ex.getMessage}' error") - val mdcData = Mdc.mdcData - after(delay, actorSystem.scheduler){ - Mdc.putMdc(mdcData) - loop(remainingIntervals.tail) - } + logger.warn(s"Retrying $label in $delay due to error: ${ex.getMessage}") + val mdcData = Mdc.mdcData + after(delay, actorSystem.scheduler){ + Mdc.putMdc(mdcData) + loop(remainingIntervals.tail) + } } } loop(intervals) @@ -57,17 +73,4 @@ trait Retries { configuration.getDurationList("http-verbs.retries.intervals").asScala.toSeq.map { d => FiniteDuration(d.toMillis, TimeUnit.MILLISECONDS) } - - private lazy val sslEngineClosedMatcher = - new SSlEngineClosedMatcher( - enabled = configuration.getBoolean("http-verbs.retries.ssl-engine-closed-already.enabled") - ) - - private class SSlEngineClosedMatcher(enabled: Boolean) { - def unapply(ex: Throwable): Boolean = - ex match { - case _: SSLException if ex.getMessage == "SSLEngine closed already" => enabled - case _ => false - } - } } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/HttpClientV2.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/HttpClientV2.scala new file mode 100644 index 00000000..2c3b3a87 --- /dev/null +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/HttpClientV2.scala @@ -0,0 +1,74 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.http.client + +import play.api.libs.ws.{BodyWritable, WSRequest} +import uk.gov.hmrc.http.{HeaderCarrier, HttpReads} + +import java.net.URL +import scala.concurrent.{ExecutionContext, Future} +import scala.reflect.runtime.universe.TypeTag + +/** This client centralises the execution of the request to ensure that the common concerns (e.g. auditing, logging, + * retries) occur, but makes building the request more flexible (by exposing play-ws). + * It also supports streaming. + */ +trait HttpClientV2 { + protected def mkRequestBuilder(url: URL, method: String)(implicit hc: HeaderCarrier): RequestBuilder + + def get(url: URL)(implicit hc: HeaderCarrier): RequestBuilder = + mkRequestBuilder(url, "GET") + + def post(url: URL)(implicit hc: HeaderCarrier): RequestBuilder = + mkRequestBuilder(url, "POST") + + def put(url: URL)(implicit hc: HeaderCarrier): RequestBuilder = + mkRequestBuilder(url, "PUT") + + def delete(url: URL)(implicit hc: HeaderCarrier): RequestBuilder = + mkRequestBuilder(url, "DELETE") + + def patch(url: URL)(implicit hc: HeaderCarrier): RequestBuilder = + mkRequestBuilder(url, "PATCH") + + def head(url: URL)(implicit hc: HeaderCarrier): RequestBuilder = + mkRequestBuilder(url, "HEAD") + + def options(url: URL)(implicit hc: HeaderCarrier): RequestBuilder = + mkRequestBuilder(url, "OPTIONS") +} + +trait RequestBuilder { + def transform(transform: WSRequest => WSRequest): RequestBuilder + + def execute[A: HttpReads](implicit ec: ExecutionContext): Future[A] + + def stream[A: StreamHttpReads](implicit ec: ExecutionContext): Future[A] + + // support functions + + def replaceHeader(header: (String, String)): RequestBuilder + + def addHeaders(headers: (String, String)*): RequestBuilder + + def withProxy: RequestBuilder + + /** `withBody` should be called rather than `transform(_.withBody)`. + * Failure to do so will lead to a runtime exception + */ + def withBody[B : BodyWritable : TypeTag](body: B): RequestBuilder +} diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/HttpClientV2Impl.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/HttpClientV2Impl.scala new file mode 100644 index 00000000..59d24858 --- /dev/null +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/HttpClientV2Impl.scala @@ -0,0 +1,324 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.http.client + +import akka.actor.ActorSystem +import akka.stream.scaladsl.Source +import akka.util.ByteString +import com.typesafe.config.Config +import play.api.Configuration +import play.api.libs.ws.{BodyWritable, EmptyBody, InMemoryBody, SourceBody, WSClient, WSProxyServer, WSRequest, WSResponse} +import play.core.parsers.FormUrlEncodedParser +import uk.gov.hmrc.http.{BadGatewayException, BuildInfo, GatewayTimeoutException, HeaderCarrier, HttpReads, HttpResponse, Retries} +import uk.gov.hmrc.play.http.BodyCaptor +import uk.gov.hmrc.play.http.ws.WSProxyConfiguration +import uk.gov.hmrc.http.hooks.{HookData, HttpHook} +import uk.gov.hmrc.http.logging.ConnectionTracing + +import java.net.{ConnectException, URL} +import java.util.concurrent.TimeoutException +import scala.concurrent.{ExecutionContext, Future, Promise} +import scala.reflect.runtime.universe.{TypeTag, typeOf} +import scala.util.{Failure, Success} + + +trait Executor { + def execute[A]( + request : WSRequest, + hookDataF: Option[Future[Option[HookData]]], + isStream : Boolean, + r : HttpReads[A] + )(implicit + hc: HeaderCarrier, + ec: ExecutionContext + ): Future[A] +} + +class HttpClientV2Impl( + wsClient : WSClient, + actorSystem: ActorSystem, + config : Configuration, + hooks : Seq[HttpHook] +) extends HttpClientV2 { + + private lazy val optProxyServer = + WSProxyConfiguration.buildWsProxyServer(config) + + private val hcConfig = + HeaderCarrier.Config.fromConfig(config.underlying) + + protected val executor = + new ExecutorImpl(actorSystem, config, hooks) + + private val clientVersionHeader = + "Http-ClientV2-Version" -> BuildInfo.version + + override protected def mkRequestBuilder( + url : URL, + method: String + )(implicit + hc: HeaderCarrier + ): RequestBuilderImpl = + new RequestBuilderImpl( + config, + optProxyServer, + executor + )( + wsClient + .url(url.toString) + .withMethod(method) + .withHttpHeaders(hc.headersForUrl(hcConfig)(url.toString) :+ clientVersionHeader : _*), + None + ) +} + + +// is final since `transform` (and derived functions) return instances of RequestBuilderImpl, and any overrides would be lost. +final class RequestBuilderImpl( + config : Configuration, + optProxyServer: Option[WSProxyServer], + executor : Executor +)( + request : WSRequest, + hookDataF: Option[Future[Option[HookData]]] +)(implicit + hc: HeaderCarrier +) extends RequestBuilder { + + override def transform(transform: WSRequest => WSRequest): RequestBuilderImpl = + new RequestBuilderImpl(config, optProxyServer, executor)(transform(request), hookDataF) + + // -- Transform helpers -- + + private def replaceHeaderOnRequest(request: WSRequest, header: (String, String)): WSRequest = { + def denormalise(hdrs: Map[String, Seq[String]]): Seq[(String, String)] = + hdrs.toList.flatMap { case (k, vs) => vs.map(k -> _) } + val hdrsWithoutKey = request.headers.filterKeys(!_.equalsIgnoreCase(header._1)).toMap // replace existing header + request.withHttpHeaders(denormalise(hdrsWithoutKey) :+ header : _*) + } + + override def replaceHeader(header: (String, String)): RequestBuilderImpl = + transform(replaceHeaderOnRequest(_, header)) + + override def addHeaders(headers: (String, String)*): RequestBuilderImpl = + transform(_.addHttpHeaders(headers: _*)) + + override def withProxy: RequestBuilderImpl = + transform(request => optProxyServer.foldLeft(request)(_ withProxyServer _)) + + private def withHookData(hookDataF: Future[Option[HookData]]): RequestBuilderImpl = + new RequestBuilderImpl(config, optProxyServer, executor)(request, Some(hookDataF)) + + // for erasure + private object IsMap { + def unapply[B: TypeTag](b: B): Option[Map[String, Seq[String]]] = + typeOf[B] match { + case _ if typeOf[B] =:= typeOf[Map[String, String]] => Some(b.asInstanceOf[Map[String, String]].map { case (k, v) => k -> Seq(v) }) + case _ if typeOf[B] =:= typeOf[Map[String, Seq[String]]] => Some(b.asInstanceOf[Map[String, Seq[String]]]) + case _ => None + } + } + + override def withBody[B : BodyWritable : TypeTag](body: B): RequestBuilderImpl = { + val hookDataP = Promise[Option[HookData]]() + val maxBodyLength = config.get[Int]("http-verbs.auditing.maxBodyLength") + val loggingContext = s"outgoing ${request.method} ${request.url} request" + transform { req => + val req2 = req.withBody(body) + req2.body match { + case EmptyBody => hookDataP.success(None) + replaceHeaderOnRequest(req2, play.api.http.HeaderNames.CONTENT_LENGTH -> "0") // rejected by Akami without a Content-Length (https://jira.tools.tax.service.gov.uk/browse/APIS-5100) + case InMemoryBody(bytes) => // we can't guarantee that the default BodyWritables have been used - so rather than relying on content-type alone, we identify form data + // by provided body type (Map) or content-type (e.g. form data as a string) + (body, req2.header("Content-Type")) match { + case (IsMap(m), _ ) => hookDataP.success(Some(HookData.FromMap(m))) + case (_ , Some("application/x-www-form-urlencoded")) => hookDataP.success(Some(HookData.FromMap(FormUrlEncodedParser.parse(bytes.decodeString("UTF-8"))))) + case _ => val auditedBody = BodyCaptor.bodyUpto(bytes, maxBodyLength, loggingContext, isStream = false).decodeString("UTF-8") + hookDataP.success(Some(HookData.FromString(auditedBody))) + } + req2 + case SourceBody(source) => val src2: Source[ByteString, _] = + source + .alsoTo( + BodyCaptor.sink( + loggingContext = loggingContext, + maxBodyLength = maxBodyLength, + withCapturedBody = body => hookDataP.success(Some(HookData.FromString(body.decodeString("UTF-8")))) + ) + ).recover { + case e => hookDataP.failure(e); throw e + } + // preserve content-type (it may have been set with a different body writeable - e.g. play.api.libs.ws.WSBodyWritables.bodyWritableOf_Multipart) + req2.header("Content-Type") match { + case Some(contentType) => replaceHeaderOnRequest(req2.withBody(src2), "Content-Type" -> contentType) + case _ => req2.withBody(src2) + } + } + }.withHookData(hookDataP.future) + } + + // -- Execution -- + + override def execute[A](implicit r: HttpReads[A], ec: ExecutionContext): Future[A] = + executor.execute(request, hookDataF, isStream = false, r) + + override def stream[A](implicit r: StreamHttpReads[A], ec: ExecutionContext): Future[A] = + executor.execute(request, hookDataF, isStream = true, r) +} + +class ExecutorImpl( + override val actorSystem: ActorSystem, // for Retries + config: Configuration, + hooks : Seq[HttpHook] +) extends Executor + with Retries + with ConnectionTracing { + + // for Retries + override val configuration: Config = config.underlying + + private val maxBodyLength = config.get[Int]("http-verbs.auditing.maxBodyLength") + + final def execute[A]( + request : WSRequest, + optHookDataF: Option[Future[Option[HookData]]], + isStream : Boolean, + httpReads : HttpReads[A] + )(implicit + hc: HeaderCarrier, + ec: ExecutionContext + ): Future[A] = { + val hookDataF = + optHookDataF match { + case None if request.body != EmptyBody => + sys.error(s"There is no audit data available. Please ensure you call `withBody` on the RequestBuilder rather than `transform(_.withBody)`") + case None => Future.successful(None) + case Some(f) => f + } + + val startAge = System.nanoTime() - hc.age + val responseF = + retryOnSslEngineClosed(request.method, request.url)( + // we do the execution since if clients are responsable for it (e.g. a callback), they may further modify the request outside of auditing etc. + if (isStream) request.stream() else request.execute() + ) + + val (httpResponseF, auditResponseF) = toHttpResponse(isStream, request, responseF) + executeHooks(isStream, request, hookDataF, auditResponseF) + httpResponseF.onComplete(logResult(hc, request.method, request.uri.toString, startAge)) + // we don't delegate the response conversion to the client + // (i.e. return Future[WSResponse] to be handled with Future.transform/transformWith(...)) + // since the transform functions require access to the request (method and url) + mapErrors(request, httpResponseF) + .map(httpReads.read(request.method, request.url, _)) + } + + // unfortunate return type - first HttpResponse is the full response, the second HttpResponse is truncated for auditing... + private def toHttpResponse( + isStream : Boolean, + request : WSRequest, + responseF: Future[WSResponse] + )(implicit ec: ExecutionContext + ): (Future[HttpResponse], Future[HttpResponse]) = { + val auditResponseF = Promise[HttpResponse]() + val loggingContext = s"outgoing ${request.method} ${request.url} response" + val httpResponseF = + for { + response <- responseF + status = response.status + headers = response.headers.mapValues(_.toSeq).toMap + } yield { + def httpResponse(body : Either[Source[ByteString, _], String]): HttpResponse = + body match { + case Left(src) => HttpResponse( + status = response.status, + bodyAsSource = src, + headers = response.headers.mapValues(_.toSeq).toMap + ) + case Right(str) => HttpResponse( + status = response.status, + body = str, + headers = response.headers.mapValues(_.toSeq).toMap + ) + } + if (isStream) { + val source = + response.bodyAsSource + .alsoTo( + BodyCaptor.sink( + loggingContext = loggingContext, + maxBodyLength = maxBodyLength, + withCapturedBody = body => auditResponseF.success(httpResponse(Right(body.decodeString("UTF-8")))) + ) + ) + .recover { + case e => auditResponseF.failure(e); throw e + } + httpResponse(Left(source)) + } else { + auditResponseF.success( + httpResponse(Right( + BodyCaptor.bodyUpto(response.body, maxBodyLength, loggingContext, isStream = false) + )) + ) + httpResponse(Right(response.body)) + } + } + (httpResponseF, auditResponseF.future) + } + + private def executeHooks( + isStream : Boolean, + request : WSRequest, + hookDataF : Future[Option[HookData]], + auditedResponseF: Future[HttpResponse] // play-auditing expects the body to be a String + )(implicit + hc: HeaderCarrier, + ec: ExecutionContext + ): Unit = { + def denormalise(hdrs: Map[String, Seq[String]]): Seq[(String, String)] = + hdrs.toList.flatMap { case (k, vs) => vs.map(k -> _) } + + def executeHooksWithHookData(hookData: Option[HookData]) = + hooks.foreach( + _.apply( + verb = request.method, + url = new URL(request.url), + headers = denormalise(request.headers), + body = hookData, + responseF = auditedResponseF + ) + ) + + hookDataF.onComplete { + case Success(hookData) => executeHooksWithHookData(hookData) + case Failure(e) => // this is unlikely, but we want best attempt at auditing + executeHooksWithHookData(None) + } + } + + protected def mapErrors( + request : WSRequest, + responseF: Future[HttpResponse] + )(implicit + ec: ExecutionContext + ): Future[HttpResponse] = + responseF.recoverWith { + case e: TimeoutException => Future.failed(new GatewayTimeoutException(s"${request.method} of '${request.url}' timed out with message '${e.getMessage}'")) + case e: ConnectException => Future.failed(new BadGatewayException(s"${request.method} of '${request.url}' failed. Caused by: '${e.getMessage}'")) + } +} diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/package.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/package.scala new file mode 100644 index 00000000..4b12f2ff --- /dev/null +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/http/client/package.scala @@ -0,0 +1,60 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.http + +import akka.stream.Materializer +import akka.stream.scaladsl.Source +import akka.util.ByteString + +import scala.annotation.implicitNotFound + +package client { + trait Streaming +} +package object client + extends StreamHttpReadsInstances { + + // ensures strict HttpReads are not passed to stream function, which would lead to stream being read into memory + // (or runtime exceptions since HttpResponse.body with throw exception for streamed responses) + @implicitNotFound("""Could not find an implicit StreamHttpReads[${A}]. + You may be missing an implicit Materializer.""") + type StreamHttpReads[A] = HttpReads[A] with Streaming +} + +trait StreamHttpReadsInstances { + def tag[A](instance: A): A with client.Streaming = + instance.asInstanceOf[A with client.Streaming] + + implicit def readEitherSource(implicit mat: Materializer, errorTimeout: ErrorTimeout): client.StreamHttpReads[Either[UpstreamErrorResponse, Source[ByteString, _]]] = + tag[HttpReads[Either[UpstreamErrorResponse, Source[ByteString, _]]]]( + HttpReads.ask.flatMap { case (method, url, response) => + HttpErrorFunctions.handleResponseEitherStream(method, url)(response) match { + case Left(err) => HttpReads.pure(Left(err)) + case Right(response) => HttpReads.pure(Right(response.bodyAsSource)) + } + } + ) + + implicit def readSource(implicit mat: Materializer, errorTimeout: ErrorTimeout): client.StreamHttpReads[Source[ByteString, _]] = + tag[HttpReads[Source[ByteString, _]]]( + readEitherSource + .map { + case Left(err) => throw err + case Right(value) => value + } + ) +} diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/BodyCaptor.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/BodyCaptor.scala new file mode 100644 index 00000000..64472732 --- /dev/null +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/BodyCaptor.scala @@ -0,0 +1,102 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.play.http + +import akka.stream.{Attributes, FlowShape, Inlet, Outlet} +import akka.stream.scaladsl.{Flow, Sink} +import akka.stream.stage._ +import akka.util.ByteString +import org.slf4j.LoggerFactory + +// based on play.filters.csrf.CSRFAction#BodyHandler + +private class BodyCaptorFlow( + loggingContext : String, + maxBodyLength : Int, + withCapturedBody: ByteString => Unit +) extends GraphStage[FlowShape[ByteString, ByteString]] { + val in = Inlet[ByteString]("BodyCaptorFlow.in") + val out = Outlet[ByteString]("BodyCaptorFlow.out") + override val shape = FlowShape.of(in, out) + + override def createLogic(inheritedAttributes: Attributes): GraphStageLogic = + new GraphStageLogic(shape) { + private var buffer = ByteString.empty + + setHandlers( + in, + out, + new InHandler with OutHandler { + override def onPull(): Unit = + pull(in) + + override def onPush(): Unit = { + val chunk = grab(in) + if (buffer.size < maxBodyLength) + buffer ++= chunk + push(out, chunk) + } + + override def onUpstreamFinish(): Unit = { + withCapturedBody(BodyCaptor.bodyUpto(buffer, maxBodyLength, loggingContext, isStream = true)) + completeStage() + } + } + ) + } +} + +object BodyCaptor { + private val logger = LoggerFactory.getLogger(getClass) + + def flow( + loggingContext : String, + maxBodyLength : Int, + withCapturedBody: ByteString => Unit // provide a callback since a Materialized value would be not be available until the flow has been run + ): Flow[ByteString, ByteString, akka.NotUsed] = + Flow.fromGraph(new BodyCaptorFlow( + loggingContext = loggingContext, + maxBodyLength = maxBodyLength, + withCapturedBody = withCapturedBody + )) + + def sink( + loggingContext : String, + maxBodyLength : Int, + withCapturedBody: ByteString => Unit + ): Sink[ByteString, akka.NotUsed] = + flow(loggingContext, maxBodyLength, withCapturedBody) + .to(Sink.ignore) + + def bodyUpto(body: String, maxBodyLength: Int, loggingContext: String, isStream: Boolean): String = + if (body.length > maxBodyLength) { + logger.warn( + s"$loggingContext ${if (isStream) "streamed body" else "body " + body.length} exceeds maxLength $maxBodyLength - truncating" + ) + body.take(maxBodyLength) + } else + body + + def bodyUpto(body: ByteString, maxBodyLength: Int, loggingContext: String, isStream: Boolean): ByteString = + if (body.length > maxBodyLength) { + logger.warn( + s"$loggingContext ${if (isStream) "streamed body" else "body " + body.length} exceeds maxLength $maxBodyLength - truncating" + ) + body.take(maxBodyLength) + } else + body +} diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/HeaderCarrierConverter.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/HeaderCarrierConverter.scala index 704a14dd..fa23bf0a 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/HeaderCarrierConverter.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/HeaderCarrierConverter.scala @@ -16,9 +16,9 @@ package uk.gov.hmrc.play.http +import play.api.http.{HeaderNames => PlayHeaderNames} import play.api.mvc.{Cookies, Headers, RequestHeader, Session} import uk.gov.hmrc.http._ -import play.api.http.{HeaderNames => PlayHeaderNames} import scala.util.Try diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequest.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequest.scala index 1b3b7205..db71615d 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequest.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequest.scala @@ -21,7 +21,7 @@ import play.api.libs.ws.{DefaultWSProxyServer, WSProxyServer, WSRequest => PlayW trait WSRequest extends WSRequestBuilder { - override def buildRequest[A]( + override def buildRequest( url : String, headers: Seq[(String, String)] ): PlayWSRequest = @@ -33,7 +33,7 @@ trait WSProxy extends WSRequest { def wsProxyServer: Option[WSProxyServer] - override def buildRequest[A](url: String, headers: Seq[(String, String)]): PlayWSRequest = + override def buildRequest(url: String, headers: Seq[(String, String)]): PlayWSRequest = wsProxyServer match { case Some(proxy) => super.buildRequest(url, headers).withProxyServer(proxy) case None => super.buildRequest(url, headers) @@ -42,25 +42,34 @@ trait WSProxy extends WSRequest { object WSProxyConfiguration { + @deprecated("Use buildWsProxyServer instead. See docs for differences.", "14.0.0") def apply(configPrefix: String, configuration: Configuration): Option[WSProxyServer] = { val proxyRequired = configuration.getOptional[Boolean](s"$configPrefix.proxyRequiredForThisEnvironment").getOrElse(true) - if (proxyRequired) Some(parseProxyConfiguration(configPrefix, configuration)) else None + if (proxyRequired) + Some( + DefaultWSProxyServer( + protocol = Some(configuration.get[String](s"$configPrefix.protocol")), + host = configuration.get[String](s"$configPrefix.host"), + port = configuration.get[Int](s"$configPrefix.port"), + principal = configuration.getOptional[String](s"$configPrefix.username"), + password = configuration.getOptional[String](s"$configPrefix.password") + ) + ) + else None } - private def parseProxyConfiguration(configPrefix: String, configuration: Configuration) = - DefaultWSProxyServer( - protocol = configuration - .getOptional[String](s"$configPrefix.protocol") - .orElse(throw ProxyConfigurationException("protocol")), - host = - configuration.getOptional[String](s"$configPrefix.host").getOrElse(throw ProxyConfigurationException("host")), - port = configuration.getOptional[Int](s"$configPrefix.port").getOrElse(throw ProxyConfigurationException("port")), - principal = configuration.getOptional[String](s"$configPrefix.username"), - password = configuration.getOptional[String](s"$configPrefix.password") - ) - - case class ProxyConfigurationException(key: String) - extends RuntimeException(s"Missing proxy configuration - key '$key' not found") + def buildWsProxyServer(configuration: Configuration): Option[WSProxyServer] = + if (configuration.get[Boolean]("http-verbs.proxy.enabled")) + Some( + DefaultWSProxyServer( + protocol = Some(configuration.get[String]("proxy.protocol")), + host = configuration.get[String]("proxy.host"), + port = configuration.get[Int]("proxy.port"), + principal = configuration.getOptional[String]("proxy.username"), + password = configuration.getOptional[String]("proxy.password") + ) + ) + else None } diff --git a/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequestBuilder.scala b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequestBuilder.scala index 1c11287c..4b2ae03a 100644 --- a/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequestBuilder.scala +++ b/http-verbs-common/src/main/scala/uk/gov/hmrc/play/http/ws/WSRequestBuilder.scala @@ -23,5 +23,5 @@ trait WSRequestBuilder extends Request { protected def wsClient: WSClient - protected def buildRequest[A](url: String, headers: Seq[(String, String)]): PlayWSRequest + protected def buildRequest(url: String, headers: Seq[(String, String)]): PlayWSRequest } diff --git a/http-verbs-common/src/test/resources/logback.xml b/http-verbs-common/src/test/resources/logback.xml index d36f5f0c..c44ac0bc 100644 --- a/http-verbs-common/src/test/resources/logback.xml +++ b/http-verbs-common/src/test/resources/logback.xml @@ -3,12 +3,13 @@ - %date [%level] [%logger] [%thread] %message%n + %date %highlight([%level]) [%logger] [%thread] %message%n + diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HeadersSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HeadersSpec.scala index a1b30415..4038a53b 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HeadersSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HeadersSpec.scala @@ -18,20 +18,19 @@ package uk.gov.hmrc.http import akka.actor.ActorSystem import com.github.ghik.silencer.silent -import com.github.tomakehurst.wiremock._ import com.github.tomakehurst.wiremock.client.WireMock._ -import com.github.tomakehurst.wiremock.core.WireMockConfiguration.wireMockConfig import com.typesafe.config.Config import org.scalatest.concurrent.{IntegrationPatience, ScalaFutures} import org.scalatest.matchers.must.Matchers import org.scalatest.wordspec.AnyWordSpecLike -import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach} +import play.api.Application +import play.api.http.{HeaderNames => PlayHeaderNames} import play.api.inject.guice.GuiceApplicationBuilder import play.api.libs.json.{JsValue, Json} import play.api.libs.ws.WSClient -import play.api.{Application, Play} import uk.gov.hmrc.http.hooks.HttpHook -import uk.gov.hmrc.play.http.ws.{PortTester, WSHttp} +import uk.gov.hmrc.play.http.ws.WSHttp +import uk.gov.hmrc.http.test.WireMockSupport import scala.concurrent.ExecutionContext.Implicits.global import uk.gov.hmrc.http.HttpReads.Implicits._ @@ -39,28 +38,11 @@ import uk.gov.hmrc.http.HttpReads.Implicits._ class HeadersSpec extends AnyWordSpecLike with Matchers - with BeforeAndAfterAll - with BeforeAndAfterEach + with WireMockSupport with ScalaFutures with IntegrationPatience { private lazy val app: Application = new GuiceApplicationBuilder().build() - private val server = new WireMockServer(wireMockConfig().port(PortTester.findPort())) - - override def beforeAll(): Unit = { - Play.start(app) - server.start() - } - - override def afterAll(): Unit = { - server.stop() - Play.stop(app) - } - - override def beforeEach(): Unit = { - server.stop() - server.start() - } @silent("deprecated") private implicit val hc: HeaderCarrier = HeaderCarrier( @@ -70,24 +52,27 @@ class HeadersSpec requestId = Some(RequestId("request-id")) ).withExtraHeaders("extra-header" -> "my-extra-header") - private lazy val client = new HttpGet with HttpPost with HttpDelete with HttpPatch with HttpPut with WSHttp { - override def wsClient: WSClient = app.injector.instanceOf[WSClient] - override protected def configuration: Config = app.configuration.underlying - override val hooks: Seq[HttpHook] = Seq.empty - override protected def actorSystem: ActorSystem = ActorSystem("test-actor-system") - } + private lazy val httpClient = + new HttpGet with HttpPost with HttpDelete with HttpPatch with HttpPut with WSHttp { + override def wsClient: WSClient = app.injector.instanceOf[WSClient] + override protected def configuration: Config = app.configuration.underlying + override val hooks: Seq[HttpHook] = Seq.empty + override protected def actorSystem: ActorSystem = ActorSystem("test-actor-system") + } "a post request" when { "with an arbitrary body" should { "contain headers from the header carrier" in { - server.stubFor( + stubFor( post(urlEqualTo("/arbitrary")) .willReturn(aResponse().withStatus(200)) ) - client.POST[JsValue, HttpResponse](s"http://localhost:${server.port()}/arbitrary", Json.obj()).futureValue + httpClient + .POST[JsValue, HttpResponse](s"$wireMockUrl/arbitrary", Json.obj()) + .futureValue - server.verify( + verify( postRequestedFor(urlEqualTo("/arbitrary")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) @@ -98,18 +83,19 @@ class HeadersSpec } "allow a user to set an extra header in the POST" in { - server.stubFor( + stubFor( post(urlEqualTo("/arbitrary")) .willReturn(aResponse().withStatus(200)) ) - client.POST[JsValue, HttpResponse]( - url = s"http://localhost:${server.port()}/arbitrary", - body = Json.obj(), - headers = Seq("extra-header-2" -> "my-extra-header-2") - ).futureValue + httpClient + .POST[JsValue, HttpResponse]( + url = s"$wireMockUrl/arbitrary", + body = Json.obj(), + headers = Seq("extra-header-2" -> "my-extra-header-2") + ).futureValue - server.verify( + verify( postRequestedFor(urlEqualTo("/arbitrary")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) @@ -123,17 +109,18 @@ class HeadersSpec "with a string body" should { "contain headers from the header carrier" in { - server.stubFor( + stubFor( post(urlEqualTo("/string")) .willReturn(aResponse().withStatus(200)) ) - client.POSTString[HttpResponse]( - url = s"http://localhost:${server.port()}/string", - body = "foo" - ).futureValue + httpClient + .POSTString[HttpResponse]( + url = s"$wireMockUrl/string", + body = "foo" + ).futureValue - server.verify( + verify( postRequestedFor(urlEqualTo("/string")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) @@ -144,18 +131,19 @@ class HeadersSpec } "allow a user to set an extra header in the POST" in { - server.stubFor( + stubFor( post(urlEqualTo("/string")) .willReturn(aResponse().withStatus(200)) ) - client.POSTString[HttpResponse]( - url = s"http://localhost:${server.port()}/string", - body = "foo", - headers = Seq("extra-header-2" -> "my-extra-header-2") - ).futureValue + httpClient + .POSTString[HttpResponse]( + url = s"$wireMockUrl/string", + body = "foo", + headers = Seq("extra-header-2" -> "my-extra-header-2") + ).futureValue - server.verify( + verify( postRequestedFor(urlEqualTo("/string")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) @@ -169,21 +157,23 @@ class HeadersSpec "with an empty body" should { "add a content length header if none is present" in { - server.stubFor( + stubFor( post(urlEqualTo("/empty")) .willReturn(aResponse().withStatus(200)) ) - client.POSTEmpty[HttpResponse](s"http://localhost:${server.port()}/empty").futureValue + httpClient + .POSTEmpty[HttpResponse](s"$wireMockUrl/empty") + .futureValue - server.verify( + verify( postRequestedFor(urlEqualTo("/empty")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) .withHeader(HeaderNames.xSessionId, equalTo("session-id")) .withHeader(HeaderNames.xRequestId, equalTo("request-id")) .withHeader("extra-header", equalTo("my-extra-header")) - .withHeader(play.api.http.HeaderNames.CONTENT_LENGTH, equalTo("0")) + .withHeader(PlayHeaderNames.CONTENT_LENGTH, equalTo("0")) ) } } @@ -191,16 +181,16 @@ class HeadersSpec "a get request" should { "contain headers from the header carrier" in { - server.stubFor( + stubFor( get(urlEqualTo("/")) .willReturn(aResponse().withStatus(200)) ) - client - .GET[HttpResponse](s"http://localhost:${server.port()}/") + httpClient + .GET[HttpResponse](s"$wireMockUrl/") .futureValue - server.verify( + verify( getRequestedFor(urlEqualTo("/")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) @@ -213,17 +203,18 @@ class HeadersSpec "a delete request" should { "contain headers from the header carrier" in { - server.stubFor( + wireMockServer.stubFor( delete(urlEqualTo("/")) .willReturn(aResponse().withStatus(200)) ) - client.DELETE[HttpResponse]( - url = s"http://localhost:${server.port()}/", - headers = Seq("header" -> "foo") - ).futureValue + httpClient + .DELETE[HttpResponse]( + url = s"$wireMockUrl/", + headers = Seq("header" -> "foo") + ).futureValue - server.verify( + wireMockServer.verify( deleteRequestedFor(urlEqualTo("/")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) @@ -237,20 +228,20 @@ class HeadersSpec "a patch request" should { "contain headers from the header carrier" in { - server.stubFor( + stubFor( patch(urlEqualTo("/")) .willReturn(aResponse().withStatus(200)) ) - client + httpClient .PATCH[JsValue, HttpResponse]( - url = s"http://localhost:${server.port()}/", + url = s"$wireMockUrl/", body = Json.obj(), headers = Seq("header" -> "foo") ) .futureValue - server.verify( + verify( patchRequestedFor(urlEqualTo("/")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) @@ -264,17 +255,18 @@ class HeadersSpec "a put request" should { "contain headers from the header carrier" in { - server.stubFor( + stubFor( put(urlEqualTo("/")) .willReturn(aResponse().withStatus(200)) ) - client.PUT[JsValue, HttpResponse]( - url = s"http://localhost:${server.port()}/", - body = Json.obj() - ).futureValue + httpClient + .PUT[JsValue, HttpResponse]( + url = s"$wireMockUrl/", + body = Json.obj() + ).futureValue - server.verify( + verify( putRequestedFor(urlEqualTo("/")) .withHeader(HeaderNames.authorisation, equalTo("authorization")) .withHeader(HeaderNames.xForwardedFor, equalTo("forwarded-for")) diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpDeleteSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpDeleteSpec.scala index 3ac89a2f..32933aa9 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpDeleteSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpDeleteSpec.scala @@ -146,16 +146,22 @@ class HttpDeleteSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testDelete.configuration) val headers = HeaderCarrier.headersForUrl(config, url, Seq("header" -> "foo")) - verify(testDelete.testHook1).apply(eqTo("DELETE"), eqTo(url"$url"), eqTo(headers), eqTo(None), respArgCaptor1)(any, any) - verify(testDelete.testHook2).apply(eqTo("DELETE"), eqTo(url"$url"), eqTo(headers), eqTo(None), respArgCaptor2)(any, any) + verify(testDelete.testHook1).apply(eqTo("DELETE"), eqTo(url"$url"), headerCaptor1, eqTo(None), respArgCaptor1)(any, any) + verify(testDelete.testHook2).apply(eqTo("DELETE"), eqTo(url"$url"), headerCaptor2, eqTo(None), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } } diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpGetSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpGetSpec.scala index 2a12f1cb..09840162 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpGetSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpGetSpec.scala @@ -142,16 +142,22 @@ class HttpGetSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testGet.configuration) val headers = HeaderCarrier.headersForUrl(config, url) - verify(testGet.testHook1).apply(eqTo("GET"), eqTo(url"$url"), eqTo(headers), eqTo(None), respArgCaptor1)(any, any) - verify(testGet.testHook2).apply(eqTo("GET"), eqTo(url"$url"), eqTo(headers), eqTo(None), respArgCaptor2)(any, any) + verify(testGet.testHook1).apply(eqTo("GET"), eqTo(url"$url"), headerCaptor1, eqTo(None), respArgCaptor1)(any, any) + verify(testGet.testHook2).apply(eqTo("GET"), eqTo(url"$url"), headerCaptor2, eqTo(None), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPatchSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPatchSpec.scala index b4f843fd..14af4044 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPatchSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPatchSpec.scala @@ -152,18 +152,22 @@ class HttpPatchSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testPatch.configuration) val headers = HeaderCarrier.headersForUrl(config, url, Seq("header" -> "foo")) - verify(testPatch.testHook1) - .apply(eqTo("PATCH"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testJson))), respArgCaptor1)(any, any) - verify(testPatch.testHook2) - .apply(eqTo("PATCH"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testJson))), respArgCaptor2)(any, any) + verify(testPatch.testHook1).apply(eqTo("PATCH"), eqTo(url"$url"), headerCaptor1, eqTo(Some(HookData.FromString(testJson))), respArgCaptor1)(any, any) + verify(testPatch.testHook2).apply(eqTo("PATCH"), eqTo(url"$url"), headerCaptor2, eqTo(Some(HookData.FromString(testJson))), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } } diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPostSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPostSpec.scala index 93499aba..d7a0ae30 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPostSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPostSpec.scala @@ -194,16 +194,22 @@ class HttpPostSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testPost.configuration) val headers = HeaderCarrier.headersForUrl(config, url) - verify(testPost.testHook1).apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testJson))), respArgCaptor1)(any, any) - verify(testPost.testHook2).apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testJson))), respArgCaptor2)(any, any) + verify(testPost.testHook1).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor1, eqTo(Some(HookData.FromString(testJson))), respArgCaptor1)(any, any) + verify(testPost.testHook2).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor2, eqTo(Some(HookData.FromString(testJson))), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } @@ -231,16 +237,22 @@ class HttpPostSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testPost.configuration) val headers = HeaderCarrier.headersForUrl(config, url) - verify(testPost.testHook1).apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromMap(Map()))), respArgCaptor1)(any, any) - verify(testPost.testHook2).apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromMap(Map()))), respArgCaptor2)(any, any) + verify(testPost.testHook1).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor1, eqTo(Some(HookData.FromMap(Map()))), respArgCaptor1)(any, any) + verify(testPost.testHook2).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor2, eqTo(Some(HookData.FromMap(Map()))), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } @@ -272,18 +284,22 @@ class HttpPostSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testPost.configuration) val headers = HeaderCarrier.headersForUrl(config, url) - verify(testPost.testHook1) - .apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor1)(any, any) - verify(testPost.testHook2) - .apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor2)(any, any) + verify(testPost.testHook1).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor1, eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor1)(any, any) + verify(testPost.testHook2).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor2, eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } @@ -311,16 +327,22 @@ class HttpPostSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testPost.configuration) val headers = HeaderCarrier.headersForUrl(config, url) - verify(testPost.testHook1).apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(None), respArgCaptor1)(any, any) - verify(testPost.testHook2).apply(eqTo("POST"), eqTo(url"$url"), eqTo(headers), eqTo(None), respArgCaptor2)(any, any) + verify(testPost.testHook1).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor1, eqTo(None), respArgCaptor1)(any, any) + verify(testPost.testHook2).apply(eqTo("POST"), eqTo(url"$url"), headerCaptor2, eqTo(None), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPutSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPutSpec.scala index ff458422..e76a901e 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPutSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/HttpPutSpec.scala @@ -170,16 +170,22 @@ class HttpPutSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testPut.configuration) val headers = HeaderCarrier.headersForUrl(config, url) - verify(testPut.testHook1).apply(eqTo("PUT"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testJson))), respArgCaptor1)(any, any) - verify(testPut.testHook2).apply(eqTo("PUT"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testJson))), respArgCaptor2)(any, any) + verify(testPut.testHook1).apply(eqTo("PUT"), eqTo(url"$url"), headerCaptor1, eqTo(Some(HookData.FromString(testJson))), respArgCaptor1)(any, any) + verify(testPut.testHook2).apply(eqTo("PUT"), eqTo(url"$url"), headerCaptor2, eqTo(Some(HookData.FromString(testJson))), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } @@ -198,7 +204,8 @@ class HttpPutSpec behave like anErrorMappingHttpCall( "PUT", - (url, responseF) => new StubbedHttpPut(responseF).PUTString[HttpResponse](url, testRequestBody, Seq.empty)) + (url, responseF) => new StubbedHttpPut(responseF).PUTString[HttpResponse](url, testRequestBody, Seq.empty) + ) behave like aTracingHttpCall("PUT", "PUT", new StubbedHttpPut(defaultHttpResponse)) { _.PUTString[HttpResponse](url, testRequestBody, Seq.empty) } @@ -213,18 +220,22 @@ class HttpPutSpec val respArgCaptor1 = ArgCaptor[Future[HttpResponse]] val respArgCaptor2 = ArgCaptor[Future[HttpResponse]] + val headerCaptor1 = ArgCaptor[Seq[(String, String)]] + val headerCaptor2 = ArgCaptor[Seq[(String, String)]] + val config = HeaderCarrier.Config.fromConfig(testPut.configuration) val headers = HeaderCarrier.headersForUrl(config, url) - verify(testPut.testHook1) - .apply(eqTo("PUT"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor1)(any, any) - verify(testPut.testHook2) - .apply(eqTo("PUT"), eqTo(url"$url"), eqTo(headers), eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor2)(any, any) + verify(testPut.testHook1).apply(eqTo("PUT"), eqTo(url"$url"), headerCaptor1, eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor1)(any, any) + verify(testPut.testHook2).apply(eqTo("PUT"), eqTo(url"$url"), headerCaptor2, eqTo(Some(HookData.FromString(testRequestBody))), respArgCaptor2)(any, any) // verifying directly without ArgumentCaptor didn't work as Futures were different instances // e.g. Future.successful(5) != Future.successful(5) respArgCaptor1.value.futureValue shouldBe dummyResponse respArgCaptor2.value.futureValue shouldBe dummyResponse + + headerCaptor1.value should contain allElementsOf(headers) + headerCaptor2.value should contain allElementsOf(headers) } } } diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/RetriesSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/RetriesSpec.scala index a5c0ac25..22f4cdb2 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/RetriesSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/RetriesSpec.scala @@ -47,7 +47,7 @@ class RetriesSpec with IntegrationPatience { import ExecutionContext.Implicits.global - "Retries" should { + "Retries.retryOnSslEngineClosed" should { "be disabled by default" in { val retries: Retries = new Retries { override protected val configuration: Config = ConfigFactory.load() @@ -56,7 +56,7 @@ class RetriesSpec @volatile var counter = 0 val resultF = - retries.retry("GET", "url") { + retries.retryOnSslEngineClosed("GET", "url") { Future.failed { counter += 1 new SSLException("SSLEngine closed already") @@ -89,7 +89,7 @@ class RetriesSpec @volatile var counter = 0 val resultF = - retries.retry("GET", "url") { + retries.retryOnSslEngineClosed("GET", "url") { Future.successful { counter += 1 counter @@ -118,7 +118,7 @@ class RetriesSpec Future.failed(new SSLException("SSLEngine closed already")) } - val _ = Try(retries.retry("GET", "url")(failingFuture).futureValue) + val _ = Try(retries.retryOnSslEngineClosed("GET", "url")(failingFuture).futureValue) val actualIntervals: List[Long] = timestamps.sliding(2).toList.map { @@ -143,7 +143,7 @@ class RetriesSpec } val resultF = - retries.retry("GET", "url") { + retries.retryOnSslEngineClosed("GET", "url") { Future.failed { new SSLException("SSLEngine closed already") } @@ -167,7 +167,7 @@ class RetriesSpec val expectedResponse = HttpResponse(404, "") val resultF = - retries.retry("GET", "url") { + retries.retryOnSslEngineClosed("GET", "url") { retries.failFewTimesAndThenSucceed( success = Future.successful(expectedResponse), exception = new SSLException("SSLEngine closed already") @@ -198,7 +198,7 @@ class RetriesSpec val resultF = for { _ <- Future.successful(Mdc.putMdc(mdcData)) - res <- retries.retry("GET", "url") { + res <- retries.retryOnSslEngineClosed("GET", "url") { // assert mdc available to block execution Option(MDC.getCopyOfContextMap).map(_.asScala.toMap).getOrElse(Map.empty) shouldBe mdcData diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/http/client/HttpClientV2Spec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/client/HttpClientV2Spec.scala new file mode 100644 index 00000000..fe0067f4 --- /dev/null +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/http/client/HttpClientV2Spec.scala @@ -0,0 +1,637 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.http.client + +import akka.actor.ActorSystem +import akka.stream.scaladsl.Source +import akka.util.ByteString +import com.github.tomakehurst.wiremock.client.WireMock +import com.github.tomakehurst.wiremock.client.WireMock.{verify => _, _} +import com.typesafe.config.ConfigFactory +import org.mockito.ArgumentMatchersSugar +import org.mockito.captor.ArgCaptor +import org.mockito.scalatest.MockitoSugar +import org.scalatest.concurrent.{IntegrationPatience, ScalaFutures} +import org.scalatest.matchers.should.Matchers +import org.scalatest.wordspec.AnyWordSpecLike +import play.api.Configuration +import play.api.libs.json.{Json, Reads, Writes} +import play.api.libs.ws.ahc.{AhcWSClient, AhcWSClientConfigFactory} +import uk.gov.hmrc.http.{HeaderCarrier, HttpReads, HttpReadsInstances, HttpResponse, Retries, StringContextOps, UpstreamErrorResponse} +import uk.gov.hmrc.http.hooks.{HookData, HttpHook} +import uk.gov.hmrc.http.test.WireMockSupport + +import java.util.concurrent.atomic.AtomicInteger +import scala.concurrent.{ExecutionContext, Future} +import scala.concurrent.ExecutionContext.Implicits.global +import scala.util.Random + +class HttpClientV2Spec + extends AnyWordSpecLike + with Matchers + with WireMockSupport + with ScalaFutures + with IntegrationPatience + with MockitoSugar + with ArgumentMatchersSugar { + + import uk.gov.hmrc.http.HttpReads.Implicits._ + + "HttpClientV2" should { + "work with json" in new Setup { + implicit val hc = HeaderCarrier() + + wireMockServer.stubFor( + WireMock.put(urlEqualTo("/")) + .willReturn(aResponse().withBody("\"res\"").withStatus(200)) + ) + + val res: Future[ResDomain] = + httpClientV2 + .put(url"$wireMockUrl/") + .withBody(Json.toJson(ReqDomain("req"))) + .execute[ResDomain] + + res.futureValue shouldBe ResDomain("res") + + wireMockServer.verify( + putRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("application/json")) + .withRequestBody(equalTo("\"req\"")) + .withHeader("User-Agent", equalTo("myapp")) + .withHeader("Http-ClientV2-Version", matching(".*")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("PUT"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromString("\"req\""))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "application/json") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe "\"res\"" + } + + "work with streams" in new Setup { + implicit val hc = HeaderCarrier() + + val requestBody = Random.alphanumeric.take(maxAuditBodyLength - 1).mkString + val responseBody = Random.alphanumeric.take(maxAuditBodyLength - 1).mkString + + wireMockServer.stubFor( + WireMock.put(urlEqualTo("/")) + .willReturn(aResponse().withBody(responseBody).withStatus(200)) + ) + + val srcStream: Source[ByteString, _] = + Source.single(ByteString(requestBody)) + + val res: Future[Source[ByteString, _]] = + httpClientV2 + .put(url"$wireMockUrl/") + .withBody(srcStream) + .stream[Source[ByteString, _]] + + res.futureValue.map(_.utf8String).runReduce(_ + _).futureValue shouldBe responseBody + + wireMockServer.verify( + putRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("application/octet-stream")) + .withRequestBody(equalTo(requestBody)) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("PUT"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromString(requestBody))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "application/octet-stream") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe responseBody + } + + "handled failed requests with streams" in new Setup { + implicit val hc = HeaderCarrier() + + val requestBody = Random.alphanumeric.take(maxAuditBodyLength - 1).mkString + val responseBody = Random.alphanumeric.take(maxAuditBodyLength - 1).mkString + + wireMockServer.stubFor( + WireMock.put(urlEqualTo("/")) + .willReturn(aResponse().withBody(responseBody).withStatus(500)) + ) + + val srcStream: Source[ByteString, _] = + Source.single(ByteString(requestBody)) + + val res: Future[Source[ByteString, _]] = + httpClientV2 + .put(url"$wireMockUrl/") + .withBody(srcStream) + .stream[Source[ByteString, _]] + + val failure = res.failed.futureValue + + failure shouldBe a[UpstreamErrorResponse] + failure.asInstanceOf[UpstreamErrorResponse].statusCode shouldBe 500 + failure.asInstanceOf[UpstreamErrorResponse].message shouldBe s"PUT of 'http://localhost:6001/' returned 500. Response body: '$responseBody'" + + wireMockServer.verify( + putRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("application/octet-stream")) + .withRequestBody(equalTo(requestBody)) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("PUT"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromString(requestBody))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "application/octet-stream") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 500 + auditedResponse.body shouldBe responseBody + } + + "truncate stream payloads for auditing if too long" in new Setup { + implicit val hc = HeaderCarrier() + + val requestBody = Random.alphanumeric.take(maxAuditBodyLength * 2).mkString + val responseBody = Random.alphanumeric.take(maxAuditBodyLength * 2).mkString + + wireMockServer.stubFor( + WireMock.put(urlEqualTo("/")) + .willReturn(aResponse().withBody(responseBody).withStatus(200)) + ) + + val srcStream: Source[ByteString, _] = + Source.single(ByteString(requestBody)) + + val res: Future[Source[ByteString, _]] = + httpClientV2 + .put(url"$wireMockUrl/") + .withBody(srcStream) + .stream[Source[ByteString, _]] + + res.futureValue.map(_.utf8String).runReduce(_ + _).futureValue shouldBe responseBody + + wireMockServer.verify( + putRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("application/octet-stream")) + .withRequestBody(equalTo(requestBody)) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("PUT"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromString(requestBody.take(maxAuditBodyLength)))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "application/octet-stream") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe responseBody.take(maxAuditBodyLength) + } + + "truncate strict payloads for auditing if too long" in new Setup { + implicit val hc = HeaderCarrier() + + val requestBody = Random.alphanumeric.take(maxAuditBodyLength * 2).mkString + val responseBody = Random.alphanumeric.take(maxAuditBodyLength * 2).mkString + + wireMockServer.stubFor( + WireMock.put(urlEqualTo("/")) + .willReturn(aResponse().withBody(responseBody).withStatus(200)) + ) + + val res: Future[HttpResponse] = + httpClientV2 + .put(url"$wireMockUrl/") + .withBody(requestBody) + .execute[HttpResponse] + + res.futureValue.body shouldBe responseBody + + wireMockServer.verify( + putRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", containing("text/plain")) // play-26 doesn't send charset + .withRequestBody(equalTo(requestBody)) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("PUT"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromString(requestBody.take(maxAuditBodyLength)))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "text/plain") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe responseBody.take(maxAuditBodyLength) + } + + "work with form data" in new Setup { + implicit val hc = HeaderCarrier() + + wireMockServer.stubFor( + WireMock.post(urlEqualTo("/")) + .willReturn(aResponse().withBody("\"res\"").withStatus(200)) + ) + + val body: Map[String, Seq[String]] = + Map( + "k1" -> Seq("v1", "v2"), + "k2" -> Seq("v3") + ) + + val res: Future[ResDomain] = + httpClientV2 + .post(url"$wireMockUrl/") + .withBody(body) + .execute[ResDomain] + + res.futureValue shouldBe ResDomain("res") + + wireMockServer.verify( + postRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("application/x-www-form-urlencoded")) + .withRequestBody(equalTo("k1=v1&k1=v2&k2=v3")) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("POST"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromMap(body))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "application/x-www-form-urlencoded") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe "\"res\"" + } + + "work with form data - custom writeable for content-type" in new Setup { + implicit val hc = HeaderCarrier() + + wireMockServer.stubFor( + WireMock.post(urlEqualTo("/")) + .willReturn(aResponse().withBody("\"res\"").withStatus(200)) + ) + + import play.api.libs.ws.{BodyWritable, InMemoryBody} + implicit val writeableOf_urlEncodedForm: BodyWritable[Map[String, Seq[String]]] = { + import java.net.URLEncoder + BodyWritable( + formData => + InMemoryBody( + ByteString.fromString( + formData.flatMap(item => item._2.map(c => s"${item._1}=${URLEncoder.encode(c, "UTF-8")}")).mkString("&") + ) + ), + "nonstandard/x-www-form-urlencoded" + ) + } + + val body: Map[String, Seq[String]] = + Map( + "k1" -> Seq("v1", "v2"), + "k2" -> Seq("v3") + ) + + val res: Future[ResDomain] = + httpClientV2 + .post(url"$wireMockUrl/") + .withBody(body) + .execute[ResDomain] + + res.futureValue shouldBe ResDomain("res") + + wireMockServer.verify( + postRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("nonstandard/x-www-form-urlencoded")) + .withRequestBody(equalTo("k1=v1&k1=v2&k2=v3")) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("POST"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromMap(body))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "nonstandard/x-www-form-urlencoded") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe "\"res\"" + } + + "work with form data - custom writeable for map type" in new Setup { + implicit val hc = HeaderCarrier() + + wireMockServer.stubFor( + WireMock.post(urlEqualTo("/")) + .willReturn(aResponse().withBody("\"res\"").withStatus(200)) + ) + + import play.api.libs.ws.{BodyWritable, InMemoryBody} + implicit val writeableOf_urlEncodedForm: BodyWritable[scala.collection.mutable.Map[String, Seq[String]]] = { + import java.net.URLEncoder + BodyWritable( + formData => + InMemoryBody( + ByteString.fromString( + formData.flatMap(item => item._2.map(c => s"${item._1}=${URLEncoder.encode(c, "UTF-8")}")).mkString("&") + ) + ), + "application/x-www-form-urlencoded" + ) + } + + val body: scala.collection.mutable.Map[String, Seq[String]] = + scala.collection.mutable.Map( + "k1" -> Seq("v1", "v2"), + "k2" -> Seq("v3") + ) + + val res: Future[ResDomain] = + httpClientV2 + .post(url"$wireMockUrl/") + .withBody(body) + .execute[ResDomain] + + res.futureValue shouldBe ResDomain("res") + + wireMockServer.verify( + postRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("application/x-www-form-urlencoded")) + .withRequestBody(equalTo("k1=v1&k1=v2&k2=v3")) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("POST"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromMap(body.toMap))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "application/x-www-form-urlencoded") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe "\"res\"" + } + + /* Note, using non-form-encoding and a non immutable Map implementation will not be escaped properly + "work with any form data" in new Setup { + implicit val hc = HeaderCarrier() + + wireMockServer.stubFor( + WireMock.post(urlEqualTo("/")) + .willReturn(aResponse().withBody("\"res\"").withStatus(200)) + ) + + import play.api.libs.ws.{BodyWritable, InMemoryBody} + implicit val writeableOf_urlEncodedForm: BodyWritable[scala.collection.mutable.Map[String, Seq[String]]] = { + import java.net.URLEncoder + BodyWritable( + formData => + InMemoryBody( + ByteString.fromString( + formData.flatMap(item => item._2.map(c => s"${item._1}=${URLEncoder.encode(c, "UTF-8")}")).mkString("&") + ) + ), + "non-standard/x-www-form-urlencoded" + ) + } + + val body: scala.collection.mutable.Map[String, Seq[String]] = + scala.collection.mutable.Map( + "k1" -> Seq("v1", "v2"), + "k2" -> Seq("v3") + ) + + val res: Future[ResDomain] = + httpClientV2 + .post(url"$wireMockUrl/") + .withBody(body) + .execute[ResDomain] + + res.futureValue shouldBe ResDomain("res") + + wireMockServer.verify( + postRequestedFor(urlEqualTo("/")) + .withHeader("Content-Type", equalTo("nonstandard/x-www-form-urlencoded")) + .withRequestBody(equalTo("k1=v1&k1=v2&k2=v3")) + .withHeader("User-Agent", equalTo("myapp")) + ) + + val headersCaptor = ArgCaptor[Seq[(String, String)]] + val responseCaptor = ArgCaptor[Future[HttpResponse]] + + verify(mockHttpHook) + .apply( + verb = eqTo("POST"), + url = eqTo(url"$wireMockUrl/"), + headers = headersCaptor, + body = eqTo(Some(HookData.FromMap(body.toMap))), + responseF = responseCaptor + )(any[HeaderCarrier], any[ExecutionContext]) + + headersCaptor.value should contain ("User-Agent" -> "myapp") + headersCaptor.value should contain ("Content-Type" -> "application/x-www-form-urlencoded") + val auditedResponse = responseCaptor.value.futureValue + auditedResponse.status shouldBe 200 + auditedResponse.body shouldBe "\"res\"" + } + */ + + "fail if call withBody on the wsRequest itself" in new Setup { + implicit val hc = HeaderCarrier() + + a[RuntimeException] should be thrownBy + httpClientV2 + .put(url"$wireMockUrl/") + .transform(_.withBody(Json.toJson(ReqDomain("req")))) + .replaceHeader("User-Agent" -> "ua2") + .execute[ResDomain] + } + + "allow overriding user-agent" in new Setup { + implicit val hc = HeaderCarrier() + + wireMockServer.stubFor( + WireMock.put(urlEqualTo("/")) + .willReturn(aResponse().withBody("\"res\"").withStatus(200)) + ) + + val res: Future[ResDomain] = + httpClientV2 + .put(url"$wireMockUrl/") + .withBody(Json.toJson(ReqDomain("req"))) + .replaceHeader("User-Agent" -> "ua2") + .execute[ResDomain] + + res.futureValue shouldBe ResDomain("res") + + wireMockServer.verify( + putRequestedFor(urlEqualTo("/")) + .withRequestBody(equalTo("\"req\"")) + .withHeader("User-Agent", equalTo("ua2")) + ) + } + + "allow custom retries" in new Setup { + val retries = new Retries { + override val actorSystem = as + override val configuration = ConfigFactory + .parseString("http-verbs.retries.intervals = [ 100.ms, 200.ms, 300.ms ]") + .withFallback(ConfigFactory.load()) + } + + implicit val hc = HeaderCarrier() + + wireMockServer.stubFor( + WireMock.put(urlEqualTo("/")) + .willReturn(aResponse().withBody("\"res\"").withStatus(502)) + ) + + val count = new AtomicInteger(0) + + val res: Future[ResDomain] = + retries.retryFor("get reqdomain"){ case UpstreamErrorResponse.WithStatusCode(502) => true }{ + count.incrementAndGet + httpClientV2 + .put(url"$wireMockUrl/") + .withBody(Json.toJson(ReqDomain("req"))) + .execute[ResDomain] + } + + res.failed.futureValue shouldBe a[UpstreamErrorResponse] + count.get shouldBe 4 + } + } + + trait Setup { + implicit val as: ActorSystem = ActorSystem("test-actor-system") + + val mockHttpHook = mock[HttpHook](withSettings.lenient) + + val maxAuditBodyLength = 30 + + val httpClientV2: HttpClientV2 = { + val config = + Configuration( + ConfigFactory.parseString( + s"""|appName = myapp + |http-verbs.auditing.maxBodyLength = $maxAuditBodyLength + |""".stripMargin + ).withFallback(ConfigFactory.load()) + ) + new HttpClientV2Impl( + wsClient = AhcWSClient(AhcWSClientConfigFactory.forConfig(config.underlying)), + as, + config, + hooks = Seq(mockHttpHook), + ) + } + } +} + +case class ReqDomain( + field: String +) + +object ReqDomain { + implicit val w: Writes[ReqDomain] = + implicitly[Writes[String]].contramap(_.field) +} + +case class ResDomain( + field: String +) + +object ResDomain extends HttpReadsInstances { + implicit val r: Reads[ResDomain] = + implicitly[Reads[String]].map(ResDomain.apply) + + implicit val hr: HttpReads[ResDomain] = readFromJson[ResDomain] +} diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/HeaderCarrierConverterSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/HeaderCarrierConverterSpec.scala index f84d7465..d2d57723 100644 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/HeaderCarrierConverterSpec.scala +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/HeaderCarrierConverterSpec.scala @@ -24,6 +24,7 @@ import play.api.inject.guice.GuiceApplicationBuilder import play.api.{Configuration, Play} import play.api.mvc._ import play.api.test.Helpers._ +import play.api.http.{HeaderNames => PlayHeaderNames} import play.api.test.{FakeHeaders, FakeRequest, FakeRequestFactory} import play.api.mvc.request.RequestFactory import uk.gov.hmrc.http._ @@ -381,7 +382,7 @@ class HeaderCarrierConverterSpec extends AnyWordSpecLike with Matchers with Befo "utilise values from cookies" should { "find the deviceID from the cookie if set in header" in { val cookieHeader = Cookies.encodeCookieHeader(Seq(Cookie(CookieNames.deviceID, "deviceIdCookie"))) - val req = FakeRequest().withHeaders(play.api.http.HeaderNames.COOKIE -> cookieHeader) + val req = FakeRequest().withHeaders(PlayHeaderNames.COOKIE -> cookieHeader) HeaderCarrierConverter.fromHeadersAndSession(req.headers, Some(req.session)) .deviceID shouldBe Some("deviceIdCookie") diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/HttpTimeoutSpec.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/HttpTimeoutSpec.scala deleted file mode 100644 index 7804b1a9..00000000 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/HttpTimeoutSpec.scala +++ /dev/null @@ -1,97 +0,0 @@ -/* - * Copyright 2022 HM Revenue & Customs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package uk.gov.hmrc.play.http - -import java.net.{ServerSocket, URI} -import java.util.concurrent.TimeoutException - -import org.scalatest.BeforeAndAfterAll -import org.scalatest.concurrent.{IntegrationPatience, ScalaFutures} -import org.scalatest.matchers.should.Matchers -import org.scalatest.wordspec.AnyWordSpecLike -import org.webbitserver.handler.{DelayedHttpHandler, StringHttpHandler} -import org.webbitserver.netty.NettyWebServer -import play.api.inject.guice.GuiceApplicationBuilder -import play.api.libs.ws.WSClient -import play.api.test.WsTestClient -import play.api.{Application, Configuration, Play} -import uk.gov.hmrc.play.http.ws.WSHttp -import uk.gov.hmrc.play.test.TestHttpCore - -import scala.concurrent.ExecutionContext - -class HttpTimeoutSpec - extends AnyWordSpecLike - with Matchers - with ScalaFutures - with IntegrationPatience - with BeforeAndAfterAll { - - import ExecutionContext.Implicits.global - - private lazy val fakeApplication: Application = - GuiceApplicationBuilder(configuration = Configuration("play.ws.timeout.request" -> "1000ms")).build() - - override def beforeAll() { - super.beforeAll() - Play.start(fakeApplication) - } - - override def afterAll() { - super.afterAll() - Play.stop(fakeApplication) - } - - WsTestClient.withClient { _ => - - "HttpCalls" should { - - "be gracefully timeout when no response is received within the 'timeout' frame" in { - val http: WSHttp = new WSHttp with TestHttpCore { - override val wsClient: WSClient = fakeApplication.injector.instanceOf[WSClient] - } - - // get an unused port - val ss = new ServerSocket(0) - ss.close() - val executor = ExecutionContext.global - val publicUri = URI.create(s"http://localhost:${ss.getLocalPort}") - val ws = new NettyWebServer(executor, ss.getLocalSocketAddress, publicUri) - try { - //starts web server - ws.add( - "/test", - new DelayedHttpHandler(executor, 2000, new StringHttpHandler("application/json", "{name:'pong'}")) - ) - ws.start().get() - - val start = System.currentTimeMillis() - - //make request to web server - http.doPost(s"$publicUri/test", "{name:'ping'}", Seq()).failed.futureValue shouldBe a [TimeoutException] - - val diff = (System.currentTimeMillis() - start).toInt - // there is test execution delay around 700ms - diff should be >= 1000 - diff should be < 2500 - } finally { - ws.stop() - } - } - } - } -} diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/ws/wireMockEndpoints.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/ws/wireMockEndpoints.scala deleted file mode 100644 index b4f22a47..00000000 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/http/ws/wireMockEndpoints.scala +++ /dev/null @@ -1,65 +0,0 @@ -/* - * Copyright 2022 HM Revenue & Customs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package uk.gov.hmrc.play.http.ws - -import java.net.ServerSocket - -import com.github.tomakehurst.wiremock.WireMockServer -import com.github.tomakehurst.wiremock.client.WireMock -import com.github.tomakehurst.wiremock.core.WireMockConfiguration._ - -import scala.util.Try - -trait WireMockEndpoints { - - val host: String = "localhost" - - val endpointPort: Int = PortTester.findPort() - val endpointMock = new WireMock(host, endpointPort) - val endpointServer: WireMockServer = new WireMockServer(wireMockConfig().port(endpointPort)) - - val proxyPort: Int = PortTester.findPort(endpointPort) - val proxyMock: WireMock = new WireMock(host, proxyPort) - val proxyServer: WireMockServer = new WireMockServer(wireMockConfig().port(proxyPort)) - - def withServers(test: => Unit) { - endpointServer.start() - proxyServer.start() - - try { - test - } finally { - Try(endpointServer.stop()) - Try(proxyServer.stop()) - } - } -} - -object PortTester { - - def findPort(excluded: Int*): Int = - (6000 to 7000).find(port => !excluded.contains(port) && isFree(port)).getOrElse(throw new Exception("No free port")) - - private def isFree(port: Int): Boolean = { - val triedSocket = Try { - val serverSocket = new ServerSocket(port) - Try(serverSocket.close()) - serverSocket - } - triedSocket.isSuccess - } -} diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/PortFinder.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/PortFinder.scala new file mode 100644 index 00000000..1e792b65 --- /dev/null +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/PortFinder.scala @@ -0,0 +1,38 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.http.test + +import scala.util.Try + +import java.net.ServerSocket + +object PortFinder { + + def findFreePort(portRange: Range, excluded: Int*): Int = + portRange + .find(port => !excluded.contains(port) && isFree(port)) + .getOrElse(throw new Exception("No free port")) + + private def isFree(port: Int): Boolean = { + val triedSocket = Try { + val serverSocket = new ServerSocket(port) + Try(serverSocket.close()) + serverSocket + } + triedSocket.isSuccess + } +} diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/WireMockSupport.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/WireMockSupport.scala new file mode 100644 index 00000000..9b83c927 --- /dev/null +++ b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/WireMockSupport.scala @@ -0,0 +1,62 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.http.test + +import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach, Suite} +import com.github.tomakehurst.wiremock._ +import com.github.tomakehurst.wiremock.client.WireMock +import com.github.tomakehurst.wiremock.core.WireMockConfiguration +import org.slf4j.{Logger, LoggerFactory} + +trait WireMockSupport + extends BeforeAndAfterAll + with BeforeAndAfterEach { + this: Suite => + + private val logger: Logger = LoggerFactory.getLogger(getClass) + + lazy val wireMockHost: String = + "localhost" + + lazy val wireMockPort: Int = + // we lookup a port ourselves rather than using `wireMockConfig().dynamicPort()` since it's simpler to provide + // it up front (rather than query the running server), and allow overriding. + PortFinder.findFreePort(portRange = 6001 to 7000) + + lazy val wireMockUrl: String = + s"http://$wireMockHost:$wireMockPort" + + lazy val wireMockServer = + new WireMockServer(WireMockConfiguration.wireMockConfig().port(wireMockPort)) + + override protected def beforeAll(): Unit = { + super.beforeAll() + wireMockServer.start() + WireMock.configureFor(wireMockHost, wireMockServer.port()) + logger.info(s"Started WireMock server on host: $wireMockHost, port: ${wireMockServer.port()}") + } + + override protected def afterAll(): Unit = { + wireMockServer.stop() + super.afterAll() + } + + override protected def beforeEach(): Unit = { + super.beforeEach() + wireMockServer.resetMappings() + } +} diff --git a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/helpers.scala b/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/helpers.scala deleted file mode 100644 index f7540016..00000000 --- a/http-verbs-common/src/test/scala/uk/gov/hmrc/play/test/helpers.scala +++ /dev/null @@ -1,107 +0,0 @@ -/* - * Copyright 2022 HM Revenue & Customs - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package uk.gov.hmrc.play.test - -import play.api.libs.json.Writes -import uk.gov.hmrc.http._ - -import scala.concurrent.{ExecutionContext, Future} - -trait TestHttpCore extends CorePost with CoreGet with CorePut with CorePatch with CoreDelete with Request { - - override def POST[I, O]( - url: String, - body: I, - headers: Seq[(String, String)])( - implicit wts: Writes[I], - rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? - - override def POSTString[O]( - url: String, - body: String, - headers: Seq[(String, String)])( - implicit rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? - - override def POSTForm[O]( - url: String, - body: Map[String, Seq[String]], - headers: Seq[(String, String)])( - implicit rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? - - override def POSTEmpty[O]( - url: String, - headers: Seq[(String, String)])( - implicit rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? - - override def GET[A]( - url: String, - queryParams: Seq[(String, String)], - headers: Seq[(String, String)])( - implicit rds: HttpReads[A], - hc: HeaderCarrier, - ec: ExecutionContext): Future[A] = - ??? - - override def PUT[I, O]( - url: String, - body: I, - headers: Seq[(String, String)])( - implicit wts: Writes[I], - rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? - - override def PUTString[O]( - url: String, - body: String, - headers: Seq[(String, String)])( - implicit rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? - - override def PATCH[I, O]( - url: String, - body: I, - headers: Seq[(String, String)])( - implicit wts: Writes[I], - rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? - - override def DELETE[O]( - url: String, - headers: Seq[(String, String)])( - implicit rds: HttpReads[O], - hc: HeaderCarrier, - ec: ExecutionContext): Future[O] = - ??? -} diff --git a/http-verbs-test-common/src/main/scala/uk/gov/hmrc/http/test/HttpClient2Support.scala b/http-verbs-test-common/src/main/scala/uk/gov/hmrc/http/test/HttpClient2Support.scala new file mode 100644 index 00000000..65c3e939 --- /dev/null +++ b/http-verbs-test-common/src/main/scala/uk/gov/hmrc/http/test/HttpClient2Support.scala @@ -0,0 +1,41 @@ +/* + * Copyright 2022 HM Revenue & Customs + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package uk.gov.hmrc.http.test + +import akka.actor.ActorSystem +import com.typesafe.config.ConfigFactory +import play.api.Configuration +import play.api.libs.ws.ahc.{AhcWSClient, AhcWSClientConfigFactory} +import uk.gov.hmrc.http.client.{HttpClientV2, HttpClientV2Impl} + +trait HttpClientV2Support { + + def mkHttpClientV2( + config: Configuration = Configuration(ConfigFactory.load()) + ): HttpClientV2 = { + implicit val as: ActorSystem = ActorSystem("test-actor-system") + + new HttpClientV2Impl( + wsClient = AhcWSClient(AhcWSClientConfigFactory.forConfig(config.underlying)), + as, + config, + hooks = Seq.empty, + ) + } + + lazy val httpClientV2: HttpClientV2 = mkHttpClientV2() +} diff --git a/http-verbs-test-common/src/main/scala/uk/gov/hmrc/http/test/HttpClientSupport.scala b/http-verbs-test-common/src/main/scala/uk/gov/hmrc/http/test/HttpClientSupport.scala index ef6e4b42..7a669455 100644 --- a/http-verbs-test-common/src/main/scala/uk/gov/hmrc/http/test/HttpClientSupport.scala +++ b/http-verbs-test-common/src/main/scala/uk/gov/hmrc/http/test/HttpClientSupport.scala @@ -17,11 +17,9 @@ package uk.gov.hmrc.http.test import akka.actor.ActorSystem -import akka.stream.{ActorMaterializer, Materializer} -import com.github.ghik.silencer.silent import com.typesafe.config.{Config, ConfigFactory} import play.api.libs.ws.WSClient -import uk.gov.hmrc.http._ +import uk.gov.hmrc.http.HttpClient import uk.gov.hmrc.http.hooks.HttpHook import uk.gov.hmrc.play.http.ws.WSHttp import play.api.libs.ws.ahc.{AhcWSClient, AhcWSClientConfigFactory} @@ -33,9 +31,6 @@ trait HttpClientSupport { new HttpClient with WSHttp { private implicit val as: ActorSystem = ActorSystem("test-actor-system") - @silent("deprecated") - private implicit val mat: Materializer = ActorMaterializer() // explicitly required for play-26 - override val wsClient: WSClient = AhcWSClient(AhcWSClientConfigFactory.forConfig(config)) override protected val configuration: Config = config override val hooks: Seq[HttpHook] = Seq.empty