Performance issue with mpOpenAPI-4.0

[jdmcclur]

When comparing the performance of hitting the /openapi endpoint with mpOpenAP1-3.1 to mpOpenAPI-4.0 I see a significant throughput regression with mpOpenAPI-4.0.

I don't totally understand what is going on, but it looks like lots of openapi internal model properties are getting initialized on every request (and put into a hashmap).

  Parent      0   0.00   0.02           0           1    J:io/smallrye/openapi/api/OpenApiConfigImpl.getConfigValue(Ljava/lang/String;Ljava/lang/Class;Ljava/util/function/Function;Ljava/util/function/Supplier;)Ljava/lang/Object;
  Parent      0   0.02   0.02           1           1    J:com/ibm/ws/webcontainer/srt/SRTServletRequest.setAttribute(Ljava/lang/String;Ljava/lang/Object;)V
  Parent      0   0.00   0.04           0           2    J:java/util/HashSet.add(Ljava/lang/Object;)Z
  Parent      0   0.00   0.07           0           4    J:com/ibm/ws/genericbnf/internal/BNFHeadersImpl.addHeader(Lcom/ibm/ws/genericbnf/internal/HeaderElement;Z)V
  Parent      0   0.00   0.07           0           4    J:io/opentelemetry/sdk/internal/AttributesMap.put(Lio/opentelemetry/api/common/AttributeKey;Ljava/lang/Object;)V
  Parent      0   0.02   0.14           1           8    J:io/opentelemetry/instrumentation/api/instrumenter/UnsafeAttributes.put(Lio/opentelemetry/api/common/AttributeKey;Ljava/lang/Object;)Lio/opentelemetry/api/common/AttributesBuilder;
  Parent      0   0.05   0.16           3           9    J:io/smallrye/openapi/model/BaseModel.setProperty(Ljava/lang/String;Ljava/lang/Object;)V
  Parent      0   0.20   0.34          11          19    J:io/smallrye/openapi/internal/models/security/AbstractSecurityRequirement$Properties.<init>()V
  Parent      0   0.22   0.36          12          20    J:io/smallrye/openapi/internal/models/info/License$Properties.<init>()V
  Parent      0   0.13   0.36           7          20    J:io/smallrye/openapi/internal/models/Paths$Properties.<init>()V
  Parent      0   0.20   0.38          11          21    J:io/smallrye/openapi/internal/models/tags/AbstractTag$Properties.<init>()V
  Parent      0   0.16   0.40           9          22    J:io/smallrye/openapi/internal/models/media/Discriminator$Properties.<init>()V
  Parent      0   0.22   0.47          12          26    J:io/smallrye/openapi/internal/models/info/Contact$Properties.<init>()V
  Parent      0   0.05   0.47           3          26    J:io/smallrye/openapi/internal/models/examples/Example$Properties.<init>()V
  Parent      0   0.25   0.49          14          27    J:io/smallrye/openapi/internal/models/servers/ServerVariable$Properties.<init>()V
  Parent      0   0.14   0.51           8          28    J:io/smallrye/openapi/internal/models/ExternalDocumentation$Properties.<init>()V
  Parent      0   0.00   0.56           0          31    J:io/smallrye/openapi/model/BaseModel.getProperties(Ljava/lang/Class;)Ljava/util/Map;
  Parent      0   0.31   0.60          17          33    J:io/smallrye/openapi/internal/models/media/XML$Properties.<init>()V
  Parent      0   0.29   0.65          16          36    J:io/smallrye/openapi/internal/models/servers/Server$Properties.<init>()V
  Parent      0   0.36   0.65          20          36    J:io/smallrye/openapi/internal/models/media/Content$Properties.<init>()V
  Parent      0   0.27   0.65          15          36    J:io/smallrye/openapi/internal/models/callbacks/Callback$Properties.<init>()V
  Parent      0   0.16   0.69           9          38    J:io/smallrye/openapi/internal/models/security/OAuthFlow$Properties.<init>()V
  Parent      0   0.18   0.70          10          39    J:io/smallrye/openapi/internal/models/responses/APIResponses$Properties.<init>()V
  Parent      0   0.43   0.70          24          39    J:io/smallrye/openapi/internal/models/media/Encoding$Properties.<init>()V
  Parent      0   0.14   0.74           8          41    J:io/smallrye/openapi/internal/models/info/Info$Properties.<init>()V
  Parent      0   0.38   0.76          21          42    J:io/smallrye/openapi/internal/models/parameters/AbstractRequestBody$Properties.<init>()V
  Parent      0   0.40   0.76          22          42    J:io/smallrye/openapi/internal/models/responses/APIResponse$Properties.<init>()V
  Parent      0   0.27   0.79          15          44    J:io/smallrye/openapi/internal/models/media/MediaType$Properties.<init>()V
  Parent      0   0.29   0.90          16          50    J:io/smallrye/openapi/internal/models/security/OAuthFlows$Properties.<init>()V
  Parent      0   0.32   0.94          18          52    J:io/smallrye/openapi/internal/models/headers/Header$Properties.<init>()V
  Parent      0   0.18   0.96          10          53    J:io/smallrye/openapi/internal/models/links/Link$Properties.<init>()V
  Parent      0   0.47   0.97          26          54    J:io/smallrye/openapi/internal/models/security/SecurityScheme$Properties.<init>()V
  Parent      0   0.11   0.99           6          55    J:com/fasterxml/jackson/databind/node/ObjectNode.set(Ljava/lang/String;Lcom/fasterxml/jackson/databind/JsonNode;)Lcom/fasterxml/jackson/databind/JsonNode;
  Parent      0   0.72   1.30          40          72    J:io/smallrye/openapi/internal/models/AbstractOpenAPI$Properties.<init>()V
  Parent      0   0.32   1.43          18          79    J:io/smallrye/openapi/internal/models/AbstractPathItem$Properties.<init>()V
  Parent      0   0.43   1.48          24          82    J:io/smallrye/openapi/internal/models/parameters/Parameter$Properties.<init>()V
  Parent      0   0.76   1.55          42          86    J:io/smallrye/openapi/internal/models/Operation$Properties.<init>()V
  Parent      0   0.97   1.77          54          98    J:io/smallrye/openapi/internal/models/Components$Properties.<init>()V
  Parent      0   0.99   3.95          55         219    J:io/smallrye/openapi/internal/models/SmallRyeOASModels.<init>()V
  Parent      0   1.43   5.40          79         299    J:io/smallrye/openapi/internal/models/media/AbstractSchema$Properties.<init>()V
 
    Self      0  11.86  34.19         657        1894    J:java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
 
   Child      0  13.52  20.88         749        1157    J:java/util/HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object;
   Child      0   1.44   1.44          80          80    J:java/util/HashMap.hash(Ljava/lang/Object;)I

Farther up the stack

    Parent      0   0.67   6.06          37         336    J:io/smallrye/openapi/internal/models/SmallRyeOASModels.<init>()V
 
    Self      0   0.67   6.06          37         336    J:io/smallrye/openapi/internal/models/media/AbstractSchema$Properties.<init>()V
 
   Child      0   1.43   5.40          79         299    J:java/util/HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;

And above that

  Parent      0   0.02   0.70           1          39    J:io/smallrye/openapi/runtime/io/info/InfoIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   0.74           0          41    J:io/smallrye/openapi/runtime/io/tags/TagIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.02   0.74           1          41    J:io/smallrye/openapi/runtime/io/media/ContentIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   0.78           0          43    J:io/smallrye/openapi/runtime/io/info/LicenseIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   0.83           0          46    J:io/smallrye/openapi/runtime/io/servers/ServerIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.07   0.94           4          52    J:io/smallrye/openapi/runtime/io/responses/APIResponsesIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   0.94           0          52    J:io/smallrye/openapi/runtime/io/security/SecurityRequirementsSetIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.04   0.97           2          54    J:io/smallrye/openapi/runtime/io/PathsIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   0.97           0          54    J:io/smallrye/openapi/runtime/io/ExternalDocumentationIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.02   1.08           1          60    J:io/smallrye/openapi/runtime/io/info/ContactIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.02   1.10           1          61    J:io/smallrye/openapi/runtime/io/security/OAuthFlowIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.02   1.12           1          62    J:io/smallrye/openapi/runtime/io/security/OAuthFlowsIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   1.12           0          62    J:io/smallrye/openapi/runtime/io/media/MediaTypeIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.04   1.14           2          63    J:io/smallrye/openapi/runtime/io/security/SecurityRequirementIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   1.17           0          65    J:io/smallrye/openapi/runtime/io/media/DiscriminatorIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   1.62           0          90    J:io/smallrye/openapi/runtime/io/ComponentsIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.00   1.82           0         101    J:io/smallrye/openapi/runtime/io/OpenAPIDefinitionIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;)V
  Parent      0   0.02   2.64           1         146    J:io/smallrye/openapi/runtime/io/OperationIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;Lorg/jboss/jandex/DotName;)V
  Parent      0   0.09  14.58           5         808    J:io/smallrye/openapi/runtime/io/MapModelIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;Lorg/jboss/jandex/DotName;Lorg/jboss/jandex/DotName;)V
 
    Self      0   0.34  35.02          19        1940    J:io/smallrye/openapi/runtime/io/ModelIO.<init>(Lio/smallrye/openapi/runtime/io/IOContext;Lorg/jboss/jandex/DotName;Lorg/jboss/jandex/DotName;)V
 
   Child      0   0.32  33.74          18        1869    J:io/smallrye/openapi/internal/models/SmallRyeOASModels.<init>()V
   Child      0   0.02   0.94           1          52    J:java/util/Set.of(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/util/Set;

[Azquelt]

@jdmcclur They did change a lot of the internal structures to be map-based, but we shouldn't be initializing many of them on every request. I did raise concerns upstream when this change was added, but was assured that they didn't see a noticeable impact, so hopefully it's something we're doing and can fix.

I can't tell much about why this is happening from the data above. If we are initializing lots of objects on every request, it should be happening within io.openliberty.microprofile.openapi20.internal.servlet.ApplicationServlet. Can you start from there and drill down into which step is taking the time? Particularly at the point where it leaves the openliberty code and enters the smallrye code. See update below.

I assume either our code is doing something stupid, or the smallrye internals are doing something stupid. We will initialize a few objects per-request because the servers and sometimes the info are set dynamically, but it shouldn't be many.

[Azquelt]

Hmm, with a look at that very last part, ModelIO does have a SmallRyeOASModels field and a Set field that I think should be static. I'll change that and see if it fixes some of the issues.

[Azquelt]

We've made a fix upstream: smallrye/smallrye-open-api#2050

I've sent you a snapshot build via slack to see if that fixes all the performance issues, or if there are more problems. I think this is the cause of all the issues you found above, but it would be good to be sure it's not hiding anything else.

[jdmcclur]

The snapshot build looks much better. The reported issue above seems to be resolved. The throughput is still slower, but looks to be a different issue.

[Azquelt]

We should be able to close this when SmallRye OpenAPI 4.0.1 is released and included in liberty.

[Azquelt]

4.0.1 is now available https://github.com/smallrye/smallrye-open-api/releases/tag/4.0.1