[Azure websubhub] Analyze performace test results for content-publishing

[ayeshLK]

Description:

$subject

Asgardeo perf test results can be hound here [1]

[1] - https://docs.google.com/spreadsheets/d/1DV2-bXqFff-UPR76sS3pLpEeohRVtAdPlmpwpEJ1EbA/edit?usp=sharing

[ayeshLK]

Analyzed the performance test results for the given parameters mentioned in the "Round 2 - After correlation log fix" sheet. The settings used were as follows:

• Number of tenants: 100
• Number of SPs per tenant: 10
• Number of users per tenant: 10

For the concurrency configurations of 1, 20, and 50, there were no instances of HTTP 502 or HTTP 503 responses. However, a few (less than 10) request timeouts were observed. These timeouts could be attributed to the low timeout value (300ms) set in the Asgardeo publisher client. To resolve these timeouts, it is recommended to update the timeout configuration in the Asgardeo publisher client to the previously agreed value of 5 seconds.

On the other hand, when using the concurrency configurations of 100 and 200, a significant number of HTTP 502 and HTTP 503 responses were observed, along with a quite a few request timeouts. The occurrence of these HTTP 502 and HTTP 503 responses can be attributed to the resource limitations in the hub pods. To improve the performance of the hub and enable support for higher concurrency settings, it is necessary to update the current resource limitations applied to the hub pods.

[ayeshLK]

Asgardeo team has conducted round-3 performance tests [1] with the suggested changes. In the mean-time relevant resource limits were updated in the azure-websubhub deployment as well. They found following observations:

• No HTTP 502 or HTTP 503 responses from the hub
• With the increased concurrency setting with 100 topics no time-out responses found.
• With increased concurrency setting with only 1 topic there were several time-out responses found.

azure-websubhub development team will complete following action items to provide a proper analysis:

• Analyze the network connection stats between the AKS cluster (in which the hub is deployed) and the Azure service-bus namespace to identify any network delays.
• Do a load-test on the Ballerina Azure service bus connector to check for any performance issues in the connector level.

[1] - https://docs.google.com/spreadsheets/d/1DV2-bXqFff-UPR76sS3pLpEeohRVtAdPlmpwpEJ1EbA/edit?usp=sharing

[ayeshLK]

Conducted a load-test on the Ballerina Azure service bus connector using following sample-code:

import ballerinax/asb;
import ballerina/os;
import ballerina/http;

final string CONNECTION_STRING = os:getEnv("CONNECTION_STRING");
configurable string TOPIC_NAME = ?;

service / on new http:Listener(9090) {
    private final asb:MessageSender producer;

    function init() returns error? {
        self.producer = check new ({
            entityType: asb:TOPIC,
            topicOrQueueName: TOPIC_NAME,
            connectionString: CONNECTION_STRING
        });
    }

    resource function post test(@http:Payload json payload) returns http:Ok|error? {
        check self.producer->send({
            body:  payload.toJsonString().toBytes(),
            contentType: asb:JSON
        });
        return {};
    }
}

Resource constraints:

• CPU: 2
• Memory: 1GB
• Ballerina Max Pool Size: 100

For the load-test we configured a request-timeout of 5 seconds (which is the current configuration for Asgardeo publisher).

Following are the results:

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1714	349	312	416	486	653	277	5006	0.058%	2.85650	0.27	0.61
5	6080	572	611	757	862	1088	277	5006	0.016%	4.44536	0.41	0.95
10	75844	648	602	870	1112	2032	277	5021	0.171%	22.29318	2.14	4.78
20	31984	645	615	804	921	1407	277	5010	0.191%	11.60340	1.12	2.49
50	43860	650	586	967	1253	2346	284	5021	0.157%	73.07077	6.99	15.67

As per the results there is <1% error rate for all the concurrency settings and all the failed-requests are due to the time-outs.