[Bug]: Azure service bus connector encounter a delay while publishing concurrent messages

[ayeshLK]

Connector Name

module/azure-servicebus (Azure Servicebus)

What happened?

Description

$subject

When publishing concurrent messages using single asb:MessageSender(which is assigned to a ASB topic) the client experiences an unexpected delay. Conducted a load-test using the Azure service bus connector within an HTTP service and it was observed that few requests have been timed-out due to the above mentioned behavior.

Following code sample was used:

import ballerinax/asb;
import ballerina/os;
import ballerina/http;

final string CONNECTION_STRING = os:getEnv("CONNECTION_STRING");
configurable string TOPIC_NAME = ?;

service / on new http:Listener(9090) {
    private final asb:MessageSender producer;

    function init() returns error? {
        self.producer = check new ({
            entityType: asb:TOPIC,
            topicOrQueueName: TOPIC_NAME,
            connectionString: CONNECTION_STRING
        });
    }

    resource function post test(@http:Payload json payload) returns http:Ok|error? {
        check self.producer->send({
            body:  payload.toJsonString().toBytes(),
            contentType: asb:JSON
        });
        return {};
    }
}

Constrained the application resources to following limits:

• CPU: 2
• Memory: 1GB
• Ballerina Max Pool Size: 100

Load test was conducted for a duration of 10 minutes. Following are the results:

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1714	349	312	416	486	653	277	5006	0.058%	2.85650	0.27	0.61
5	6080	572	611	757	862	1088	277	5006	0.016%	4.44536	0.41	0.95
10	75844	648	602	870	1112	2032	277	5021	0.171%	22.29318	2.14	4.78
20	31984	645	615	804	921	1407	277	5010	0.191%	11.60340	1.12	2.49
50	43860	650	586	967	1253	2346	284	5021	0.157%	73.07077	6.99	15.67

Note: All the errors in the results are due to request timeouts. The configured request timeout is 5 seconds

Related issues

• [Azure websubhub] Analyze performace test results for content-publishing #4508

Versions

• Ballerina SL 2201.4.1

[ayeshLK]

Before Fix (ASB v3.8.0)

CPU: 0.5 | Memory: 1g | MaxPoolSize: 100

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1294	463	409	615	732	1207	293	5026	0.232%	2.15489	0.21	0.46
10	6856	835	816	1020	1227	2213	300	5023	0.467%	11.41883	1.18	2.43
50	40439	706	728	894	1013	1434	303	5038	0.064%	67.30299	6.29	14.38

CPU: 1 | Memory: 1g | MaxPoolSize: 100

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1452	412	407	509	584	788	294	5024	0.069%	2.42028	0.23	0.52
10	7855	729	780	909	1025	1433	306	5032	0.013%	13.08576	1.21	2.80
50	44932	634	653	844	915	1281	302	5025	0.009%	74.82095	6.89	16.00

CPU: 2 | Memory: 1g | MaxPoolSize: 100

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1400	428	408	511	647	1239	295	5027	0.071%	2.33314	0.22	0.50
10	7899	725	751	898	946	1325	301	5024	0.013%	13.15459	1.21	2.81
50	43074	662	734	849	911	1116	305	5023	0.005%	71.73930	6.59	15.34

After Fix

After doing this fix [1] conducted a perf-test for the same setup. Following are the results for different configuration settings.

CPU: 0.5 | Memory: 1g | MaxPoolSize: 100

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1429	419	408	507	595	742	295	5026	0.140%	2.38159	0.22	0.51
10	7595	754	814	901	1002	1231	298	5024	0.066%	12.64739	1.17	2.70
50	38803	735	748	934	1102	1699	314	5028	0.093%	64.62062	6.02	13.81

CPU: 1 | Memory: 1g | MaxPoolSize: 100

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1435	417	408	509	583	721	299	5067	0.139%	2.39108	0.23	0.51
10	7275	788	818	921	1025	1229	304	5028	0.014%	12.10916	1.10	2.59
50	42005	679	745	873	940	1113	303	5032	0.010%	69.94364	6.37	14.96

CPU: 2 | Memory: 1g | MaxPoolSize: 100

Concurrency	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
1	1446	414	408	507	583	809	300	5087	0.069%	2.41053	0.22	0.52
10	7387	776	816	898	963	1219	307	5024	0.014%	12.29912	1.12	2.63
50	43105	662	748	834	900	1025	304	5023	0.002%	71.77539	6.52	15.35

[1] - #4982

[ayeshLK]

After comparing and contrasting the results it is observed that in lower resource level the ASB connector is performing in an acceptable rate after the improvements. Hence, closing this issue.