Use thread-safe cache in thread_local_caching_allocator #6539

Pansysk75 · 2024-09-07T00:47:34Z

No description provided.

StellarBot · 2024-09-07T00:57:52Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	--	--	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-07T00:47:36+00:00
HPX Commit	`d27ac2e`	`e972af2`
Clustername	rostam	rostam
Datetime	2024-03-18T09:18:04.949759-05:00	2024-09-06T19:55:11.519858-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Envfile

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-07T00:47:36+00:00
HPX Commit	`d27ac2e`	`e972af2`
Clustername	rostam	rostam
Datetime	2024-03-18T09:19:53.062988-05:00	2024-09-06T19:57:01.633845-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Envfile

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	--	--	--
Stream Benchmark - Scale	--	--	---
Stream Benchmark - Triad	--	--	--
Stream Benchmark - Copy	--	--	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-07T00:47:36+00:00
HPX Commit	`d27ac2e`	`e972af2`
Clustername	rostam	rostam
Datetime	2024-03-18T09:20:13.002391-05:00	2024-09-06T19:57:19.484829-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Envfile

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

StellarBot · 2024-09-07T22:47:49Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	--	--	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-07T22:36:42+00:00
HPX Commit	`d27ac2e`	`b751e72`
Clustername	rostam	rostam
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5
Envfile
Datetime	2024-03-18T09:18:04.949759-05:00	2024-09-07T17:45:10.927119-05:00
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-07T22:36:42+00:00
HPX Commit	`d27ac2e`	`b751e72`
Clustername	rostam	rostam
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5
Envfile
Datetime	2024-03-18T09:19:53.062988-05:00	2024-09-07T17:47:00.957806-05:00
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	--	--	--
Stream Benchmark - Scale	--	--	---
Stream Benchmark - Triad	--	--	--
Stream Benchmark - Copy	--	--	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-07T22:36:42+00:00
HPX Commit	`d27ac2e`	`b751e72`
Clustername	rostam	rostam
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5
Envfile
Datetime	2024-03-18T09:20:13.002391-05:00	2024-09-07T17:47:18.556829-05:00
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

codacy-production · 2024-09-08T01:01:32Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.07%	✅ 88.89%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`d9b5f6c`)	223405	190053	85.07%
Head commit (`c008395`)	191295 (-32110)	162869 (-27184)	85.14% (+0.07%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6539)	9	8	88.89%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

hkaiser

The remaining problems are unrelated. LGTM! Thanks!

libs/core/allocator_support/include/hpx/allocator_support/thread_local_caching_allocator.hpp

StellarBot · 2024-09-09T17:27:39Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	--	--	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-09T17:15:16+00:00
HPX Commit	`d27ac2e`	`ed89bf1`
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Clustername	rostam	rostam
Envfile
Datetime	2024-03-18T09:18:04.949759-05:00	2024-09-09T12:25:01.395820-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-09T17:15:16+00:00
HPX Commit	`d27ac2e`	`ed89bf1`
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Clustername	rostam	rostam
Envfile
Datetime	2024-03-18T09:19:53.062988-05:00	2024-09-09T12:26:52.095834-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	--	--	--
Stream Benchmark - Scale	--	--	---
Stream Benchmark - Triad	--	--	--
Stream Benchmark - Copy	--	--	--

Info

Property	Before	After
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-09T17:15:16+00:00
HPX Commit	`d27ac2e`	`ed89bf1`
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Clustername	rostam	rostam
Envfile
Datetime	2024-03-18T09:20:13.002391-05:00	2024-09-09T12:27:09.764152-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

hkaiser · 2024-09-09T18:35:28Z

@Pansysk75 we still have the circular dependency between the concurrency and allocator modules. Also, some of the header tests are failing (most likely caused by the circular dependency issue)

Pansysk75 · 2024-09-09T18:54:08Z

@Pansysk75 we still have the circular dependency between the concurrency and allocator modules. Also, some of the header tests are failing (most likely caused by the circular dependency issue)

Hmm I suppose that means the allocator_support module is higher in the module hierarchy than the concurrency module?
I'm not exactly sure what fixing this would involve.

Oh, also, we should probably make all accesses to members of allocated_cache thread safe... Not sure why I ignored that. Also, this now makes me dislike this solution even more.

I opened an issue #6540 to document this.

hkaiser · 2024-09-09T22:12:05Z

@Pansysk75 we still have the circular dependency between the concurrency and allocator modules. Also, some of the header tests are failing (most likely caused by the circular dependency issue)

Hmm I suppose that means the allocator_support module is higher in the module hierarchy than the concurrency module? I'm not exactly sure what fixing this would involve.

The concurrency module is higher up in the hierarchy, see: https://hpx-docs.stellar-group.org/latest/report/v1.10.0/core/allocator_support.html

Oh, also, we should probably make all accesses to members of allocated_cache thread safe... Not sure why I ignored that. Also, this now makes me dislike this solution even more.

Yes, the deallocated and allocated members should be atomic. But we should combine both variables into one counter for that.

I opened an issue #6540 to document this.

Thanks!

StellarBot · 2024-09-11T23:45:24Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	--	--	--

Info

Property	Before	After
HPX Commit	`d27ac2e`	`3c74e4b`
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-11T23:01:22+00:00
Envfile
Datetime	2024-03-18T09:18:04.949759-05:00	2024-09-11T18:42:36.321504-05:00
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	--

Info

Property	Before	After
HPX Commit	`d27ac2e`	`3c74e4b`
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-11T23:01:22+00:00
Envfile
Datetime	2024-03-18T09:19:53.062988-05:00	2024-09-11T18:44:26.972373-05:00
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	--	--	--
Stream Benchmark - Scale	--	--	--
Stream Benchmark - Triad	--	--	--
Stream Benchmark - Copy	--	--	---

Info

Property	Before	After
HPX Commit	`d27ac2e`	`3c74e4b`
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-11T23:01:22+00:00
Envfile
Datetime	2024-03-18T09:20:13.002391-05:00	2024-09-11T18:44:44.719698-05:00
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

codacy-production · 2024-09-12T01:30:08Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.07%	✅ 84.21%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`d9b5f6c`)	223405	190053	85.07%
Head commit (`f8d92b0`)	191295 (-32110)	162869 (-27184)	85.14% (+0.07%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6539)	38	32	84.21%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

… modules

StellarBot · 2024-09-12T16:52:39Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	--	--	--

Info

Property	Before	After
HPX Commit	`d27ac2e`	`1bb0796`
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-12T16:43:09+00:00
Clustername	rostam	rostam
Envfile
Datetime	2024-03-18T09:18:04.949759-05:00	2024-09-12T11:50:01.583868-05:00
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	--

Info

Property	Before	After
HPX Commit	`d27ac2e`	`1bb0796`
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-12T16:43:09+00:00
Clustername	rostam	rostam
Envfile
Datetime	2024-03-18T09:19:53.062988-05:00	2024-09-12T11:51:51.740992-05:00
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	--	--	--
Stream Benchmark - Scale	--	-	--
Stream Benchmark - Triad	--	--	--
Stream Benchmark - Copy	--	--	--

Info

Property	Before	After
HPX Commit	`d27ac2e`	`1bb0796`
HPX Datetime	2024-03-18T14:00:30+00:00	2024-09-12T16:43:09+00:00
Clustername	rostam	rostam
Envfile
Datetime	2024-03-18T09:20:13.002391-05:00	2024-09-12T11:52:09.519499-05:00
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/18.1.5/bin/clang++ 18.1.5

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

hkaiser · 2024-09-13T17:42:34Z

@Pansysk75 I think this is good to go as a stop-gap measure circumventing the underlying problem from happening. What do you think?

Pansysk75 · 2024-09-13T18:27:33Z

@Pansysk75 I think this is good to go as a stop-gap measure circumventing the underlying problem from happening. What do you think?

Sure, one last thing, why is this check implemented like that? Our underlying data structure has a .size() member function, is it any different to use that?

hpx/libs/core/allocator_support/include/hpx/allocator_support/thread_local_caching_allocator.hpp

Lines 95 to 104 in f8d92b0

    
           void deallocate(pointer p, size_type n) noexcept 
        
           { 
        
               data.push(std::make_pair(p, n)); 
        
               if (++deallocated > 2 * (allocated + 16)) 
        
               { 
        
                   clear_cache(); 
        
                   allocated = 0; 
        
                   deallocated = 0; 
        
               } 
        
           }

libs/core/allocator_support/include/hpx/allocator_support/thread_local_caching_allocator.hpp

            {
                throw std::bad_array_new_length();
            }
-            return cache().allocate(n);
+
+            auto [p, _] = detail::allocate_from_cache(num_cache);


Pansysk75 requested a review from hkaiser as a code owner September 7, 2024 00:47

hkaiser added type: enhancement compiler: gcc compiler: clang category: data-structures labels Sep 7, 2024

hkaiser added this to the 1.11.0 milestone Sep 7, 2024

Use thread-safe cache in thread_local_caching_allocator

5ed299c

Pansysk75 force-pushed the fix-thread-local-caching-allocator branch from 30370cb to 5ed299c Compare September 7, 2024 22:36

hkaiser previously approved these changes Sep 9, 2024

View reviewed changes

hkaiser reviewed Sep 9, 2024

View reviewed changes

libs/core/allocator_support/include/hpx/allocator_support/thread_local_caching_allocator.hpp Outdated Show resolved Hide resolved

Fix race condition

c008395

Pansysk75 dismissed hkaiser’s stale review via c008395 September 9, 2024 17:15

hkaiser force-pushed the fix-thread-local-caching-allocator branch from 459cf67 to bd454ff Compare September 11, 2024 23:01

Resolve circular dependency between concurrency and allocator_support…

f8d92b0

… modules

hkaiser force-pushed the fix-thread-local-caching-allocator branch from bd454ff to f8d92b0 Compare September 12, 2024 16:43

hkaiser force-pushed the fix-thread-local-caching-allocator branch 3 times, most recently from fc10910 to a3e95e9 Compare September 17, 2024 00:31

github-advanced-security bot found potential problems Sep 17, 2024

View reviewed changes

libs/core/allocator_support/include/hpx/allocator_support/thread_local_caching_allocator.hpp Fixed Show fixed Hide fixed

hkaiser force-pushed the fix-thread-local-caching-allocator branch 3 times, most recently from 73abaa9 to 6478b44 Compare September 21, 2024 19:58

github-advanced-security bot found potential problems Sep 21, 2024

View reviewed changes

hkaiser force-pushed the fix-thread-local-caching-allocator branch 4 times, most recently from 5e3c656 to e12bc1c Compare September 25, 2024 19:12

Factoring out allocator cache

64988ae

hkaiser force-pushed the fix-thread-local-caching-allocator branch from e12bc1c to 64988ae Compare September 25, 2024 19:32

Use thread-safe cache in thread_local_caching_allocator #6539

Are you sure you want to change the base?

Use thread-safe cache in thread_local_caching_allocator #6539

Conversation

Pansysk75 commented Sep 7, 2024

StellarBot commented Sep 7, 2024

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

StellarBot commented Sep 7, 2024

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

codacy-production bot commented Sep 8, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

hkaiser left a comment

Choose a reason for hiding this comment

StellarBot commented Sep 9, 2024

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

hkaiser commented Sep 9, 2024

Pansysk75 commented Sep 9, 2024 • edited Loading

hkaiser commented Sep 9, 2024 • edited Loading

StellarBot commented Sep 11, 2024

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

codacy-production bot commented Sep 12, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

StellarBot commented Sep 12, 2024

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

hkaiser commented Sep 13, 2024

Pansysk75 commented Sep 13, 2024 • edited Loading

codacy-production bot commented Sep 8, 2024 •

edited

Loading

Pansysk75 commented Sep 9, 2024 •

edited

Loading

hkaiser commented Sep 9, 2024 •

edited

Loading

codacy-production bot commented Sep 12, 2024 •

edited

Loading

Pansysk75 commented Sep 13, 2024 •

edited

Loading