Logging Optimizations #1714
Logging Optimizations #1714
Conversation
… to nil. Fix UDP multicast test on Mac
Codecov Report
@@ Coverage Diff @@
## master #1714 +/- ##
==========================================
- Coverage 78.22% 78.17% -0.06%
==========================================
Files 235 236 +1
Lines 18188 18251 +63
==========================================
+ Hits 14228 14267 +39
- Misses 3960 3984 +24
Continue to review full report at Codecov.
|
|
The difference this makes is huge. My CPUs no longer have any stress. |
| @@ -196,23 +196,23 @@ static VALUE packet_initialize(int argc, VALUE *argv, VALUE self) | |||
| packet_name = argv[1]; | |||
| default_endianness = symbol_BIG_ENDIAN; | |||
| description = Qnil; | |||
| buffer = rb_str_new2(""); | |||
| buffer = Qnil; | |||
There was a problem hiding this comment.
Wow optimizations in the C code. Nice! Simply don't make the allocation up front ... you think this made a big difference too? Memory?
There was a problem hiding this comment.
The buffer changes didn't make as much of a difference for the use cases we've been looking at because the packets aren't that big. If you had 50,000 large packets, it would make a significant difference.
| return @buffer.length if @buffer | ||
|
|
||
| return 0 | ||
| allocate_buffer() |
There was a problem hiding this comment.
This name is a little misleading and I deleted my previous comment. It's basically allocate buffer if needed. Maybe get_local_buffer() (I see you use the term local below).
There was a problem hiding this comment.
I'll change to allocate_buffer_if_needed
| def allocate_buffer | ||
| unless @buffer | ||
| @buffer = ZERO_STRING * @defined_length | ||
| @buffer.force_encoding(ASCII_8BIT_STRING) |
| else | ||
| raise "No buffer available for method_missing" | ||
| return read(name.to_s) |
There was a problem hiding this comment.
So we always have @buffer now?
There was a problem hiding this comment.
Effectively. It will allocate if needed.
| @@ -350,10 +350,10 @@ module Cosmos | |||
| end | |||
|
|
|||
| describe "read_item", no_ext: true do | |||
| it "complains if no buffer given" do | |||
| it "works if no buffer given" do | |||
There was a problem hiding this comment.
Because a buffer is created internally right?
|
For the specific use case we've been looking at, it reduced the number of threads from more than the system could handle (100,000+) down to 110. CPU utilization seems to be about 75% less as well, but that is hard to really quantify as it couldn't handle it before. |
Make all packet log writers share cycle thread. With large numbers of packets, this makes a huge difference in the number of threads needed to run COSMOS.
Default packet buffer to nil. This improves everywhere where a System is generated. Packets will not allocate space for their buffers until needed. Will greatly reduce RAM allocation in situations where there are large numbers of defined but unused packets.
Fix UDP multicast test on Mac. Macs do not like Multicast specifying port 0.