Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Data Buffer compression ports and components #1959

Open
timcanham opened this issue Apr 10, 2023 · 1 comment
Open

Implement Data Buffer compression ports and components #1959

timcanham opened this issue Apr 10, 2023 · 1 comment

Comments

@timcanham
Copy link
Collaborator

F´ Version 3.x
Affected Component new component

Feature Description

Add a set of data compression ports and a bzip2 based compression component.

Requirements:

  1. input Fw::Buffer port for incoming uncompressed buffers
  2. output Fw::Buffer port for outgoing compressed buffers
  3. Component would not require additional Fw::Buffer allocation; compress in place or use temporary working buffer.

Rationale

This will allow the DpWriter component (and others) to optionally compress data that is in Fw::Buffer memory.

@matt392code
Copy link
Contributor

compression-component.txt
Proposed BZip2Compressor component for F' that meets the requirements. Here's a breakdown of the key features:

  1. Port Configuration:
  • Input port bufferIn of type Fw::BufferSend for receiving uncompressed data
  • Output port bufferOut of type Fw::BufferSend for sending compressed data
  • Additional standard F' ports for events, telemetry, and time
  1. Buffer Handling:
  • Compresses data in-place by using a temporary working buffer
  • Properly sizes temporary buffer according to bzip2 requirements (input + 1% + 600 bytes)
  • Copies compressed data back to original buffer
  • No additional permanent buffer allocation required
  1. Telemetry & Events:
  • Tracks compression statistics including:
    • Average compression ratio
    • Number of buffers processed
  • Events for compression start, completion, and errors
  • Detailed formatting for monitoring compression performance
  1. Error Handling:
  • Comprehensive error checking and reporting
  • Warning events for compression failures
  • Memory management with proper cleanup

To integrate this component:

  1. Add to your topology:
instance Compression.BZip2Compressor compressor
  1. Connect the ports:
connect dpWriter.bufferOut -> compressor.bufferIn
connect compressor.bufferOut -> yourNextComponent.bufferIn
  1. Build dependencies:
  • Add bzip2 library to your build configuration
  • Include the component in your CMakeLists.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants