Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve StreamingFeatherWriter to support file rotation and better me management #1954

Merged
merged 4 commits into from
Sep 24, 2024

Conversation

graceyangfan
Copy link
Contributor

@graceyangfan graceyangfan commented Sep 21, 2024

Pull Request

This PR adds file rotation functionality to the StreamingFeatherWriter class and improves time management and file naming. #1945
Key changes include:
Implementation of file rotation based on size, time interval, and scheduled dates.
Use of a Clock object for time management, improving flexibility and testability.
Addition of timestamps to filenames for better organization and identification.
Tracking of file sizes and creation times to support rotation functionality and provide more information.
Addition of methods to query current file information and next rotation time.

Type of change

Delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

How has this change been tested?

The code undergoes testing in both live trading and backtesting processes. It functions correctly in live trading. However, in backtesting, there is an issue with the initialization of _next_rotation_time because the clock component is not initialized immediately, but it does not affect its usage.

@graceyangfan graceyangfan changed the title Improve StreamingFeatherWriter to support file rotation and better ti… Improve StreamingFeatherWriter to support file rotation and better me management Sep 21, 2024
@faysou
Copy link
Collaborator

faysou commented Sep 21, 2024

While you're working on this, maybe you could update this function that converts a feather file to a parquet catalogue.

https://github.com/nautechsystems/nautilus_trader/blob/develop/nautilus_trader/persistence/catalog/parquet.py#L759

@cjdsellers
Copy link
Member

Hey @graceyangfan

Thanks for the contribution!

Looks like there are just some tests failing in tests/unit_tests/persistence/test_streaming.py.

@limx0
Copy link
Collaborator

limx0 commented Sep 21, 2024

+1, nice one @graceyangfan !

@cjdsellers cjdsellers merged commit 1a98cfe into nautechsystems:develop Sep 24, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants