Op#3648 reduce memory consumption in track parser #554

KueblerJelle · 2024-09-18T07:31:21Z

OP#3648

add lazy version of track dataset and detection parser

Detection parser can accept a stream of file contents to lazily load files only when they are needed

… made to TrackDataset in subclass

…ode duplication with python track parser make some functionality reusable through classmethod use bzip 2 json stream reader to sort files by start date

renamed BulkPythonDetectionParser back to PythonDetectionParser extracted methods for creating PythonDetections and PythonTracks as static methods on module level

i have some old ottrk files for testing that could no longer be parsed a new metadata format fixer fixes the old date string format by transforming it to timestamp format TODO: review versions of fixer

clean up streaming parser and add more doc strings

found potential memory leak: delete det_list in StreamOttrkParser.parser() OttrkVideoParser crashed in a manual test because the metadata format did not match -> apply format fixer

…n_application.py

also reduced code duplication in cli.py and main_application.py also fixed/adapted tests

…b.com/OpenTrafficCam/OTAnalytics into OP#3648-create-streaming-track-parser

fixed test_cli patches

briemla

Looks good. As this is a larger PR, I have few questions and comments.

OTAnalytics/application/datastore.py

OTAnalytics/plugin_parser/export.py

OTAnalytics/plugin_parser/otvision_parser.py

tests/OTAnalytics/plugin_ui/test_cli.py

OTAnalytics/plugin_ui/customtkinter_gui/dummy_viewmodel.py

OTAnalytics/plugin_ui/cli.py

…t is defined by the EventListExporter implementation

…o not yet produce an output file also tested export mode FLUSH, with same behavior as OVERWRITE

…trk or otdet

…on CliMode

…n no more track data is available

…tection parser (similar to test_otvision_parser)

… windows

…which expects the file not to exist previously the test case: test_export seems to have created that file/dir

KueblerJelle and others added 30 commits December 12, 2023 18:00

first attempts for streaming track parser

ed0c9f0

add lazy version of track dataset and detection parser

renamed lazy to streaming

6e3c81f

Merge branch 'main' into OP#3648-create-streaming-track-parser

5c2d677

Added functionality to parse detections as stream for multiple files

b3944ed

Detection parser can accept a stream of file contents to lazily load files only when they are needed

Merge branch 'main' into OP#3648-create-streaming-track-parser

66eb2d1

Adapted changes to DetectionParser: added id_generator

723f4c8

Reduced code duplication by inheriting OttrkParser, adapted additions…

ed30083

… made to TrackDataset in subclass

Add test of old parser to compare ram usage

2efca9d

Merge branch 'main' into OP#3648-create-streaming-track-parser

8939dd2

Merge branch 'main' into OP#3648-create-streaming-track-parser

9ca3fde

Merge branch 'main' into OP#3648-create-streaming-track-parser

e65ec68

refactored stream detection / track parser -> requires reduction of c…

5c5bfc9

…ode duplication with python track parser make some functionality reusable through classmethod use bzip 2 json stream reader to sort files by start date

added doc strings to streaming_parser.py

b1e627e

cleaned up otvision_parser

f01ed28

renamed BulkPythonDetectionParser back to PythonDetectionParser extracted methods for creating PythonDetections and PythonTracks as static methods on module level

added metadata format fixer for recorded start date format

328b291

i have some old ottrk files for testing that could no longer be parsed a new metadata format fixer fixes the old date string format by transforming it to timestamp format TODO: review versions of fixer

move stateless reusable methods from PythonTrackDataset to module level

26950b5

clean up streaming parser and add more doc strings

Added more doc strings to streaming_parser.py

ec9c022

fixed bugs in ottrk_parser and streaming_parser

2b7d87a

found potential memory leak: delete det_list in StreamOttrkParser.parser() OttrkVideoParser crashed in a manual test because the metadata format did not match -> apply format fixer

add stream version of OTAnalyticsCli and code to instantiate it in ma…

cec9362

…n_application.py

Merge branch 'main' into OP#3648-create-streaming-track-parser

4ab4171

apply streaming track parser in cli loop

7f23cd8

Use always all sections to create events

c5fd43a

Track export does not work with streaming parser

09a3fd9

Add ijson as dependency

d283752

Remove dependency to track repository in ModeTagger

a6c1216

Added cli-mode to cli args to start either bulk or stream processing

bd19151

also reduced code duplication in cli.py and main_application.py also fixed/adapted tests

Merge branch 'OP#3648-create-streaming-track-parser' of https://githu…

3c545de

…b.com/OpenTrafficCam/OTAnalytics into OP#3648-create-streaming-track-parser

fixed cli mode in ArgparseCliParser.parser()

21b6436

fixed MetadataFixer for recorded start date -> catch TypeError

c940583

fixed test_cli patches

Merge branch 'main' into OP#3648-create-streaming-track-parser

ed26a52

Merge branch 'main' into OP#3648-create-streaming-track-parser

7a67ed3

briemla changed the title ~~Op#3648 remove memory consumption in track parser~~ Op#3648 reduce memory consumption in track parser Oct 22, 2024

briemla requested changes Oct 25, 2024

View reviewed changes

ar0305 and others added 18 commits November 14, 2024 11:11

the "format" attribute of the EventExportSpecification is not used, i…

359d9a3

…t is defined by the EventListExporter implementation

added test case for CsvExport to test incremental export modes that d…

821ac6d

…o not yet produce an output file also tested export mode FLUSH, with same behavior as OVERWRITE

allow MetadataFixers to pick which formats version they depend on: ot…

4aaf9a5

…trk or otdet

removed debug print

ebb25cd

removed obsolete parentheses

de27653

use 'with' context manager to read bzip2 json file

da93fa7

fixed test of MetadataFixes

613f1b3

fixed, typo, fstring error and print statement

5037e86

remove parameterize in regression_otanalytics: ..._recreate_test_data

0c06933

reduced cod duplication in test_cli when setting up Cli object based …

2239207

…on CliMode

fixed missing OTAnalytics prefix in import

719060e

removed commented out code

e3b6be6

end generator function in StreamOttrkParser explicitly via return whe…

124eeed

…n no more track data is available

added test cases for streaming ottrk parser and the used streaming de…

0c2ea41

…tection parser (similar to test_otvision_parser)

Merge branch 'main' into OP#3648-create-streaming-track-parser

bfbe226

restore precommit config of update-type-stub which caused problems on…

8efcefd

… windows

use other output file name in TestCsvExport.test_increment_no_export …

d21bda1

…which expects the file not to exist previously the test case: test_export seems to have created that file/dir

Merge branch 'main' into OP#3648-create-streaming-track-parser

6ce7e47

briemla previously approved these changes Dec 2, 2024

View reviewed changes

briemla added 2 commits December 2, 2024 14:35

Merge branch 'main' into OP#3648-create-streaming-track-parser

c4bc805

Fix whitespaces

8fccf06

briemla dismissed their stale review via 8fccf06 December 3, 2024 12:36

Fix whitespaces

4aa4f42

briemla enabled auto-merge December 3, 2024 12:37

briemla approved these changes Dec 3, 2024

View reviewed changes

briemla merged commit 8ab1c26 into main Dec 3, 2024
6 checks passed

briemla deleted the OP#3648-create-streaming-track-parser branch December 3, 2024 12:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Op#3648 reduce memory consumption in track parser #554

Op#3648 reduce memory consumption in track parser #554

KueblerJelle commented Sep 18, 2024 •

edited by briemla

Loading

briemla left a comment

Op#3648 reduce memory consumption in track parser #554

Op#3648 reduce memory consumption in track parser #554

Conversation

KueblerJelle commented Sep 18, 2024 • edited by briemla Loading

briemla left a comment

Choose a reason for hiding this comment

KueblerJelle commented Sep 18, 2024 •

edited by briemla

Loading