Spark to MongoDB #12
Closed
Spark to MongoDB #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replaced the SingletonMeta pattern with a standard class implementation. Introduced thread-safe methods for loading, saving, and configuring data paths. This simplifies the design and improves readability while maintaining concurrency safety.
Introduce logging to provide detailed information on MongoDB connection success or failure. Replaced `print` statements with `logger` to standardize output and improve traceability. This enhances debugging and monitoring capabilities for the database connection process.
Reorganized and encapsulated functionality into the `DataIdMapper` class for better maintainability and clarity. Removed redundant functions and replaced print statements with logging for consistent logging practices. Updated the invoker to use the refactored `DataIdMapper` methods for data normalization.
Break down main functions into smaller, reusable modules for better readability and maintainability. Introduced error handling, logging improvements, type hints, and modernized list comprehensions. Simplified Spark data loading, preprocessing, aggregation, and configuration setup with enhanced structure.
Eliminated the explicit read mode ("r") when opening the config file, as it is the default behavior in Python. This simplifies the code without altering its functionality.
MacMat01
added
documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes significant improvements and refactoring across multiple files to enhance the configuration management, data processing, and logging functionalities. The most important changes include refactoring the
ConfigurationManagerclass, updating the geoJSON files, and improving the data aggregation and storage process.Configuration Management:
config/configuration_manager.py: Refactored theConfigurationManagerclass to simplify the configuration loading and data path setup. Removed the singleton pattern and added methodsload_configandconfigure_data_pathsfor better modularity.GeoJSON Updates:
data/geojson/FirstFloor.geojson: Updated room IDs to be integers and added new sensor features with coordinates. [1] [2] [3]data/geojson/SecondFloor.geojson: Addediot_device_idplaceholders and new sensor features with coordinates. [1] [2] [3] [4]Data Aggregation and Storage:
src/analysis/analyzer.py: Enhanced data aggregation by addingaggregate_single_dataframeandaggregate_by_minute_windowfunctions. Introducedstore_geojson_to_dbandstore_single_geojson_to_dbfor saving aggregated data to MongoDB. Improved logging throughout the analysis process.Logging Improvements:
main.py: Improved logging setup and refactored the main execution flow into modular functions for better readability and error handling.src/database/db_config.py: Added logging for MongoDB connection attempts and errors.Type Annotations and Code Cleanup:
src/analysis/analysis_preprocessor.py: Added type annotations to therun_preprocessfunction and refactored it for clarity.src/analysis/spark_data_reader.py: Added type annotations and refactored methods for better readability and error handling.