Skip to content

Home

Jump to bottom
AMProduction edited this page Sep 5, 2023 · 6 revisions

Welcome to the SearchMyData-2.0 App wiki! The web version of the SearchMyData App

License: MIT CodeQL Build and Push Docker Image to Docker Hub Docker Image Version (latest semver) Docker Image Size with architecture (latest by date/latest semver)

Apache Airflow Python Flask MongoDB Docker GitHub Actions

1. Description

The app gives the possibility to perform a search into the Ukrainian government Open data portal datasets.
At this moment (September 2023) integrated Information on missing citizens, Information about people hiding from the authorities, Unified register of debtors, Unified State Register of Legal Entities, Individual Entrepreneurs and Public Associations (temporarily unavailable) and Integrated Unified State Register of Lustrated Persons.

2. Requirements

Recommended Operating Systems

  • Windows: 10 or newer
  • MAC: OS X v10.7 or higher
  • Linux

Hardware requirements

  • Processor: 2 gigahertz (GHz) or faster processor or SoC
  • RAM: 8+ GB
  • SSD

Prerequisites

Docker

Airflow is tested with:

  • Python: 3.8, 3.9, 3.10, 3.11
  • Databases:
    • PostgreSQL: 11, 12, 13, 14, 15
    • MySQL: 5.7, 8
    • SQLite: 3.15.0+
    • MSSQL(Experimental): 2017, 2019
  • Kubernetes: 1.23, 1.24, 1.25, 1.26, 1.27

3. Changelog

v2.0

The main pull requests

The main changes

  • Migration to Flask
  • UI improvements
  • Refactoring to implement PEP 8
  • Update the issue templates
  • Create CODE_OF_CONDUCT.md
  • Search results save to PDF via html2pdf library
  • The docker-compose.yml added
  • Migrated to Airflow
  • Refactoring ETL pipeline
  • Create SECURITY.md
  • Doc strings updated
  • Refactored get_collections_info() function
  • Dynamic display year in the footer of the pages
  • Read the DB connection string from the .env file
  • HTML structure changed
  • Refactored search results representation
  • Code beautification
  • Added CSRF app protection
  • Added WTForms support

v2.1a

The main pull requests

The main changes

  • Dockerfile added
  • The docker-compose.yml modified
  • Added GitHub actions to build and push images to Docker hub
  • Return documents per page and page numbers
  • Added the paginations to the template
  • Added process page_number from the URL
  • Added processing NEXT and PREVIOUS buttons
  • Docstring updated. Added documents found count. SearchForm modified
  • Added saving the search query to the session variable

v2.2b

The main pull requests

The main changes

  • Create PULL_REQUEST_TEMPLATE.md
  • Create CONTRIBUTING.md
  • Refactored the service collection processing
  • Added: debtors_model, lustrated_persons_model, missing_persons_model, wanted_persons_model
  • Added mapping for ObjectID field in the models. Added processing Missing Persons register through the ORM
  • Added processing all registers through the ORM
  • Refactored /result route
  • Change import in routes.py
  • Code prettified
  • readme.md updated
  • changelog.md added

v2.3

The main pull requests

The main changes

4. How to use

Use docker-compose.yml to start services.
The ENV variables:

  • FLASK_DEBUG. True or False. The built-in Werkzeug development server provides a debugger which shows an interactive traceback in the browser when an unhandled error occurs during a request. This debugger should only be used during development.
  • FLASK_APP. The environment variable is the name of the module to import at flask run. Usually main.py.
  • SECRET_KEY. A secret key that will be used for securely signing the session cookie and can be used for any other security related needs by extensions or your application. It should be a long random bytes or str.
  • MONGO_URI. The standard URI connection scheme.
  • MONGO_INITDB_DATABASE. The DB name.
  • DOCUMENTS_PER_PAGE. The count of search result records per page.
  • APP_NAME. In our case searchmydata2.
  • TAG. The app version v2.2b.

5. Miscellaneous

Developed in PyCharm - The Python IDE for Professional Developers.
License kindly provided by JetBrains Community Support Team
JetBrains Logo (Main) logo PyCharm logo PyCharm logo

Copyright 2023 AMProduction

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Clone this wiki locally