Organization Parser

Description

Utilities and Telegram bot for parsing data from Places API by Yandex.

This can help you to search for companies and collect data about them on request to the Yandex database of organizations.

For using cli utils and bot you need to get API key from here.

(Remember that the free key allows you to make up to 500 calls to the Yandex API per day.)

Main technologies:

• python 3.11
• poetry
• aiogram 3.0
• aiogram_dialog 2.0
• httpx and Beautiful Soup
• sqlalchemy 2.0
• alembic
• aiosqlite
• docker

Demo

You can try to using that bot here.

Installation

For running and building app you need to get source code of this repo:

git clone https://github.com/andy-takker/organization_parser

or you can get public docker image from Docker hub for ARM64 (here).

Configuration

An example of the settings is in the file .env.dev.

cp .env.dev .env

TELEGRAM_BOT_TOKEN  # Your telegram bot token. Required
SQLITE_DB_PATH      # Path to file sqlite db. Optional, default: ./bot.sqlite3

Running

You can run bot in two ways: with docker or natively as is

Docker

docker build . --tag organization_parser_bot:latest
docker run --env-file .env -it organization_parser_bot:latest

Local

For local running I recommend use venv and you need to install poetry.

python -m venv .venv
source .venv/bin/activate  # for unix systems
pip install -U pip poetry 
poetry install --no-root
alembic -c ./src/alembic.ini upgrade head # create db and all tables
TELEGRAM_BOT_TOKEN=XXX PYTHONPATH=. python ./src/bot/main.py

Database

Project database is SQLite with async driver - aiosqlite.

alembic is used to manage the database version. To automatically create a new migration, run

alembic -c ./src/alembic.ini revision --autogenerate -m "New migration"

To update database to last actual version use

alembic -c ./src/alembic.ini upgrade head

Using

1. At first, send /start

2. For registration in bot you need set Yandex API Key to use the API. The key looks in the UUID format. You can get key here

3. After that you can make requests:

   • set place where need search
   • set query (what you looking for)
   • set radius of searching

4. Click next to get parsed CSV file.

5. Finish!