Create parser, update admin panel and fix bugs (#22)

* Initial commit * Initial commit * Create dockerfiles for services (#1) * Create .dockerignore * Move the file with the sample secrets * Update packets * Create imports for migrations * Create a migration script * Fix bug in template * Create docker-compose config * Create dockerfile * Update main * Add new dependencies * Update settings * Create the alembic of config * Create a model connection and a database connection * Create the alembic of config * Update readme * API, models & repositories. * Some bugs fixed after Alex review. * User model edited. * Code refactoring and handlers refinement * Create registration form, sending messages to the coordinator and help command * add requirements.txt * Admin panel has been added. * Fix config * Update dockerfile * Update admin panel. * Create requests to the service to receive data (#6) * Update settings * Update validate for password * Move packets with the schemas * Create base of exceptions for requests * Create requests to the service to receive data * Update protocols * Update schemas * Пофиксил ошибки и немного навел порядок. * env.example * Add redis settings to the bot. * Add menu and change logic bot (#7) * Add menu * test * Add menu and improve bot logic * For test * Fix users service and add deafault settings for admin. * test * 4tests * add 12 tests * del test_routes * Update migrations * Fix bugs * Create method for checking meeting * Move constants in file * Update db * Update imports * Create an error description return * Update endpoints * Create timezone and update methods * Update models * Update uow * Optimize database queries * Update schemas * Update provider * Update repositories * Update modules * Update models * Fix bugs * Fix bugs * Update schemas * Create validate for date * Update imports * Update dockerfile * Update imports * Plug api to bot * Create new migrations * Update packets * Create new functional * Add the ability to create administrators * Add the ability to create administrators * Update config for pytest * Create decorator * Update main * Move files * Create password hash * Add docstrings * Update env.example * Update .dockerignore * Update scripts * Update dockerfile * Create requirements for prod * Create readme * Update settings * Update requirements * Update the admin widget * Update dockerfile * Update dependencies * Update dependencies * Update migrations * Update models * Fix bug * Update requirements * Update settings * Update env.example * Update validate * Update validate * Update meeting of service * Update validation * Fix bug * Update validate * Fix bug * Fix bug * Update readme * Add tests for user endpoints. * Change pytest.ini. * Add test for meetings. * Fixed bugs in tests. * Fix date * Fix date * Fix date * Fix date * New feature logging (#16) * logging * meeting and users finish * Update logging and move files * Update dockerfile * Update scripts * Create workflow * Create docker-compose for prod * Update scripts * Update deploy.yml * Update main * Create Dockerfile * Create nginx config * Update docker compose config * Update env.example * Create webhooks * Update nginx config * Update settings * Update nginx config * Update constants * Create new custom action * Create parser * Update req * Update users * Update parser * Update parser --------- Co-authored-by: pavlovvitaliy <me@pavlovvitaliy.ru> Co-authored-by: Boichuk Julia <juliaboichuk503@gmail.com> Co-authored-by: juliaboichuk <37875454+juliaboichuk@users.noreply.github.com> Co-authored-by: Александр Русанов <shurik.82rusanov@yandex.ru> Co-authored-by: Александр Русанов <130507152+shurikman82@users.noreply.github.com>
Studio-Yandex-Practicum · Feb 6, 2024 · 45e11b4 · 45e11b4
1 parent 46e3dec
commit 45e11b4
Show file tree

Hide file tree

Showing 15 changed files with 385 additions and 11 deletions.
diff --git a/.env.example b/.env.example
@@ -22,3 +22,7 @@ ADMIN_MIDDLEWARE_SECRET=1234567890
 MANAGER_CHAT_ID=CHAT_ID
 URL=http://localhost:80800
 REDIS_HOST=localhost
+WEBHOOK_PATH=/path
+WEBHOOK_URI=https://url
+WEB_SERVER_HOST=0.0.0.0
+WEB_SERVER_PORT=6000
diff --git a/.github/workflows/check_pull_requests.yml b/.github/workflows/check_pull_requests.yml
@@ -0,0 +1,60 @@
+name: Check pull request
+
+on:
+  pull_request:
+    types:
+      - labeled
+      - unlabeled
+      - synchronize
+      - opened
+      - edited
+      - ready_for_review
+      - reopened
+      - unlocked
+    branches:
+      - dev
+      - developer
+      - main
+      - master
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: password
+          POSTGRES_DB: users
+        ports:
+          - 5432:5432
+        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.11
+      - name: Install Dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ./backend/requirements.txt
+      - name: Run tests
+        env:
+          DB_URL: postgresql+asyncpg://postgres:password@postgresql:5432/users
+        run: |
+          cd backend/
+          python -m pytest
+
+  send_message_deploy:
+    runs-on: ubuntu-latest
+    needs: tests
+    steps:
+      - name: Send message
+        uses: appleboy/telegram-action@master
+        with:
+          to: ${{ secrets.TELEGRAM_TO }}
+          token: ${{ secrets.TELEGRAM_TOKEN }}
+          message: PR прошёл тесты!
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -0,0 +1,147 @@
+name: Deploy
+
+on:
+  push:
+    branches:
+      - main
+      - master
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: password
+          POSTGRES_DB: users
+        ports:
+          - 5432:5432
+        options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: 3.11
+      - name: Install Dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r ./backend/requirements.txt
+      - name: Run tests
+        env:
+          DB_URL: postgresql+asyncpg://postgres:password@postgresql:5432/users
+        run: |
+          cd backend/
+          python -m pytest
+
+  build_and_push_backend:
+    name: Push Docker image to Dockerhub
+    runs-on: ubuntu-latest
+    needs: tests
+    if: ${{ github.event_name == 'push' }}
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+      - name: Login to Docker
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+      - name: Push to DockerHub
+        uses: docker/build-push-action@v4
+        with:
+          context: ./backend/
+          push: true
+          tags: wolfmtk/sozidateli_backend
+
+  build_and_push_bot:
+    name: Push Docker image to Dockerhub
+    runs-on: ubuntu-latest
+    needs: tests
+    if: ${{ github.event_name == 'push' }}
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+      - name: Login to Docker
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+      - name: Push to DockerHub
+        uses: docker/build-push-action@v4
+        with:
+          context: ./bot/
+          push: true
+          tags: wolfmtk/sozidateli_bot
+
+  build_and_push_gateway:
+    name: Push Docker image to Dockerhub
+    runs-on: ubuntu-latest
+    needs: tests
+    if: ${{ github.event_name == 'push' }}
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+      - name: Login to Docker
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+      - name: Push to DockerHub
+        uses: docker/build-push-action@v4
+        with:
+          context: ./nginx/
+          push: true
+          tags: wolfmtk/sozidateli_gateway
+
+  deploy:
+    runs-on: ubuntu-latest
+    needs:
+      - build_and_push_backend
+      - build_and_push_bot
+      - build_and_push_gateway
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v3
+      - name: Copy docker-compose.yml via ssh
+        uses: appleboy/scp-action@master
+        with:
+          host: ${{ secrets.HOST }}
+          username: ${{ secrets.USER }}
+          key: ${{ secrets.SSH_KEY }}
+          password: ${{ secrets.SSH_PASSWORD }}
+          source: "docker-compose.production.yml"
+          target: "sozidateli"
+      - name: Executing remote ssh commands to deploy
+        uses: appleboy/ssh-action@master
+        with:
+          host: ${{ secrets.HOST }}
+          username: ${{ secrets.USER }}
+          key: ${{ secrets.SSH_KEY }}
+          password: ${{ secrets.SSH_PASSWORD }}
+          script: |
+            cd sozidateli
+            sudo docker compose -f docker-compose.production.yml pull
+            sudo docker compose -f docker-compose.production.yml down
+            sudo docker compose -f docker-compose.production.yml up -d
+            sudo docker image prune -fa
+
+  send_message_deploy:
+    runs-on: ubuntu-latest
+    needs: deploy
+    steps:
+      - name: Send message
+        uses: appleboy/telegram-action@master
+        with:
+          to: ${{ secrets.TELEGRAM_TO }}
+          token: ${{ secrets.TELEGRAM_TOKEN }}
+          message: Деплой успешно выполнен!
diff --git a/backend/app/admin/meetings.py b/backend/app/admin/meetings.py
@@ -9,17 +9,21 @@
     HasMany,
     TextAreaField,
 )
+from starlette_admin import row_action
 from starlette_admin.contrib.sqla.ext.pydantic import ModelView
 from starlette_admin.exceptions import FormValidationError
 
+from app.application.repositories.meetings import MeetingRepository
 from app.core.constants import ZONEINFO
+from app.core.parser import parser_date
 
 
 class MeetingView(ModelView):
     """Модель для отображения собраний в админке."""
 
     identity = "meeting"
 
+    row_actions = ['view', 'make_published', 'edit', 'delete']
     fields = [
         DateTimeField("date", label="Дата и время"),
         BooleanField("is_open", label="Закрыто/Открыто"),
@@ -30,6 +34,33 @@ class MeetingView(ModelView):
     sortable_fields = ["date", "is_open"]
     fields_default_sort = ["date", ("is_open", True)]
 
+    @row_action(
+        name="make_published",
+        text="Получить дату с сайта",
+        icon_class="fas fa-check-circle",
+        submit_btn_text="Yes, proceed",
+        submit_btn_class="btn-success",
+        action_btn_class="btn-info",
+        form="""
+            <form>
+                <div class="mt-3">
+                    <input type="text" 
+                           class="form-control" 
+                           name="example-text-input" 
+                           placeholder="Enter value">
+                </div>
+            </form>
+            """,
+    )
+    async def make_published_row_action(self, request: Request, pk: Any) -> str:
+        date = await parser_date()
+        if date:
+            session = request.state.session
+            await MeetingRepository(session).add_one(date=date)
+            await session.commit()
+            return "Дата успешно получена с сайта!"
+        return "Дату не удалось получить с сайта, добавьте вручную!"
+
     async def validate(self, request: Request, data: dict[str, Any]) -> None:
         """Валидация полей."""
         errors: dict[str, str] = dict()
@@ -40,7 +71,7 @@ async def validate(self, request: Request, data: dict[str, Any]) -> None:
         if data["date"] and data["date"].replace(
                 tzinfo=ZoneInfo(ZONEINFO)
         ).timestamp() < dt.datetime.now(
-                tz=ZoneInfo(ZONEINFO)
+            tz=ZoneInfo(ZONEINFO)
         ).timestamp():
             errors["date"] = "Дата собрания не может быть меньше текущей."
 

diff --git a/backend/app/admin/users.py b/backend/app/admin/users.py
@@ -13,7 +13,6 @@
 from starlette_admin.exceptions import FormValidationError
 
 from app.application.repositories.users import UserRepository
-from app.domain.models import User
 from app.domain.models.enums import AssistanceSegment
 
 
@@ -53,7 +52,7 @@ class UserView(ModelView):
         ),
     ]
     label = "Участники"
-    sortable_fields = [User.meeting]
+    sortable_fields = ["name", "assistance_segment"]
 
     async def validate(self, request: Request, data: dict[str, Any]) -> None:
         """Валидация полей."""

diff --git a/backend/app/core/constants.py b/backend/app/core/constants.py
@@ -3,3 +3,19 @@
 ZONEINFO = "Europe/Moscow"
 USERNAME_LENGTH = 4
 PASSWORD_LENGTH = 6
+URL = "https://sozidateli.spb.ru/volonteru/"
+DATE_PATTERN = r"\d{2}\s\w+\s\w+\s\d{2}\.\d{2}"
+MONTH = {
+    'января': 1,
+    'февраля': 2,
+    'марта': 3,
+    'апреля': 4,
+    'мая': 5,
+    'июня': 6,
+    'июля': 7,
+    'августа': 8,
+    'сентября': 9,
+    'октября': 10,
+    'ноября': 11,
+    'декабря': 12
+}
diff --git a/backend/app/core/parser.py b/backend/app/core/parser.py
@@ -0,0 +1,30 @@
+import datetime as dt
+import re
+from http import HTTPStatus
+
+import httpx
+from lxml import html
+
+from app.core.constants import DATE_FORMAT, URL, MONTH, DATE_PATTERN
+
+
+async def parser_date() -> None | dt.datetime:
+    """Парсер даты с сайта."""
+    async with httpx.AsyncClient() as client:
+        response = await client.get(URL)
+        if response.status_code != HTTPStatus.OK:
+            return None
+        tree = html.fromstring(response.text)
+        text = tree.xpath(
+            "//div[@class='e-con-inner']"
+            "//p[@class='has-black-color has-white-background-color "
+            "has-text-color has-background']//strong",
+        )
+        try:
+            date_text = re.findall(DATE_PATTERN, text[0].text)[0].split()
+            date = (f'{date_text[0]} {MONTH[date_text[1]]} '
+                    f'{dt.datetime.now().year} '
+                    f'{date_text[-1].replace(".", ":")}:00')
+            return dt.datetime.strptime(date, DATE_FORMAT)
+        except (IndexError, KeyError):
+            return None
diff --git a/backend/prod.txt b/backend/prod.txt
@@ -26,3 +26,4 @@ starlette==0.35.1
 starlette-admin==0.13.1
 typing_extensions==4.9.0
 uvicorn==0.27.0.post1
+lxml==5.1.0
diff --git a/backend/requirements.txt b/backend/requirements.txt
@@ -39,3 +39,4 @@ starlette-admin==0.13.1
 tomli==2.0.1
 typing_extensions==4.9.0
 uvicorn==0.25.0
+lxml==5.1.0
diff --git a/bot/app/core/settings.py b/bot/app/core/settings.py
@@ -10,3 +10,7 @@ class Settings:
     manager_chat_id: int = os.getenv('MANAGER_CHAT_ID')
     url: str = os.getenv('URL', 'http://localhost:8000')
     redis_host: str = os.getenv("REDIS_HOST", "localhost")
+    WEBHOOK_PATH: str | None = os.getenv('WEBHOOK_PATH')
+    WEBHOOK_URI: str | None = os.getenv('WEBHOOK_URI')
+    WEB_SERVER_HOST: str = os.getenv('WEB_SERVER_HOST', 'localhost')
+    WEB_SERVER_PORT: int = os.getenv('WEB_SERVER_PORT', 8000)