[CVAT-M2] Oracles M2 API (#2094)

* Fix job event handling when no assignments exist * Implement boxes from points task creation * Refactor cloud storage api * Implement job downloading * Estimate max bbox * Little refactoring * Copy some shared code from excor to recor * Refactor task creation and export * Implement boxes from points validation * Integrate SDK updates * Fix escrow access * [Exchange/Recoring oracles] Add GCS support * Remove local dev mocks * Remove some extra changes * Update tests * Update some tests * Remove extra launcher change * Refactor code * Add google-cloud-storage dependency * Fix exception classes * Set env variables in docker-compose.test.yml * [CVAT] Points to boxes task (#1560) * Fix job event handling when no assignments exist * Implement boxes from points task creation * Refactor cloud storage api * Implement job downloading * Estimate max bbox * Little refactoring * Copy some shared code from excor to recor * Refactor task creation and export * Implement boxes from points validation * Integrate SDK updates * Fix escrow access * Remove local dev mocks * Remove some extra changes * Update tests * Update some tests * Remove extra launcher change * Use virtual hosted bucket style * Fix creating CS in CVAT * Fix tests * Add basic implementation for skeletons from boxes * Refactor and fix some errors * Implement job uploading * Implement downloading * Fix extra gt boxes in merged results from excor * Implement skeleton matching * Fix labels in merged annotations * Update project events handling in excor * Improve oks sigma comment * Remove local testing assets * Fix label mapping * Add extra skeleton validations in manifest * Update .env template * Update several comments * Fix failing tests * Update enum name * Add more annotation validations for skeletons * Extend input annotations validation * Fix quality computation, refactor * Unify assignment accuracy checks between different job types * Refactor GT downloading for validation * Add gcs support - fixes (#1) * Update code formatting * Update poetry lock * Refactor * Use dict for gcs file contents * Update class name * Add backward compatibility for data bucket env var * Remove extra changes * Remove extra import * Fix enum name * Update bucket access info parsing * Fix error type in cloud provider parsing * Update tests * Add job annotation mode param into assignment links * Add CD trigger for experimental cvat oracles * Update code formatting * Fix test * Format code * Fix bucket uses * Fix result sending * Remove extra changes * Fix extra code * Remove unused code * Align enum naming convention * Update oracle-webhook endpoints * Update webhooks for m2 * Update imports * Use relative paths in bucket in task creation * Improve error messages * Improve skeleton and bbox validation * Fix escrow manifest downloading * Update tests * Fix tests * Implement general image bans * Fix import * Make default gt ban threshold more strict * Fix comparison for absent points * Fix comparison for omitted points in jobs * Finish escrows with too many unverifiable assignments * Check if an increased healthcheck interval will fix unhealthy container (#1700) * Fix validations for boxes from points task creation * Clean code * disable roi estimation for unreliable cases * Fix manifest parsing * Remove m0 launcher stubs * Remove m0 rep or stubs * Make max assignment time optional in manifest * Make label type fully optional in manifest * Fix test * Enable and fix a disabled m1 task creation test * Fix RecOr tests * Update poetry lock * Rename rejected_jobs to assignments * Update imports * Fix dict traversal * [Dependent][CVAT-M2] Oracles M2 API - endpoints (#1649) * Upgrade deps, upgrade pydantic to v2 and fastapi to 0.109 * Migrate code to pydantic v2 * Add filtering and pagination utilities * Add draft implementation for updated excor endpoints, refactor enum names * Fixes and refactoring * Update endpoint response fields and filters * Implement response field selection * Add new endpoints * Support JWT auth * Update .env template * Move type definition * Refactor pagination * Add page size env var * Add jwt lib project dependency * Fix authorization header integration with swagger * Adopt pagination changes * Fix assignment status filter * Add job status field and filter * Add redis and rate limiting for stats * Update recording update deps, align versions * Remove job_title from /job * Update poetry lock * Refactor some code * Add resign endpoint * Take updated_at from created_at for active assignments * Update register endpoint api * Add qualifications stubs in API * Remove deprecated max_time from manifest * Add qualifications into job responses * Update code formatting * Update deprecated method .dict() uses * Update deprecated parse_obj calls on pydantic models * Update comment * Remove human app signature header from /register * Fix throttling implementation * Add new filters in API * Update fastapi to 0.111.1 * Update email and method name in /register * Add missing chain_id filter in list_assignments * Rename amount to total in stats * Don't allow updating user * [Exchange Oracle] Update lock file * [Exchange Oracle] Mark @field_validator as a classmethod decorator * [Exchange Oracle] Fix Ruff violations * [Exchange Oracle] Remove deprecated param * [Recording Oracle] Fix ruff violations * [Recording Oracle] Update poetry.lock * Add redis auth * Refactor some code, allow empty wallet_address in JWT if role is human app * Fix CI checks * Update dev env configs * [Exchange oracle] return 401 instead of 403 when there is no authorization header * [Exchange oracle] list assignments: make filter params optional && resign assignment: make assignment_id a request body param * [Exchange oracle] Update api tests * [Exchange oracle] Update integraton/services/test_exchange.py * Use renamed enum members * t * Update poetry.lock * t * Small fixes * Import StrEnum from strenum directly * [Exchange oracle] Fix pagination * [Exchange oracle] GET /job fields param: support json array * [Exchange oracle] more tests * [Exchange oracle] Fix sorting * [Exchange oracle] RecordingOracleEvent_SubmissionRejected::RejectedAssignmentInfo::task_id -> assignment_id * [Recording oracle] RecordingOracleEvent_SubmissionRejected::RejectedAssignmentInfo::task_id -> assignment_id * Remove todo * [Exchange oracle] Fix missed updated_at in GET /job response * [Exchange oracle] Add description and examples for JobsFilter::fields * Update packages/examples/cvat/exchange-oracle/src/endpoints/filtering.py * Remove extra file after merge * Improve comment * Refactor some code * Improve comment * Improve comment * Remove outdated comment * Fix formatting * Apply comments * Fix invalid updated_after filtering for assignments * Fix invalid assignment status filter and refactor job status filter * Make docker images lighter for oracles * Refactor some code * Extend list assignments test * Fix linter errors * Touch parents updated_at when updating a child && update tests * Update packages/examples/cvat/recording-oracle/src/handlers/process_intermediate_results.py * Update packages/examples/cvat/exchange-oracle/src/crons/cvat/state_trackers.py * Fix ordering value register in API * Allow ordering in any register * Unify parent update mechanism on ORM models * Unify touch in cvat service * Remove touch methods from models * Update tests * Add dummy joints * Remove .vscode dir * Update cryptography to 43.0.1 GHSA-h4gh-qq45-vh27 * Replace python-jose with pyjwt GHSA-cjwg-qfpm-7377 GHSA-6c5p-j8vq-pqhj --------- Co-authored-by: maya <maria@cvat.ai> Co-authored-by: Ivan <ivan@hmt.ai> Co-authored-by: Dzeranov <sergey.dzeranov@gmail.com> Co-authored-by: Arseny Boykov <36469655+Bobronium@users.noreply.github.com> Co-authored-by: Arseny Boykov <appkiller16@gmail.com>
humanprotocol · Oct 8, 2024 · 1eb2b81 · 1eb2b81
1 parent cb7153a
commit 1eb2b81
Show file tree

Hide file tree

Showing 78 changed files with 4,234 additions and 960 deletions.
diff --git a/packages/examples/cvat/exchange-oracle/Dockerfile b/packages/examples/cvat/exchange-oracle/Dockerfile
@@ -4,12 +4,13 @@ WORKDIR /app
 
 RUN apt-get update -y && \
     apt-get install -y jq ffmpeg libsm6 libxext6 && \
-    pip install poetry
+    pip install --no-cache poetry
 
 COPY pyproject.toml poetry.lock ./
 
 RUN poetry config virtualenvs.create false \
-    && poetry install --no-interaction --no-ansi --no-root
+    && poetry install --no-interaction --no-ansi --no-root \
+    && poetry cache clear pypi --all
 
 COPY . .
 

diff --git a/.../examples/cvat/exchange-oracle/alembic/versions/2cbc85686054_add_assignment_updated_at.py b/.../examples/cvat/exchange-oracle/alembic/versions/2cbc85686054_add_assignment_updated_at.py
@@ -0,0 +1,88 @@
+"""add-assignment-updated-at
+
+Revision ID: 2cbc85686054
+Revises: 0f3fb7bfcbcf
+Create Date: 2024-09-24 19:18:23.248579
+
+"""
+
+import enum
+
+import sqlalchemy as sa
+from sqlalchemy import Column, DateTime, Enum, String
+from sqlalchemy.orm import Session, declarative_base
+from sqlalchemy.sql import func
+
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "2cbc85686054"
+down_revision = "0f3fb7bfcbcf"
+branch_labels = None
+depends_on = None
+
+
+Base = declarative_base()
+
+
+class AssignmentStatuses(str, enum.Enum):
+    created = "created"
+    completed = "completed"
+    expired = "expired"
+    rejected = "rejected"
+    canceled = "canceled"
+
+
+class Assignment(Base):
+    # Represents the model before the transaction is applied
+
+    __tablename__ = "assignments"
+    id = Column(String, primary_key=True, index=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), onupdate=func.now(), server_default=func.now())
+    expires_at = Column(DateTime(timezone=True), nullable=False)
+    completed_at = Column(DateTime(timezone=True), nullable=True, server_default=None)
+    status = Column(
+        String,
+        Enum(AssignmentStatuses),
+        server_default=AssignmentStatuses.created.value,
+        nullable=False,
+    )
+
+
+def define_initial_updated_at():
+    bind = op.get_bind()
+    session = Session(bind=bind)
+
+    session.query(Assignment).filter(
+        Assignment.updated_at == None,
+        Assignment.status.in_(
+            [AssignmentStatuses.expired, AssignmentStatuses.rejected, AssignmentStatuses.canceled]
+        ),
+    ).update({Assignment.updated_at: Assignment.expires_at})
+
+    session.query(Assignment).filter(
+        Assignment.updated_at == None,
+        Assignment.status == AssignmentStatuses.completed,
+    ).update({Assignment.updated_at: Assignment.completed_at})
+
+    session.query(Assignment).filter(
+        Assignment.updated_at == None,
+        # fallback for invalid entries above + handling of status == "created"
+    ).update({Assignment.updated_at: Assignment.created_at})
+
+
+def upgrade() -> None:
+    # This operation can fail because of null values, it's intentional.
+    # Invalid entries in the DB must be fixed in order to proceed
+    op.alter_column("assignments", "created_at", nullable=False)
+
+    op.add_column("assignments", sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True))
+    define_initial_updated_at()
+    op.alter_column("assignments", "updated_at", server_default=sa.text("now()"), nullable=False)
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column("assignments", "updated_at")
+    # ### end Alembic commands ###
diff --git a/packages/examples/cvat/exchange-oracle/docker-compose.dev.yml b/packages/examples/cvat/exchange-oracle/docker-compose.dev.yml
@@ -1,7 +1,6 @@
-version: '3.8'
-
 services:
   postgres:
+    container_name: oracle-postgres
     image: postgres:14.4
     restart: on-failure
     environment:
@@ -15,5 +14,81 @@ services:
       - 5431:5432
     command: ["postgres", "-c", "log_statement=all"]
 
+  redis:
+    container_name: oracle-redis
+    image: redis:7.2.3-alpine
+    restart: always
+    command: [
+      "redis-server",
+      "--save", "60", "100",
+      "--appendonly", "yes",
+      # Uncomment if you want to configure a password
+      # "--requirepass", "password",
+      # "--user", "username", "on", ">password", "~*", "allcommands",
+      # "--user", "default", "off", "nopass", "nocommands",
+    ]
+    volumes:
+      - redis:/data
+    ports:
+      - 6380:6379
+    networks:
+      - test-network
+
+  minio:
+    container_name: minio
+    image: minio/minio:RELEASE.2022-05-26T05-48-41Z
+    environment:
+      MINIO_ROOT_USER: dev
+      MINIO_ROOT_PASSWORD: devdevdev
+    volumes:
+      - ./tests/utils/datasets:/tmp/datasets
+    entrypoint: 'sh'
+    ports:
+      - 9010:9010
+      - 9011:9011
+    command:
+      -c "
+      cp -r /tmp/datasets /data &&
+      minio server /data --console-address ':9011' --address ':9010'"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:9010/minio/health/live"]
+      interval: 5s
+      timeout: 5s
+      retries: 3
+    networks:
+      - test-network
+      - cvat-human-bridge
+
+  minio-mc:
+    container_name: minio-mc
+    image: minio/mc
+    depends_on:
+      minio:
+        condition: service_healthy
+    entrypoint: >
+      /bin/sh -c "
+      /usr/bin/mc config host add myminio http://minio:9010 dev devdevdev;
+      /usr/bin/mc anonymous set public myminio/manifests;
+      /usr/bin/mc mb myminio/excor-data;
+      /usr/bin/mc anonymous set public myminio/excor-data;
+      /usr/bin/mc mb myminio/recor-data;
+      /usr/bin/mc anonymous set public myminio/recor-data;
+      /usr/bin/mc mb myminio/datasets;
+      /usr/bin/mc anonymous set public myminio/datasets;
+      /usr/bin/mc mb myminio/launcher;
+      /usr/bin/mc anonymous set public myminio/launcher;
+      "
+    networks:
+      - test-network
+
 volumes:
-  postgres:
+  postgres:
+  redis:
+
+networks:
+  test-network:
+  cvat-human-bridge:
+    name: cvat-human-bridge
+    external: true
+    # Create with:
+    # docker network create -o "com.docker.network.bridge.enable_icc=true"  -o "com.docker.network.driver.mtu=1500" -o "com.docker.network.bridge.enable_ip_masquerade=true" cvat-human-bridge
diff --git a/packages/examples/cvat/exchange-oracle/docker-compose.test.yml b/packages/examples/cvat/exchange-oracle/docker-compose.test.yml
@@ -1,5 +1,3 @@
-version: '3.8'
-
 services:
   postgres:
     image: postgres:14.4
@@ -14,6 +12,20 @@ services:
     ports:
       - 5432:5432
 
+  redis:
+    container_name: oracle-redis
+    image: redis:7.2.3-alpine
+    restart: always
+    command: [
+      "redis-server",
+      "--save", "60", "100",
+      "--appendonly", "yes",
+    ]
+    ports:
+      - 6379:6379
+    networks:
+      - test-network
+
   minio:
     container_name: minio
     image: minio/minio:RELEASE.2022-05-26T05-48-41Z
@@ -72,9 +84,12 @@ services:
       STORAGE_USE_SSL: False
       STORAGE_PROVIDER: 'aws'
       ENABLE_CUSTOM_CLOUD_HOST: Yes
+      REDIS_HOST: 'redis'
     depends_on:
       postgres:
         condition: service_started
+      redis:
+        condition: service_started
       minio-mc:
         condition: service_completed_successfully
     # Used to reference localhost since minio:9000 is rejected by the SDK URL validator

diff --git a/packages/examples/cvat/exchange-oracle/dockerfiles/test.Dockerfile b/packages/examples/cvat/exchange-oracle/dockerfiles/test.Dockerfile
@@ -4,13 +4,14 @@ WORKDIR /app
 
 RUN apt-get update -y && \
     apt-get install -y jq ffmpeg libsm6 libxext6 && \
-    pip install poetry
+    pip install --no-cache poetry
 
 COPY pyproject.toml poetry.lock ./
 
 RUN poetry config virtualenvs.create false \
-    && poetry install --no-interaction --no-ansi --no-root
+    && poetry install --no-interaction --no-ansi --no-root \
+    && poetry cache clear pypi --all
 
 COPY . .
 
-CMD ["pytest", "-W", "ignore::DeprecationWarning"]
+CMD ["pytest", "-W", "ignore::DeprecationWarning", "-v"]