Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: the @bentoml.task decorator's cancel functionality does not respond properly. #5141

Open
oogou11 opened this issue Dec 20, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@oogou11
Copy link

oogou11 commented Dec 20, 2024

Reproduction Code

import bentoml
import typing as t
import time

@bentoml.service(workers=1)
class ChunkTask:

    @bentoml.task
    def chunk_task(self) -> t.Dict:
        time.sleep(10)
        return {"code": "Ok"}

Observed Symptoms

When attempting to cancel a task, regardless of the task's current state:

  • The application hangs for an extended period.
  • Eventually, it returns a 500 Internal Server Error.

Initial Diagnosis

The issue seems to be related to SQLite database operations in the task status management functionality. Specifically, the following method appears to hang when executing self._conn.execute:

async def set_status(self, task_id: str, status: ResultStatus) -> None:
    await self._conn.execute(
        "UPDATE result SET status = ?, updated_at = ? WHERE task_id = ? AND status = ?",
        (
            status.value,
            datetime.datetime.now(tz=datetime.timezone.utc),
            task_id,
            ResultStatus.IN_PROGRESS.value,
        ),
    )
    await self._conn.commit()

Additional Notes

The issue might stem from SQLite's locking behavior, particularly during concurrent updates or uncommitted transactions.

Environment

  • Python Version: 3.11.x
  • BentoML Version: >=1.3.15
  • Operating System:
    • Windows
    • Ubuntu 22.04.5 LTS
@oogou11 oogou11 added the bug Something isn't working label Dec 20, 2024
@oogou11 oogou11 closed this as completed Dec 20, 2024
@oogou11 oogou11 reopened this Dec 20, 2024
@frostming
Copy link
Contributor

We do not support interrupting a running task. The cancel endpoint only has effect on pending tasks(scheduled but not started yet).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants