[FEATURE] Automatic task distribution #5000
Labels
type: enhancement
Indicates new feature requests
Milestone
Comments
This was referenced Jul 1, 2024
jfcalvo
added a commit
that referenced
this issue
Jul 4, 2024
…nly (#5148) # Description Add changes to `responses_submitted` relationship to avoid problems with existent `responses` relationship and avoid a warning message that SQLAlchemy was reporting. Refs #5000 **Type of change** - Improvement (change adding some improvement to an existing functionality) **How Has This Been Tested** - [x] Warning is not showing anymore. - [x] Test are passing. **Checklist** - I added relevant documentation - follows the style guidelines of this project - I did a self-review of my code - I made corresponding changes to the documentation - I confirm My changes generate no new warnings - I have added tests that prove my fix is effective or that my feature works - I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)
jfcalvo
added a commit
that referenced
this issue
Jul 12, 2024
…riable setting (#5213) # Description On the distribution task testing effort meeting we found some errors raised by SQLite when the database was locked because too many writes were executed concurrently. Increasing the timeout for SQLite reduces this problem, specifically changing the connection `timeout` parameter from the default value of `5` seconds to `30` seconds make the problem totally disappear locally. In this PR I have added a new `ARGILLA_DATABASE_SQLITE_TIMEOUT` environment variable that allow us to set a value for this. With a default value of `15` seconds. The idea is to test this change on Spaces using a value of `30` seconds. Refs #5000 **Type of change** - Improvement (change adding some improvement to an existing functionality) **How Has This Been Tested** - Tested locally using SQLite and running 20 parallel response creations in bulk. **Checklist** - I added relevant documentation - I followed the style guidelines of this project - I did a self-review of my code - I made corresponding changes to the documentation - I confirm My changes generate no new warnings - I have added tests that prove my fix is effective or that my feature works - I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)
1 task
jfcalvo
added a commit
that referenced
this issue
Jul 12, 2024
…_DATABASE_POSTGRESQL_MAX_OVERFLOW` (#5220) # Description After testing a high number of concurrent requests using PostgreSQL I received the following error: ``` QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 ``` This PR add the following two environment variables so we can configure the pool size and max overflow. Refs #5000 **Type of change** - Improvement (change adding some improvement to an existing functionality) **How Has This Been Tested** - [x] Manually testing with PostgreSQL. **Checklist** - I added relevant documentation - I followed the style guidelines of this project - I did a self-review of my code - I made corresponding changes to the documentation - I confirm My changes generate no new warnings - I have added tests that prove my fix is effective or that my feature works - I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)
2 tasks
jfcalvo
added a commit
that referenced
this issue
Jul 16, 2024
# Description After investigate timeouts for PostgreSQL I have found that timeouts should not affect errors when a SERIALIZABLE transactions is rollbacked because another concurrent update error is raised. So the only way to support concurrent updates with PostgreSQL and SERIALIZABLE transactions is to capture errors and retry the transaction. This code has the following changes: * Start using `backoff` library to retry any of the possible CRUD context functions updating responses and record statuses, using SERIALIZABLE database sessions. * This change has the side effect of working with PostgreSQL and SQLite at the same time. * I have set a fixed time of 15 seconds as maximum time for retrying with exponential backoff. * I have moved search engine updates outside of the transaction block. * This should mitigate errors on high concurrency scenarios for PostgreSQL and SQLite: * For SQLite we have the additional setting to set a timeout if necessary. * I have changed `DEFAULT_DATABASE_SQLITE_TIMEOUT` value to `5` seconds so the backoff logic will handle possible problems with locked database errors and SQLite. Refs #5000 **Type of change** - Improvement (change adding some improvement to an existing functionality) **How Has This Been Tested** - [x] Manually testing with PostgreSQL and SQLite, running benchmarks using 20 concurrent requests. - [x] Running test suite for PostgreSQL and SQLite. **Checklist** - I added relevant documentation - I followed the style guidelines of this project - I did a self-review of my code - I made corresponding changes to the documentation - I confirm My changes generate no new warnings - I have added tests that prove my fix is effective or that my feature works - I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Backend
statuscolumn torecordstable #5058statuscolumn when a response is created/updated/upserted/deleted #5069statuscolumn forrecordstable #5155SDK
distributionattribute to be assigned when creating datasets #5033distributionsettings when updating datasets #5034statusattribute to records #5141UI
The text was updated successfully, but these errors were encountered: