Feat: add a new info_hash URL query array to the torrent list endpoint #728
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We need to parse URL query parameter arrays. For example: http://127.0.0.1:1212/api/v1/torrents?token=MyAccessToken&info_hash=9c38422213e30bff212b30c360d26f9a02136422&info_hash=2b66980093bc11806fab50cb3cb41835b95a0362 ```rust pub struct QueryParams { /// The offset of the first page to return. Starts at 0. #[serde(default, deserialize_with = "empty_string_as_none")] pub offset: Option<u32>, /// The maximum number of items to return per page. #[serde(default, deserialize_with = "empty_string_as_none")] pub limit: Option<u32>, /// A list of infohashes to retrieve. #[serde(default, rename = "info_hash")] pub info_hashes: Vec<String>, } ```
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #728 +/- ##
===========================================
+ Coverage 73.36% 73.41% +0.05%
===========================================
Files 144 144
Lines 9277 9315 +38
===========================================
+ Hits 6806 6839 +33
- Misses 2471 2476 +5 ☔ View full report in Codecov by Sentry. |
josecelano
force-pushed
the
725-api-add-scrape-endpoint
branch
from
838129a to
8e53525
Compare
josecelano
force-pushed
the
725-api-add-scrape-endpoint
branch
from
8e53525 to
2729ea2
Compare
The torrents endppint allow getting a list of torrents provifing the infohashes: http://127.0.0.1:1212/api/v1/torrents?token=MyAccessToken&info_hash=9c38422213e30bff212b30c360d26f9a02136422&info_hash=2b66980093bc11806fab50cb3cb41835b95a0362 It's like the tracker "scrape" request. The response JSON is the same as the normal torrent list: ```json [ { "info_hash": "9c38422213e30bff212b30c360d26f9a02136422", "seeders": 1, "completed": 0, "leechers": 0 }, { "info_hash": "2b66980093bc11806fab50cb3cb41835b95a0362", "seeders": 1, "completed": 0, "leechers": 0 } ] ```
josecelano
force-pushed
the
725-api-add-scrape-endpoint
branch
from
2729ea2 to
d39bfc2
Compare
|
ACK d39bfc2 |
3 tasks
josecelano
added a commit
to torrust/torrust-index
that referenced
this pull request
Mar 13, 2024
af7150b feat: [#469] import torrent stats using multiple torrents tracker API endpoint (Jose Celano) 16cbea8 feat: [#469] import torrent statistics in batches (Jose Celano) feffd09 feat: [#469] add update datetime for tracker stasts importation (Jose Celano) Pull request description: Currently, the Index imports statistics for all torrents every hour (1 hour is the default value in the configuration). We need to import stats for all torrents because we allow users to sort torrents by torrent stats (number of seeders and leechers). This PR improves a little bit the process. - [x] Add a new field (`updated_at`) to the table `torrust_torrent_tracker_stats` with the datetime when the stats were imported from the tracker. This is for logging purposes but it also helps to import torrents in batches. Regarding logging, it could help to check that the cronjob is running correctly. - [x] We get all torrents (`get_all_torrents_compact`) from the database. That could be big array of infohashes. We could obtain the 50 records that have not been updated for the longest time and run the importation every 100 milliseconds. We request the tracker API every 100 milliseconds getting 50 torrents. Those values can be adjusted in the future. - [x] A [new filter was added to the tracker API to get statistics for a list of torrents with one request](torrust/torrust-tracker#728). We can use it instead of getting one torrent at a time. **Pros:** - With millions of torrents we don't need to load all of them into memory. - The new field `updated_at` helps to monitor the importation process. - We get torrent stats for 50 torrents in one request instead of one request per torrent. **Cons:** - Every 100 milliseconds we run a query to check which torrent stats are pending to update. ACKs for top commit: josecelano: ACK af7150b Tree-SHA512: af1632282419457e20cc86e447b65d36c8e52dbff47e5c79cc1802fc6f67c759d572568f2846f65d4d5540049240ea82246df21d773ed1e6a285bde681fb423b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
API endpoint: http://127.0.0.1:1212/api/v1/torrents?token=MyAccessToken&info_hash=9c38422213e30bff212b30c360d26f9a02136422&info_hash=2b66980093bc11806fab50cb3cb41835b95a0362
Added a new query parameter
info_hashwhich is an array. You can specify directly the list of torrents you want to get.The JSON result is the same:
[ { "info_hash": "9c38422213e30bff212b30c360d26f9a02136422", "seeders": 1, "completed": 0, "leechers": 0 }, { "info_hash": "2b66980093bc11806fab50cb3cb41835b95a0362", "seeders": 1, "completed": 0, "leechers": 0 } ]It contains torrent's stats.