Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for not retrieving all items when response has multiple pages of items #318

Merged
merged 6 commits into from
Dec 13, 2024

Conversation

renoyjohnm
Copy link
Contributor

This pull request includes significant changes to the get_items_by_name function in tabcmd/commands/server.py to support pagination and improve item retrieval. Additionally, comprehensive unit tests have been added to ensure the function's correctness under various scenarios.

@renoyjohnm renoyjohnm requested a review from jacalata November 25, 2024 22:27
@renoyjohnm renoyjohnm self-assigned this Nov 25, 2024
Copy link

github-actions bot commented Nov 25, 2024

Coverage

Coverage Report
FileStmtsMissCoverMissing
tabcmd
   __main__.py101010 0%
   tabcmd.py141414 0%
   version.py633 50%
tabcmd/commands
   commands.py101010 0%
   constants.py771515 81%
   server.py1331919 86%
tabcmd/commands/auth
   session.py3604242 88%
tabcmd/commands/datasources_and_workbooks
   datasources_and_workbooks_command.py1271616 87%
   datasources_workbooks_views_url_parser.py1401010 93%
   delete_command.py601616 73%
   export_command.py1142525 78%
   get_url_command.py1274747 63%
   publish_command.py822424 71%
   runschedule_command.py2177 67%
tabcmd/commands/extracts
   create_extracts_command.py4288 81%
   decrypt_extracts_command.py2722 93%
   delete_extracts_command.py3766 84%
   encrypt_extracts_command.py2722 93%
   extracts.py2022 90%
   reencrypt_extracts_command.py2722 93%
   refresh_extracts_command.py481313 73%
tabcmd/commands/group
   create_group_command.py2955 83%
   delete_group_command.py2722 93%
tabcmd/commands/project
   create_project_command.py4688 83%
   delete_project_command.py3544 89%
   publish_samples_command.py3044 87%
tabcmd/commands/site
   create_site_command.py3455 85%
   delete_site_command.py2822 93%
   edit_site_command.py3822 95%
   list_command.py441010 77%
   list_sites_command.py2922 93%
tabcmd/commands/user
   create_site_users.py571010 82%
   create_users_command.py601010 83%
   delete_site_users_command.py4355 88%
   user_data.py2213131 86%
tabcmd/execution
   _version.py222 0%
   global_options.py1452323 84%
   localize.py691111 84%
   logger_config.py4266 86%
   tabcmd_controller.py4166 85%
TOTAL270744184% 

@jacalata
Copy link
Contributor

Some sites have thousands of items. What's the performance going to be like in that case?

@renoyjohnm
Copy link
Contributor Author

Some sites have thousands of items. What's the performance going to be like in that case?

This issue is specifically is about fetching all items filtered to the provided item name & doesn't fetch all items of a content type from the site. Yes, there could be multiple pages of items with the same name (like in the case of customer issue) & could be low of occurrence given the filter query but I believe we should be fetching all the results from the multiple pages of items despite the performance hit instead of just relying on the first page of items which could lead to unintended consequences. Please let me know of your thoughts on it, Thanks

@jacalata
Copy link
Contributor

The point of the bug is that we want to return more than the first page, yes. But what is the performance like? Does it appear to hang?

@renoyjohnm
Copy link
Contributor Author

The point of the bug is that we want to return more than the first page, yes. But what is the performance like? Does it appear to hang?

Not much in the limited testing I performed but it could, depending on the network latency for the requests, another quick option be to bump the default value of TSC_PAGE_SIZE to 500 to reduce the number of API calls for fetching all the pages of items, thoughts?

@jacalata
Copy link
Contributor

that does sound like a great quick fix. We could do that and then take a bit more time on this one.

@renoyjohnm renoyjohnm merged commit ec730ae into development Dec 13, 2024
18 checks passed
@jacalata jacalata deleted the rjohn/fix_not_retrieving_all_pages_pagination branch January 3, 2025 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants