Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Timeout Configuration Option for Solr Harvests + multiple PIDs and Deep harvest #107

Closed
alliomeria opened this issue Jul 6, 2022 · 4 comments
Assignees
Labels
AMI Import Plugins Gets data from somewhere, puts data into a strawberry basked enhancement New feature or request

Comments

@alliomeria
Copy link
Contributor

What?

Sometimes you need a little more time for harvesting data-heavy solr documents (👀❤️ Mega Metadata Fans). Current AMI default is 5 seconds.

Per @DiegoPino, an option to adjust to a greater amount of time, such as 10 seconds (or even more+), will be helpful to have available via the administrative interface.

In the meantime:

You can manually update your local copy of SolrImporter.php.

  • Found at /web/modules/contrib/ami/src/Plugin/ImporterAdapter/SolrImporter.php in your Archipelago Deployment.
  • Find all (3) instances of:
    • $adapter = new SolariumCurl(); // or any other adapter implementing AdapterInterface
  • Replace all (3) instances with:
    • $adapter = new SolariumCurl(); // or any other adapter implementing AdapterInterface $adapter->setTimeout(10);
  • Save and test the updated 10 second setTimeout option; Adjust higher if needed.
@alliomeria alliomeria added enhancement New feature or request AMI Import Plugins Gets data from somewhere, puts data into a strawberry basked labels Jul 6, 2022
@DiegoPino
Copy link
Member

So adding this here as per @alliomeria debugging

This is what happens when we hit a timeout on a very slow server with infinite number of fields

Error message
		Solr Error: Solr HTTP error: HTTP request failed, Operation timed out after 10005 milliseconds with 1794131 bytes received
		Warning: Undefined array key "processed" in Drupal\ami\Plugin\ImporterAdapter\SolrImporter::finishfetchFromSolr() (line 1145 of modules/contrib/ami/src/Plugin/ImporterAdapter/SolrImporter.php).
		Warning: Trying to access array offset on value of type null in Drupal\ami\Plugin\ImporterAdapter\SolrImporter::finishfetchFromSolr() (line 1145 of modules/contrib/ami/src/Plugin/ImporterAdapter/SolrImporter.php).
		Warning: Undefined variable $finished in _batch_process() (line 355 of core/includes/batch.inc).
		Warning: Undefined array key "processed" in Drupal\ami\Plugin\ImporterAdapter\SolrImporter::finishfetchFromSolr() (line 1145 of modules/contrib/ami/src/Plugin/ImporterAdapter/SolrImporter.php).
		Warning: Trying to access array offset on value of type null in Drupal\ami\Plugin\ImporterAdapter\SolrImporter::finishfetchFromSolr() (line 1145 of modules/contrib/ami/src/Plugin/ImporterAdapter/SolrImporter.php).

Means we can also, in case of this react better, still dump what we managed to get into CSV and avoid total warning but still let the user know at which ROW number this failed and give an error that allow the harvest to be tried again

@DiegoPino DiegoPino changed the title Add Timeout Configuration Option for Solr Harvests Add Timeout Configuration Option for Solr Harvests + multiple PIDs and Deep harvest Jul 26, 2022
@DiegoPino
Copy link
Member

This also touches #47

@alliomeria
Copy link
Contributor Author

@DiegoPino Noting per our discussion--consider also removing the option to uncheck / hide Auto UUID generation option for the Solr Importer set options.

@DiegoPino
Copy link
Member

Implemented and also improved on

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AMI Import Plugins Gets data from somewhere, puts data into a strawberry basked enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants