Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job should fail if Collect yields no files/file items #190

Closed
jpmckinney opened this issue Dec 1, 2021 · 3 comments
Closed

Job should fail if Collect yields no files/file items #190

jpmckinney opened this issue Dec 1, 2021 · 3 comments
Labels
bug Something isn't working component: orchestration

Comments

@jpmckinney
Copy link
Member

I'll leave #29 open for longer term quality assurance.

Presently, if Collect yields only errors, then Process never finishes.

I'll have a look at Process to see if it can finish despite receiving no workable data (might be in "completable" function).

That said, once all tasks are complete, the job status should be failed.

@jpmckinney
Copy link
Member Author

jpmckinney commented Apr 27, 2024

Can test with Buenos Aires, currently. Edit: And now with nigeria_portal (all requests were 503).

@jpmckinney
Copy link
Member Author

  • Kingfisher Collect increments kingfisher_process_expected_files_count for both File and FileError. Let's say only FileError's are sent.
  • Kingfisher Process creates a CollectionFile for FileError's. It creates a note, but otherwise it doesn't create any further rows.
  • Kingfisher Process's compilable function would return True if expected_files_count were 0. As it's not, it then checks the collection's data_type. Since only FileError's were received, the data_type isn't set, so it returns False.

@jpmckinney
Copy link
Member Author

Okay, I've made changes to Collect and Process such that errors aren't counted.

We can now see our first failed empty collection: https://data.open-contracting.org/admin/data_registry/job/2190/change/?_changelist_filters=status__exact%3DRUNNING

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working component: orchestration
Projects
None yet
Development

No branches or pull requests

1 participant