Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(prepro): correctly handle empty ndjson response #3072

Closed
wants to merge 2 commits into from

Conversation

corneliusroemer
Copy link
Contributor

@corneliusroemer corneliusroemer commented Oct 24, 2024

Avoids this error when no data available:

DEBUG:urllib3.connectionpool:http://loculus-backend-service:8079 "POST /cchf/extract-unprocessed-data HTTP/11" 200 610
DEBUG:charset_normalizer:Encoding detection: ascii is most likely the one.
Traceback (most recent call last):
  File "/opt/conda/bin/prepro", line 8, in <module>
    sys.exit(cli_entry())
             ^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/loculus_preprocessing/__main__.py", line 16, in cli_entry
    run(config)
  File "/opt/conda/lib/python3.12/site-packages/loculus_preprocessing/prepro.py", line 748, in run
    etag, unprocessed = fetch_unprocessed_sequences(etag, config)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/loculus_preprocessing/backend.py", line 110, in fetch_unprocessed_sequences
    return response.headers["ETag"], parse_ndjson(response.text)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/loculus_preprocessing/backend.py", line 78, in parse_ndjson
    json_object = json.loads(json_str_processed)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

auto-merge was automatically disabled October 24, 2024 23:07

Pull request was closed

@corneliusroemer
Copy link
Contributor Author

corneliusroemer commented Oct 24, 2024

Actually the error was because the backend errored:

Exception: Failed to parse JSON: An unexpected error occurred while streaming, aborting the stream: java.sql.BatchUpdateException: Batch entry 0 INSERT INTO sequence_entries_preprocessed_data (accession, pipeline_version, processing_status, started_processing_at, “version”) VALUES ((‘PP_000KBM9’), (‘2’::int8), (‘IN_PROCESSING’), (‘2024-10-24 21:49:16.448672+00’), (‘2’::int8)) was aborted: ERROR: duplicate key value violates unique constraint “sequence_entries_preprocessed_data_pkey”

So we should catch and reraise with the full error message instead which I've added to:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant