Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Importer exits with error code 1 when no transactions are found #9357

Closed
4 tasks done
diogotcorreia opened this issue Oct 17, 2024 · 4 comments
Closed
4 tasks done
Labels
fixed Bugs that are fixed (in a coming release).

Comments

@diogotcorreia
Copy link

Support guidelines

I've found a bug and checked that ...

Description

When importing from GoCardless, if an account has transactions but they are all out of the filter range (or if they don't have transactions in the last 90 days at all), the data importer will exit with error code 1.
I've upgraded from 1.4.2 to 1.5.6 and this started happening (I vaguely remember this being a problem in the past that was fixed, so this might be a regression).

I'm aware that this might be intended behavior. If that's the case, feel free to dismiss this issue as I have a workaround for it (see additional information section).

Debug information

Debug information generated at 2024-10-17 10:01:58 Europe/Amsterdam for Firefly III Data Importer version 1.5.6.

System information
ItemValue
Version1.5.6
SystemPHP 8.3.12, Linux, fpm-fcgi
App information
ItemValue
TimezoneEurope/Amsterdam, Europe/Stockholm
Environmentproduction
Debug modefalse, cache 'file'
Log leveldebug, stack
Display errorsOff, ALL errors
BCscale12
Trusted proxies
User information
ItemValue
User agentMozilla/5.0 (X11; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0

Expected behaviour

If there are no transactions, I would still expect the exit code to be 0 if there was no actual error talking to GoCardless.

Steps to reproduce

Probably not very useful, but this is my JSON config for that specific account:

{
    "date": "",
    "default_account": 1,
    "delimiter": "comma",
    "headers": false,
    "rules": true,
    "skip_form": false,
    "add_import_tag": true,
    "roles": [],
    "do_mapping": [],
    "mapping": [],
    "duplicate_detection_method": "classic",
    "ignore_duplicate_lines": false,
    "ignore_duplicate_transactions": true,
    "unique_column_index": 0,
    "unique_column_type": "external-id",
    "version": 3,
    "flow": "nordigen",
    "identifier": "0",
    "connection": "0",
    "ignore_spectre_categories": false,
    "map_all_data": false,
    "accounts": {
        "REDACTED": 1
    },
    "date_range": "partial",
    "date_range_number": 5,
    "date_range_unit": "d",
    "date_not_before": "2022-02-23",
    "date_not_after": "2022-02-23",
    "nordigen_country": "REDACTED",
    "nordigen_bank": "REDACTED",
    "nordigen_requisitions": {
        "REDACTED": "REDACTED"
    },
    "conversion": false
}

If there are no transactions in the last 5 days, exit code of the importer is 1.

Additional info

Log (slightly redacted):

[2024-10-16 13:45:55] production.DEBUG: ImportFile: directory "/persist/firefly-auto-import-configs"
[2024-10-16 13:45:55] production.DEBUG: ImportFile: file      "REDACTED_BANK-nordigen.json"
[2024-10-16 13:45:55] production.DEBUG: ImportFile: importable "/persist/firefly-auto-import-configs/REDACTED_BANK-nordigen.json"
[2024-10-16 13:45:55] production.DEBUG: ImportFile: JSON       "/persist/firefly-auto-import-configs/REDACTED_BANK-nordigen.json"
[2024-10-16 13:45:55] production.DEBUG: Now in updateDateRange()
[2024-10-16 13:45:55] production.DEBUG: Range is partial, after is NULL, dateNotBefore will be calculated.
[2024-10-16 13:45:55] production.DEBUG: dateNotBefore is now "2024-10-11"
Going to convert from file /persist/firefly-auto-import-configs/REDACTED_BANK-nordigen.json using configuration /persist/firefly-auto-import-configs/REDACTED_BANK-nordigen.json and flow "nordigen".
[2024-10-16 13:45:55] production.DEBUG: Now in App\Console\AutoImports::startConversion
[2024-10-16 13:45:55] production.DEBUG: Going to generate conversion routine identifier.
[2024-10-16 13:45:55] production.DEBUG: Attempt #1 results in "conversion-NL8J7w38Xt2s"
[2024-10-16 13:45:55] production.INFO: Job identifier is "conversion-NL8J7w38Xt2s"
[2024-10-16 13:45:55] production.DEBUG: Now in startOrFindConversion(conversion-NL8J7w38Xt2s)
[2024-10-16 13:45:55] production.DEBUG: File does not exist or error, create a new one.
[2024-10-16 13:45:55] production.DEBUG: Return status. {"status":"waiting_to_start","errors":[],"warnings":[],"messages":[]}
[2024-10-16 13:45:55] production.DEBUG: Now in setConversionStatus(conv_running)
[2024-10-16 13:45:55] production.DEBUG: Found "conversion-NL8J7w38Xt2s" in the session
[2024-10-16 13:45:55] production.DEBUG: Now in startOrFindConversion(conversion-NL8J7w38Xt2s)
[2024-10-16 13:45:55] production.DEBUG: Conversion status is "waiting_to_start"
[2024-10-16 13:45:55] production.DEBUG: Now in storeConversionStatus(conversion-NL8J7w38Xt2s): conv_running
[2024-10-16 13:45:55] production.DEBUG: Messages: 0, warnings: 0, errors: 0
[2024-10-16 13:45:55] production.DEBUG: Set identifier to "conversion-NL8J7w38Xt2s"
[2024-10-16 13:45:55] production.DEBUG: Set identifier to "conversion-NL8J7w38Xt2s"
[2024-10-16 13:45:55] production.DEBUG: Now in App\Services\Nordigen\Conversion\RoutineManager::start
[2024-10-16 13:45:55] production.DEBUG: The GoCardless API URL is https://bankaccountdata.gocardless.com
[2024-10-16 13:45:55] production.DEBUG: Call transaction processor download.
[2024-10-16 13:45:55] production.DEBUG: Now in App\Services\Nordigen\Conversion\Routine\TransactionProcessor::download
[2024-10-16 13:45:55] production.DEBUG: Found 1 accounts to download from.
[2024-10-16 13:45:55] production.DEBUG: Going to download transactions for account #0 "REDACTED"
[2024-10-16 13:45:55] production.DEBUG: Now in App\Services\Nordigen\Services\AccountInformationCollector::collectInformation
[2024-10-16 13:45:55] production.DEBUG: Get account details is DISABLED.
[2024-10-16 13:45:55] production.DEBUG: Get account balance is DISABLED.
[2024-10-16 13:45:55] production.DEBUG: Now in App\Services\Nordigen\Services\AccountInformationCollector::getBasicDetails(REDACTED)
[2024-10-16 13:45:55] production.DEBUG: Request parameters will be set to:
[2024-10-16 13:45:55] production.DEBUG: authenticatedGet(https://bankaccountdata.gocardless.com/api/v2/accounts/REDACTED)
[2024-10-16 13:45:55] production.DEBUG: Rate limit: 100
[2024-10-16 13:45:55] production.DEBUG: Rate limit remaining: 82
[2024-10-16 13:45:55] production.DEBUG: Rate limit reset: 26
[2024-10-16 13:45:55] production.DEBUG: Now in pauseForRateLimit
[2024-10-16 13:45:55] production.DEBUG: Rate limit: 82 requests left, and 26s before the limit resets.
[2024-10-16 13:45:55] production.DEBUG: Save the account success limits? NO
[2024-10-16 13:45:55] production.DEBUG: Response for basic information request: {"id":"REDACTED","created":"REDACTED","last_accessed":"REDACTED","iban":"REDACTED","institution_id":"REDACTED","status":"READY","owner_name":""}
[2024-10-16 13:45:55] production.DEBUG: Set new IBAN from basic details.
[2024-10-16 13:45:55] production.DEBUG: Done downloading information for debug purposes.
[2024-10-16 13:45:55] production.DEBUG: Request parameters will be set to:
[2024-10-16 13:45:55] production.DEBUG: authenticatedGet(https://bankaccountdata.gocardless.com/api/v2/accounts/REDACTED/transactions/)
[2024-10-16 13:45:57] production.DEBUG: Rate limit: 100
[2024-10-16 13:45:57] production.DEBUG: Rate limit remaining: 81
[2024-10-16 13:45:57] production.DEBUG: Rate limit reset: 25
[2024-10-16 13:45:57] production.DEBUG: Account success rate limit: 10
[2024-10-16 13:45:57] production.DEBUG: Account success rate limit remaining: 8
[2024-10-16 13:45:57] production.DEBUG: Account success rate limit reset: 40392
[2024-10-16 13:45:57] production.DEBUG: Now in pauseForRateLimit
[2024-10-16 13:45:57] production.DEBUG: Rate limit: 81 requests left, and 25s before the limit resets.
[2024-10-16 13:45:57] production.DEBUG: Save the account success limits? YES
[2024-10-16 13:45:57] production.DEBUG: Account success limit: 8 requests left, and 11h13m12s before the limit resets.
[2024-10-16 13:45:57] production.DEBUG: Downloaded [12:12] transactions
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array {"bookingDate":"2024-10-04","valueDate":"2024-10-04","transactionAmount":{"amount":"REDACTED","currency":"EUR"},"remittanceInformationUnstructured":"REDACTED","internalTransactionId":"REDACTED","key":"booked"}
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array REDACTED
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Nordigen transaction from array {"bookingDate":"2024-08-05","valueDate":"2024-08-05","transactionAmount":{"amount":"REDACTED","currency":"EUR"},"remittanceInformationUnstructured":"REDACTED","internalTransactionId":"REDACTED","key":"booked"}
[2024-10-16 13:45:57] production.DEBUG: Downloaded transaction with ID "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: GetTransactionsResponse: count 12 transaction(s)
[2024-10-16 13:45:57] production.INFO: Going to filter downloaded transactions. Original set length is 12
[2024-10-16 13:45:57] production.INFO: Will not grab transactions before "2024-10-11 00:00:00"
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "2024-10-04 00:00:00" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "REDACTED" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.DEBUG: Returning book date
[2024-10-16 13:45:57] production.DEBUG: Skip transaction because "2024-08-05 00:00:00" is before "2024-10-11 00:00:00".
[2024-10-16 13:45:57] production.INFO: After filtering, set is 0 transaction(s)
[2024-10-16 13:45:57] production.DEBUG: Done downloading transactions for account 0 "REDACTED"
[2024-10-16 13:45:57] production.DEBUG: Done with download
[2024-10-16 13:45:57] production.DEBUG: Rate limit for account REDACTED: 8 request(s) left, 40392 second(s)
[2024-10-16 13:45:57] production.DEBUG: Generating Firefly III transactions.
[2024-10-16 13:45:57] production.DEBUG: Nordigen: Defer account search to trait.
[2024-10-16 13:45:57] production.DEBUG: Now in collectAllTargetAccounts()
[2024-10-16 13:45:57] production.DEBUG: Now in collectAccounts("asset")
[2024-10-16 13:45:57] production.DEBUG: No access token in hasAccessToken() session, will return config variable.
[2024-10-16 13:45:57] production.DEBUG: No base url in getBaseUrl() session, will return config variable.
[2024-10-16 13:45:58] production.DEBUG: Found 16 accounts of type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #7 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #11 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #9 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #4 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #13 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #15 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #5 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Collected account IBAN "REDACTED" (asset) under ID #5
[2024-10-16 13:45:58] production.DEBUG: Processing account #82 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #147 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #1 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #3 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #29 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Collected account IBAN "REDACTED" (asset) under ID #29
[2024-10-16 13:45:58] production.DEBUG: Processing account #33 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #55 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Processing account #128 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Collected account IBAN "REDACTED" (asset) under ID #128
[2024-10-16 13:45:58] production.DEBUG: Processing account #129 ("REDACTED") with type "asset"
[2024-10-16 13:45:58] production.DEBUG: Collected account IBAN "REDACTED" (asset) under ID #129
[2024-10-16 13:45:58] production.DEBUG: Collected 16 accounts of type "asset"
[2024-10-16 13:45:58] production.DEBUG: Now in collectAccounts("liabilities")
[2024-10-16 13:45:58] production.DEBUG: No access token in hasAccessToken() session, will return config variable.
[2024-10-16 13:45:58] production.DEBUG: No base url in getBaseUrl() session, will return config variable.
[2024-10-16 13:45:58] production.DEBUG: Found 0 accounts of type "liabilities"
[2024-10-16 13:45:58] production.DEBUG: Collected 0 accounts of type "liabilities"
[2024-10-16 13:45:58] production.DEBUG: Nordigen: Collected 4 accounts.
[2024-10-16 13:45:58] production.DEBUG: Add message about rate limits.
[2024-10-16 13:45:58] production.DEBUG: Add message about rate limits for account 1.
[2024-10-16 13:45:58] production.DEBUG: Found NO Firefly III account to report on, will not report rate limit.
[2024-10-16 13:45:58] production.WARNING: Downloaded nothing, will return nothing.
[2024-10-16 13:45:58] production.ERROR: [c] Add error to index #0: No transactions were downloaded from GoCardless. You may be rate limited or something else went wrong.
[2024-10-16 13:45:58] production.DEBUG: Add error on index #0 (line no. 1): No transactions were downloaded from GoCardless. You may be rate limited or something else went wrong.
[2024-10-16 13:45:58] production.DEBUG: Now in storeConversionStatus(conversion-NL8J7w38Xt2s): conv_running
[2024-10-16 13:45:58] production.DEBUG: Messages: 0, warnings: 0, errors: 1
[2024-10-16 13:45:58] production.ERROR: [a] Zero transactions!
[2024-10-16 13:45:58] production.DEBUG: Now in setConversionStatus(conv_done)
[2024-10-16 13:45:58] production.DEBUG: Found "conversion-NL8J7w38Xt2s" in the session
[2024-10-16 13:45:58] production.DEBUG: Now in startOrFindConversion(conversion-NL8J7w38Xt2s)
[2024-10-16 13:45:58] production.DEBUG: Conversion status is "conv_running"
[2024-10-16 13:45:58] production.DEBUG: Now in storeConversionStatus(conversion-NL8J7w38Xt2s): conv_done
[2024-10-16 13:45:58] production.DEBUG: Messages: 0, warnings: 0, errors: 1
Conversion index (error) 0: No transactions were downloaded from GoCardless. You may be rate limited or something else went wrong.
Too many errors in the data conversion (1), exit.
[2024-10-16 13:45:58] production.DEBUG: Created event ImportedTransactions with filtering (2)
[2024-10-16 13:45:58] production.DEBUG: Array contains 0 message(s)
[2024-10-16 13:45:58] production.DEBUG: Array contains 0 warning(s)
[2024-10-16 13:45:58] production.DEBUG: Array contains 1 error(s)
[2024-10-16 13:45:58] production.DEBUG: Now in sendReportOverMail
[2024-10-16 13:45:58] production.INFO: Will send report message.
[2024-10-16 13:45:58] production.DEBUG: Messages count: 0
[2024-10-16 13:45:58] production.DEBUG: Warnings count: 0
[2024-10-16 13:45:58] production.DEBUG: Errors count  : 1
[2024-10-16 13:45:58] production.DEBUG: If no error below this line, mail was sent!
[2024-10-16 13:45:58] production.DEBUG: If no error above this line, mail was sent!
[2024-10-16 13:45:58] production.ERROR: Could not complete import from file "REDACTED_BANK-nordigen.json".
[2024-10-16 13:45:58] production.ERROR: Too many errors in the data conversion.

I've fixed this on my side by making the no transactions error a warning instead: diogotcorreia/dotfiles@399f824#diff-19cd1f20eb3f5b530f5ad59217e4f13e68b6fced9575624b2787b4fcb725b795 (file 0001-no-transactions-warning-instead-of-error.diff).

@JC5
Copy link
Member

JC5 commented Oct 17, 2024

Hey, thanks for opening an issue!

I remember somebody wanted specifically to exit 1 when no no transactions were found or imported or whatever so that's what I did. It made sense at the time.

I could change to exit code 2 or something like that, I'll think about it!

@diogotcorreia
Copy link
Author

I can see this being useful in certain situations yeah.
For context, I'm running this every day across all my accounts and I have monitoring in place for when services exit with error (non-zero), so it was triggering an alert 😅
For checking if transactions were imported or not, I have the email reports.

Thanks for the quick response, let me know if you need anything else from my side

@JC5
Copy link
Member

JC5 commented Oct 18, 2024

The importer:import command and importer:auto-import will exit 73 when nothing was imported. The importer:auto-import command may also exit 1 in the same situation, since it can process multiple files (and another file may have had an import that was perfectly fine, or had another error).

Exit codes are documented here:

https://docs.firefly-iii.org/how-to/data-importer/advanced/cli/

@JC5 JC5 added this to the importer-v1.5.7 milestone Oct 18, 2024
@JC5 JC5 moved this to Done in Data Importer Oct 18, 2024
@JC5 JC5 added the fixed Bugs that are fixed (in a coming release). label Oct 18, 2024
@JC5 JC5 closed this as completed Nov 9, 2024
Copy link
Contributor

github-actions bot commented Nov 9, 2024

Hi there! This is an automatic reply. Share and enjoy

This issue is now 🔒 closed. Please be aware that closed issues are not monitored by the developer of Firefly III.

  • If the original bug is not actually fixed, please open a new issue. Refer to this issue for clarity.
  • Follow-up questions must be posted in a new discussion
  • Further replies to this issue may get no response.

If there is more to discuss, please open a new issue or discussion.

Thank you for your contributions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
fixed Bugs that are fixed (in a coming release).
Projects
Status: Done
Development

No branches or pull requests

2 participants