N26 importer creates duplicate entries

[peterjeschke]

I still couldn't pinpoint the exact source for this problem, but maybe someone more insight into this issue. Sometimes, the n26 importer creates duplicate transactions of already existing transactions. At first, only one transaction is shown (that is correct) but after running the importer a few times, a second transaction is created. This happens frequently with relatively recent transactions, but a few days ago, it happened with transactions that were three months old. It doesn't happen with every transaction either.

My hunch is that n26 changes the payee and memo fields, but I couldn't pinpoint it exactly. I have attached one instance of the duplicated transation:

[gitviola]

Hi Peter, I’m also facing this issue and I find it a bit frustrating. The reason for it is the YNAB api which changes the timestamp by a few milliseconds. That means that their data is inconsitend. The change happens when a transaction gets processed.

The only solution I can think of is to add an option to the N26 dumper to not import not-processed transactions. That would mean that if you pay something today it might be imported during the next 2-3 days. Would that work for you?

[peterjeschke]

I don't I understand what you mean by transation processing. How does YNAB process these transactions and why does it only affect N26? Or does the n26 api changes the timestamp?

[gitviola]

Yes correct the N26 api changed the timestamp. They have a timezone mismatch which I already handle but they also sometimes change the seconds/milliseconds which I can’t handle

[peterjeschke]

Is the difference between the old and the changed timestamps large? Maybe it is possible to match entries if they have the same price and very close timestamps? Am Sa., 3. Nov. 2018, 17:00 hat Martin Schurig <notifications@github.com> geschrieben:

…

Yes correct the N26 api changed the timestamp. They have a timezone mismatch which I already handle but they also sometimes change the seconds/milliseconds which I can’t handle — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#36 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAlhES4SHiW9TjeImzBnYNx9rbVGtXUFks5urb2agaJpZM4YLkJx> .

[gitviola]

The problem is that I don't know the timestamp that it was before if the transaction comes in. I calculate a transaction-hash based on the time for each transaction I get through their API. I don't use the combination of Payee + amount because they're too unreliable (payee changes very often from pending to processed transactions). So I do this for every transaction and if the hash differs then YNAB treats it as a new transaction. What I was thinking of was to always round the time but it's going to be the same problem because if we have a time like 15:46:29 (with milliseconds) and when it's processed it's becoming 15:46:59 or even 15:47:01 the hash will still be wrong 😞

[manuelgrabowski]

I had the same issue. Reading this discussion, my first thought was that it might be worth using only "HH:mm" from the timestamp for the hash… I guess it would be pretty rare to have two transactions in the same minute with the same amount (I'm only guessing the amount is part of the hash, though). But as you wrote, even "rounding" to "HH:mm" could still result in deviations, so it's not perfect either. Has N26 ever commented on their shitty API output? Don't they have a unique identifier for each transaction in their API output?

[manuelgrabowski]

I've thought about it some more and actually went a bit into the code and their API – sadly not coming up with anything you haven't already mentioned. Seeing how even the payee changes after processing of transactions, I'd say that only sending processed transactions to YNAB would be preferable. I know that many people like how much more "instant" N26 is especially when it comes to pending CC charges, but as you suggested it could be made as an option. Maybe I'll find some time over the holidays to try and add it.

(Possibly it could even be an option with multiple values, as opposed to a boolean – so instead of setting "only_processed_transactions" to true or false, you could have a "handle_timestamp_problems" with "not|only_processed|approximate_date" or something like that, with the third options being what I suggested above. While not as exact as only using processed transactions, I still think it would probably catch most cases without causing much trouble.)

[gitviola]

@manuelgrabowski thanks for adding your thoughts here!! I just created a PR #38 that adds a config flag to the N26 dumper.

Because by next year's autumn all banks should be required to have a normalized API (PSD2), I would suggest to only introduce a boolean as you suggest first. Also I would like to be on the safe side when it comes to transactions being imported (I rather want people to have duplicates than transactions that don't show up at all).

Let me know what you think about the PR. I wonder if the config flag should be true by default or if we want to keep it turned off by default. My idea behind false was to not change the current behavior of the script. What do you think?

[manuelgrabowski]

That PSD2 thing sounds interesting, neat! I agree with your reasoning about the default value for the new flag, and apart from the small typo I found the PR looks great to me! 🎉