File Verification

[c0fe]

Is there an option to verify the files uploaded compared to the source, such as a file hash?

[AndyScherzinger]

Hi @c0fe

I am not aware of any (which doesn't mean there isn't something like that), I just know that the app is using the etag to detect changes.

So cc @tobiasKaminsky (Android) and @rullzer @MorrisJobke @LukasReschke @schiessle for server team.

[rullzer]

No such things at the moment.

[c0fe]

Can this be put up as a feature?

[AndyScherzinger]

Hi @c0fe, in generall evrything is up for discussion so yes, there is a chance to have such a feature added and since you already open up the (more important) server issue being the prerequisite nextcloud/server#1381 let's first start the discussion, decision and detailing on the server side and go on from there.

[tobiasKaminsky]

I always thought that during a (chunked) upload the transferred parts are verified that they are transferred correctly.
So I do not see the need of an additional check after uploading?

[rullzer]

@tobiasKaminsky how is that verified?

[tobiasKaminsky]

Uhh. It was just a thought that the transferred tcp packages are verified?

[iav]

It was just a thought that the transferred tcp packages are verified?

But time-to time, rarely, it's possible - look for example to fresh error in linux kernel bugs like https://ucrtoday.ucr.edu/39030.
Remember about checksum-control in modern filesystems like zfs and btrfs, remember about concept of ab ECC type of nenory modules - and about lack of support ECC functionality on a non-server harware.

It's inperfect world. Errors occcures. If can check for a checksum - better to check (if there are resources for it).

[c0fe]

I would like to add that a lot of supposed ECC memory is still ECC prone albeit at a lower frequency with the exception of GSKill memory which was tested and proved.

[rullzer]

@tobiasKaminsky well sure tcp stuff is handled by your kernel. But still there can be bugs etc. Or other ways for your file to get corrupted. Ideally you would want to be able to request the checksum of a file. This is of course very resource intensive.

[c0fe]

@rullzer so do you think this would be a possibility to generate the checksum on Android and have it sent to the server for verification?

[ZarVladimirII]

As this is still open I want to give it a push.
Since using nextcloud (Only from my Wi-Fi to the server at my brothers) I get very often the 'File already exists notification' and to manually decide which file to keep.
Also after moving my mom to a new phone it has 1000+ duplicates, of course they were already uploaded with the old phone.

A hash comparison would eliminate every notification where it is identical. It´s identical -> there is nothing to do. No need to bother someone.
Same name already exists -> cancel. Is not the same functionality. Could still be a different file.

Every backup/syncing software is able to use checksums e.g. 'FreeFileSync', 'Syncthing'

Could eliminate: #10572,#11047, #7378, and similiar situations/bugs
#6911 (comment) agrees
Makes less relevant: #11077

[qguv]

…I get very often the 'File already exists notification' and to manually decide which file to keep.

+1, a hash comparison would save a lot of manual tasks for these

[joshtrichards]

Since this will have to be driven server-side it's worth an update:

• File Verification server#1381 (comment) was closed out awhile ago
• there is a new related discussion here: Checksums on files / reliable up/download server#11138

[Siress]

See my feedback here: #11974 (comment)

"Performance issues are tolerable; data loss is not."
It's a shame, because NextCloud looked like a really beautiful, mature technology - but it lacks essential performance for a cloud app.