Corruption of persistent database file cause by sudden lost of power

[thanhvtruong]

It looks like when the persistent file is being save there is a tiny amount of time where a sudden power lost will cause the persistent database file to be corrupted.

Thousands of sudden power lost on our system were performed we notice the following:

• Mosquitto start up with "invalid argument" and "database" read error.
• mosquitto.db and mosquitto.db.new both existed in /var/lib/mosquitto
• Both mosquitto.db and mosquitto.db.new have the same inode. (suggest that a rename has occurred)

We have a similar problem with another application that we build and reading into Linux documentation, we found out that flushing or closing a file is not enough to write the content to disk. An fsync need to be perform to confirm that the content is written to disk.

You can see the documentation in the man page (man close, under "note" second paragraph) on Fedora 23.

"A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a filesystem to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored, use fsync(2). (It will depend on the disk hardware at this point.)"

[ralight]

Thanks for the report, I've added code to call fsync() on the file and on its directory. Could you please confirm whether this fixes the problem for you?

[kcallin]

Recommend calling fflush before fsync to ensure that application buffers are completely flushed to kernel buffer before being flushed to disk. It's awful hard to reproduce this, but between thanhvtruong and myself we started a long-term test series to mechanically verify.

I do not believe the directory sync is required; the rename logic should work as-is.

I opened a pull requrest for these changes and will update as the long-term tests progress.

[ralight]

Thanks very much for your work on this, I'm closing this now based on your pull request.