Investigate O_SYNC / fsync effects; update counters during buffer flush

[a-j-wood]

See what happens if we fsync() and/or use O_SYNC, which would be useful when writing to USB devices or other things that tend to hang for ages after we've finished.

[a-j-wood]

Correspondence from 2018:

From: Andrew Wood
To: Kalin KOZHUHAROV me.kalin@gmail.com
Subject: Re: Message from Contact Form (PV)

Thanks for your email. Sorry for the late reply.

PV is not aware of what's happening when you see this, because in this case
the operating system is buffering the data, so PV thinks it's finished. So
PV will be writing to the output, but it's not really going direct to the
disk, it's going to an area of memory that the operating system maintains.
The operating system then handles the slow writing of data out of that area
to the actual disk. When PV goes to close its output, when it thinks it's
finished, the operating system makes PV wait until the operating system's
write buffer is empty.

One way around this would be to add an option to PV which tells it to set
the output to bypass all operating system buffering. This would decrease
performance, but increase accuracy because none of the activity described
above would happen any more.

This is on the TODO list as "option to enable O_DIRECT" so it might appear
in a future version.

Sent from: Kalin KOZHUHAROV

Hello,
I have been using pv for hm.. 10 years? for various things, mostly processing large files. Great!
One thing that sometimes bother me is that with slow output (e.g. redirecting to a file on slow disk), the counters reach 100% and stop while the output is still "in flight".
Will it be possible to continue updating the counters (you can leave the 100% out) while the pv is waiting to exit?
Initially I thought of simply issuing sync(1) and waiting, but there is no guarantee the next in pipeline is doing diskio.

[a-j-wood]

Additional correspondence - May 2022

Sent from: Andrew

When I use pv to write an iso file to USB, it goes to 100% in 2 seconds followed by many minutes of waiting before the process finishes and I return to my terminal prompt.

sudo sh -c "pv /mnt/somepath/to/the/image.iso > /dev/sda"
1.19GiB 0:00:02 [ 535MiB/s] [=====================================================================>] 100%

pv --version
pv 1.6.20 - Copyright 2015 Andrew Wood andrew.wood@ivarch.com

arch linux 5.17.7

[a-j-wood]

This is partially covered by the new "--sync" and "--direct-io" options introduced in 1.7.17, but that will still lock up pv's display if the disk write takes a long time.