Use mbuffer to smooth things out

[lkowolowski]

Wrap the transfer in buffers to make it less bursty

[Jehops]

Thanks for this.

Could you elaborate about the problem this is solving? I'm not keen on adding external dependencies unless the benefit is substantial and mbuffer is not part of the default installation on FreeBSD and the GNU/Linux distributions that I checked out.

[lkowolowski]

Could you elaborate about the problem this is solving? I'm not keen on adding external dependencies unless the benefit is substantial and mbuffer is not part of the default installation on FreeBSD and the GNU/Linux distributions that I checked out.

Sure. I don't know about Linux, but you're right that its not part of the FreeBSD base. The notion is that is helps keep the network stream as fast as possible, while reading from/to storage can ebb and flow. Its not uncommon for a TCP stream, once it slows down due to congestion, to never (or very slowly) ramp back up. This is less noticeable on fast local networks but is very noticeable over more latent connections such as internet.

[maxatome]

While I am sure using mbuffer can greatly improve the transfer, I think one should test whether it exists or not before using it, so it does not become a hard dependency.
Just my 2 cents.

[Jehops]

Sorry for the delay. Adding a check to see if mbuffer is installed and then optionally using it sounds like a reasonable approach. Package maintainers can then choose to add mbuffer as a dependency. I could make it an option for the FreeBSD port. I'm swamped with a few things, but will get to this soon.

Thanks,

Joe

[Jehops]

Apologies for the delay. I created a branch with optional mbuffer support.

https://github.com/Jehops/zap/compare/mbuffer?expand=1

I'm still on the fence due to the concerns expressed in the commit log. Do you have suggestions for tuning the memory usage?

[maxatome]

What about a MBUFFER_OPTS env var with default values? like

: ${MBUFFER_OPTS:='-s 128k -m 1G'}

so the user can easily override it.

[maxatome]

Can't we simplify the whole thing by providing some variables and a zfs_send function like:

mbuf=$(command -v mbuffer)
mbuf_pipe=
if [ -n "$mbuf" ]; then
    : ${MBUFFER_OPTS:='-s 128k -m 1G -q'}
    mbuf_pipe="mbuffer $MBUFFER_OPTS | "
fi

zfs_send() {
    if [ -n "$mbuf" ]; then
        zfs send "$@" | mbuffer $MBUFFER_OPTS
    else
        zfs send "$@"
    fi
}

then use zfs_send everywhere zfs send is used and use $mbuf_pipe in ssh command strings just before zfs recv … as in:

ssh "$sshto" "sh -c '${mbuf_pipe}zfs recv -Fu $v_opt -d $rloc'"

as well as in all echo messages:

echo "zfs send -Lep $C_opt $lsnap | ${mbuf_pipe}ssh $sshto \"sh -c '${mbuf_pipe}zfs recv -Fu

[lkowolowski]

Can't we simplify the whole thing by providing some variables and a zfs_send function like:

mbuf=$(command -v mbuffer)
mbuf_pipe=
if [ -n "$mbuf" ]; then
    : ${MBUFFER_OPTS:='-s 128k -m 1G -q'}
    mbuf_pipe="mbuffer $MBUFFER_OPTS | "
fi

zfs_send() {
    if [ -n "$mbuf" ]; then
        zfs send "$@" | mbuffer $MBUFFER_OPTS
    else
        zfs send "$@"
    fi
}

then use zfs_send everywhere zfs send is used and use $mbuf_pipe in ssh command strings just before zfs recv … as in:

ssh "$sshto" "sh -c '${mbuf_pipe}zfs recv -Fu $v_opt -d $rloc'"

as well as in all echo messages:

echo "zfs send -Lep $C_opt $lsnap | ${mbuf_pipe}ssh $sshto \"sh -c '${mbuf_pipe}zfs recv -Fu

I like this. Probably what I should've submitted in the first place.
The only other thing I can think of is that we could dynamically set the -m option by calculating a percentage of installed memory, with a max of 1g. Seems like this might help lower memory systems that still want to use ZFS.

MEM=`free | awk '/mem_phys/ {print $3}'`
PERCENT=".1"
BUFFER=`echo " scale=0; ( ${MEM} * ${PERCENT} )/1" | bc -l`
if [ "${BUFFER}" -gt 1024000000 ] ; then
	BUFFER="1g"
fi

[maxatome]

IMHO computing values using current free memory amount should be let to the user. Wiith MBUFFER_OPTS var, he can do it easily without involving zap internal rules. Note that free command is not available in FreeBSD bare release.

The default MBUFFER_OPTS value can just be -q as mbuffer automatically compute default values for -k and -m (2% of available memory says the manual).

[Jehops]

I'll give this a shot hopefully over the weekend.

I had a conversation about this with a colleague, @wahjava, and he had another good suggestion. We could make it even more generic to allow the user to choose any filter. He mentioned pv, which he says can throttle speeds and also displays the transfer rate. User could then set something like ZAP_FILTER='mbuffer...'.

[Jehops]

Hello both,

Does the implementation in the filter branch meet your needs?

https://github.com/Jehops/zap/compare/filter?expand=1

Any testing you can do would be appreciated.

Joe

[lkowolowski]

I'm good with this.

[maxatome]

Same here, thanks!

[Jehops]

Thanks both. I will merge the filter branch after a few days of running it through my backups with different filters.