Clean up influx_inspect export command and improve performance

[mark-rushakoff]

• Rebased/mergable
• Tests pass
• CHANGELOG.md updated

I cleaned up and reorganized the influx_inspect export command code and added tests and benchmarks, and then I made exports go quite a bit faster.

(This benchmark is updated from a commit message to include the EscapeStringField optimization.)

benchmark                                   old ns/op     new ns/op     delta
BenchmarkExportTSMFloats_100s_250vps-4      23206480      10279106      -55.71%
BenchmarkExportTSMInts_100s_250vps-4        17995000      5762310       -67.98%
BenchmarkExportTSMBools_100s_250vps-4       17067605      4235467       -75.18%
BenchmarkExportTSMStrings_100s_250vps-4     51858849      27720601      -46.55%
BenchmarkExportWALFloats_100s_250vps-4      23459937      10436297      -55.51%
BenchmarkExportWALInts_100s_250vps-4        18747150      6236062       -66.74%
BenchmarkExportWALBools_100s_250vps-4       17988273      4814358       -73.24%
BenchmarkExportWALStrings_100s_250vps-4     55168996      29045761      -47.35%

benchmark                                   old allocs     new allocs     delta
BenchmarkExportTSMFloats_100s_250vps-4      201442         51738          -74.32%
BenchmarkExportTSMInts_100s_250vps-4        201442         51728          -74.32%
BenchmarkExportTSMBools_100s_250vps-4       201441         51638          -74.37%
BenchmarkExportTSMStrings_100s_250vps-4     404067         76938          -80.96%
BenchmarkExportWALFloats_100s_250vps-4      250322         75627          -69.79%
BenchmarkExportWALInts_100s_250vps-4        250323         75617          -69.79%
BenchmarkExportWALBools_100s_250vps-4       250321         75527          -69.83%
BenchmarkExportWALStrings_100s_250vps-4     452830         100729         -77.76%

benchmark                                   old bytes     new bytes     delta
BenchmarkExportTSMFloats_100s_250vps-4      5170539       2351789       -54.52%
BenchmarkExportTSMInts_100s_250vps-4        5143189       2331276       -54.67%
BenchmarkExportTSMBools_100s_250vps-4       3724951       2143780       -42.45%
BenchmarkExportTSMStrings_100s_250vps-4     17136727      6062059       -64.63%
BenchmarkExportWALFloats_100s_250vps-4      4487868       1468109       -67.29%
BenchmarkExportWALInts_100s_250vps-4        4458395       1452359       -67.42%
BenchmarkExportWALBools_100s_250vps-4       2838719       1258755       -55.66%
BenchmarkExportWALStrings_100s_250vps-4     16714986      5286940       -68.37%

Also, after improving those benchmarks, I did a time-filtered export on
a 450MB TSM file to a 21GB plain text output, with and without the
bufio.BufferedWriter.

Without buffering, it took about 263s, and with buffering, it took about
60s, for a delta of about -77%.

[e-dard]

Just a small nit, otherwise LGTM 👍

[e-dard]

Can this not just be w := bufio.... and then we don't need the var w = bw below?

[mark-rushakoff]

w needs to be typed as an IO Writer so the gzip writer can be conditionally assigned to w later. In fact I need to double check whether it performs better with the buffered writer in front of the gzip writer. I suspect it will.

…

On Mon, Dec 19, 2016 at 4:30 AM Edd Robinson ***@***.***> wrote: ***@***.**** approved this pull request. Just a small nit, otherwise LGTM 👍 ------------------------------ In cmd/influx_inspect/export/export.go <#7741 (review)> : > if err != nil { return err } - defer w.Close() + defer f.Close() + + // Because calling (*os.File).Write is relatively expensive, + // and we don't *need* to sync to disk on every written line of export, + // use a sized buffered writer so that we only sync the file every megabyte. + bw := bufio.NewWriterSize(f, 1024*1024) Can this not just be w := bufio.... and then we don't need the var w = bw below? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7741 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARvV1bBrt_w4KrcOlX4z1Bt85i1rxuaks5rJnjtgaJpZM4LQC9j> .

[mark-rushakoff]

Using my earlier test case of a 450MB TSM file that, when time-filtered, exports to a 21GB plain text file or a 650MB gzipped text file, I found:

time influx_inspect export -datadir . -waldir . -start '2016-12-13T10:59:49Z' -compress -out ./s4.2.txt.gz

# 1.1.1
real	5m7.206s
user	5m8.761s
sys	0m7.027s

# buf -> gzw -> f
real	3m53.893s
user	3m51.619s
sys	0m5.935s

# gzw -> buf -> f (behavior as committed in PR)
real	3m52.385s
user	3m54.782s
sys	0m1.984s

So the real time is about the same whether the buffer writer is before or after the gzip writer, but putting the gzip writer before the buffered writer seems to result in fewer file writes, which seems like the better win.

[jwilder]

Nice speedup!