EnableRewind causes data loss on files larger than the buffer threshold

[DavidObando]

Multipart posts containing files larger than the default buffering threshold (currently 30 KB) used by EnableRewind will result in data loss.

The data loss I've seen is usually at the beginning of the file, and usually around 28 KB long. For example, uploading a file that on the client side is 100 KB, the server will save ~72.3 KB on a file. As a proof of concept, when I enlarged the buffering threshold to 200 KB the server was able to save the entire 100 KB file. The solution of course isn't to enlarge the buffer threshold but to make it so we don't lose the bits from the first buffer frame, as we apparently are.

A self-contained repro can be found at: https://github.com/DavidObando/BugRepros/tree/master/2016-03-31/EnableRewindBug. The repro will save the client-side file and the server-side file in C:\Temp\ for easy comparison.

[DavidObando]

/cc: @yuwaMSFT @sivagms

[davidfowl]

/cc @Tratcher

[Tratcher]

Hmm, there was a perf change in this code last week, I wonder if it regressed something...
bd60507#diff-6bee66e850efacd8fe96daae334fa1c5R195

[davidfowl]

Revert it and test.

[Tratcher]

Yes, that was it. I'll send a PR after I add some tests.