improves memory usage of add

[whyrusleeping]

Most of the memory leak in add was because the dag editor is given an in memory dagservice, and it adds all of the intermediate nodes to it. This makes it remove an intermediary node if its about to be changed and readded.

Memory pressure is reduced a good amount, but its not 'good' yet. The other changes I want to make need to be made on top dev0.4.0

License: MIT
Signed-off-by: Jeromy jeromyj@gmail.com

[rht]

Can this be captured with mean memory usage from command time -f '%K' ipfs add -r random-dir?

[rht]

@whyrusleeping here is a quick script to check

TIME() {
    command time -f '%K' $*
}

perf() {
    mkdir -p foo
    random-files -q --depth=1 --files="$1" --filesize=1000 foo #1KB
    TIME ipfs add -r -q foo >hashname  # preferably --no-pin
    ipfs pin rm -r $(tail -n1 hashname)
    rm hashname
    rm -r foo
}

echo -n "" >outdata
for i in $(seq 10 50 5000)
do
    perf $i 2>>outdata
done

[whyrusleeping]

@rht yeah, thats roughly what i was doing to test locally (minus it being a pretty shell script). The issue i noticed is that its not very consistent, so i'm hesitant to have strict tests around memory consumption, that said, We can probably make something like t0300-memory.sh and have things like this in it.

[jbenet]

👍 to having mem tests.

• rerunning tests as they halted.

[whyrusleeping]

looks like i broke something... wheeeeee

[whyrusleeping]

@jbenet i can add a script that checks memory consumption here, but making it into a test is difficult, as every different os and computer will report different numbers. My linux desktop with 32GB ram running a given ipfs test will use more memory than my laptop with 8GB ram just due to the way memory is allocated, not to mention differences across osx, windows, linux and the various architectures of each.

[whyrusleeping]

closing in favor of #1965