pserver and trainer: memory increases over time.

[gongweibao]

No description provided.

[gongweibao]

GPU memory increases also:

The default exception handler is LOG(FATAL).enforce allocating <= available failed, 9129863414 > 5486083840
 at [/paddle/paddle/fluid/platform/gpu_info.cc:119]
PaddlePaddle Call Stacks:
0       0x7f30f059c248p paddle::platform::GpuMaxChunkSize() + 5080
1       0x7f30ef6fa85cp void* paddle::memory::Alloc<paddle::platform::CUDAPlace>(paddle::platform::CUDAPlace, unsigned long) + 476
2       0x7f30ef652fd8p paddle::framework::Tensor::PlaceholderImpl<paddle::platform::CUDAPlace>::PlaceholderImpl(paddle::platform::CUDAPlace, unsigned long, std::type_index) + 72
3       0x7f30ef659660p paddle::framework::Tensor::mutable_data(boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_>, std::type_index) + 336
4       0x7f30f036573cp paddle::operators::detail::VariableResponse::CopyLodTensorData(google::protobuf::io::CodedInputStream*, paddle::platform::DeviceContext const&, paddle::framework::DDim const&, int) + 1548
5       0x7f30f0366cc9p paddle::operators::detail::VariableResponse::Parse(paddle::operators::detail::Source*) + 2329
6       0x7f30f03670fap paddle::operators::detail::VariableResponse::Parse(grpc::ByteBuffer const&) + 90
7       0x7f30f036ab65p paddle::operators::detail::DeserializeFromByteBuffer(grpc::ByteBuffer const&, paddle::platform::DeviceContext const&, paddle::framework::Scope const*, paddle::framework::Variable**) + 117
8       0x7f30f0350ab6p paddle::operators::detail::ProcGetResponse(paddle::operators::detail::VarHandle const&, grpc::ByteBuffer const&) + 54
9       0x7f30f0351927p paddle::operators::detail::RPCClient::Proceed() + 551
10      0x7f30f03536e0p
11      0x7f30f0103cfep std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) + 46
12      0x7f31404b2a99p
13      0x7f30f0351497p
14      0x7f30f0436c54p paddle::framework::ThreadPool::TaskLoop() + 1012
15      0x7f3136cb5c80p
16      0x7f31404ab6bap
17      0x7f31401e141dp clone + 109
 at [/paddle/paddle/fluid/operators/detail/sendrecvop_utils.cc:192]