Optimizing data parallel Fuse-Allreduce-Overlapping

[JZ-LIANG]

PR types

Performance optimization

PR changes

Others

Describe

Update1: Update the synchronization in DP-Overlapping from stream-synchronization to event-wait-record, which might reduce the synchronization overhead in scheduling.

Update2: Improve the after-allreduce-sync to allow a better fully Overlapping.
There two synchronization need by DP-Overlapping:

1. before-allreduce-sync: allreduce comm need to wait gradient computation.
2. after-allreduce-sync: the later usage of gradient should wait allreduce to be finished.
   BUT when combining Overlapping and Fusing, we lose the dependencies of allreduced gradients in the data flow graph since the Coalescence of gradients fusing.

The Common solution is to conduct the after-allreduce-sync right after fuse-allreduce (like before this PR and in Paddle Parallel Executor). But this way would lead to insufficiency of overlapping:
The CPU timeline: there is a "wait" right after nccl-sumarray

The GPU timeline: insufficiency of overlapping, the allreduce cloud only overlap with SumArray computation, but not with LayerNorm Backward.

In this PR, we resolute the exactly data flow dependency after Fusing, and instead performming the after-allreduce-sync immediately after allreduce, we put that wait where it should be (and as late as possible), which favor a better fully Overlapping.
The CPU timeline: the "right after" after-allreduce-sync is move to where it should be

The GPU timeline: better fully overlapping of allreduce and the later LayerNorm Backward kernel.

GPT3-1.3B-dp8， gbz=64	Mem (MB)	tokens/sec
Before PR	25014	88503
After	25252	95420

to reproduce the performance you need another two PR:
support exe ctx in Comm op 48308
disable redundant dependency and prior comm op in standalone exe 48454

[aoyulong]

LGTM