Are the depth embedding used at every transformer block of TreeGen or only at the input?

[brando90]

Hi Zeyu,

I was wondering, do you use a depth embedding that is a function of the block transformer number b? Or at the very least always apply the depth embedding at the beginning of every input block or only at the very very very beginning (i.e. only once)?

Thanks in advance for the patience!

[zysszy]

Hello,

We use depth embedding at the beginning of every input block.

Zeyu