[S-02-2] UOF + DDPG Implementation

[CUN-bjy]

to-do-list

• Implementation of DDPG + HER only. adding ddpg+her only universal-option-framwork-pytorch#3
• Implementation of high-level policy and DIOL. implementing DIOL universal-option-framwork-pytorch#4
• add AAES adding AAES universal-option-framwork-pytorch#5
• add Abstract Demonstration.

Results

• ropiens/universal-option-framwork-pytorch

[CUN-bjy]

Discussion

1. MountainCarContinuous has some dependencies between time-series actions or states-actions.

• using HAC(Hierarchical Actor Critic), this gap can be handled because all states(position and velocity) could be subgoals.
• on the other hand, using UOF, the gap couldn't be handled because we cannot choose the optimal subgoals(we don't know exactly about the domain information of environments)

2. MountainCarContinuous has just one final goal and a small range of start position.

• it makes UOF harder to converge. Since UOF just uses some subgoal(even we don't know exactly this goal is good or not), we cannot get various states and achieved goals for the hindsight framework.

3. MountainCarContinuous requires harder and continuous hierarchical reasoning.

• In UOF paper, the algorithm was tested only block-stacking manipulation problem with various start positions and goal positions. In this environment, the agent's state is not affecting its behavior.
• We cannot handle MountainCarContinuous only using discrete (and unknown) subgoals

Closing this issue,

• we need to test on block-stacking manipulation problem.
• UOF is not good enough for general hierarchical framwork.