Refactor LLM pretraining examples

[maanug-nv]

What does this PR do ?

Simplify LLM pretraining example scripts by moving common logic into a new TrainerBuilder and exp_manager.

Collection: [Note which collection this PR will affect]

Changelog

• add a TrainerBuilder type that hides some common logic for setting up a Trainer
• move logic to handle resume_from_checkpoint arg to exp_manager Current disabling resume_from_checkpoint logic and leaving a TODO comment since its current behavior is not a desirable user experience. Will make a separate PR to improve this.
• utilize above refactors to reduce length of Megatron LLM pretraining example scripts

Usage

• You can potentially add a usage example below

# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation
• Refactor

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[maanug-nv]

Rebased to include PTL 2.0 changes from #6433

[athitten]

@maanug-nv where are we taking care of the below lines then:

    if cfg.model.resume_from_checkpoint is not None:
        trainer.ckpt_path = cfg.model.resume_from_checkpoint
    logging.info(f'Resuming training from checkpoint: {trainer.ckpt_path}')

Since the pre training scripts were assigning the checkpoint to trainer.ckpt_path if we passed a checkpoint path for resume_from_checkpoint under model in config.

[maanug-nv]

Yes, so initially I moved those lines exactly as is to this place in exp_manager.py. After testing and discussing with @titu1994 , having those lines here (or in pretraining as they were before/are currently on main) has some undesirable behavior, details below. I wanted to keep this PR purely refactor (thought that would get it merged faster), so I'll correct the behavior in another PR. I can uncomment these lines if you prefer.

if 'resume_from_checkpoint' is set, that checkpoint is always used despite what is in the log dir. What makes more sense is that 'resume_from_checkpoint' is used if no log_dir is present, but log_dir takes priority if present.

[ericharper]

LGTM. Thanks!

@maanug-nv will address resume_from_checkpoint in a follow up PR