Need to log TFLOPs

[stas00]

We need to log TFLOPS to TB and logs.

The code is already in https://github.com/microsoft/DeepSpeedExamples/

we just need to backport it to our repo. Here is what's needed:

./Megatron-LM-v1.1.5-ZeRO3/megatron/training.py:from megatron.utils import report_memory, flops_calculator
./Megatron-LM-v1.1.5-ZeRO3/megatron/training.py:        flops_calculator(model, args, elapsed_time)
./Megatron-LM-v1.1.5-ZeRO3/megatron/utils.py:def flops_calculator(model, args, iteration_time):
./Megatron-LM-v1.1.5-ZeRO3/megatron/utils.py:    giga_flops_per_model_per_train_step = approx_parameters_in_billions * args.batch_size * args.seq_length * 2.0 * 4.0
./Megatron-LM-v1.1.5-ZeRO3/megatron/utils.py:    effective_tera_flops_per_gpu = giga_flops_per_model_per_train_step / (iteration_time * 1000.0 * gpus_per_model)
./Megatron-LM-v1.1.5-ZeRO3/megatron/utils.py:    print_rank_0(f"Effective Tera Flops per GPU: {round(effective_tera_flops_per_gpu, 2)} and total parameters {round(approx_parameters_in_billions, 3)} B")

[jaketae]

I will be looking at this issue once #204 is sorted!

[stas00]

@jaketae, I was working on a different set of graphs and TFLOPs was just a multiple of a graph I added, so this Issue will be closed once #210 is merged.