The trained LLaMA-MLP-MoE (2/8) cannot generate the correct output

[cnlinxi]

Hello authors. I tried to train LLaMA-MLP-MoE (2/8). After two stages of training, the model cannot output normal sentences. The inference script is as follows:

model_dir = ""
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_dir, torch_dtype=torch.bfloat16, trust_remote_code=True)
model.eval()
model.cuda()

input_text = "Could you recommend me some mystery novels?"
input_text = f"<|start_header_id|>user<|end_header_id|>\n\n{input_text}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"
inputs = tokenizer(input_text, return_tensors="pt")
input_ids = inputs["input_ids"].cuda()

pred = model.generate(input_ids, max_length=200, temperature=1.0, do_sample=True, use_cache=True)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

The output is:

Could you recommend me some mystery novels?assistant

 to.The old answer and you, can的 is a, ),2 and,的 is is as。 the : for2 to in the is of the as。  we and "). the.

， of1 a for and is is you，,).

 a is on: in and a1, : with that    ( of. the    of to of and to to for the2 of. and =) of

Where the error occurred?

Thank you for your reply

[cnlinxi]

I follow supervised_fine_tuning/LLaMA-MoE-v2.md and dataset is:

First-stage

OpenHermes-2.5
SlimOrca
sharegpt_gpt4
lima
Infinity-Instruct

Two-stage

Infinity-Instruct
MetaMathQA

[cnlinxi]

By the way, can you show the training loss and trend?
The loss I tried was always around 7.8 during second stage and could not be reduced.
Thank you for your reply.