进行MFA音素对齐时出错

[ms903x1]

系统：wsl2-ubuntu20.4
python版本：3.8.13

之前用库里的pinyin.dict和pinyin_eng.dict进行MFA音素对齐时提示字典错误，去MFA官网下了Mandarin (China) MFA dictionary v2.0.0后不报字典错误了，但是运行一段时间以后报了以下错误，文件夹里也没有生成TextGrid文件

错误：
INFO - Accumulating statistics...
100%|████████████████████████████████████████████████████████████▉| 841/842 [00:22<00:00, 37.72it/s]
INFO - Training complete!
INFO - Compiling training graphs...
100%|███████████████████████████████████████████████████████████▉| 841/842 [00:02<00:00, 337.80it/s]
INFO - Generating alignments...
100%|████████████████████████████████████████████████████████████▉| 841/842 [00:29<00:00, 28.54it/s]
INFO - Initializing training for lda...
100%|███████████████████████████████████████████████████████████▉| 841/842 [00:01<00:00, 707.74it/s]
ERROR - There was an error in the run, please see the log.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/ms903/anaconda3/envs/py38/bin/mfa", line 11, in
sys.exit(main())
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/command_line/mfa.py", line 1071, in main
run_train_acoustic_model(args, unknown)
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/command_line/train_acoustic_model.py", line 112, in run_train_acoustic_model
train_acoustic_model(args, unknown_args)
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/command_line/train_acoustic_model.py", line 36, in train_acoustic_model
trainer.train()
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/acoustic_modeling/trainer.py", line 394, in train
trainer.train()
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/acoustic_modeling/base.py", line 533, in train
self.initialize_training()
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/acoustic_modeling/base.py", line 266, in initialize_training
self._trainer_initialization()
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/acoustic_modeling/lda.py", line 453, in _trainer_initialization
self.lda_acc_stats()
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/site-packages/montreal_forced_aligner/acoustic_modeling/lda.py", line 442, in lda_acc_stats
shutil.copyfile(
File "/home/ms903/anaconda3/envs/py38/lib/python3.8/shutil.py", line 264, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/home/ms903/Documents/MFA/split_train_acoustic_model/lda/lda.mat'

[jerryuhoo]

用不同的字典可能在后面会有问题，用pinyin.dict和pinyin_eng.dict会报什么错？最近没时间搞这个了，之前没固定版本，可能MFA更新了？如果用之前的MFA2最开始的几个版本试试呢？

[ms903x1]

用不同开始的字典可能会在后面出现问题，用pinyin.dict和pinyin_eng.dict会报什么？最近没时间搞这个了，在没有固定版本之前，可能MFA更新了？用之前的MFA2最差的几句一个版本呢？

用pinyin.dict和pinyin_eng.dict 的错误是：
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
ERROR - There was an error in the run, please see the log.
DictionaryError:

Error parsing line 0 of MFA/pinyin.dict: Did not find any tabs, please ensure that your dictionary has tabs between
words and their pronunciations.

OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
ArgumentError:

Could not find the corpus directory /home/ms903/data/wav/speaker_name.

OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
ERROR - There was an error in the run, please see the log.
DictionaryError:

Error parsing line 0 of /home/ms903/Downloads/VTuberTalk/MFA/pinyin.dict: Did not find any tabs, please ensure that
your dictionary has tabs between words and their pronunciations.

[ms903x1]

我用MFA官网的模型加字典成功生成了TextGrid，进行训练的时候爆显存了，调什么参数能避免爆显存？
我的显卡是1070

报错：
Exception in main training loop: (Fatal) Operator transpose2 raises an paddle::memory::allocation::BadAlloc exception.
The exception content is
:ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 145.312500MB memory on GPU 0, 7.999878GB memory has been allocated and available memory is only 0.000000B.

Please check whether there is any other process using GPU 0.

1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model.
   If the above ways do not solve the out of memory problem, you can try to use CUDA managed memory. The command is export FLAGS_use_cuda_managed_memory=false.
   (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:87)
   . (at /paddle/paddle/fluid/imperative/tracer.cc:307)

Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/training/trainer.py", line 149, in run
update()
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/training/updaters/standard_updater.py", line 109, in update
self.update_core(batch)
File "train/models/fastspeech2/fastspeech2_updater.py", line 77, in update_core
spk_emb=spk_emb)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "train/models/fastspeech2/fastspeech2.py", line 557, in forward
tone_id=tone_id)
File "train/models/fastspeech2/fastspeech2.py", line 718, in _forward
zs, _ = self.decoder(hs, h_masks)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/encoder.py", line 361, in forward
xs, masks = self.encoders(xs, masks)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/repeat.py", line 25, in forward
args = m(*args)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/encoder_layer.py", line 95, in forward
x = residual + self.dropout(self.feed_forward(x))
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/multi_layer_conv.py", line 67, in forward
return self.w_2(self.dropout(x).transpose([0, 2, 1])).transpose(
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layers/nn.py", line 5692, in transpose
out, _ = _C_ops.transpose2(x, 'axis', perm)
Trainer extensions will try to handle the extension. Then all extensions will finalize.Traceback (most recent call last):
File "train/exps/fastspeech2/train.py", line 233, in
main()
File "train/exps/fastspeech2/train.py", line 229, in main
train_sp(args, config)
File "train/exps/fastspeech2/train.py", line 170, in train_sp
trainer.run()
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/training/trainer.py", line 198, in run
six.reraise(*exc_info)
File "/usr/local/lib/python3.7/dist-packages/six.py", line 719, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/training/trainer.py", line 149, in run
update()
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/training/updaters/standard_updater.py", line 109, in update
self.update_core(batch)
File "train/models/fastspeech2/fastspeech2_updater.py", line 77, in update_core
spk_emb=spk_emb)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "train/models/fastspeech2/fastspeech2.py", line 557, in forward
tone_id=tone_id)
File "train/models/fastspeech2/fastspeech2.py", line 718, in _forward
zs, _ = self.decoder(hs, h_masks)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/encoder.py", line 361, in forward
xs, masks = self.encoders(xs, masks)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/repeat.py", line 25, in forward
args = m(*args)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/encoder_layer.py", line 95, in forward
x = residual + self.dropout(self.feed_forward(x))
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call
return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddlespeech/t2s/modules/transformer/multi_layer_conv.py", line 67, in forward
return self.w_2(self.dropout(x).transpose([0, 2, 1])).transpose(
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layers/nn.py", line 5692, in transpose
out, _ = _C_ops.transpose2(x, 'axis', perm)
SystemError: (Fatal) Operator transpose2 raises an paddle::memory::allocation::BadAlloc exception.
The exception content is
:ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 145.312500MB memory on GPU 0, 7.999878GB memory has been allocated and available memory is only 0.000000B.

Please check whether there is any other process using GPU 0.

1. If yes, please stop them, or start PaddlePaddle on another GPU.
2. If no, please decrease the batch size of your model.
   If the above ways do not solve the out of memory problem, you can try to use CUDA managed memory. The command is export FLAGS_use_cuda_managed_memory=false.
   (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:87)
   . (at /paddle/paddle/fluid/imperative/tracer.cc:307)

[jerryuhoo]

我用MFA官网的模型加字典成功生成了TextGrid，进行训练的时候爆显存了，调什么参数能避免爆显存？ 我的显卡是1070

train/conf里调整batch size