rasa test nlu cross validation fails with autoconfiguration

[tmbo]

Rasa version: master

Rasa SDK version (if used & relevant): -

Rasa X version (if used & relevant): -

Python version: 3.6

Operating system (windows, osx, ...): osx

Issue:

• create a config using the autoconfiguration feature (running rasa init should get you a failing config):

# Configuration for Rasa NLU.
# https://rasa.com/docs/rasa/nlu/components/
language: en

pipeline:
# # No configuration for the NLU pipeline was provided. The following default pipeline was used to train your model.
# # If you'd like to customize it, uncomment and adjust the pipeline.
# # See https://rasa.com/docs/rasa/tuning-your-model for more information.
#   - name: WhitespaceTokenizer
#   - name: RegexFeaturizer
#   - name: LexicalSyntacticFeaturizer
#   - name: CountVectorsFeaturizer
#   - name: CountVectorsFeaturizer
#     analyzer: char_wb
#     min_ngram: 1
#     max_ngram: 4
#   - name: DIETClassifier
#     epochs: 100
#     constrain_similarities: true
#   - name: EntitySynonymMapper
#   - name: ResponseSelector
#     epochs: 100
#     constrain_similarities: true
#   - name: FallbackClassifier
#     threshold: 0.3
#     ambiguity_threshold: 0.1

• run rasa test nlu --cross-validation

Error (including full traceback):

2021-05-05 18:40:50 INFO     rasa.cli.test  - Test model using cross validation.
InvalidConfigException: Can not train an empty pipeline. Make sure to specify a proper pipeline in the configuration using the 'pipeline' key.

Remove the comments (and hence not using autoconfig anymore) leads to the validation working just fine.

Definition of Done

• rasa test nlu uses autoconfig to interpolate config files with empty pipeline keys (e.g. by using the TrainingDataImporter)