invalid domain in model breaks Rasa Open Source server

[wochinge]

Problem:
When Rasa X serves (for whatever reason) a model with an invalid domain (also NLU model?) then the entire Rasa Open Source server crashes and only comes up when re-starting (at which point it will break again 🤔 )

Possible Solution

line 95 in agent.py needs to be part of the try: to catch this thing

Example stack trace

2020-06-09 08:52:02 ERROR    pykwalify.core  - validation.invalid
2020-06-09 08:52:02 ERROR    pykwalify.core  -  --- All found errors ---
2020-06-09 08:52:02 ERROR    pykwalify.core  - ["Value '{'airtravel_form': {'travel_departure': [{'entity': 'city', 'role': 'from', 'type': 'from_entity'}, {'entity': 'iata', 'role': 'from', 'type': 'from_entity'}, {'entity': 'city', 'type': 'from_entity'}, {'entity': 'iata', 'type': 'from_entity'}, {'intent': 'inform', 'type': 'from_text'}], 'travel_destination': [{'entity': 'city', 'role': 'to', 'type': 'from_entity'}, {'entity': 'iata', 'role': 'to', 'type': 'from_entity'}, {'entity': 'city', 'type': 'from_entity'}, {'entity': 'iata', 'type': 'from_entity'}, {'intent': 'inform', 'type': 'from_text'}], 'travel_flight_class': [{'intent': 'affirm', 'type': 'from_intent', 'value': 'economy'}, {'intent': 'deny', 'type': 'from_intent', 'value': 'business'}, {'entity': 'travel_flight_class', 'type': 'from_entity'}]}}' is not of type 'str'. Path: '/forms/0'"]
2020-06-09 08:52:02 ERROR    rasa.core.agent  - Could not load model due to Failed to validate yaml file. Please make sure the file is correct and all mandatory parameters are specified; to do so, take a look at the errors logged during validation previous to this exception..
[2020-06-09 08:52:02 +0000] [1] [ERROR] Experienced exception while trying to serve
Traceback (most recent call last):
  File "/opt/venv/lib/python3.7/site-packages/rasa/utils/validation.py", line 58, in validate_yaml_schema
    c.validate(raise_exception=True)
  File "/opt/venv/lib/python3.7/site-packages/pykwalify/core.py", line 167, in validate
    error_msg=u'.\n - '.join(self.validation_errors)))
pykwalify.errors.SchemaError: <SchemaError: error code 2: Schema validation failed:
 - Value '{'airtravel_form': {'travel_departure': [{'entity': 'city', 'role': 'from', 'type': 'from_entity'}, {'entity': 'iata', 'role': 'from', 'type': 'from_entity'}, {'entity': 'city', 'type': 'from_entity'}, {'entity': 'iata', 'type': 'from_entity'}, {'intent': 'inform', 'type': 'from_text'}], 'travel_destination': [{'entity': 'city', 'role': 'to', 'type': 'from_entity'}, {'entity': 'iata', 'role': 'to', 'type': 'from_entity'}, {'entity': 'city', 'type': 'from_entity'}, {'entity': 'iata', 'type': 'from_entity'}, {'intent': 'inform', 'type': 'from_text'}], 'travel_flight_class': [{'intent': 'affirm', 'type': 'from_intent', 'value': 'economy'}, {'intent': 'deny', 'type': 'from_intent', 'value': 'business'}, {'entity': 'travel_flight_class', 'type': 'from_entity'}]}}' is not of type 'str'. Path: '/forms/0'.: Path: '/'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/domain.py", line 128, in from_yaml
    validate_yaml_schema(yaml, DOMAIN_SCHEMA_FILE)
  File "/opt/venv/lib/python3.7/site-packages/rasa/utils/validation.py", line 61, in validate_yaml_schema
    "Failed to validate yaml file. "
rasa.utils.validation.InvalidYamlFileError: Failed to validate yaml file. Please make sure the file is correct and all mandatory parameters are specified; to do so, take a look at the errors logged during validation previous to this exception.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.7/site-packages/sanic/app.py", line 1167, in run
    serve(**server_settings)
  File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 892, in serve
    trigger_events(before_start, loop)
  File "/opt/venv/lib/python3.7/site-packages/sanic/server.py", line 668, in trigger_events
    loop.run_until_complete(result)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/run.py", line 258, in load_agent_on_start
    action_endpoint=endpoints.action,
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/agent.py", line 247, in load_agent
    model_server,
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/agent.py", line 63, in load_from_server
    await _update_model_from_server(model_server, agent)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/agent.py", line 125, in _update_model_from_server
    _load_and_set_updated_model(agent, model_directory, new_model_fingerprint)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/agent.py", line 95, in _load_and_set_updated_model
    domain = Domain.load(domain_path)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/domain.py", line 100, in load
    other = cls.from_path(path)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/domain.py", line 110, in from_path
    domain = cls.from_file(path)
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/domain.py", line 123, in from_file
    return cls.from_yaml(rasa.utils.io.read_file(path))
  File "/opt/venv/lib/python3.7/site-packages/rasa/core/domain.py", line 130, in from_yaml
    raise InvalidDomain(str(e))
rasa.core.domain.InvalidDomain: Failed to validate yaml file. Please make sure the file is correct and all mandatory parameters are specified; to do so, take a look at the errors logged during validation previous to this exception.

[wochinge]

It happens rarely but then it is a major blocker. Hence, I gave it a prio high label.