Description

This phase will focus exclusively on NLU (natural language understanding).
NLU engines perform two tasks:

1. Intent matching,
2. entity extraction.

How can developers who have no experience in NLU attack these subjects? What data can they use for training and testing of NLU engines? How can a developer or even a user easily fix a defect (i.e., the utterance is routed to the wrong intent)?

We want to help developers understand NLU engines through example engines, benchmark these examples, and deep dive into current popular engines.

Developers can either clean their own data set with the tools provided, or they can use the comprehensive, quality NLU data set from Secret Sauce AI.

High-level user stories

As a developer,

• I want a data set I can convert to other formats, that contains many annotated utterances so that I can use them for my domains/skills.
• I want several NLU engines benchmarked so that I can pick the right one for me.
• I want easy to follow examples of how an NLU engine works so that I have a basic understanding.

Prototype deliverables

NLU-engine-prototype-benchmarks repo

• Find the best possible intent and entity dataset across many domains (skills)
• Explore possible current solutions for NLG that include grammar agreement
• Create notebook using the most basic engine components (intent and entity extraction) to demonstrate the process and benchmark solutions
• Deep dive into a popular NLU engine (i.e. Snips), write a dataset convertor, and benchmark results
• Create a data cleaning pipeline
• Make an intent and entity classifier with DistilBERT and benchmark the results
• Write an article with a summary, links to the dataset, notebooks, models, etc.

DoD (definition of done)

See NLU Engine Prototype Benchmark milestones

KPI—intent and entity tagging

• clean data set
• intent f1 score for the whole dataset (all intents)
• entity extraction (entity tagging) f1 ones for the whole dataset
• intent and entity extraction per domain f1 scores
• documentation
  • prototype engine
  • onboarding
  • results
  • cleaning tool
  • DistilBERT engine
• document