This document explains how to create a feature branch in your GitHub repository, how to make updates to your LUIS app and to save changes in source control, how to raise a pull request (PR) to merge the updates back into the master branch and how to execute continuous integration workflows.
In this example, we follow the GitHub flow branching strategy which is a simple and effective branching strategy. Broadly you will follow this flow:
- Developer creates a feature branch and does the feature/work in that branch.
- When done, the developer does a push of their changes and raises a pull request from their feature branch to master. The developer works on the updates using a LUIS app that they create solely to support the work in the feature branch.
- The continuous integration workflow is triggered automatically by the pull request and runs as a quality gate check. It builds a temporary LUIS app from the source in the PR, runs unit tests against it and then deletes it at the end of the run.
- If the workflow completes successfully, and reviewers approve the pull request, the developer merges the PR into master.
- The merge to master automatically triggers the full CI/CD workflow which:
- Creates a new LUIS app version in the master LUIS app from the merged source. This LUIS app will be created if it does not already exist, such as on first run of the CI/CD workflow.
- Runs unit test against it. If the tests fail, GitHub fails the workflow and notifies repository members by email.
- If unit tests pass, creates a GitHub Release of the repository.
- Runs LUIS quality tests to determine and publish the F-measure of the new LUIS app version.
This document explains how to run through these steps. Note that all steps involving git operations are shown using the Git Bash shell. Developers may have their own preference for using GUI tools or tooling built into developer tools such as Visual Studio Code to achieve the same result.
You will be performing the following steps to make updates to the LUIS app:
- Making updates to a LUIS app in a feature branch
You will need to install the following tools:
-
Bot Framework CLI - this includes tools for working with LUIS apps. Follow these instructions to install the Bot Framework CLI tools.
-
NLU.DevOps - you use these tools for running tests against a LUIS model. Follow these instructions to install the NLU.DevOps tools.
If you followed the setup instruction for this sample you will have cloned your repository from GitHub to your own machine. To set up a feature branch:
-
After the GitHub repository is cloned, navigate to the project directory:
$ cd my-LUIS-DevOps-sample -
Create a feature branch and check it out:
$ git checkout -B update-luis-sample
Now that you are working inside the feature branch, you can make your updates to the LUIS app source, unit tests, and model verification tests. If this was a brand new project, you would need to create the LUDown representation of the first version of your LUIS app and the JSON files for testing and check them in. We use the LUDown format to define a LUIS app since it can be maintained in a source control system and is human readable which enables the reviewing process because of its legibility.
In this sample, the LUDown for a sample application and the test files are provided:
- luis-app/model.lu - the LUDown encoding of the LUIS app
- luis-app/tests/unittests.json - the unit tests
- luis-app/tests/verificationtests.json - the verification tests
For minor updates to a LUIS app, it is possible to edit the model.lu file directly. However, for most feature branch development, it is easier and more practical to make updates using the LUIS Portal.
To make updates using the LUIS Portal:
-
Sign into the LUIS portal for your authoring and publishing region:
- LUIS authoring portal - https://www.luis.ai
- LUIS authoring portal (Europe) - https://eu.luis.ai
- LUIS authoring portal (Asia) - https://au.luis.ai
Important: If you are an existing LUIS user and have not yet migrated your account to use an Azure resource authoring key rather than an email, you should consider doing this now. If you do not migrate your account, you will not be able to select LUIS Authoring resources in the portal and it will not be possible to follow all the steps described in this solution walkthrough. See Migrate to an Azure resource authoring key for more information.
-
Select the Azure subscription and the authoring resource you want to use while you are working in the feature branch (note that this does not have to be the same subscription and/or authoring resource that you configured for the GitHub Actions workflows):
-
You will now create a LUIS app that you will use just for the work in this feature branch:
- Click New app for conversation. In the dropdown, click Import as LU.
- Select the /luis-app/model.lu file in this repo in the Import new app dialog, set a suitable name for the app such as DEV-update-luis-sample and click Done.
-
Now you can make your changes to the app. For the purposes of this sample, you will add a new training utterance to the None intent.
-
From the Intents editor, click on the None intent.
-
Enter a new example utterance, for example: Great that we can do DevOps with LUIS and hit Enter.
-
Click Train at the top of the page.
-
When training is complete, then click Publish, select Staging Slot and then click Done.
-
The CI workflows are setup to perform automated unit testing of the LUIS model when you raise your PR, and again when your changes are merged to master. The test utterances and the expected responses are defined in the luis-app/tests/unittests.json file.
It is good development practice for the developer to run all the unit tests manually during feature development to make sure that no problems have been introduced before checking in changes. For this, we use the NLU.DevOps tool, https://github.com/NLU.DevOps/.
You need to enter settings for the test target LUIS app - the app you have just created - in an appsettings.local.json file. In the luis-app folder, a sample appsettings.sample.json has been provided. Rename this file to appsettings.local.json and then enter the required values:
{
"luisAppId": "***your LUIS App Id***",
"luisEndpointKey": "****your LUIS endpoint key***",
"luisPredictionResourceName": "***your LUIS prediction resource name***",
"luisIsStaging": true
}-
You can find the LUIS App Id for your app by clicking the Manage tab in the LUIS Portal. The App ID is shown on the Application Information page.
-
To get the luisEndpointKey value, go to the Azure Resources page on the Manage tab in the LUIS Portal, click on the Prediction resources tab and copy the Primary Key of your Azure prediction resource.
-
To get the luisPredictionResourceName, enter the name of your prediction resource, for example LUISDevOpsResource-Prediction.
-
Save this file.
-
In your console terminal, set your current working directory to the luis-app folder.
-
Run the unit tests defined in /tests/unittests.json against the LUIS app you just published:
dotnet nlu test -s luisV3 -u .\tests\unittests.json -o results.json -
The
nlu testcommand runs your test utterances against the LUIS app endpoint and returns the results in the results.json file. In order to compare these new results with the expected results (defined in the tests/unittests.json file), use thenlu comparecommand:dotnet nlu compare -e .\tests\unittests.json -a results.json --unit-test -
This reports which tests have passed and failed:
To find out more about unit testing with NLU.DevOps, read Testing an NLU model and Analyzing NLU model results.
Now that the changes have been applied to the LUIS app and you have tested it, you must download the updated LUIS app from the LUIS Portal and check in your changes and raise the PR.
-
Download the LUIS app version and convert to LUDown either at the command line or using the LUIS portal:
-
To download the app version at the command line:
-
Use the following command:
$ bf luis:version:export --appId {your-appId} --versionId "0.1" --endpoint {your-authoring-endpoint} --subscriptionKey {your-authoring-key} --out model.jsonNote: The App ID and Authoring Key are the same values you entered in the appsettings.local.json file. You can get the Authoring Endpoint from the Manage tab, Azure Resources page for your app in the Azure portal.
-
Convert the JSON file to LUDown:
$ bf luis:convert -i model.json -o model.lu
-
-
Alternatively, to export the app from the LUIS portal:
- In the LUIS Portal, click the Manage tab at the top of the page, and then go to the Versions page.
- Select the latest version (there will only be one, version 0.1, if you have been following this solution walkthrough), click Export and then click Export as LU.
- Take the downloaded file (currently named as {LUIS App Id}_v0.1.lu) and rename it to model.lu
-
-
Copy the new model.lu file to luis-app/model.lu in your project, replacing the existing version of that file.
-
Add your changed file(s) to the commit and commit the changes:
$ git add -A
$ git commit -m 'Updated the none intent' -
Push the changes up to the remote - the GitHub repository:
$ git push --set-upstream origin update-luis-sample -
In order to raise the Pull Request for these changes, we switch to the GitHub portal.
- Go to your repository on github.com. Click on the Pull requests tab.
- You should see a notification that you recently pushed the update-luis-sample branch. Click on the green Compare & pull request button.
-
In the Open a pull request page, click the green Create pull request button near the bottom of the page.
-
You will see that after the pull request is raised, the status screen shows that completion of the pull request is blocked because of Review required. In addition, after a few seconds, you will see an additional status display Some checks haven't completed yet and underneath that the status of the LUIS-CI /Build and Test LUIS model (pull_request) CI workflow as it runs, triggered by the raising of the pull request.
- You can click on the Details link next to the Build workflow status to see the workflow stages as they execute.
- Read more about the internals and operation of the workflow in the document GitHub Actions workflow with NLU.DevOps.
-
When the workflow has completed, the All checks have passed status shows as green since the CI workflow ran and the unit tests passed. The workflow has performed its function of a quality gate on the changes in the pull request. However, merging of the pull request is still blocked since at least 1 approving review is required by reviewers.
- On a real project, the changes should be reviewed and approved. However, for this sample, you can override this requirement since you are the Administrator of this repository.
- Click on the Merge pull request button.
- On the next page, check Use your administrator privileges to merge this pull request.
- Click Confirm merge.
If you click on the Actions tab immediately after you merge your pull request, you will see that the full CI/CD workflow has already been triggered and is executing by the push to master. The first job of this workflow is similar to the one that executed for the pull request, but it has some important differences in operation:
- It builds a new LUIS app version from the source that has been merged. It uses the LUIS app that is dedicated to the master branch, the name of which is set in the environment variables at the top of the .github/workflows/luis_ci.yaml file. The app is created if it does not already exist, such as on first run of the workflow.
- It runs the job LUIS Build and Test which builds the new LUIS master app version and runs unit tests against it and if the tests pass it creates a GitHub release for the new version.
- If the LUIS Build and Test job completes successfully, it runs the LUIS CD job . This job publishes the app version to the Production endpoint. It is a simple example of a CD (Continuous Delivery) workflow.
- It also runs the LUIS F-measure testing job which runs LUIS verification tests (equivalent to using the batch testing capability in the LUIS portal).
The LUIS F-measure testing job is currently configured to run concurrently with the Create LUIS Release job and runs purely as an advisory and does not block the LUIS CD job from running should it fail. It is provided as an example of how to perform model quality testing within the workflow. See Enabling the F-measure testing capability for details of how to enable this feature. - If the workflow fails, the repository contributors and the author of the pull request are notified by email and must determine the failing tests and resolve the code failures that caused the workflow failure.
Note: This solution uses GitVersion to increment the version number on every build. It looks at your git history on every commit to calculate what the version currently is, and calculates the new version number automatically using semantic versioning. You can instruct GitVersion to manually increment the major, minor or patch version using commit messages, or by setting the next-version property in the GitVersion.yml file in the root of your repository. Read more about GitVersion Version Incrementing in the GitVersion documentation.
After the workflow has completed successfully, if you look on the home page of your repository, you can see that a new release has been created:
When you click on this release flag, you can see the details of the new release, which is the zipped source code from the repository and a file called luis_latest_version.json.If you click on luis_latest_version.json you can download it and open it in a text editor. This gives you data about the new LUIS app version the workflow has created.
To view the LUIS app version that the workflow has created:
- Go to the LUIS Portal to the My apps page, select your Azure subscription and the LUIS authoring resource you configured for the workflow during Provisioning Azure resources while setting up this solution.
- You will see the LUISDevOps-master app that the workflow has created. This name of this app is defined in the Environment variables at the top of luis_ci.yaml.
- Select the LUISDevOps-master app and go to the Manage tab, and then to Versions, you can see the LUIS app version that the workflow has built and tested.
Important: If you are an existing LUIS user and have not yet migrated your account to use an Azure resource authoring key rather than an email, you should consider doing this now. If you do not migrate your account, you will not be able to select LUIS Authoring resources in the portal and it will not be possible to find apps in the portal that have been created using Azure LUIS Authoring resources, such as the app created by this workflow. See Migrate to an Azure resource authoring key for more information.
You can test out the new LUIS app version by sending a prediction request from the LUIS portal using the Test button on the task bar:
-
In the LUIS portal, ensure that the LUISDevOps-master app is open.
-
Click the Test button on the taskbar.
-
Enter a test utterance, for example: great now i can do devops with my luis apps. Press Enter.
-
Click the Inspect link to view the response from the authoring endpoint.
-
Click Compare with published to send the request to the published endpoint. If you see the message Please publish your model first, click Additional Settings underneath the Published heading and select Production in the Publish slot dropdown.
-
You will see the prediction response returned from the LUIS service for the Production slot for this app. You can click Show JSON view if you want to view the JSON response from the service:
-
Try other utterances, such as: i want my vacation to start on july 4th and last for 10 days.
For the next steps, find out how to adapt this repository to your own project:
- Next: Adapting this repository to your own project.
See the following documents for more information on this template and the engineering practices it demonstrates: