Please follow the steps below to set up the Azure environment
Clone or download this repository and navigate to the project's root directory.
Before you can upload assests to the Synapse Workspace you will need to add your IP address:
- Go to the Synapse resouce you created in the previous step.
- Navigate to
Networking
underSecurity
on the left hand side of the page. - At the top of the screen click
+ Add client IP
- Your IP address should now be visible in the IP list and click
Save
at the top
In order to perform the necessary actions in Synapse workspace, you will need to grant more access.
- Go to the Azure Data Lake Storage Account for your Synapse Workspace
- Note: you will see two storage accounts created in the resource group, be sure to select the Synapse Workspace Storage account and not the machine learning storage account
- Go to the
Access Control (IAM) > + Add > Add role assignment
- Now search and select the
Storage Blob Data Contributor
role and click "Next" - Click "+ Select members", search and select your username and click "Select"
- Click
Review and assign
at the bottom
- Launch the Synapse workspace Synapse Workspace
- Select the
subscription
andworkspace
name you are using for this solution accelerator - In Synapse Studio, navigate to the
Data
Hub - Select
Linked
- Under the category
Azure Data Lake Storage Gen2
you'll see an item with a name likexxxxx(xxxxx- Primary)
- Select the container named
Data (Primary)
, select "New Folder" entersourcedata
and select "Create" - Select the
sourcedata
folder, selectUpload
and select following sample data files downloaded from Data folderresidents_source1.csv
residents_source2.csv
leases.csv
payments.csv
surveys.csv
workorders.csv
properties.csv
- Launch the Synapse workspace Synapse Workspace
- Go to
Develop
, click the+
, and clickImport
to select all notebooks from this repository's folder - For each of the notebooks, select
Attach to > spark1
in the top dropdown - Configure the parameters in the following 6 notebooks and publish the changes
- Note: Only change the parameters but do not run all the notebooks. You will run pipelines that will run notebooks 1-5 in later steps after the Customer Insights set up.
-
00_prepare_sourcedata_for_ci.ipynb
data_lake_account_name = '' # Synapse ADLS file_system_name = 'data' synapse_workspace_name = ''
-
1_preparedata_from_ci.ipynb
data_lake_account_name = '' # Synapse ADLS file_system_name = 'data' resident_file_name = "residents.csv"
-
2_train_model.ipynb
data_lake_account_name = "" # Synapse ADLS file_system_name = "data" table_name = "c360_data.prepareddata" #AML workspace details subscription_id = "" resource_group = "" workspace_name = ""
-
3_preparedata_for_inference.ipynb
data_lake_account_name = '' # Synapse ADLS file_system_name = 'data' resident_file_name = "residents.csv"
-
4_run_batch_inference.ipynb
data_lake_account_name = '' # Synapse ADLS file_system_name = 'data' table_name = "c360_data.preparedinferencedata" #AML workspace details subscription_id = "" resource_group = "" workspace_name = ""
-
5_prepare_predictionsdata_for_ci.ipynb
data_lake_account_name = '' # Synapse ADLS file_system_name = 'data' synapse_workspace_name = ''
- Run the following notebook
00_prepare_sourcedata_for_ci.ipynb
In order to perform the necessary actions in Customer Insights, you will need to grant more access.
- Go to the Azure Data Lake Storage Account for your Synapse Workspace
- Go to the
Access Control (IAM) > + Add > Add role assignment
- Now search and select the
Storage Blob Data Contributor
role and click "Next" - Click "+ Select members", search and select
Dynamics 365 AI for Customer Insights
and click "Select" - Click
Review and assign
at the bottom
- Note: if you cannot find the
Dynamics 365 AI for Customer Insights
service principle, Follow the steps here to grant theDynamics 365 AI for Customer Insights
access to the ADLS Gen 2 storage account you are using for this solution.
- Navigate to the CI Set Up Documentation and follow the steps to set up the Customer Insights Environment.
- Launch the Azure Machine Learning studio AML Studio
- Go to the
Notebooks
tab in the AML Studio and upload theAML Notebooks
folder - Go to the
Compute
tab in the AML Studio and click on theCompute Instances
- Click
New
, selectStandard_DS11_v2
for Virtual Machine Size and click onCreate
to create a new compute instance - Once the state of compute instance changes from
Creating
toRunning
, click onJupyter
under Applications to launch the compute instance - Run the following notebooks in order:
CreateTrainingPipeline.ipynb
CreateInferencingPipeline.ipynb
(This step publishes a pipeline that will be used in Customer Insights set up)
In order to run the pipelines, you will need to grant more access.
- In the Azure Portal, select the Azure Machine Learning Service created above
- Go to the
Access Control (IAM) > + Add > Add role assignment
- Now click the Role dropdown and select
Contributor
- Search for your Synapse Workspace and add
- Click
Save
at the bottom
In this step you will create the Training Pipeline
- Note: You will need to manually connect each object in the pipeline.
- Launch the Synapse workspace Synapse Workspace
- Go to
Integration
, click+
and clickPipeline
- Under Properties provide the pipeline with the following name
TrainingPipeline
- Click the "{}" button at the top right corner to open the Code window
- Copy and paste the contents of TrainingPipeline.json
- Click OK to apply
- Click "Publish All" > "Publish"
- Trigger the pipeline
In this step you will create the Inferencing Pipeline
- Note: You will need to manually connect each object in the pipeline.
- Launch the Synapse workspace Synapse Workspace
- Go to
Integration
, click+
and clickPipeline
- Under Properties provide the pipeline with the following name
InferencingPipeline
- Click the "{}" button at the top right corner to open the Code window
- Copy and paste the contents of InferencingPipeline.json
- Click OK to apply
- Click "Publish All" > "Publish"
- Trigger the pipeline
- Launch the Synapse workspace Synapse Workspace
- Configure the parameters and Run the notebooks in the following order
5_prepare_predictionsdata_for_ci.ipynb
- Navigate to Step 4 in the CI Set Up Documentation