Consumption workflow - Survey and Observation( with and without rubric) created using KP for Manage Learn Usecase.

[aks30]

Purpose:

Come up with a workflow and solution to utilize observation and surveys created using VDN to be consumed within current and upcoming Manage Learn programs via the current consumption channels enabled for Manage Learn users (Portal and App)

Context:

So far Observations(with and w/o rubric) and Survey are created using Manage Learn APIs exposed for the implementation team(the partners raise tickets for the same and give the necessary data in a CSV). These questions are then consumed using Manage learn player both on the app and portal.

We have during the past few months enabled creation of observation and survey using the creation portal(KP) thereby moving towards having all the questions in one common storage on the platform as well as removing the dependency on the implementation team. We are also working to migrate all the old and current observation and survey questions to KP. Going forward all new question sets will be created in KP only.

This document talks about how these question sets created using KP will be consumed for Manage Learn programs and usecase via the existing portal and app.

Solution:

Once a question set in VDN is identified to be used within a Manage Learn program we will duplicate the same question set in VDN and change the visibility to Private. It will delink the original question set to the one being used in the program so that any change in the original template doesn’t affect an ongoing program and within a program, the meta of it can be modified e.g. name, description, etc.

At present, the consumption side APIs for observation and survey are being used in Manage learn programs get the data from MongoDB. With all the new question sets now being created using KP, we will now switch the current APIs to get data from the questionSet read API transform the same(for backward compatibility) and pass it in API response. The transformed structure will be stored in Redis for faster retrieval for repeated access.

Migration Plan here

Plan for transforming the KP response into MongoDB response format as to not alter consumption API responses:

1. Build a component that converts the KP response to MongoDB structure
2. After transformation, the transformed data will be stored in the Redis with a key as mongoId:do_id.
3. The TTL for the data in Redis will be 1 day.
4. If there are any updates in the question set the data in Redis will be removed.
5. This component will be invoked whenever the question set API is called to get the transformed data.
6. This component will be written in the ml survey service.

Implementation Design:

1) New APIs in Manage Learn to map a questionSet from KP to a program ===> questionsetId, programId

Repo to change: ml-survey-service
Api Endpoint: program/observation/create

API Request:

{	
      "createdFor":["string"],     // channel name  ,mandatory
      "name":"string", // Name of the questionset, optional
      “questionsetId” : “String”,	// do_id of the questionset to be copied ,mandatory
      “programId”:”String” //  (mongo object id) ,/mandatory
}

Success Response:

{
   "id": "api.program.observation.create",
   "ver": "1.0",
   "ts": "2022-06-21T05:44:02ZZ",
   "params": {
       "resmsgid": "db600cbd-d9e2-4f8d-b9c6-93b8ec462baf",
       "msgid": null,
       "err": null,
       "status": "successful",
       "errmsg": null
   },
   "responseCode": "OK",
   "result": {
       "node_id": {
           "do_1135353079207444481964": "do_1135642336673136641573",
“do_1135642336673136641573”: “mongoObjectId”
       },
       "versionKey": "1655790242598"
   }
}

Error Response:

{
   "id": "api.program.observation.create",
   "ver": "3.0",
   "ts": "2022-06-21T07:24:47ZZ",
   "params": {
       "resmsgid": "ccac2c3f-8242-4754-a350-694d210592f2",
       "msgid": null,
       "err": "ERR_INVALID_REQUEST",
       "status": "failed",
       "errmsg": "Please provide valid value for List(createdBy)"
   },
   "responseCode": "CLIENT_ERROR",
   "result": {
       "messages": null
   }
}

Endpoints to use
Copy_Questionset: https://dock.sunbirded.org/api/questionset/v1/copy/{questionsetId to copy}
Update_Questionset: https://dock.sunbirded.org/api/questionset/v1/update
Publish_Questionset: https://dock.sunbirded.org/api/questionset/v1/publish/{copiedQuestionsetdo_id}
Read_Questionset: https://dock.sunbirded.org/api/questionset/v1/hierarchy/{$copiedQuestionsetdo_id}?mode=edit

Implementation:

• If the program doesn’t exist don’t do any further steps.
• Create a copy of the observation using Copy_Questionset
• Read the questionset data and update
• We have to update the visibility as private for parent and children and call the update api Update_Questionset
• Publish the copied questionset Publish_Questionset
• solutionMongoId = Create a mongo observation object with the fields (name, description, migratedId: copied observation do id , createdBy, createdat, updatedBy, updatedat, programId, entityType) in solutions collection
• programId has to be mapped with mongoObjectId
• Read the program collection with the programId
• Add solutionMongoId to the components [] fields
• Update the program with the solution mongo objectId.

2) New Manage learn API to Update any QuestionSet already mapped to a program ===> solutionObjectId (from request params) and updated data (from request body) ===>

Repo to change: ml-survey
Api Endpoint: program/observation/update/:solutionId
Assumption: only name or description will be updated

Request:

Params: solutionId 	//mongo objectId  , mandatory
Body:
{
     “request”: {
          “name”:  ”string” ,	//updated questionset name 		//optional
          “description”: “string”,//updated questionset description	// optional
     }
}

Success Response:

	{
                   "id": "api.program.observation.update",
                   "ver": "1.0",
                   "ts": "2022-06-21T05:44:02ZZ",
                   "params": {
                       "resmsgid": "db600cbd-d9e2-4f8d-b9c6-93b8ec462baf",
                       "msgid": null,
                       "err": null,
                       "status": "successful",
                       "errmsg": null
                   },
                   "responseCode": "OK",
                   "result": {
                                    “msg“: “Solution updated successfully”
                    }
               }

Error Response :

    {
        "id": "api.program.observation.create",
        "ver": "3.0",
        "ts": "2022-06-21T07:24:47ZZ",
        "params": {
            "resmsgid": "ccac2c3f-8242-4754-a350-694d210592f2",
            "msgid": null,
            "err": "ERR_INVALID_REQUEST",
            "status": "failed",
            "errmsg": "Please provide valid value for List(createdBy)"
        },
        "responseCode": "CLIENT_ERROR",
        "result": {
            "messages": null
        }
    }

End points to use:
Update_Questionset: https://dock.sunbirded.org/api/questionset/v1/update
Publish_Questionset: https://dock.sunbirded.org/api/questionset/v1/publish/{copiedQuestionsetdo_id}

Implementation:

1. Read the solutions collection by solutionObjectId
2. Update the questionset by calling Update_Questionset
3. Publish the updated questionset by calling Publish_Questionset
4. If no error
5. Update the fields with requested data in mongo
6. Map the fields to be updated in KP and get the migratedId
7. Else throw Error : unable to update observation: solutionId

Below list of Manage Learn APIs which need to change to read data from Questionset read API or redis :

• observations/assessment
• observationSubmissions/create
• observationSubmissions/rate
• observationSubmissions/multiRate
• observationSubmissions/update
• surveys/details
• solutions/details
• solutions/questions

Cc - @vijiurs, @kiranharidas187 @aishwaryashikshalokam, @priyanka-TL, @VISHNUDAS-tunerlabs, @Shakthieshwari, @Prateek-slokam, @MeghnaSL, @joffinjoy, @githubpiyush

[surendrasinghs]

@maheshkumargangula, @vinukumar-vs and @vrayulu can you please help Akash with the API design proposed above?

[vinukumar-vs]

@aks30 As we discussed, For the below points are you making changes in the existing Knowledge service or building your own middleware to handle this?

1. Build a component that converts the KP response to MongoDB structure
2. After transformation, the transformed data will be stored in the Redis with a key as mongoId:do_id.
3. The TTL for the data in Redis will be 1 day.

In your discussion I was suggesting to use middleware, but in the above approach you have mentioned as component & using the same microservice redis DB to store mogodbID details. Think through & update the approach in the design wiki(design diagram will help to understand better)

Can you update the same in the desing wiki along with the above implementation desing details.
It will be easy to comment in the design wiki for specific details rather than discussing them in this thread.

Also as I mentioned you have to detail are you planning for single migration or phase-wise migration. Based on your approach you have to plan for mobile backward compatibility. Please update these in the design wiki.

Based on your migration plan, On the consumption side you have to enhance the QUML-player to render all other types of questions. Please document this as well.

@vrayulu @maheshkumargangula @surendrasinghs

[aks30]

Hi @vinukumar-vs at present the consumption side use APIs of ml-survey-service for Observations and Survey capability. With this change being planned the current set of APIs will switch from being original responders to becoming a middleware that gets the questions from Knowlg service and passes it on the consumption side in between transforming it to maintain backward compatibility.

We created a Design Wiki for the same here - https://project-sunbird.atlassian.net/l/cp/Fy81CViA. Do check and lest us know your comments.

Also, we will be going with single-phase migration only. For the point of enhancing the current QuML player to support rendering new question types created for observations and surveys, we are tracking that item in the backlog and shall work on the implementation plan for it once we are taking up it to develop as part of a release.