feat: add update assistant for storage migrations (#690)

Signed-off-by: Timo Glastra <timo@animo.id>

demo/src/Faber.ts

@@ -1,5 +1,5 @@
 import type { ConnectionRecord } from '@aries-framework/core'
-import type { CredDef, Schema } from 'indy-sdk-react-native'
+import type { CredDef, Schema } from 'indy-sdk'
 import type BottomBar from 'inquirer/lib/ui/bottom-bar'
 
 import { AttributeFilter, CredentialPreview, ProofAttributeInfo, utils } from '@aries-framework/core'

docs/migration/0.1-to-0.2.md

@@ -0,0 +1,75 @@
+# Migrating from AFJ 0.1.0 to 0.2.x
+
+## Breaking Code Changes
+
+> TODO
+
+## Breaking Storage Changes
+
+The 0.2.0 release is heavy on breaking changes to the storage format. This is not what we intend to do with every release. But as there's not that many people yet using the framework in production, and there were a lot of changes needed to keep the API straightforward, we decided to bundle a lot of breaking changes in this one release.
+
+Below all breaking storage changes are explained in as much detail as possible. The update assistant provides all tools to migrate without a hassle, but it is important to know what has changed.
+
+See [Updating](./updating.md) for a guide on how to use the update assistant.
+
+The following config can be provided to the update assistant to migrate from 0.1.0 to 0.2.0:
+
+```json
+{
+  "v0_1ToV0_2": {
+    "mediationRoleUpdateStrategy": "<mediationRoleUpdateStrategy>"
+  }
+}
+```
+
+### Credential Metadata
+
+The credential record had a custom `metadata` property in pre-0.1.0 storage that contained the `requestMetadata`, `schemaId` and `credentialDefinition` properties. Later a generic metadata API was added that only allows objects to be stored. Therefore the properties were moved into a different structure.
+
+The following pre-0.1.0 structure:
+
+```json
+{
+  "requestMetadata": <value of requestMetadata>,
+  "schemaId": "<value of schemaId>",
+  "credentialDefinitionId": "<value of credential definition id>"
+}
+```
+
+Will be transformed into the following 0.2.0 structure:
+
+```json
+{
+  "_internal/indyRequest": <value of requestMetadata>,
+  "_internal/indyCredential": {
+    "schemaId": "<value of schemaId>",
+    "credentialDefinitionId": "<value of credential definition id>"
+  }
+}
+```
+
+Accessing the `credentialDefinitionId` and `schemaId` properties will now be done by retrieving the `CredentialMetadataKeys.IndyCredential` metadata key.
+
+```ts
+const indyCredential = credentialRecord.metadata.get(CredentialMetadataKeys.IndyCredential)
+
+// both properties are optional
+indyCredential?.credentialDefinitionId
+indyCredential?.schemaId
+```
+
+### Mediation Record Role
+
+The role in the mediation record was always being set to `MediationRole.Mediator` for both mediators and recipients. This didn't cause any issues, but would return the wrong role for recipients.
+
+In 0.2 a check is added to make sure the role of a mediation record matches with actions (e.g. a recipient can't grant mediation), which means it will throw an error if the role is not set correctly.
+
+Because it's not always possible detect whether the role should actually be mediator or recipient, a number of configuration options are provided on how the role should be updated using the `v0_1ToV0_2.mediationRoleUpdateStrategy` option:
+
+- `allMediator`: The role is set to `MediationRole.Mediator` for both mediators and recipients
+- `allRecipient`: The role is set to `MediationRole.Recipient` for both mediators and recipients
+- `recipientIfEndpoint` (**default**): The role is set to `MediationRole.Recipient` if their is an `endpoint` configured on the record. The endpoint is not set when running as a mediator. There is one case where this could be problematic when the role should be recipient, if the mediation grant hasn't actually occurred (meaning the endpoint is not set). This is probably the best approach
+  otherwise it is set to `MediationRole.Mediator`
+- `doNotChange`: The role is not changed
+
+Most agents only act as either the role of mediator or recipient, in which case the `allMediator` or `allRecipient` configuration is the most appropriate. If your agent acts as both a recipient and mediator, the `recipientIfEndpoint` configuration is the most appropriate. The `doNotChange` options is not recommended and can lead to errors if the role is not set correctly.

docs/migration/updating.md

@@ -0,0 +1,121 @@
+# Updating
+
+- [Update Strategies](#update-strategies)
+- [Backups](#backups)
+
+## Update Strategies
+
+There are three options on how to leverage the update assistant on agent startup:
+
+1. Manually instantiating the update assistant on agent startup
+2. Storing the agent storage version outside of the agent storage
+3. Automatically update on agent startup
+
+### Manually instantiating the update assistant on agent startup
+
+When the version of the storage is stored inside the agent storage, it means we need to check if the agent needs to be updated on every agent startup. We'll initialize the update assistant and check whether the storage is up to date. The advantage of this approach is that you don't have to store anything yourself, and have full control over the workflow.
+
+```ts
+import { UpdateAssistant, Agent } from '@aries-framework/core'
+
+// or @aries-framework/node
+import { agentDependencies } from '@aries-framework/react-native'
+
+// First create the agent
+const agent = new Agent(config, agentDependencies)
+
+// Then initialize the update assistant with the update config
+const updateAssistant = new UpdateAssistant(agent, {
+  v0_1ToV0_2: {
+    mediationRoleUpdateStrategy: 'allMediator',
+  },
+})
+
+// Initialize the update assistant so we can read the current storage version
+// from the wallet. If you manually initialize the wallet you should do this _before_
+// calling initialize on the update assistant
+// await agent.wallet.initialize(walletConfig)
+await updateAssistant.initialize()
+
+// Check if the agent is up to date, if not call update
+if (!(await updateAssistant.isUpToDate())) {
+  await updateAssistant.update()
+}
+
+// Once finished initialize the agent. You should do this on every launch of the agent
+await agent.initialize()
+```
+
+### Storing the agent storage version outside of the agent storage
+
+When the version of the storage is stored outside of the agent storage, you don't have to initialize the `UpdateAssistant` on every agent agent startup. You can just check if the storage version is up to date and instantiate the `UpdateAssistant` if not. The advantage of this approach is that you don't have to instantiate the `UpdateAssistant` on every agent startup, but this does mean that you have to store the storage version yourself.
+
+When a wallet has been exported and later imported you don't always have the latest version available. If this is the case you can always rely on Method 1 or 2 for updating the wallet, and storing the version yourself afterwards. You can also get the current version by calling `await updateAssistant.getCurrentAgentStorageVersion()`. Do note the `UpdateAssistant` needs to be initialized before calling this method.
+
+```ts
+import { UpdateAssistant, Agent } from '@aries-framework/core'
+
+// or @aries-framework/node
+import { agentDependencies } from '@aries-framework/react-native'
+
+// The storage version will normally be stored in e.g. persistent storage on a mobile device
+let currentStorageVersion: VersionString = '0.1'
+
+// First create the agent
+const agent = new Agent(config, agentDependencies)
+
+// We only initialize the update assistant if our stored version is not equal
+// to the frameworkStorageVersion of the UpdateAssistant. The advantage of this
+// is that we don't have to initialize the UpdateAssistant to retrieve the current
+// storage version.
+if (currentStorageVersion !== UpdateAssistant.frameworkStorageVersion) {
+  const updateAssistant = new UpdateAssistant(agent, {
+    v0_1ToV0_2: {
+      mediationRoleUpdateStrategy: 'recipientIfEndpoint',
+    },
+  })
+
+  // Same as with the previous strategy, if you normally call agent.wallet.initialize() manually
+  // you need to call this before calling updateAssistant.initialize()
+  await updateAssistant.initialize()
+
+  await updateAssistant.update()
+
+  // Store the version so we can leverage it during the next agent startup and don't have
+  // to initialize the update assistant again until a new version is released
+  currentStorageVersion = UpdateAssistant.frameworkStorageVersion
+}
+
+// Once finished initialize the agent. You should do this on every launch of the agent
+await agent.initialize()
+```
+
+### Automatically update on agent startup
+
+This is by far the easiest way to update the agent, but has the least amount of flexibility and is not configurable. This means you will have to use the default update options to update the agent storage. You can find the default update config in the respective version migration guides (e.g. in [0.1-to-0.2](/docs/migration/0.1-to-0.2.md)).
+
+```ts
+import { UpdateAssistant, Agent } from '@aries-framework/core'
+
+// or @aries-framework/node
+import { agentDependencies } from '@aries-framework/react-native'
+
+// First create the agent, setting the autoUpdateStorageOnStartup option to true
+const agent = new Agent({ ...config, autoUpdateStorageOnStartup: true }, agentDependencies)
+
+// Then we call initialize, which under the hood will call the update assistant if the storage is not update to date.
+await agent.initialize()
+```
+
+## Backups
+
+Before starting the update, the update assistant will automatically create a backup of the wallet. If the migration succeeds the backup won't be used. If the backup fails, another backup will be made of the migrated wallet, after which the backup will be restored.
+
+The backups can be found at the following locations. The `backupIdentifier` is generated at the start of the update process and generated based on the current timestamp.
+
+- Backup path: `${agent.config.fileSystem.basePath}/afj/migration/backup/${backupIdentifier}`
+- Migration backup: `${agent.config.fileSystem.basePath}/afj/migration/backup/${backupIdentifier}-error`
+
+> In the future the backup assistant will make a number of improvements to the recovery process. Namely:
+>
+> - Do not throw an error if the update fails, but rather return an object that contains the status, and include the backup paths and backup identifiers.

package.json

@@ -53,6 +53,7 @@
     "ts-jest": "^27.0.3",
     "ts-node": "^10.0.0",
     "tsconfig-paths": "^3.9.0",
+    "tsyringe": "^4.6.0",
     "typescript": "~4.3.0",
     "ws": "^7.4.6"
   },

packages/core/src/agent/Agent.ts

@@ -23,8 +23,11 @@ import { LedgerModule } from '../modules/ledger/LedgerModule'
 import { ProofsModule } from '../modules/proofs/ProofsModule'
 import { MediatorModule } from '../modules/routing/MediatorModule'
 import { RecipientModule } from '../modules/routing/RecipientModule'
+import { StorageUpdateService } from '../storage'
 import { InMemoryMessageRepository } from '../storage/InMemoryMessageRepository'
 import { IndyStorageService } from '../storage/IndyStorageService'
+import { UpdateAssistant } from '../storage/migration/UpdateAssistant'
+import { DEFAULT_UPDATE_CONFIG } from '../storage/migration/updates'
 import { IndyWallet } from '../wallet/IndyWallet'
 import { WalletModule } from '../wallet/WalletModule'
 import { WalletError } from '../wallet/error'
@@ -59,9 +62,13 @@ export class Agent {
   public readonly dids: DidsModule
   public readonly wallet: WalletModule
 
-  public constructor(initialConfig: InitConfig, dependencies: AgentDependencies) {
-    // Create child container so we don't interfere with anything outside of this agent
-    this.container = baseContainer.createChildContainer()
+  public constructor(
+    initialConfig: InitConfig,
+    dependencies: AgentDependencies,
+    injectionContainer?: DependencyContainer
+  ) {
+    // Take input container or child container so we don't interfere with anything outside of this agent
+    this.container = injectionContainer ?? baseContainer.createChildContainer()
 
     this.agentConfig = new AgentConfig(initialConfig, dependencies)
     this.logger = this.agentConfig.logger
@@ -70,10 +77,18 @@ export class Agent {
     this.container.registerInstance(AgentConfig, this.agentConfig)
 
     // Based on interfaces. Need to register which class to use
-    this.container.registerInstance(InjectionSymbols.Logger, this.logger)
-    this.container.register(InjectionSymbols.Wallet, { useToken: IndyWallet })
-    this.container.registerSingleton(InjectionSymbols.StorageService, IndyStorageService)
-    this.container.registerSingleton(InjectionSymbols.MessageRepository, InMemoryMessageRepository)
+    if (!this.container.isRegistered(InjectionSymbols.Wallet)) {
+      this.container.register(InjectionSymbols.Wallet, { useToken: IndyWallet })
+    }
+    if (!this.container.isRegistered(InjectionSymbols.Logger)) {
+      this.container.registerInstance(InjectionSymbols.Logger, this.logger)
+    }
+    if (!this.container.isRegistered(InjectionSymbols.StorageService)) {
+      this.container.registerSingleton(InjectionSymbols.StorageService, IndyStorageService)
+    }
+    if (!this.container.isRegistered(InjectionSymbols.MessageRepository)) {
+      this.container.registerSingleton(InjectionSymbols.MessageRepository, InMemoryMessageRepository)
+    }
 
     this.logger.info('Creating agent with config', {
       ...initialConfig,
@@ -162,6 +177,29 @@ export class Agent {
       )
     }
 
+    // Make sure the storage is up to date
+    const storageUpdateService = this.container.resolve(StorageUpdateService)
+    const isStorageUpToDate = await storageUpdateService.isUpToDate()
+    this.logger.info(`Agent storage is ${isStorageUpToDate ? '' : 'not '}up to date.`)
+
+    if (!isStorageUpToDate && this.agentConfig.autoUpdateStorageOnStartup) {
+      const updateAssistant = new UpdateAssistant(this, DEFAULT_UPDATE_CONFIG)
+
+      await updateAssistant.initialize()
+      await updateAssistant.update()
+    } else if (!isStorageUpToDate) {
+      const currentVersion = await storageUpdateService.getCurrentStorageVersion()
+      // Close wallet to prevent un-initialized agent with initialized wallet
+      await this.wallet.close()
+      throw new AriesFrameworkError(
+        // TODO: add link to where documentation on how to update can be found.
+        `Current agent storage is not up to date. ` +
+          `To prevent the framework state from getting corrupted the agent initialization is aborted. ` +
+          `Make sure to update the agent storage (currently at ${currentVersion}) to the latest version (${UpdateAssistant.frameworkStorageVersion}). ` +
+          `You can also downgrade your version of Aries Framework JavaScript.`
+      )
+    }
+
     if (publicDidSeed) {
       // If an agent has publicDid it will be used as routing key.
       await this.walletService.initPublicDid({ seed: publicDidSeed })
@@ -174,10 +212,13 @@ export class Agent {
       })
     }
 
-    // Start transports
-    const allTransports = [...this.inboundTransports, ...this.outboundTransports]
-    const transportPromises = allTransports.map((transport) => transport.start(this))
-    await Promise.all(transportPromises)
+    for (const transport of this.inboundTransports) {
+      await transport.start(this)
+    }
+
+    for (const transport of this.outboundTransports) {
+      await transport.start(this)
+    }
 
     // Connect to mediator through provided invitation if provided in config
     // Also requests mediation ans sets as default mediator

packages/core/src/agent/AgentConfig.ts

@@ -112,4 +112,8 @@ export class AgentConfig {
   public get connectionImageUrl() {
     return this.initConfig.connectionImageUrl
   }
+
+  public get autoUpdateStorageOnStartup() {
+    return this.initConfig.autoUpdateStorageOnStartup ?? false
+  }
 }

packages/core/src/modules/credentials/repository/CredentialRecord.ts

@@ -10,11 +10,11 @@ import { AriesFrameworkError } from '../../../error'
 import { BaseRecord } from '../../../storage/BaseRecord'
 import { uuid } from '../../../utils/uuid'
 import {
-  CredentialPreviewAttribute,
-  IssueCredentialMessage,
   OfferCredentialMessage,
+  IssueCredentialMessage,
   ProposeCredentialMessage,
   RequestCredentialMessage,
+  CredentialPreviewAttribute,
 } from '../messages'
 import { CredentialInfo } from '../models/CredentialInfo'
 

packages/core/src/storage/BaseRecord.ts

@@ -1,4 +1,4 @@
-import { Exclude, Type } from 'class-transformer'
+import { Exclude, Transform, TransformationType } from 'class-transformer'
 
 import { JsonTransformer } from '../utils/JsonTransformer'
 import { MetadataTransformer } from '../utils/transformers'
@@ -24,10 +24,14 @@ export abstract class BaseRecord<
 
   public id!: string
 
-  @Type(() => Date)
+  @Transform(({ value, type }) =>
+    type === TransformationType.CLASS_TO_PLAIN ? value.toISOString(value) : new Date(value)
+  )
   public createdAt!: Date
 
-  @Type(() => Date)
+  @Transform(({ value, type }) =>
+    type === TransformationType.CLASS_TO_PLAIN ? value.toISOString(value) : new Date(value)
+  )
   public updatedAt?: Date
 
   @Exclude()

packages/core/src/storage/index.ts

@@ -1 +1,2 @@
 export * from './didcomm'
+export * from './migration'