Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for postgres wallet type #699

Merged
merged 16 commits into from May 10, 2022
Merged

feat: add support for postgres wallet type #699

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
004a7ed
feat: add postgres plugin and required methods
sairanjit Apr 7, 2022
2ed6194
refactor: add extension for libindystrgpostgres file based on system
sairanjit Apr 11, 2022
085f966
Merge branch 'hyperledger:main' into feat/add-postgres-support
May 2, 2022
3907d17
fix: test helpers for postgres plugin
sairanjit May 2, 2022
58cab5a
Merge branch 'main' into feat/add-postgres-support
May 2, 2022
d4586d9
refactor: create a separate file for postgres tests
sairanjit May 2, 2022
d5c5ccf
refactor: resolve pr comments
sairanjit May 2, 2022
fc3c7ab
Merge branch 'main' into feat/add-postgres-support
May 2, 2022
a29b72a
Merge branch 'feat/add-postgres-support' of https://github.com/sairan…
sairanjit May 2, 2022
c7122c3
refactor: move postgres related types into node package
sairanjit May 2, 2022
e1749ca
fix: refactor import
sairanjit May 2, 2022
b33231c
Merge branch 'main' into feat/add-postgres-support
May 2, 2022
f81d5bd
fix: removed type
sairanjit May 2, 2022
e41be46
Merge branch 'feat/add-postgres-support' of https://github.com/sairan…
sairanjit May 2, 2022
dfa7487
refactor: make function synchronous
sairanjit May 2, 2022
d8077f3
Merge branch 'main' into feat/add-postgres-support
TimoGlastra May 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ RUN apt-get update -y && apt-get install -y \
apt-transport-https \
curl \
# Only needed to build indy-sdk
build-essential
build-essential \
git \
libzmq3-dev libsodium-dev pkg-config libssl-dev

# libindy
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE7709D068DB5E88
Expand All @@ -28,6 +30,19 @@ RUN apt-get update -y && apt-get install -y --allow-unauthenticated \
# Install yarn seperately due to `no-install-recommends` to skip nodejs install
RUN apt-get install -y --no-install-recommends yarn

# postgres plugin setup
# install rust and set up rustup
RUN curl https://sh.rustup.rs -sSf | bash -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

# clone indy-sdk and build postgres plugin
RUN git clone https://github.com/hyperledger/indy-sdk.git
WORKDIR /indy-sdk/experimental/plugins/postgres_storage/
RUN cargo build --release

# set up library path for postgres plugin
ENV LD_LIBRARY_PATH="/indy-sdk/experimental/plugins/postgres_storage/target/release"

FROM base as final

# AFJ specifc setup
Expand Down
3 changes: 2 additions & 1 deletion packages/core/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ export { MessageSender } from './agent/MessageSender'
export type { AgentDependencies } from './agent/AgentDependencies'
export type { InitConfig, OutboundPackage, EncryptedMessage } from './types'
export { DidCommMimeType } from './types'
export type { FileSystem } from './storage/FileSystem'
export type { FileSystem, WalletStorageCreds } from './storage/FileSystem'
export { WalletScheme } from './storage/FileSystem'
export { BaseRecord } from './storage/BaseRecord'
export { InMemoryMessageRepository } from './storage/InMemoryMessageRepository'
export { Repository } from './storage/Repository'
Expand Down
26 changes: 26 additions & 0 deletions packages/core/src/storage/FileSystem.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,34 @@
import type { WalletStorageConfig } from 'indy-sdk'

export interface FileSystem {
readonly basePath: string

exists(path: string): Promise<boolean>
write(path: string, data: string): Promise<void>
read(path: string): Promise<string>
downloadToFile(url: string, path: string): Promise<void>
loadPostgresPlugin?(storageConfig: WalletStorageConfig, storageCreds: WalletStorageCreds): Promise<boolean>
}

export enum WalletScheme {
DatabasePerWallet = 'DatabasePerWallet',
MultiWalletSingleTable = 'MultiWalletSingleTable',
MultiWalletSingleTableSharedPool = 'MultiWalletSingleTableSharedPool',
}

export interface StorageConfig {
url: string
wallet_scheme?: WalletScheme
path?: string | undefined
}

export interface WalletStorageCreds {
[key: string]: unknown
}

export interface StorageCreds {
account: string
password: string
admin_account: string
admin_password: string
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The postgres plugin is related to the environment and the wallet type (in this case indy-sdk). I'm not sure the file system is the best place to add this, mainly because the filesystem stays the same if we support something different than indy-sdk.

Maybe we can keep it a bit more generic. Maybe later we can integrate more, but for now maybe can approach it like this:

import { Agent } from '@aries-framework/core'
import { agentDependencies, loadPostgresPlugin } from '@aries-framework/node'

async function run() {
  const storageConfig = {}
  const storageCreds = {}

  await loadPostgresPlugin(storageConfig, storageCreds)

  const agent = new Agent({
    walletConfig: {
      id: 'id',
      key: 'key',
      storageConfig,
      storageCreds,
      storageType: 'postgres'
  })
}

run()

10 changes: 10 additions & 0 deletions packages/core/src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ import type { DidCommService } from './modules/dids/domain/service/DidCommServic
import type { IndyPoolConfig } from './modules/ledger/IndyPool'
import type { AutoAcceptProof } from './modules/proofs'
import type { MediatorPickupStrategy } from './modules/routing'
import type { WalletStorageCreds } from './storage/FileSystem'
import type { WalletStorageConfig } from 'indy-sdk'

export const enum KeyDerivationMethod {
/** default value in indy-sdk. Will be used when no value is provided */
Expand All @@ -20,6 +22,9 @@ export interface WalletConfig {
id: string
key: string
keyDerivationMethod?: KeyDerivationMethod
storageType?: WalletStorageType
storageConfig?: WalletStorageConfig
storageCreds?: WalletStorageCreds
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can chamge the interface of this to be not 100% what indy expects, but more how we would want the user of the framework to provide the values. Mainly with the addition of other wallet types I'm not sure if this approach fits nicely. I was thinking about something like this:

// defined in types.ts
export interface WalletConfig {
  id: string
  key: string
  keyDerivationMethod?: KeyDerivationMethod
  storage?: {
    type: string
    [key: string]: unknown
  }
}

// defined in @aries-framework/node package
interface IndyPostgresStorageConfig {
  type: 'postgres',
  config: {
    // .. parameters needed for postgres plugin
  },
   credentials: {
    // .. parameters needed for postgres plugin
  }
}

It can then be used like this:

import { IndyPostgresStorageConfig } from '@aries-framework/core'

const storageConfig: IndyPostgresStorageConfig = {
  type: 'postgres',
  // other required fields
}

await loadPostgresPlugin()

const agent = new Agent({
  walletConfig: {
     // other required fields
    storage: storageConfig
  }
})

await agent.initialize()

}

export interface WalletConfigRekey {
Expand Down Expand Up @@ -47,6 +52,11 @@ export enum DidCommMimeType {
V1 = 'application/didcomm-envelope-enc',
}

export enum WalletStorageType {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can also be removed now

Default = 'default',
Postgres = 'postgres_storage',
}

export interface InitConfig {
endpoints?: string[]
label: string
Expand Down
69 changes: 54 additions & 15 deletions packages/core/src/wallet/IndyWallet.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import { Lifecycle, scoped } from 'tsyringe'

import { AgentConfig } from '../agent/AgentConfig'
import { AriesFrameworkError } from '../error'
import { WalletStorageType } from '../types'
import { JsonEncoder } from '../utils/JsonEncoder'
import { isIndyError } from '../utils/indyError'

Expand All @@ -33,6 +34,14 @@ export class IndyWallet implements Wallet {
public constructor(agentConfig: AgentConfig) {
this.logger = agentConfig.logger
this.indy = agentConfig.agentDependencies.indy

const { walletConfig, fileSystem } = agentConfig

if (walletConfig?.storageType === WalletStorageType.Postgres) {
if (walletConfig.storageConfig && walletConfig.storageCreds && fileSystem?.loadPostgresPlugin) {
fileSystem?.loadPostgresPlugin(walletConfig.storageConfig, walletConfig.storageCreds)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the wallet storage is set to postgres, I think we should throw an error if the required parameters are not set. I think we should also throw an error if loadPostgresPlugin is not defined and wallet storage type is postgres.

}
}

public get isProvisioned() {
Expand Down Expand Up @@ -67,6 +76,41 @@ export class IndyWallet implements Wallet {
return this.walletConfig.id
}

private walletStorageConfig(walletConfig: WalletConfig): Indy.WalletConfig {
const walletStorageConfig: Indy.WalletConfig = {
id: walletConfig.id,
storage_type: walletConfig.storageType,
}

if (walletConfig.storageConfig) {
walletStorageConfig.storage_config = walletConfig.storageConfig
}

return walletStorageConfig
}

private walletCredentials(
walletConfig: WalletConfig,
rekey?: string,
rekeyDerivation?: KeyDerivationMethod
): Indy.OpenWalletCredentials {
const walletCredentials: Indy.OpenWalletCredentials = {
key: walletConfig.key,
key_derivation_method: walletConfig.keyDerivationMethod,
}
if (rekey) {
walletCredentials.rekey = rekey
}
if (rekeyDerivation) {
walletCredentials.rekey_derivation_method = rekeyDerivation
}
if (walletConfig.storageCreds) {
walletCredentials.storage_credentials = walletConfig.storageCreds
}

return walletCredentials
}

/**
* @throws {WalletDuplicateError} if the wallet already exists
* @throws {WalletError} if another error occurs
Expand All @@ -84,11 +128,7 @@ export class IndyWallet implements Wallet {
this.logger.debug(`Creating wallet '${walletConfig.id}' using SQLite storage`)

try {
await this.indy.createWallet(
{ id: walletConfig.id },
{ key: walletConfig.key, key_derivation_method: walletConfig.keyDerivationMethod }
)

await this.indy.createWallet(this.walletStorageConfig(walletConfig), this.walletCredentials(walletConfig))
this.walletConfig = walletConfig

// We usually want to create master secret only once, therefore, we can to do so when creating a wallet.
Expand Down Expand Up @@ -139,7 +179,11 @@ export class IndyWallet implements Wallet {
throw new WalletError('Wallet rekey undefined!. Please specify the new wallet key')
}
await this._open(
{ id: walletConfig.id, key: walletConfig.key, keyDerivationMethod: walletConfig.keyDerivationMethod },
{
id: walletConfig.id,
key: walletConfig.key,
keyDerivationMethod: walletConfig.keyDerivationMethod,
},
walletConfig.rekey,
walletConfig.rekeyDerivationMethod
)
Expand All @@ -162,13 +206,8 @@ export class IndyWallet implements Wallet {

try {
this.walletHandle = await this.indy.openWallet(
{ id: walletConfig.id },
{
key: walletConfig.key,
rekey: rekey,
key_derivation_method: walletConfig.keyDerivationMethod,
rekey_derivation_method: rekeyDerivation,
}
this.walletStorageConfig(walletConfig),
this.walletCredentials(walletConfig, rekey, rekeyDerivation)
)
if (rekey) {
this.walletConfig = { ...walletConfig, key: rekey, keyDerivationMethod: rekeyDerivation }
Expand Down Expand Up @@ -224,8 +263,8 @@ export class IndyWallet implements Wallet {

try {
await this.indy.deleteWallet(
{ id: this.walletConfig.id },
{ key: this.walletConfig.key, key_derivation_method: this.walletConfig.keyDerivationMethod }
this.walletStorageConfig(this.walletConfig),
this.walletCredentials(this.walletConfig)
)
} catch (error) {
if (isIndyError(error, 'WalletNotFoundError')) {
Expand Down
70 changes: 69 additions & 1 deletion packages/core/tests/agents.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,20 @@ import { SubjectInboundTransport } from '../../../tests/transport/SubjectInbound
import { SubjectOutboundTransport } from '../../../tests/transport/SubjectOutboundTransport'
import { Agent } from '../src/agent/Agent'

import { waitForBasicMessage, getBaseConfig } from './helpers'
import { waitForBasicMessage, getBaseConfig, getBasePostgresConfig } from './helpers'

const aliceConfig = getBaseConfig('Agents Alice', {
endpoints: ['rxjs:alice'],
})
const bobConfig = getBaseConfig('Agents Bob', {
endpoints: ['rxjs:bob'],
})
const alicePostgresConfig = getBasePostgresConfig('AgentsAlice', {
endpoints: ['rxjs:alice'],
})
const bobPostgresConfig = getBasePostgresConfig('AgentsBob', {
endpoints: ['rxjs:bob'],
})

describe('agents', () => {
let aliceAgent: Agent
Expand Down Expand Up @@ -77,3 +83,65 @@ describe('agents', () => {
expect(aliceAgent.isInitialized).toBe(true)
})
})

describe('postgres agents', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can extract this to a separate postgres.test.ts?

let aliceAgent: Agent
let bobAgent: Agent
let aliceConnection: ConnectionRecord
let bobConnection: ConnectionRecord

afterAll(async () => {
await bobAgent.shutdown()
await bobAgent.wallet.delete()
await aliceAgent.shutdown()
await aliceAgent.wallet.delete()
})

test('make a connection between agents', async () => {
const aliceMessages = new Subject<SubjectMessage>()
const bobMessages = new Subject<SubjectMessage>()

const subjectMap = {
'rxjs:alice': aliceMessages,
'rxjs:bob': bobMessages,
}

aliceAgent = new Agent(alicePostgresConfig.config, alicePostgresConfig.agentDependencies)
aliceAgent.registerInboundTransport(new SubjectInboundTransport(aliceMessages))
aliceAgent.registerOutboundTransport(new SubjectOutboundTransport(subjectMap))
await aliceAgent.initialize()

bobAgent = new Agent(bobPostgresConfig.config, bobPostgresConfig.agentDependencies)
bobAgent.registerInboundTransport(new SubjectInboundTransport(bobMessages))
bobAgent.registerOutboundTransport(new SubjectOutboundTransport(subjectMap))
await bobAgent.initialize()

const aliceConnectionAtAliceBob = await aliceAgent.connections.createConnection()
const bobConnectionAtBobAlice = await bobAgent.connections.receiveInvitation(aliceConnectionAtAliceBob.invitation)

aliceConnection = await aliceAgent.connections.returnWhenIsConnected(aliceConnectionAtAliceBob.connectionRecord.id)
bobConnection = await bobAgent.connections.returnWhenIsConnected(bobConnectionAtBobAlice.id)

expect(aliceConnection).toBeConnectedWith(bobConnection)
expect(bobConnection).toBeConnectedWith(aliceConnection)
})

test('send a message to connection', async () => {
const message = 'hello, world'
await aliceAgent.basicMessages.sendMessage(aliceConnection.id, message)

const basicMessage = await waitForBasicMessage(bobAgent, {
content: message,
})

expect(basicMessage.content).toBe(message)
})

test('can shutdown and re-initialize the same agent', async () => {
expect(aliceAgent.isInitialized).toBe(true)
await aliceAgent.shutdown()
expect(aliceAgent.isInitialized).toBe(false)
await aliceAgent.initialize()
expect(aliceAgent.isInitialized).toBe(true)
})
})
33 changes: 33 additions & 0 deletions packages/core/tests/helpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import { SubjectInboundTransport } from '../../../tests/transport/SubjectInbound
import { SubjectOutboundTransport } from '../../../tests/transport/SubjectOutboundTransport'
import { agentDependencies } from '../../node/src'
import {
WalletScheme,
LogLevel,
AgentConfig,
AriesFrameworkError,
Expand All @@ -47,6 +48,7 @@ import {
import { Attachment, AttachmentData } from '../src/decorators/attachment/Attachment'
import { AutoAcceptCredential } from '../src/modules/credentials/CredentialAutoAcceptType'
import { DidCommService } from '../src/modules/dids'
import { WalletStorageType } from '../src/types'
import { LinkedAttachment } from '../src/utils/LinkedAttachment'
import { uuid } from '../src/utils/uuid'

Expand Down Expand Up @@ -82,6 +84,37 @@ export function getBaseConfig(name: string, extraConfig: Partial<InitConfig> = {
return { config, agentDependencies } as const
}

export function getBasePostgresConfig(name: string, extraConfig: Partial<InitConfig> = {}) {
const config: InitConfig = {
label: `Agent: ${name}`,
walletConfig: {
id: `Wallet${name}`,
key: `Key${name}`,
storageConfig: { url: 'localhost:5432', wallet_scheme: WalletScheme.DatabasePerWallet },
storageCreds: {
account: 'postgres',
password: 'postgres',
admin_account: 'postgres',
admin_password: 'postgres',
},
storageType: WalletStorageType.Postgres,
},
publicDidSeed,
autoAcceptConnections: true,
indyLedgers: [
{
id: `pool-${name}`,
isProduction: false,
genesisPath,
},
],
logger: new TestLogger(LogLevel.error, name),
...extraConfig,
}

return { config, agentDependencies } as const
}

export function getAgentConfig(name: string, extraConfig: Partial<InitConfig> = {}) {
const { config, agentDependencies } = getBaseConfig(name, extraConfig)
return new AgentConfig(config, agentDependencies)
Expand Down
6 changes: 5 additions & 1 deletion packages/node/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,16 @@
"dependencies": {
"@aries-framework/core": "0.1.0",
"express": "^4.17.1",
"ffi-napi": "^4.0.3",
"indy-sdk": "^1.16.0-dev-1636",
"node-fetch": "^2.6.1",
"ws": "^7.5.3"
"ref-napi": "^3.0.3",
"ws": "^7.5.3",
"os": "^0.1.2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a node core module, we don't need to add it to the dependencies

Suggested change
"os": "^0.1.2"

},
"devDependencies": {
"@types/express": "^4.17.13",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing @types/ref-napi

"@types/ffi-napi": "^4.0.5",
"@types/node": "^15.14.4",
"@types/node-fetch": "^2.5.10",
"@types/ws": "^7.4.6",
Expand Down
Loading