Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(application-system): using aws-sdkv3 instead of v2 and refactoring #15842

Draft
wants to merge 49 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
bbfa6be
initial setup
HjorturJ Aug 30, 2024
79b2c0e
splitting things up for for dependency injection
HjorturJ Sep 2, 2024
d001f45
removing unused imports
HjorturJ Sep 2, 2024
ca7e5dd
removing attachmentservice from imports in modules, replaced in provi…
HjorturJ Sep 2, 2024
b6b57b6
Changing a bit in the front end for no reason
HjorturJ Sep 2, 2024
169fedc
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 2, 2024
274d562
chore: nx format:write update dirty files
andes-it Sep 2, 2024
30e0684
Adjusting s3 injection
HjorturJ Sep 2, 2024
60030bd
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 2, 2024
7c48047
adjusting custom resolver for dependencies
HjorturJ Sep 3, 2024
c4fa5e5
changing header again
HjorturJ Sep 3, 2024
17baf26
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 3, 2024
af4e146
chore: nx format:write update dirty files
andes-it Sep 3, 2024
6a6d165
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 3, 2024
6ff11cb
Refactoring a whole bunch of s3 and attachment implementations to con…
HjorturJ Sep 3, 2024
4149d73
fixing dependencies
HjorturJ Sep 4, 2024
1ffb5a7
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 4, 2024
e095824
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 4, 2024
06c9e83
chore: nx format:write update dirty files
andes-it Sep 4, 2024
0484c07
Removing unused code
HjorturJ Sep 4, 2024
23e38cb
merge conflict
HjorturJ Sep 4, 2024
aba23de
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 4, 2024
4aeb516
Fixing tests
HjorturJ Sep 4, 2024
41bd0f8
chore: nx format:write update dirty files
andes-it Sep 4, 2024
ff644be
Adding tests and some finishing touches on refactoring
HjorturJ Sep 9, 2024
bcf4864
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 9, 2024
aa6eb97
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 9, 2024
bac9be2
chore: nx format:write update dirty files
andes-it Sep 9, 2024
e43ddda
Reverting front end change to deploy feature branch
HjorturJ Sep 10, 2024
7fbb61b
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 10, 2024
4b1f492
Refactoring even more into the aws service and simplifying
HjorturJ Sep 11, 2024
e93a11c
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 11, 2024
daa76e3
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 11, 2024
624f471
deleting commented out file
HjorturJ Sep 11, 2024
5235604
chore: nx format:write update dirty files
andes-it Sep 11, 2024
29346d4
Fixing some error logging and variable name readability
HjorturJ Sep 12, 2024
8a77e89
merge conflict fixed
HjorturJ Sep 12, 2024
1e87112
reverted changes to a file that will be deleted on main anyway
HjorturJ Sep 12, 2024
6bc5086
resolving merge conflicts on a deleted file
HjorturJ Sep 12, 2024
6aaaaa6
chore: nx format:write update dirty files
andes-it Sep 12, 2024
ef28472
Fixing imports, adding and fixing tests
HjorturJ Sep 12, 2024
06c3717
Merge branch 'main' of github.com:island-is/island.is into fix/attach…
HjorturJ Sep 12, 2024
c6b20f0
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 12, 2024
80313ef
chore: nx format:write update dirty files
andes-it Sep 12, 2024
6ab5a91
Removing random crap input from test i was testing hehe
HjorturJ Sep 13, 2024
0f81840
merge conflict
HjorturJ Sep 13, 2024
30a68c5
upgrading credential provider to be the same version as our s3 client…
HjorturJ Sep 13, 2024
dd2894c
upgrading aes sdk sqs too
HjorturJ Sep 13, 2024
794fc87
fixing tests after upgrading packages
HjorturJ Sep 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ export const Header: FC<React.PropsWithChildren<unknown>> = () => {
}, [location])

return (
<Box background="white">
<Box background="red600">
<GridContainer>
<UIHeader
info={
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
import { getValueViaPath } from '@island.is/application/core'
import { ApplicationWithAttachments as Application } from '@island.is/application/types'
import { S3 } from 'aws-sdk'
import AmazonS3URI from 'amazon-s3-uri'
import { ApplicationWithAttachments } from '@island.is/application/types'
import { logger } from '@island.is/logging'
import { Injectable } from '@nestjs/common'
import { Inject, Injectable } from '@nestjs/common'
import { S3Service } from './s3.service'
import { ConfigService } from '@nestjs/config'
import {
BaseTemplateApiApplicationService,
BaseTemplateAPIModuleConfig,
} from '../../../types'

export interface AttachmentData {
key: string
Expand All @@ -14,13 +18,16 @@ export interface AttachmentData {

@Injectable()
export class AttachmentS3Service {
private readonly s3: AWS.S3
constructor() {
this.s3 = new S3()
}
constructor(
@Inject(S3Service) private s3Service: S3Service,
@Inject(ConfigService)
private readonly configService: ConfigService<BaseTemplateAPIModuleConfig>,
@Inject(BaseTemplateApiApplicationService)
private readonly applicationService: BaseTemplateApiApplicationService,
) {}

public async getFiles(
application: Application,
application: ApplicationWithAttachments,
attachmentAnswerKeys: string[],
): Promise<AttachmentData[]> {
const attachments: AttachmentData[] = []
Expand All @@ -43,7 +50,7 @@ export class AttachmentS3Service {
name: string
}>,
answerKey: string,
application: Application,
application: ApplicationWithAttachments,
): Promise<AttachmentData[]> {
return await Promise.all(
answers.map(async ({ key, name }) => {
Expand All @@ -58,30 +65,73 @@ export class AttachmentS3Service {
return { key: '', fileContent: '', answerKey, fileName: '' }
}
const fileContent =
(await this.getApplicationFilecontentAsBase64(url)) ?? ''
(await this.s3Service.getFileContentAsBase64(url)) ?? ''

return { key, fileContent, answerKey, fileName: name }
}),
)
}

private async getApplicationFilecontentAsBase64(
fileName: string,
): Promise<string | undefined> {
const { bucket, key } = AmazonS3URI(fileName)
const uploadBucket = bucket
try {
const file = await this.s3
.getObject({
Bucket: uploadBucket,
Key: key,
})
.promise()
const fileContent = file.Body as Buffer
return fileContent?.toString('base64')
} catch (error) {
logger.error(error)
return undefined
public async getAttachmentContentAsBase64(
application: ApplicationWithAttachments,
attachmentKey: string,
): Promise<string> {
const fileContent =
await this.s3Service.getFileFromApplicationWithAttachments(
application,
attachmentKey,
)
return fileContent?.Body?.transformToString('base64') || ''
}

public async getAttachmentContentAsBlob(
application: ApplicationWithAttachments,
attachmentKey: string,
): Promise<Blob> {
const file = await this.s3Service.getFileFromApplicationWithAttachments(
application,
attachmentKey,
)
const fileArrayBuffer = await file?.Body?.transformToByteArray()
return new Blob([fileArrayBuffer as ArrayBuffer], {
type: file?.ContentType,
})
}

public async getAttachmentUrl(
key: string,
expiration: number,
): Promise<string> {
if (expiration <= 0) {
return Promise.reject('expiration must be positive')
}

const bucket = this.configService.get('attachmentBucket') as
| string
| undefined

if (bucket == undefined) {
return Promise.reject('could not find s3 bucket')
}

return this.s3Service.getSignedUrlPromise(bucket, key, expiration)
}

public async addAttachment(
application: ApplicationWithAttachments,
fileName: string,
buffer: Buffer,
uploadParameters?: {
ContentType?: string
ContentDisposition?: string
ContentEncoding?: string
},
): Promise<string> {
return this.applicationService.saveAttachmentToApplicaton(
application,
fileName,
buffer,
uploadParameters,
)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import { Test, TestingModule } from '@nestjs/testing'
import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3'
import { mockClient } from 'aws-sdk-client-mock'
import { Readable } from 'stream'
import { sdkStreamMixin } from '@smithy/util-stream'
import { logger, LOGGER_PROVIDER } from '@island.is/logging'
import { S3Service } from './s3.service'

const s3Mock = mockClient(S3Client)

describe('S3Service', () => {
let s3Service: S3Service

beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
S3Service,
{
provide: LOGGER_PROVIDER,
useClass: jest.fn(() => ({
error: () => ({}),
})),
},
{
provide: S3Client,
useValue: s3Mock,
},
],
}).compile()

s3Service = module.get<S3Service>(S3Service)

s3Mock.reset()
})

it('should return a file from a bucket if one exists', async () => {
const expectedResult = 'hello world'

const stream = new Readable()
stream.push(expectedResult)
stream.push(null) // end of stream
const sdkStream = sdkStreamMixin(stream)

s3Mock.on(GetObjectCommand).resolvesOnce({ Body: sdkStream })

const getObjectResult = await s3Service.getFileFromBucket('bucekt', 'key')
if (!getObjectResult) fail('getObjectResult was undefined')

const contentResults = await getObjectResult.Body?.transformToString()
if (!contentResults) fail('raw transform result was undefined or empty')

expect(contentResults).toStrictEqual(expectedResult)
})

it('should return undefined from a bucket when an error is thrown', async () => {
s3Mock.on(GetObjectCommand).rejectsOnce()

const getObjectResult = await s3Service.getFileFromBucket('bucekt', 'key')

expect(getObjectResult).toBeUndefined()
})
})
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
import {
GetObjectCommand,
GetObjectCommandOutput,
S3Client,
} from '@aws-sdk/client-s3'
import AmazonS3URI from 'amazon-s3-uri'
import { Inject, Injectable } from '@nestjs/common'
import { ApplicationWithAttachments } from '@island.is/application/types'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'
import { LOGGER_PROVIDER, type Logger } from '@island.is/logging'

@Injectable()
export class S3Service {
constructor(
@Inject(S3Client) private s3Client: S3Client,
@Inject(LOGGER_PROVIDER) protected readonly logger: Logger,
) {}

public async getSignedUrlPromise(
bucket: string,
key: string,
expiration: number,
): Promise<string> {
try {
const command = new GetObjectCommand({ Bucket: bucket, Key: key })
const url = await getSignedUrl(this.s3Client, command, {
expiresIn: expiration,
})
return url ?? ''
} catch (error) {
this.logger.error('Error occurred while fetching file from S3')
this.logger.error(error)
return ''
}
}

public async getFileFromBucket(
bucket: string,
key: string,
): Promise<GetObjectCommandOutput | undefined> {
try {
const result = await this.s3Client.send(
new GetObjectCommand({
Bucket: bucket,
Key: key,
}),
)

return result
} catch (error) {
this.logger.error('Error occurred while fetching file from S3')
this.logger.error(error)
return undefined
}
}

public async getFileFromApplicationWithAttachments(
application: ApplicationWithAttachments,
attachmentKey: string,
): Promise<GetObjectCommandOutput | undefined> {
const fileName = (
application.attachments as {
[key: string]: string
}
)[attachmentKey]

const { bucket, key } = AmazonS3URI(fileName)
const uploadBucket = bucket
try {
const result = await this.s3Client.send(
new GetObjectCommand({
Bucket: uploadBucket,
Key: key,
}),
)

return result
} catch (error) {
this.logger.error('Error occurred while fetching file from S3')
this.logger.error(error)
return undefined
}
}

public async getFileContentAsBase64(
fileName: string,
): Promise<string | undefined> {
const { bucket, key } = AmazonS3URI(fileName)
const uploadBucket = bucket
try {
const { Body } = await this.s3Client.send(
new GetObjectCommand({
Bucket: uploadBucket,
Key: key,
}),
)

return await Body?.transformToString('base64')
} catch (error) {
this.logger.error('Error occurred while fetching file from S3')
this.logger.error(error)
return undefined
}
}

public async getFileContentAsBase64FromBucket(
bucket: string,
key: string,
): Promise<string | undefined> {
try {
const { Body } = await this.s3Client.send(
new GetObjectCommand({
Bucket: bucket,
Key: key,
}),
)

return await Body?.transformToString('base64')
} catch (error) {
this.logger.error('Error occurred while fetching file from S3')
this.logger.error(error)
return undefined
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import { SharedTemplateApiService } from './shared.service'
import { SmsModule } from '@island.is/nova-sms'
import { PaymentModule } from '@island.is/application/api/payment'
import { AttachmentS3Service } from './services'
import { S3Service } from './services/s3.service'
import { S3Client } from '@aws-sdk/client-s3'

export class SharedTemplateAPIModule {
static register(config: BaseTemplateAPIModuleConfig): DynamicModule {
Expand All @@ -34,9 +36,14 @@ export class SharedTemplateAPIModule {
provide: BaseTemplateApiApplicationService,
useClass: config.applicationService,
},
{
provide: S3Client,
useValue: new S3Client(),
},
S3Service,
AttachmentS3Service,
],
exports: [SharedTemplateApiService, AttachmentS3Service],
exports: [SharedTemplateApiService, S3Service, AttachmentS3Service],
}
}
}
Loading
Loading