Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(application-system): using aws-sdkv3 instead of v2 and refactoring #15842

Draft
wants to merge 49 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
bbfa6be
initial setup
HjorturJ Aug 30, 2024
79b2c0e
splitting things up for for dependency injection
HjorturJ Sep 2, 2024
d001f45
removing unused imports
HjorturJ Sep 2, 2024
ca7e5dd
removing attachmentservice from imports in modules, replaced in provi…
HjorturJ Sep 2, 2024
b6b57b6
Changing a bit in the front end for no reason
HjorturJ Sep 2, 2024
169fedc
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 2, 2024
274d562
chore: nx format:write update dirty files
andes-it Sep 2, 2024
30e0684
Adjusting s3 injection
HjorturJ Sep 2, 2024
60030bd
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 2, 2024
7c48047
adjusting custom resolver for dependencies
HjorturJ Sep 3, 2024
c4fa5e5
changing header again
HjorturJ Sep 3, 2024
17baf26
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 3, 2024
af4e146
chore: nx format:write update dirty files
andes-it Sep 3, 2024
6a6d165
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 3, 2024
6ff11cb
Refactoring a whole bunch of s3 and attachment implementations to con…
HjorturJ Sep 3, 2024
4149d73
fixing dependencies
HjorturJ Sep 4, 2024
1ffb5a7
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 4, 2024
e095824
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 4, 2024
06c9e83
chore: nx format:write update dirty files
andes-it Sep 4, 2024
0484c07
Removing unused code
HjorturJ Sep 4, 2024
23e38cb
merge conflict
HjorturJ Sep 4, 2024
aba23de
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 4, 2024
4aeb516
Fixing tests
HjorturJ Sep 4, 2024
41bd0f8
chore: nx format:write update dirty files
andes-it Sep 4, 2024
ff644be
Adding tests and some finishing touches on refactoring
HjorturJ Sep 9, 2024
bcf4864
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 9, 2024
aa6eb97
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 9, 2024
bac9be2
chore: nx format:write update dirty files
andes-it Sep 9, 2024
e43ddda
Reverting front end change to deploy feature branch
HjorturJ Sep 10, 2024
7fbb61b
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 10, 2024
4b1f492
Refactoring even more into the aws service and simplifying
HjorturJ Sep 11, 2024
e93a11c
Merge branch 'main' into fix/attachment-and-s3--consolidation
HjorturJ Sep 11, 2024
daa76e3
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 11, 2024
624f471
deleting commented out file
HjorturJ Sep 11, 2024
5235604
chore: nx format:write update dirty files
andes-it Sep 11, 2024
29346d4
Fixing some error logging and variable name readability
HjorturJ Sep 12, 2024
8a77e89
merge conflict fixed
HjorturJ Sep 12, 2024
1e87112
reverted changes to a file that will be deleted on main anyway
HjorturJ Sep 12, 2024
6bc5086
resolving merge conflicts on a deleted file
HjorturJ Sep 12, 2024
6aaaaa6
chore: nx format:write update dirty files
andes-it Sep 12, 2024
ef28472
Fixing imports, adding and fixing tests
HjorturJ Sep 12, 2024
06c3717
Merge branch 'main' of github.com:island-is/island.is into fix/attach…
HjorturJ Sep 12, 2024
c6b20f0
Merge branch 'fix/attachment-and-s3--consolidation' of github.com:isl…
HjorturJ Sep 12, 2024
80313ef
chore: nx format:write update dirty files
andes-it Sep 12, 2024
6ab5a91
Removing random crap input from test i was testing hehe
HjorturJ Sep 13, 2024
0f81840
merge conflict
HjorturJ Sep 13, 2024
30a68c5
upgrading credential provider to be the same version as our s3 client…
HjorturJ Sep 13, 2024
dd2894c
upgrading aes sdk sqs too
HjorturJ Sep 13, 2024
794fc87
fixing tests after upgrading packages
HjorturJ Sep 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,7 @@ export class TemplateApiApplicationService extends BaseTemplateApiApplicationSer
const s3key = `${application.id}/${attachmentKey}`
const url = await this.awsService.uploadFile(
buffer,
uploadBucket,
s3key,
{ bucket: uploadBucket, key: s3key },
uploadParameters,
)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -712,10 +712,10 @@ export class EndorsementListService {
await this.uploadFileToS3(fileBuffer, filename, fileType)

// Generate presigned URL with 60 minutes expiration
const url = await this.awsService.getPresignedUrl(
environment.exportsBucketName,
filename,
)
const url = await this.awsService.getPresignedUrl({
bucket: environment.exportsBucketName,
key: filename,
})
return { url }
} catch (error) {
this.logger.error(`Failed to export list ${listId}`, { error })
Expand Down Expand Up @@ -780,8 +780,7 @@ export class EndorsementListService {
try {
await this.awsService.uploadFile(
fileBuffer,
environment.exportsBucketName,
filename,
{ bucket: environment.exportsBucketName, key: filename },
{
ContentType: fileType === 'pdf' ? 'application/pdf' : 'text/csv',
},
Expand Down
44 changes: 24 additions & 20 deletions libs/application/api/files/src/lib/file.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,19 +51,23 @@ export class FileService {
async generatePdf(application: Application, pdfType: PdfTypes) {
this.validateApplicationType(application.typeId)

const fileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
const key = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
const bucket = this.getBucketName()

if ((await this.awsService.fileExists(bucket, fileName)) === false) {
if ((await this.awsService.fileExists({ bucket, key })) === false) {
const content = await this.createFile(application, pdfType)
await this.awsService.uploadFile(content, bucket, fileName, {
ContentEncoding: 'base64',
ContentDisposition: 'inline',
ContentType: 'application/pdf',
})
await this.awsService.uploadFile(
content,
{ bucket, key },
{
ContentEncoding: 'base64',
ContentDisposition: 'inline',
ContentType: 'application/pdf',
},
)
}

return await this.awsService.getPresignedUrl(bucket, fileName)
return await this.awsService.getPresignedUrl({ bucket, key })
}

async uploadSignedFile(
Expand All @@ -79,11 +83,10 @@ export class FileService {
.waitForSignature(DokobitFileName[pdfType], documentToken)
.then(async (file) => {
const s3FileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
await this.awsService.uploadFile(
Buffer.from(file, 'binary'),
await this.awsService.uploadFile(Buffer.from(file, 'binary'), {
bucket,
s3FileName,
)
key: s3FileName,
})
})
.catch((error) => {
if (error.code === DokobitErrorCodes.NoMobileSignature) {
Expand Down Expand Up @@ -128,9 +131,9 @@ export class FileService {

const bucket = this.getBucketName()

const fileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
const key = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`

return await this.awsService.getPresignedUrl(bucket, fileName)
return await this.awsService.getPresignedUrl({ bucket, key })
}

private async createFile(application: Application, pdfType: PdfTypes) {
Expand All @@ -147,9 +150,11 @@ export class FileService {
pdfType: PdfTypes,
) {
const bucket = this.getBucketName()
const s3FileName = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
const s3File = await this.awsService.getFile(bucket, s3FileName)
const fileContent = s3File.Body?.toString('binary')
const key = `${BucketTypePrefix[pdfType]}/${application.id}.pdf`
const fileContent = await this.awsService.getFileContent(
{ bucket, key },
'binary',
)

const { phoneNumber, name, title } = this.getSigningOptionsFromApplication(
application,
Expand Down Expand Up @@ -223,8 +228,7 @@ export class FileService {
}

async getAttachmentPresignedURL(fileName: string) {
const { bucket, key } = AmazonS3URI(fileName)
const url = await this.awsService.getPresignedUrl(bucket, key)
const url = await this.awsService.getPresignedUrl(fileName)
return { url }
}

Expand All @@ -250,7 +254,7 @@ export class FileService {

try {
this.logger.info(`Deleting attachment ${s3key} from bucket ${bucket}`)
await this.awsService.deleteObject(bucket, s3key)
await this.awsService.deleteObject({ bucket, key: s3key })
result = {
...result,
deleted: result.deleted++,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,14 @@
import { getValueViaPath } from '@island.is/application/core'
import { ApplicationWithAttachments as Application } from '@island.is/application/types'
import { S3 } from 'aws-sdk'
import AmazonS3URI from 'amazon-s3-uri'
import { ApplicationWithAttachments } from '@island.is/application/types'
import { logger } from '@island.is/logging'
import { Injectable } from '@nestjs/common'
import { Inject, Injectable } from '@nestjs/common'
import { ConfigService } from '@nestjs/config'
import {
BaseTemplateApiApplicationService,
BaseTemplateAPIModuleConfig,
} from '../../../types'
import AmazonS3URI from 'amazon-s3-uri'
import { AwsService } from '@island.is/nest/aws'

export interface AttachmentData {
key: string
Expand All @@ -14,13 +19,16 @@ export interface AttachmentData {

@Injectable()
export class AttachmentS3Service {
private readonly s3: AWS.S3
constructor() {
this.s3 = new S3()
}
constructor(
@Inject(AwsService) private awsService: AwsService,
@Inject(ConfigService)
private readonly configService: ConfigService<BaseTemplateAPIModuleConfig>,
@Inject(BaseTemplateApiApplicationService)
private readonly applicationService: BaseTemplateApiApplicationService,
) {}

public async getFiles(
application: Application,
application: ApplicationWithAttachments,
attachmentAnswerKeys: string[],
): Promise<AttachmentData[]> {
const attachments: AttachmentData[] = []
Expand All @@ -43,45 +51,94 @@ export class AttachmentS3Service {
name: string
}>,
answerKey: string,
application: Application,
application: ApplicationWithAttachments,
): Promise<AttachmentData[]> {
return await Promise.all(
answers.map(async ({ key, name }) => {
const url = (
application.attachments as {
[key: string]: string
}
)[key]

const url = this.getFilenameFromApplication(application, key)
HjorturJ marked this conversation as resolved.
Show resolved Hide resolved
if (!url) {
logger.info('Failed to get url from application state')
return { key: '', fileContent: '', answerKey, fileName: '' }
}
const fileContent =
(await this.getApplicationFilecontentAsBase64(url)) ?? ''
(await this.awsService.getFileContent(url, 'base64')) ?? ''

return { key, fileContent, answerKey, fileName: name }
}),
)
}

private async getApplicationFilecontentAsBase64(
fileName: string,
): Promise<string | undefined> {
const { bucket, key } = AmazonS3URI(fileName)
const uploadBucket = bucket
try {
const file = await this.s3
.getObject({
Bucket: uploadBucket,
Key: key,
})
.promise()
const fileContent = file.Body as Buffer
return fileContent?.toString('base64')
} catch (error) {
logger.error(error)
return undefined
public async getAttachmentContentAsBase64(
application: ApplicationWithAttachments,
attachmentKey: string,
): Promise<string> {
const { bucket, key } = AmazonS3URI(
this.getFilenameFromApplication(application, attachmentKey),
)
const fileContent = await this.awsService.getFile({ bucket, key })
return fileContent?.Body?.transformToString('base64') || ''
}

public async getAttachmentContentAsBlob(
application: ApplicationWithAttachments,
attachmentKey: string,
): Promise<Blob> {
const { bucket, key } = AmazonS3URI(
this.getFilenameFromApplication(application, attachmentKey),
)
const file = await this.awsService.getFile({ bucket, key })
const fileArrayBuffer = await file?.Body?.transformToByteArray()
return new Blob([fileArrayBuffer as ArrayBuffer], {
type: file?.ContentType,
})
}

public async getAttachmentUrl(
key: string,
expiration: number,
): Promise<string> {
if (expiration <= 0) {
return Promise.reject('expiration must be positive')
}

const bucket = this.configService.get('attachmentBucket') as
| string
| undefined

if (bucket == undefined) {
return Promise.reject('could not find s3 bucket')
}

return this.awsService.getPresignedUrl({ bucket, key }, expiration)
}

public async addAttachment(
application: ApplicationWithAttachments,
fileName: string,
buffer: Buffer,
uploadParameters?: {
ContentType?: string
ContentDisposition?: string
ContentEncoding?: string
},
): Promise<string> {
return this.applicationService.saveAttachmentToApplicaton(
application,
fileName,
buffer,
uploadParameters,
)
}

private getFilenameFromApplication(
application: ApplicationWithAttachments,
attachmentKey: string,
): string {
const fileName = (
application.attachments as {
[key: string]: string
}
)[attachmentKey]
return fileName
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { DynamicModule } from '@nestjs/common'
import { ConfigModule } from '@nestjs/config'
import { EmailModule } from '@island.is/email-service'
import { ApplicationApiCoreModule } from '@island.is/application/api/core'
import { AwsModule } from '@island.is/nest/aws'
import { AwsModule, AwsService } from '@island.is/nest/aws'
import {
BaseTemplateAPIModuleConfig,
BaseTemplateApiApplicationService,
Expand Down Expand Up @@ -35,8 +35,9 @@ export class SharedTemplateAPIModule {
useClass: config.applicationService,
},
AttachmentS3Service,
AwsService,
],
exports: [SharedTemplateApiService, AttachmentS3Service],
exports: [SharedTemplateApiService, AttachmentS3Service, AwsService],
}
}
}
Loading
Loading