feat: introduce CRC32 checksums to storage:uploadData API #13649

eppjame · 2024-07-26T17:47:21Z

Description of changes

Introduce CRC32 utility, which depends on crc-32 NPM package
Add CRC32 checksum calculations and headers to both single-part and multi-part uploads

Issue #, if available

No applicable issue

Description of how you validated changes

Manually tested in a sample application
New and updated unit tests

Checklist

PR description included
yarn test passes
Unit Tests are changed or added
Relevant documentation is changed or added (and PR referenced)

Checklist for repo maintainers

Verify E2E tests for existing workflows are working as expected or add E2E tests for newly added workflows
New source file paths included in this PR have been added to CODEOWNERS, if appropriate

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…t threshold to 5GB

… CRC32

AllanZhengYP

Give a 1st round review over it. Here are some questions/nits.

AllanZhengYP · 2024-07-26T18:48:15Z

packages/storage/src/providers/s3/apis/uploadData/multipart/initialUpload.ts

+	const crc32List: ArrayBuffer[] = [];
+	const dataChunker = getDataChunker(data, size);
+	for (const { data: checkData } of dataChunker) {
+		const arrayBuffer = (await calculateContentCRC32(checkData))


Nit: arrayBuffer could make people mistake it to the actual payload.

Suggested change

const arrayBuffer = (await calculateContentCRC32(checkData))

const checksumArrayBuffer = (await calculateContentCRC32(checkData))

packages/storage/src/providers/s3/apis/uploadData/multipart/initialUpload.ts

packages/storage/src/providers/s3/apis/uploadData/multipart/uploadCache.ts

AllanZhengYP · 2024-07-26T19:15:25Z

packages/storage/src/providers/s3/utils/crc32.ts

+		internalSeed = crc32.str(content, internalSeed);
+	} else if (ArrayBuffer.isView(content)) {
+		internalSeed =
+			crc32.buf(new Uint8Array(content.buffer), internalSeed) >>> 0;


Sanity check, we don't need to worry about the buffer offset here right?

This is based on the md5 utility, where a Blob is constructed with the ArrayBufferView directly. Do you know if the Blob constructor cares about the byteOffset internally?

I think the problem is .buffer here. You can create BufferView on different offset of a same longer buffer, which is referenced by content.buffer, for example:

const buf = new ArrayBuffer(100); // buf 100 bytes const view1 = new DataView(buf, 10); // view1 10 bytes. const view1Buf = view1.buffer; // view1Buf 100 bytes.

I think here you do need to set the offset of your uint8 array. Same to other occurances.

AllanZhengYP · 2024-07-26T19:17:42Z

packages/storage/src/providers/s3/utils/crc32.ts

+		internalSeed =
+			crc32.buf(new Uint8Array(content.buffer), internalSeed) >>> 0;
+	} else if (content instanceof ArrayBuffer) {
+		internalSeed = crc32.buf(new Uint8Array(content), internalSeed) >>> 0;


Nit: Some explanation of why right shifting 0 is used here.

It converts the result to an unsigned 32-bit integer. The library has similar example in bin/crc32.njs.

Got it. Can you add this reference to the code?

AllanZhengYP · 2024-07-26T19:21:04Z

packages/storage/src/providers/s3/utils/crc32.ts

Can we add more extensive test suite to this file?

Let's reduce the impact to RN by default. Can you add a crc32.native.ts and make sure that file exports no-op function?

This has been added.

AllanZhengYP · 2024-07-26T19:25:49Z

packages/storage/src/providers/s3/utils/crc32.ts

+		await content.stream().pipeTo(
+			new WritableStream<Uint8Array>({
+				write(chunk) {
+					internalSeed = crc32.buf(chunk, internalSeed) >>> 0;
+				},
+			}),
+		);


I'm a bit worried about the availability of WritableStream API. Assuming it's to handle Blob content. Is using .stream() and WritableStream the only way here?

Blob stream and WritableStream are well supported APIs. The downside of not using streams is we become limited on how much data we can send to the CRC32 utility in one go. The consumer would have to mindful of memory usage, and split objects accordingly.

I think this comment dovetails to my other comment on React Native. I'm OK with this API if crc32.ts only going to be run on web.

AllanZhengYP · 2024-07-26T19:27:46Z

packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts

+					data: twoPartsPayload,
+				});
+				await multipartUploadJob();
+				expect(calculateContentCRC32).toHaveBeenCalledTimes(5); // (final crc32 calculation = 3 (2 part calculation + combine the 2 checksums)) + (2 part calculation)


Nit: This comment is confusing, what about elaborating it more?

Will update this comment.

Hopefully this is more clear:

The object we are testing will be split into 2 parts.

finalCrc32 is calculated at the start of multipart upload which requires 3 calculateContentCRC32 calls

There are 2 parts, 1 call each to produce 2 hashes

1 more call to combine the 2 calculated hashes

While uploading parts, 2 calls will be made to calculateContentCRC32

There are 2 parts, for every part, 1 call to calculateContentCRC32 will be made.

Combined will be 5 calls to calculateContentCRC32.

AllanZhengYP · 2024-07-26T19:31:50Z

packages/storage/src/providers/s3/utils/crc32.ts

+	checksumArrayBuffer: ArrayBuffer;
+	checksum: string;


Nit: Why do we need checksum in both string and buffer format? Can we use string for both per-part checksum and use string to calculate final checksum?

Tried it before, calculating combined crc32 from string does not produce the desired result.

packages/storage/src/providers/s3/utils/crc32.ts

jimblanc

Approach looks solid

jimblanc · 2024-07-30T15:54:09Z

packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts


 jest.mock('@aws-amplify/core');
 jest.mock('../../../../../src/providers/s3/utils/client/s3data');

+jest.mock('../../../../../src/providers/s3/utils/crc32', () => ({
+	...jest.requireActual('../../../../../src/providers/s3/utils/crc32'),


Do we need requireActual here? If only calculateContentCRC32 is exported, just mocking the module in its entirety should work. This might also allow us to remove the pollyfill stuff above.

Yes, requireActual is not required here, I will remove it. In these tests we are running the calculateContentCRC32 function and checking the resulting hash in some places. It is only mocked for the 100gb test for performance reasons.

jimblanc · 2024-07-30T16:01:51Z

packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts

+	mockCalculateContentCRC32.mockResolvedValue(undefined);
+};
+const mockCalculateContentCRC32Reset = () => {
+	mockCalculateContentCRC32.mockReset();


mockRestore might be appropriate here

jimblanc · 2024-07-30T16:43:16Z

packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts

+		}, 0);
+	});
+
+const mockCalculateContentCRC32Mock = () => {


Minor nit

Suggested change

const mockCalculateContentCRC32Mock = () => {

const mockCalculateContentCRC32 = () => {

jimblanc · 2024-07-30T16:48:16Z

packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts

@@ -621,24 +724,29 @@ describe('getMultipartUploadHandlers with key', () => {

 	describe('pause() & resume()', () => {


Sanity check: Why did this test change? Behavior change?

This test was starting and resuming multipart upload instantly and I believe it was not testing the scenario correctly even though it passed. It also started to fail after we introduced async md5 or crc32 in the uploadPartExecutor.

The result of this change looks good to me.

jimblanc · 2024-07-30T16:49:11Z

packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts

 			const { multipartUploadJob, onPause, onResume } =
 				getMultipartUploadHandlers({
-					key: defaultKey,
+					path: defaultKey,


Shouldn't this still be using key as per the test suite getMultipartUploadHandlers with key?

Yes, this should be key, good catch.

jimblanc · 2024-07-30T18:25:55Z

packages/storage/src/providers/s3/apis/uploadData/multipart/uploadPartExecutor.ts

+					ContentMD5:
+						crc32 === undefined && isObjectLockEnabled
+							? await calculateContentMd5(data)
+							: undefined,


Nit: Can we consolidate this logic with the new checks up on line 60 to do all of our checksum calculation in one place?

jimblanc · 2024-07-30T18:29:48Z

packages/storage/src/providers/s3/apis/uploadData/putObjectJob.ts

@@ -48,6 +49,7 @@ export const putObjectJob =
 			onProgress,
 		} = uploadDataOptions ?? {};

+		const ChecksumCRC32 = await calculateContentCRC32(data);


Nit:

Suggested change

const ChecksumCRC32 = await calculateContentCRC32(data);

const checksumCRC32 = await calculateContentCRC32(data);

Also ditto about consolidating Md5 calculation logic up here

jimblanc · 2024-07-30T18:32:42Z

packages/storage/src/providers/s3/utils/client/s3data/completeMultipartUpload.ts

@@ -87,7 +89,7 @@ const serializeCompletedPartList = (input: CompletedPart): string => {
 		throw new Error(`${INVALID_PARAMETER_ERROR_MSG}: ${input}`);
 	}

-	return `<Part><ETag>${input.ETag}</ETag><PartNumber>${input.PartNumber}</PartNumber></Part>`;
+	return `<Part><ETag>${input.ETag}</ETag><PartNumber>${input.PartNumber}</PartNumber>${input.ChecksumCRC32 ? `<ChecksumCRC32>${input.ChecksumCRC32}</ChecksumCRC32>` : ''}</Part>`;


Nit: Line getting a little bit long, would prefer to do this separately and inserting into the result

jimblanc · 2024-07-30T18:43:29Z

packages/storage/src/providers/s3/utils/crc32.ts

+	};
+};
+
+const padZeros = (input: string) => {


Nit: Could just use padStart() for this

jimblanc · 2024-07-30T18:44:58Z

yarn.lock

Could you regen this file with yarn to avoid changing all of our registry URLs?

AllanZhengYP

No blocking comments from me. Feel free to address them in a separate PR

AllanZhengYP · 2024-07-31T22:58:00Z

packages/interactions/package.json

@@ -89,13 +89,13 @@
 			"name": "Interactions (default to Lex v2)",
 			"path": "./dist/esm/index.mjs",
 			"import": "{ Interactions }",
-			"limit": "52.66 kB"
+			"limit": "54.05 kB"


Nit: can you double check if this indeed needs to bump up? (By check the bundle size GitHub action)

AllanZhengYP · 2024-07-31T23:02:02Z

packages/storage/__tests__/providers/s3/apis/uploadData/multipartHandlers.test.ts

@@ -621,24 +724,29 @@ describe('getMultipartUploadHandlers with key', () => {

 	describe('pause() & resume()', () => {


The result of this change looks good to me.

AllanZhengYP · 2024-07-31T23:06:22Z

packages/storage/package.json

@@ -108,6 +109,7 @@
 	"devDependencies": {
 		"@aws-amplify/core": "6.3.6",
 		"@aws-amplify/react-native": "1.1.3",
+		"@types/node": "20.14.12",


The dev dependency here should not affect the developer since they are not used by user's app. We enabled the RN integ test in the feature branch. We should be protected from such type conflict @jimblanc

AllanZhengYP · 2024-07-31T23:11:07Z

packages/storage/src/providers/s3/utils/crc32.ts

+	await blob.stream().pipeTo(
+		new WritableStream<Uint8Array>({
+			write(chunk) {
+				internalSeed = crc32.buf(chunk, internalSeed) >>> 0;


Nit: add a inline doc for this bit operation?

AllanZhengYP · 2024-07-31T23:13:48Z

packages/storage/__tests__/providers/s3/utils/client/S3/cases/completeMultipartUpload.ts

Is this option also added to listParts & putObject? If so we need to make sure they are covered as well

AllanZhengYP · 2024-07-31T23:20:16Z

packages/storage/src/providers/s3/apis/uploadData/multipart/uploadHandlers.ts

@@ -171,6 +179,7 @@ export const getMultipartUploadHandlers = (
 					onPartUploadCompletion,
 					onProgress: concurrentUploadsProgressTracker.getOnProgressListener(),
 					isObjectLockEnabled: resolvedS3Options.isObjectLockEnabled,
+					useCRC32Checksum: Boolean(inProgressUpload.finalCrc32),


Some thoughts for future improvement. Right now we have no idea if CRC32 checksum fails silently. We need to be deterministic. For example, if we detect the runtime is web but useCRC32Checksum is false, we need to throw.

cc @eppjame

AllanZhengYP · 2024-07-31T23:22:13Z

packages/storage/src/providers/s3/apis/uploadData/multipart/uploadPartExecutor.ts

+			}
+			const contentMD5 =
+				// check if checksum exists. ex: should not exist in react native
+				!checksumCRC32 && isObjectLockEnabled


Nit: we need to add a unit test when CRC32 checksume exists, we don't send MD5 even when isObjectLockEnabled is enabled.

AllanZhengYP · 2024-07-31T23:25:14Z

packages/storage/src/providers/s3/apis/uploadData/multipart/initialUpload.ts

+	for (const { data: checkData } of dataChunker) {
+		const checksumArrayBuffer = (await calculateContentCRC32(checkData))
+			?.checksumArrayBuffer;
+		if (checksumArrayBuffer === undefined) return undefined;


Is this to handle RN no-op function? Some inline document will be helpful.

I'd prefer to have other flags for RN runtime instead of reqiring reading 1st chunk of data. You can address in follow-up PR.

AllanZhengYP

No blocking comments from me. Feel free to address them in a separate PR

jimblanc

LGTM, agree with Allan's feedback which can be addressed in a followup

* chore: enable storage-browser preid release * chore: sync main (#13478) * release(required): Parsing custom oAuth in amplify_outputs (#13474) * update parseAmplify logic * revert custom oAuth from gen1 config * update bundle size * chore(release): Publish [skip release] - @aws-amplify/adapter-nextjs@1.2.3 - @aws-amplify/analytics@7.0.35 - @aws-amplify/api@6.0.37 - @aws-amplify/api-graphql@4.1.6 - @aws-amplify/api-rest@4.0.35 - @aws-amplify/auth@6.3.5 - aws-amplify@6.3.6 - @aws-amplify/core@6.3.2 - @aws-amplify/datastore@5.0.37 - @aws-amplify/datastore-storage-adapter@2.1.37 - @aws-amplify/geo@3.0.35 - @aws-amplify/interactions@6.0.34 - @aws-amplify/notifications@2.0.35 - @aws-amplify/predictions@6.1.10 - @aws-amplify/pubsub@6.1.9 - @aws-amplify/storage@6.4.6 - tsc-compliance-test@0.1.39 * chore(release): update API docs [skip release] --------- Co-authored-by: ashika112 <155593080+ashika112@users.noreply.github.com> Co-authored-by: aws-amplify-bot <aws@amazon.com> Co-authored-by: Jim Blanchard <jim.l.blanchard@gmail.com> * feat(storage): add delimiter support (#13480) * feat: add delimiter input/output types * feat: pass Delimiter parameter to ListObjectsV2 API * chore: add unit tests * chore: bump bunde size * chore: address feedback * chore: fix build * chore: address feedback * chore: address feedback * chore: address feedback * chore: enable storage-browser preid release (#13524) chore: fix npm dist tag to be storage-browser * feat(storage): add base types for storage browser (#13528) * feat(storage): add base types * fix(storage): address feedbacks Co-authored-by: israx <70438514+israx@users.noreply.github.com> * feat(storage): add creds store scaffolding and update types (#13558) * feat(storage): add cred store lru implementation (#13561) * refactor(storage): decouple utils from Amplify singleton (#13562) * feat: add config constructor * refactor: remove singleton reference from storage utils * refactor: update storage utils * chore: update upload api * chore: address feedback * chore: fix unit tests * chore: remove singleton reference * chore: add license headers * chore: address feedback * chore: update bundle size * chore: address feedback * chore: update bundle size * feat(storage): add cred store creation implementation (#13575) Co-authored-by: israx <70438514+israx@users.noreply.github.com> * feat(storage): use WeakMap for store registry (#13586) * feat(storage): use WeakMap for store registry * chore(storage): export storage browser utils from @aws-amplify/storage/storage-browser * doc(storage): add disclaimer * feat(storage): Added getDataAccess & listCallerAccessGrant clients (#13582) * feat(storage): add an adapter interface for storage browser (#13576) --------- Co-authored-by: Jim Blanchard <jim.l.blanchard@gmail.com> * Revert "refactor(storage): decouple utils from Amplify singleton (#13562)" (#13597) This reverts commit 1079e7f. * feat(storage): simplify the location cred provider option input (#13601) Remove the un-essetnial validation of per-API's location credentials provider input scope and permission for now. * feat: Implement getLocationCredentials handler & integrate with adapter (#13600) * feat(storage): implement listLocations API and creation handler (#13602) * chore: expose path storage-browser from scoped package (#13611) chore: expose path storage-browser from scoped package Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * feat(storage): enables location credentials provider (#13605) * feat: add location credentials provider * chore: add unit tests * chore: address feedback * chore: add locationCredentialsOption to copy * chore: remove casting types * chore: assert idenitity id * chore: avoid export common options interface * chore: address feedback * chore: fix test * chore: address feedback * address feedback * chore: clean-up types * chore: add test * chore: update api bundlesize * feat(storage): resolve merge issue with multibucket * chore: update bundle size for config change and s3 multibucket * chore: address feedbacks * feat(storage): introduce preventOverwrite option to uploadData via HeadObject (#13640) * feat(storage): introduce preventOverwrite operation to uploadData via HeadObject * fix: add missing license and remove dependency on core in preventOverwrite validator * chore: update storage:uploadData bundle size * feat: move existing object validation to before completeMultipartUpload * fix: increase storage:uploadData bundle size * fix(storage): export storage-browser types for TS v4.2+ (#13647) * chore(storage-browser): export store and credentials related types, update createListLocationsHandler (#13660) * feat(storage): support force refresh location credentials (#13589) * feat(storage): require temporary creds for storage browser interfaces (#13664) * feat: introduce CRC32 checksums to storage:uploadData API (#13649) Co-authored-by: Donny Wu <wuuxi@amazon.com> * fix(storage-browser): listCallerAccessGrantsDeserializer not parsing multiple AccessGrant instances (#13671) * fix(storage-browser): listCallerAccessGrantsDeserializernot parsing multiple AccessGrant instances * chore: add unit tests for single and multiple grants --------- Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * chore(storage-browser): expose additional input output types (#13682) * chore(storage-browser): expose additional internal types * address feedback * remove 'applicationArn' from listCallerAccessGrant unit test * Update packages/storage/src/storageBrowser/apis/listCallerAccessGrants.ts Co-authored-by: Caleb Pollman <cpollman1@gmail.com> --------- Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> Co-authored-by: Caleb Pollman <cpollman1@gmail.com> * chore: add ui to storage browser co-owner * chore: update bundle size * chore: enable storage-browser integ test (#13698) Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * chore(storage): update s3 control model (#13705) * chore(storage): update s3 control model * fix: move permission validation to custom client * chore: formatting code * chore(storage-browser): pin crc-32 dep at 1.2.2 (#13752) * chore(storage-browser): pin crc-32 dep at 1.2.2 * chore: update lock file * chore(storage): add durability check for urls in put object (#13746) chore: add durability check for presigned urls in put object * fix(storage-browser): missing error wrapping for s3 control responses (#13779) * feat: validate corroborating elements in response, and compare echoed elements between request/response (#13764) * feat: add list call response validation * fix: add input key to test mock * feat: align list response validations with design * fix: increment bundle size for storage:list --------- Co-authored-by: Donny Wu <wuuxi@amazon.com> * feat(storage): add support for conditional headers to copy, and validate serialization (#13772) * feat: add notModifiedSince and eTag to copy * missing tests for durability helpers * add tests for integrity helpers * feat: add URL validation and tests for copyObject * chore: increase bundle size of storage:copy * feat: clean-up copy header validation logic * fix: revert copy option interface name changes --------- Co-authored-by: Donny Wu <wuuxi@amazon.com> * feat: adding object url checks (#13810) * chore: durability checks for create & complete multipart (#13809) * chore: durability checks for create & complete multipart * fix: parsePayload mock path * chore: enable e2e and tagged release (#13848) * chore: enable e2e and tagged release * core: update tag name * Revert "core: update tag name" This reverts commit c994f51. --------- Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * chore: Setup storage internals route (#13858) * chore: increse bundle size * chore: Refactor contents of `storage-browser/` path into `internals/` (#13859) * feat(storage): export Default Part Size constant (#13851) * chore: adds a flag to uploadData api output * chore: update comments for the flag * chore: revert some naming changes * chore: revert and make internal paramter optional with a default value * chore: update bundle size * chore: update to expose constant instead of bool in return type * chore: update export tests * chore: increase bundle size * chore: Setup storage internals route (#13858) * chore: Refactor contents of `storage-browser/` path into `internals/` (#13859) * fix: multipart upload storage crc32 integrity (#13878) * fix: multipart upload storage crc32 integrity * address comments Co-authored-by: AllanZhengYP <zheallan@amazon.com> * fix linter --------- Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> Co-authored-by: AllanZhengYP <zheallan@amazon.com> * feat(storage): internal GetProperties API (#13869) --------- Co-authored-by: Jim Blanchard <jim.l.blanchard@gmail.com> * feat(storage): add new internal remove api (#13880) feat(internal-remove): add new internal remove api Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * chore: Add internal getUrl API (#13882) * feat(storage): internals list API (#13874) * feat(storage): add new internal downloadData api (#13887) * feat(internal-remove): add new internal downloadData api * code cleanup * code cleanup * chore: fix ts doc --------- Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * feat: optional checksum algorithm for upload (#13849) * feat: opt in checksum * fix: revert local prettier suggestion * fix: up size limit for storage upload data * feat: react native crc32 * fix: up bundle size limit and fix typo * feat: add documentation for checksumAlgorithm * fix: update bundle size limit * fix: update bundle size limit * fix: address pr feedbacks * fix: bundle-size limit --------- Co-authored-by: AllanZhengYP <zheallan@amazon.com> * feat(storage): add remaining copy changes for internals (#13889) * chore: add remaining copy changes for internals * chore; explicitly call internal API with params * chore: copy return type * chore: feedback changes * feat(storage): internal uploadData implementation (#13888) * feat(storage): internal uploadData implementation * chore: update bundle size * fix: address feedbacks * revert: optional checksum algorithm for upload (#13849) (#13910) This reverts commit 02cb08a. * fix(storage): internals list function not able to decide which output types to use (#13915) chore: add function overloading to list * feat(storage): Add API support for Expected Bucket Owner (#13914) * Update Top/Internal API for expected bucket owner feat --------- Co-authored-by: JoonWon Choi <joonwonc@amazon.com> * chore: sync main with storage browser integrity branch (#13928) * chore(release): Set core metadata [skip release] * chore(release): Publish [skip release] - @aws-amplify/adapter-nextjs@1.2.21 - @aws-amplify/analytics@7.0.51 - @aws-amplify/api@6.0.53 - @aws-amplify/api-graphql@4.4.0 - @aws-amplify/api-rest@4.0.51 - @aws-amplify/auth@6.5.1 - aws-amplify@6.6.4 - @aws-amplify/core@6.4.4 - @aws-amplify/datastore@5.0.53 - @aws-amplify/datastore-storage-adapter@2.1.53 - @aws-amplify/geo@3.0.51 - @aws-amplify/interactions@6.0.50 - @aws-amplify/notifications@2.0.51 - @aws-amplify/predictions@6.1.26 - @aws-amplify/pubsub@6.1.26 - @aws-amplify/storage@6.6.9 - tsc-compliance-test@0.1.56 * chore(release): Update API docs [skip release] * chore: update contributing guide vite troubleshooting (#13826) * docs: update contributing guide vite troubleshooting * Update CONTRIBUTING.md --------- Co-authored-by: AllanZhengYP <zheallan@amazon.com> * chore(ai): add UpdateConversation enum (#13920) * chore(release): Set core metadata [skip release] * chore(release): Publish [skip release] - @aws-amplify/adapter-nextjs@1.2.22 - @aws-amplify/analytics@7.0.52 - @aws-amplify/api@6.0.54 - @aws-amplify/api-graphql@4.4.1 - @aws-amplify/api-rest@4.0.52 - @aws-amplify/auth@6.5.2 - aws-amplify@6.6.5 - @aws-amplify/core@6.4.5 - @aws-amplify/datastore@5.0.54 - @aws-amplify/datastore-storage-adapter@2.1.54 - @aws-amplify/geo@3.0.52 - @aws-amplify/interactions@6.0.51 - @aws-amplify/notifications@2.0.52 - @aws-amplify/predictions@6.1.27 - @aws-amplify/pubsub@6.1.27 - @aws-amplify/storage@6.6.10 - tsc-compliance-test@0.1.57 * chore(release): Update API docs [skip release] * fix(storage): multipart upload is firing on 0 bytes data (#13927) * chore: update bundle size --------- Co-authored-by: erinleigh90 <106691284+erinleigh90@users.noreply.github.com> Co-authored-by: aws-amplify-bot <aws@amazon.com> Co-authored-by: Parker Scanlon <69879391+scanlonp@users.noreply.github.com> Co-authored-by: AllanZhengYP <zheallan@amazon.com> Co-authored-by: Danny Banks <djb@amazon.com> Co-authored-by: ashika112 <155593080+ashika112@users.noreply.github.com> Co-authored-by: Hui Zhao <10602282+HuiSF@users.noreply.github.com> * feat: Add expectedBucketOwner to remaining internal APIs (#13932) feat: Add expectedBucketOwner to remaining internal APIs. * feat(storage): add advanced option to disable upload cache (#13931) * feat(storage): set allowedByApp to listCallerAccessGrants (#13934) * Storage Browser Default Auth (#13866) * first draft poc * upadtes * add listPaths API * update new file structure * fix types * refactor types and utils * update tests * fix test * fix bundle size test * update the listLocation handler * rename util * update Path type * fix missed type * chore(ci): add new storage-gen2-internal e2e test (#13916) Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * chore(storage): remove credential store and managed auth config factory (#13944) * feat: optional checksum algorithm for upload (#13939) Co-authored-by: AllanZhengYP <zheallan@amazon.com> * feat: preventOverwrite with if-none-match header (#13954) Co-authored-by: AllanZhengYP <zheallan@amazon.com> * Feat: Add pagination to Amplify Default Auth storage Browser (#13897) * update the listLocation handler * implement memoization * add pagination logic * update usergroup logic & test * update getPaginated Locations * fix failing test * test(storage): refactor unit tests for public & internal facade (#13955) * Chore: Remove createAmplifyAdapter & refactor (#13974) remove createAmplifyAdapter & refactor * update bundle size * fix bundle size & test * Fix: retry failure in storage retryDecider (#13977) fix retry failure * feat: validate uploaded parts before completing upload (#13763) Co-authored-by: Donny Wu <wuuxi@amazon.com> Co-authored-by: Allan Zheng <zheallan@amazon.com> * feat(storage): add customEndpoint to internal apis in advanced options (#13961) * feat: add baseEndpoint to advanced options * feat: add baseEndpoint to customEndpoint * feat: thread baseEndpoint through resolved config to endpoint resolver * add customEndpoint advanced option to internals storage data-plane apis * add customEndpoint advanced option to internals storage control-plane apis * fix unit test * code cleanup * increase bundle size * wire up customEndpoint on copy API * increase the bundle size * add customEndpoint unit tests for all data and control apis * increase bundle size * update ts docs * add additional error unit tests for endpointResolver * add unit tests for internals/ apis * code cleanup * address feedback * add comment for ForcePathStyleEndpointNotSupported ErrorCode * increase bundle size * remove docs links from error recovery message --------- Co-authored-by: Erin Beal <erinleig@amazon.com> Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> * fix(core): support endpoint resolver accepting both input and config(#13985) * feat(storage): allow checksum algo for internal upload API (#14002) * feat: allow setting encoding type for list calls * refactor: move to list internal * fix: add license * fix(storage): bug in copy unmodified since to use UTC (#14025) fix to use UTC and test * Revert "chore: enable e2e and tagged release (#13848)" This reverts commit 3dfde04. * revert tagged release --------- Co-authored-by: Allan Zheng <zheallan@amazon.com> Co-authored-by: israx <70438514+israx@users.noreply.github.com> Co-authored-by: ashika112 <155593080+ashika112@users.noreply.github.com> Co-authored-by: aws-amplify-bot <aws@amazon.com> Co-authored-by: Jim Blanchard <jim.l.blanchard@gmail.com> Co-authored-by: Ashwin Kumar <ashwinkumar2468@gmail.com> Co-authored-by: Ashwin Kumar <ashwsrir@amazon.com> Co-authored-by: Jamie Epp <168486127+eppjame@users.noreply.github.com> Co-authored-by: Jamie Epp <eppjame@amazon.com> Co-authored-by: Caleb Pollman <cpollman@amazon.com> Co-authored-by: Donny Wu <wuuxi@amazon.com> Co-authored-by: Caleb Pollman <cpollman1@gmail.com> Co-authored-by: Sergio Castillo Yrizales <yriz@amazon.com> Co-authored-by: Joon Choi <requiemdeciel@gmail.com> Co-authored-by: JoonWon Choi <joonwonc@amazon.com> Co-authored-by: erinleigh90 <106691284+erinleigh90@users.noreply.github.com> Co-authored-by: Parker Scanlon <69879391+scanlonp@users.noreply.github.com> Co-authored-by: Danny Banks <djb@amazon.com> Co-authored-by: Hui Zhao <10602282+HuiSF@users.noreply.github.com> Co-authored-by: ashika112 <akasivis@amazon.com> Co-authored-by: Erin Beal <erinleig@amazon.com> Co-authored-by: Pranav Malewadkar <pranavml@amazon.com> Co-authored-by: Pranav Malewadkar <malewadkar.1@osu.edu>

eppjame and others added 18 commits May 9, 2024 14:53

Integrate CRC-32 checksum into Single-Part Upload

05950c9

feat: stream file bytes for checksum calculation, increase single-par…

92b14bf

…t threshold to 5GB

Merge branch 'aws-amplify:main' into main

f35f97b

feat: multipart upload crc32 and test fixes

ee875e1

Merge branch 'aws-amplify:main' into main

1da61fb

feat: multipart upload crc32 and test fixes

5eb7a30

Merge branch 'main' into multi-part-upload

31c35bc

feat: crc32 tests and cleanup

ba15441

fix: restore MD5 usage in single-part upload if CRC32 is not available

912f8fa

feat: update multi part upload tests

e61d5a0

feat: add crc32 fallback and caching

9486e74

feat: add md5 test for multipart upload

a110dfd

Merge branch 'staging' into multi-part-upload

559aca8

fix: add crc32 values to tests

eafb78b

feat: add license to crc32

a5a236d

fix: update size limits

092ff82

chore: upgrade @types/node devDependency for storage module

bfbc0a1

chore: increase bundle size limit for interactions module

d10743e

eppjame requested review from a team as code owners July 26, 2024 17:47

fix: do not include CRC32 in uploadPart if upload wasn't created with…

81f07e8

… CRC32

AllanZhengYP reviewed Jul 26, 2024

View reviewed changes

jimblanc reviewed Jul 30, 2024

View reviewed changes

wuuxigh and others added 7 commits July 30, 2024 15:15

feat: add dedicated crc32 tests and native variant

5f41c9e

fix: simplify crc-32 calculation by converting everything to Blob

7e73233

fix: add missing test changes for putObjectJob

0815fae

fix: resume and pause tests

c6f82c2

fix: refactoring upload checksum to address comments

58cf72c

fix: refactoring types and renaming variables to address comments

b6d9608

Merge branch 'storage-browser/main' into multi-part-upload

e4c665e

fix: reduce impact from crc-32 introduction on yarn.lock

577b6e8

AllanZhengYP approved these changes Jul 31, 2024

View reviewed changes

jimblanc approved these changes Aug 1, 2024

View reviewed changes

jimblanc merged commit a06c2f9 into aws-amplify:storage-browser/main Aug 1, 2024
28 checks passed

	const arrayBuffer = (await calculateContentCRC32(checkData))
	const checksumArrayBuffer = (await calculateContentCRC32(checkData))

	const mockCalculateContentCRC32Mock = () => {
	const mockCalculateContentCRC32 = () => {

		@@ -621,24 +724,29 @@ describe('getMultipartUploadHandlers with key', () => {

		describe('pause() & resume()', () => {

	const ChecksumCRC32 = await calculateContentCRC32(data);
	const checksumCRC32 = await calculateContentCRC32(data);

feat: introduce CRC32 checksums to storage:uploadData API #13649

feat: introduce CRC32 checksums to storage:uploadData API #13649

Conversation

eppjame commented Jul 26, 2024

Description of changes

Issue #, if available

Description of how you validated changes

Checklist

Checklist for repo maintainers

AllanZhengYP left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AllanZhengYP Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimblanc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimblanc Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AllanZhengYP left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AllanZhengYP left a comment

Choose a reason for hiding this comment

jimblanc left a comment

Choose a reason for hiding this comment

AllanZhengYP Jul 30, 2024 •

edited

Loading

jimblanc Jul 30, 2024 •

edited

Loading