Next.js application returns "complete request must include the checksum for each part" for MPU

[zshzbh]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Original reported from @Mette1982 - #6810 (comment)

This works indeed perfectly on a NodeJs environment but if I execute the same in a NextJs (executed in the browser) app for instance it no longer works.

The user created a small NextJs app to demonstrate my issue using the code sample you provided => https://github.com/Mette1982/aws-s3-error

You only have to change values in the https://github.com/Mette1982/aws-s3-error/blob/main/src/app/upload.tsx file.

Dependencies -

        "@aws-sdk/client-s3": "^3.730.0",
        "@aws-sdk/lib-storage": "^3.730.0",
        "next": "15.1.5",
        "react": "^19.0.0",
        "react-dom": "^19.0.0"

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/lib-storage; @aws-sdk/client-s3

Which JavaScript Runtime is this issue in?

Browser

Details of the browser/Node.js/ReactNative version

20.18

Reproduction Steps

Whole next js is in - https://github.com/Mette1982/aws-s3-error/blob/main/src/app/upload.tsx file.
The file that handle upload requeset -

'use client'

import {ChecksumAlgorithm, S3Client} from "@aws-sdk/client-s3";
import {Upload} from "@aws-sdk/lib-storage";
import {useState} from "react";

export default function UploadCmp() {

    const [text, setText] = useState<string[]>([]);
    const handleFileChange = async (e: React.ChangeEvent<HTMLInputElement>) => {
        if (e.target.files) {

            try {
                const file = Array.from(e.target.files)[0];

                const parallelUploads3 = new Upload({
                    client: new S3Client({
                        region: "us-east-1",
                        credentials: {
                            accessKeyId: "xxx",
                            secretAccessKey: "xxxx",
                        },
                    }),
                    params: {
                        Bucket: "new-bucket-maggie-ma",
                        Key: "large-file.txt",
                        Body: file,
                        // Ensure proper content type
                        ContentType: "text/plain",
                        ChecksumAlgorithm: ChecksumAlgorithm.CRC32,
                    },
                    //  part size 5MB
                    partSize: 1024 * 1024 * 5,
                    queueSize: 1,
                    leavePartsOnError: true,
                });

                // Log upload progress
                parallelUploads3.on("httpUploadProgress", (progress) => {
                    const progressText = `Uploaded ${progress.loaded} of ${progress.total} bytes`;

                    setText(prevState => [...prevState, progressText]);
                    console.log(progressText);
                });

                await parallelUploads3.done();

                const doneMessage = "File uploaded successfully";
                setText(prevState => [...prevState, doneMessage]);
                console.log(doneMessage);
            } catch (error: any) {
                setText(prevState => [...prevState, "Upload error see console"]);
                console.error("Upload error:", error);
                // Log more details about the error
                if (error.message) console.error("Error message:", error.message);
                if (error.code) console.error("Error code:", error.code);

            }
        }
    };


    return (
        <div>
            <input type="file" onChange={handleFileChange}/>
            <div>
                {text.map((value, index) => (
                    <div key={index}>{value}</div>
                ))}
            </div>
        </div>
    );
}

Observed Behavior

Got error

Failed to load resource: the server responded with a status of 400 (Bad Request)
node_modules_next_dist_client_523921._.js:994 Upload error: InvalidRequest: The upload was created using a crc32 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.
    at de_InvalidRequestRes (http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_client-s3_885f07._.js:6345:23)
    at de_CommandError (http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_client-s3_885f07._.js:6280:25)
    at async http://localhost:3000/_next/static/chunks/node_modules_@smithy_b64847._.js:2035:32
    at async http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_35ca9f._.js:2281:28
    at async http://localhost:3000/_next/static/chunks/node_modules_@smithy_b64847._.js:9279:54
    at async http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_35ca9f._.js:1823:32
    at async http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_35ca9f._.js:1864:24
    at async http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_35ca9f._.js:3664:34
    at async Upload.__doMultipartUpload (http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_35ca9f._.js:512:22)
    at async Upload.done (http://localhost:3000/_next/static/chunks/node_modules_@aws-sdk_35ca9f._.js:295:16)
    at async handleFileChange (http://localhost:3000/_next/static/chunks/src_app_97e46c._.js:60:17)
error @ node_modules_next_dist_client_523921._.js:994
node_modules_next_dist_client_523921._.js:994 Error message: The upload was created using a crc32 checksum. The complete request must include the checksum for each part. It was missing for part 1 in the request.

Expected Behavior

expect it to work successfully

Possible Solution

No response

Additional Information/Context

No response

[kritzware]

Running into exactly the same issue on 3.731.1, using very similar code but in the eu-central-1 region.

[zym163]

@zshzbh Reduce the version to 3.717.0，effective

[zshzbh]

Can you try 3.735 to see if this issue persists?

[cullylarson]

I'm using 3.744.0 and getting the same error. us-east-2

[Mette1982]

Can you try 3.735 to see if this issue persists?

Hi zshzbh I tested https://github.com/Mette1982/aws-s3-error with the version you provided and still get the same error.

I update both client-s3 and lib-storage to that version.
"@aws-sdk/client-s3": "^3.735.0",
"@aws-sdk/lib-storage": "^3.735.0"

[zshzbh]

Hey guys,

I tried on my end with version 3.735 and the latest version 3.744 and I'm still seeing the error.

I was working on the workaround and my workaround works-

I'm sharing my workaround with CRC32 calculation to match AWS's expected format. I'm using @Mette1982 's repo and did some changes in upload.tsx. The strategy is to split files into smaller chunks, generate checksums for each chunk, and perform uploads of these chunks with their associated checksums for data integrity

In upload.tsx -

"use client";

import {
  ChecksumAlgorithm,
  CompleteMultipartUploadCommand,
  CreateMultipartUploadCommand,
  S3Client,
  UploadPartCommand,
} from "@aws-sdk/client-s3";
import { useState } from "react";

// CRC32 lookup table
const crc32Table = new Uint32Array(256);
for (let i = 0; i < 256; i++) {
  let crc = i;
  for (let j = 0; j < 8; j++) {
    crc = (crc >>> 1) ^ ((crc & 1) ? 0xEDB88320 : 0);
  }
  crc32Table[i] = crc;
}

// Modified CRC32 calculation to match AWS's expected format
const calculateCRC32 = (chunk: Uint8Array): string => {
  let crc = 0xFFFFFFFF;
  
  // Process each byte
  for (const byte of chunk) {
    crc = (crc >>> 8) ^ crc32Table[(crc ^ byte) & 0xFF];
  }
  
  // Finalize CRC
  crc = ~crc >>> 0; // Ensure unsigned 32-bit

  // Convert to Base64
  const buffer = new ArrayBuffer(4);
  new DataView(buffer).setUint32(0, crc, false); // big-endian
  return Buffer.from(buffer).toString('base64');
};

export default function UploadCmp() {
  const [text, setText] = useState<string[]>([]);

  const handleFileChange = async (e: React.ChangeEvent<HTMLInputElement>) => {
    if (e.target.files) {
      try {
        const file = Array.from(e.target.files)[0];
        const client = new S3Client({
            region: "us-east-1",
            credentials: {
              accessKeyId: "xxxxx",
              secretAccessKey: "xxxx",
            },
        });

        // Step 1: Create multipart upload
        const createResponse = await client.send(
          new CreateMultipartUploadCommand({
            Bucket: "test-s3-upload-xxx",
            Key: "large-file.txt",
            ChecksumAlgorithm: ChecksumAlgorithm.CRC32,
          })
        );

        const uploadId = createResponse.UploadId;
        const partSize = 5 * 1024 * 1024; // 5MB parts
        const parts: { PartNumber: number; ETag: string; ChecksumCRC32: string }[] = [];
        
        // Read file in chunks
        const fileSize = file.size;
        let partNumber = 1;
        let bytesProcessed = 0;

        while (bytesProcessed < fileSize) {
          const end = Math.min(bytesProcessed + partSize, fileSize);
          const chunk = await file.slice(bytesProcessed, end).arrayBuffer();
          const chunkUint8 = new Uint8Array(chunk);
          
          // Calculate CRC32 for the chunk
          const checksum = calculateCRC32(chunkUint8);
          
          // Upload part with checksum
          const uploadResponse = await client.send(
            new UploadPartCommand({
              Bucket: "test-s3-upload-xx",
              Key: "large-file.txt",
              UploadId: uploadId,
              PartNumber: partNumber,
              Body: chunkUint8,
              ChecksumAlgorithm: ChecksumAlgorithm.CRC32,
              ChecksumCRC32: checksum,
            })
          );

          parts.push({
            PartNumber: partNumber,
            ETag: uploadResponse.ETag!,
            ChecksumCRC32: checksum,
          });

          // Update progress
          bytesProcessed = end;
          partNumber++;

          const progressText = `Uploaded ${bytesProcessed} of ${fileSize} bytes`;
          setText((prevState) => [...prevState, progressText]);
          console.log(progressText);
        }

        // Complete multipart upload
        await client.send(
          new CompleteMultipartUploadCommand({
            Bucket: "test-s3-upload-xx",
            Key: "large-file.txt",
            UploadId: uploadId,
            MultipartUpload: {
              Parts: parts.map(({ PartNumber, ETag, ChecksumCRC32 }) => ({
                PartNumber,
                ETag,
                ChecksumCRC32,
              })),
            },
          })
        );

        const doneMessage = "File uploaded successfully";
        setText((prevState) => [...prevState, doneMessage]);
        console.log(doneMessage);
      } catch (error: any) {
        setText((prevState) => [...prevState, "Upload error see console"]);
        console.error("Upload error:", error);
        if (error.message) console.error("Error message:", error.message);
        if (error.code) console.error("Error code:", error.code);
      }
    }
  };

  return (
    <div>
      <input type="file" onChange={handleFileChange} />
      <div>
        {text.map((value, index) => (
          <div key={index}>{value}</div>
        ))}
      </div>
    </div>
  );
}

---

I added this CRC32 Implementation :

const crc32Table = new Uint32Array(256);
// ... table initialization
const calculateCRC32 = (chunk: Uint8Array): string => {
    // Calculates CRC32 checksum and returns base64 encoded string
}

This implements a CRC32 checksum calculator that AWS S3 uses for data integrity verification.

And in UploadCmp() , I first initiates multipart upload -

const createResponse = await client.send(
  new CreateMultipartUploadCommand({...})
);

And splits file into 5MB chunks and uploads each part

while (bytesProcessed < fileSize) {
  // Calculate checksum for chunk
  // Upload part with checksum
  // Track progress
}

And then completes multipart upload

await client.send(
  new CompleteMultipartUploadCommand({...})
);

While investigating this issue, please let me know if this workaround works for you.

[bobbydams]

Running into the exactly the same issue. I downgraded to 3.726.1 the release just before their announcement about this change. And that worked for me.

{
  "dependencies": {
      "@aws-sdk/client-s3": "^3.726.1",
      "@aws-sdk/lib-storage": "^3.726.1"
  }
}

[charlieforward9]

@bobbydams you meant to write

{
  "dependencies": {
      "@aws-sdk/client-s3": "3.726.1",
      "@aws-sdk/lib-storage": "3.726.1"
  }
}

(without the ^)

[marcorichetta]

I got the same error with v3.750.0, using the [example codehttps://github.com/Mette1982/aws-s3-error/blob/main/src/app/upload.tsx?rgh-link-date=2025-01-17T17%3A24%3A03.000Z).

Only workaround that worked for me was pinning versions to 3.726.1 and removing the ChecksumAlgorithm param.

[kuhe]

I was able to execute this code in a browser as of the latest AWS SDK version:

import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";

const s3 = new S3Client({ region, credentials });

  // log requests
  s3.middlewareStack.add(
    (next, context) => async (args) => {
      const headers = {
        ...args.request.headers,
      };
      delete headers["x-amz-security-token"];
      delete headers.authorization;
      // console.log(context.commandName, "outgoing headers", headers);
      const r = await next(args);
      console.log(context.commandName, r.response.statusCode, r.response.headers);
      return r;
    },
    {
      step: "finalizeRequest",
    }
  );

  // generate a stream in 8kb chunks
  function stream(size) {
    async function* generate() {
      while (size > 0) {
        yield "a".repeat(8 * 1024);
        size -= 8 * 1024;
      }
    }

    return new ReadableStream({
      async start(controller) {
        for await (const chunk of generate()) {
          controller.enqueue(chunk);
        }
        controller.close();
      },
    });
  }


// execute MPU
  const params = {
    Bucket: "...",
    Key: "...",
    Body: stream(6 * 1024 * 1024), // 6 MB
    ChecksumAlgorithm: "CRC32",
  };

  await new Upload({
    client: s3,
    params,
  }).done();

This is the CORS policy on my test bucket, did you enable this?

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "PUT",
            "POST",
            "DELETE",
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "ETag",
            "x-amz-checksum-crc32"
        ]
    }
]