ERROR: getObject - RangeError: Invalid typed array length

[loretoparisi]

I get this error when downloading large files (2GB)

The error was

 {
  "message": "Invalid typed array length",
  "code": "RangeError",
  "time": "2017-05-24T08:16:11.462Z",
  "statusCode": 200,
  "retryable": false,
  "retryDelay": 72.22244454768195
}

the stack trace is

_initModules error:
RangeError: Invalid typed array length
    at Buffer.Uint8Array (native)
    at FastBuffer (buffer.js:11:5)
    at createUnsafeBuffer (buffer.js:38:12)
    at allocate (buffer.js:181:12)
    at Function.Buffer.allocUnsafe (buffer.js:141:10)
    at new Buffer (buffer.js:78:19)
    at Object.concat (//webservice/node_modules/aws-sdk/lib/util.js:154:16)
    at Request.HTTP_DONE (/webservice/node_modules/aws-sdk/lib/event_listeners.js:341:36)
    at Request.callListeners (/webservice/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (//webservice/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (//webservice/node_modules/aws-sdk/lib/request.js:678:14)
    at IncomingMessage.onEnd (//webservice/node_modules/aws-sdk/lib/event_listeners.js:255:26)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)

[loretoparisi]

Is possible that this is due to exceeding buffer size, according to:

The Buffer can't bigger than 2147483647 bytes. please use stream to process it.

See nodejs/node#6560.

[chrisradek]

@loretoparisi
Are you using the SDK in node.js or the browser?

If you are using node.js, then you can download the object from S3 using a stream, instead of buffering it into memory. See http://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/requests-using-stream-objects.html for details.

If you are using the browser to download large files, then you would want to generate a presigned url and embed that in your page for the user to download.

[loretoparisi]

@chrisradek Thank you, I'm using within node.js

root@69715783975e:/musixmatch# node -v 
v6.10.3

and Ubuntu 16.0.4 LTS.

I'm not doing specific buffering, I'm just using the libray as-it-is:

/**
     * Download from dataset backend
     * @param params object
     * @param callback function
     */
    S3Dataset.prototype.download = function(params,callback) {
        var self=this;
        
        var options = {
            fileType: 'utf-8'
        };
        var documentKey=Util.replace(self._options.s3.Key, options.fileName);  
        self.s3Store.getObject({ Key:  documentKey })
        //...

By the way I'm going to try the suggested examples:

var s3 = new AWS.S3({apiVersion: '2006-03-01'});
var params = {Bucket: 'myBucket', Key: 'myImageFile.jpg'};
var file = require('fs').createWriteStream('/path/to/file.jpg');
s3.getObject(params).createReadStream().pipe(file);

And back here with results.

[loretoparisi]

@chrisradek I have tried this solution

        var file = fs.createWriteStream(options.filePath);
        file.on('close', function(){
            if(self.logger) self.logger.info("S3Dataset file download saved to %s", options.filePath );
            return callback(null,done);
        });
        s3.getObject({ Key:  documentKey }).createReadStream().on('error', function(err) {
            if(self.logger) self.logger.error("S3Dataset download error key:%s error:%@", options.fileName, error);
            return callback(error);
        }).pipe(file);

but nothing happens i.e. the error or close callbacks are not being fired.

[loretoparisi]

The last approach I have tried was this one via httpData, since I'm successfully using this one when downloading and piping gunzip stream data as well:

        var s3=self.s3Store.GetInstance();
        var file = fs.createWriteStream(options.filePath);
        s3.getObject({ Bucket: this._options.s3.Bucket, Key: documentKey })
        .on('error', function(error) {
            if(self.logger) self.logger.error("S3Dataset download error key:%s error:%@", options.fileName, error);
            return callback(error);
        })
        .on('httpData', function(chunk) { file.write(chunk); })
        .on('httpDone', function() { 
            file.end(); 
            if(self.logger) self.logger.info("S3Dataset file download saved to %s", options.filePath );
            return callback(null,done);
        })
        .send();

And this time it worked, so despite of I do not know why the createReadStream,createWriteStream way does not work, I'm going via httpData at this time:

-rw-r--r--  1 loretoparisi  staff   2,1G 25 Mag 13:04 /root/my_model.bin

[srchase]

@loretoparisi

It looks like you got this working. Let us know if there are any additional details you can add regarding the original issue you encountered.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.