-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Apache Avro library failed to parse file - when uploading avro file #1379
Comments
Thanks for the bug catch-- PR incoming. I think the problem you're having is because you're uploading the actual zip file. You'll have to extract the .avro file first before BigQuery can handle it. |
Also, in case you find it useful, we have a built-in event emitter to the Job object: job
.on('error', console.log)
.on('complete', function(metadata) {
console.log('job completed', metadata);
}); |
no, It isn't the zip file.. I extracted it.. On Mon, Jun 13, 2016 at 5:45 PM, Stephen Sawchuk notifications@github.com
|
Okay, sorry that I missed that. After making the fix you pointed out to add support for AVRO files, I was able to import the .avro file successfully: fs.createReadStream('./local-avro-file.avro')
.pipe(table.createWriteStream(metadata))
.on('complete', function(job) {
job
.on('error', console.log)
.on('complete', function(metadata) {
console.log('job completed', metadata);
});
}); Can you try it this way to help track down where the problem is coming from? |
PR sent with the AVRO fix: #1380. |
Yes, it works now. Thanks!! a) This works only for a table which has the same schema, b) In general: how do I use gcloud for uploading jsons with their schema - On Mon, Jun 13, 2016 at 6:20 PM, Stephen Sawchuk notifications@github.com
|
A) Good question. The official API request we make when uploading to a new table is Jobs: insert, specifically a
The UI handles detecting a schema from the AVRO file when dumping to a new table, so I'm not sure why we aren't able to reproduce the same behavior. B) I'm not sure those can be done in a single operation. I believe the table has to first be created before loading data. If I took your question too literally, and you just want to know how to load JSON data into an empty table, it should look like this: table.import('./data.json', {
schema: 'name:string, servings:integer, cookingTime:float, quick:boolean'
}, function(err) {}); If you want the schema inferred from the JSON file itself, it seems like that's either not supported by the API or we're using it wrong. @jgeewax anyone from BigQuery who can help with these questions? |
@DinaWork - do you mind taking this question to StackOverflow? I think the right eyes from BigQuery members will be able to see it over there, this channel must be off their radar. Please cross-link in your post. I'll be sure to implement any changes to make how we're using BigQuery better for users. |
here is a reference to the issue
http://stackoverflow.com/questions/37790813/apache-avro-library-failed-to-parse-file-nodejs
also had to add avro formats in gcloud/lib/bigquery files.
The text was updated successfully, but these errors were encountered: