BigQuery on python. upload_from_file job not done

[patitonav]

I am running a test to upload a .json file into an bigquery table and i cant get the job done.
Here is my code :

def stream_data2(dataset_name, table_name, json_data):
    bigquery_client = bigquery.Client()
    dataset = bigquery_client.dataset(dataset_name)
    table = dataset.table(table_name)

    # Reload the table to get the schema.
    table.reload()
	
    job = table.upload_from_file(
        json_data, source_format='NEWLINE_DELIMITED_JSON', 
        encoding='UTF-8',
        create_disposition='CREATE_IF_NEEDED',
        write_disposition='WRITE_APPEND')
    pprint.pprint(job)


with open('clarin.json', 'rb') as json_data:
    stream_data2(DATASET_NAME, TABLE_NAME, json_data)

The job ended in fail with this error:

> Errors:
> file-00000000: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. (error code: invalid)
> file-00000000: JSON parsing error in row starting at position 0: . No such field: custom. (error code: invalid)
> 

This has no sense becouse i create the table with a json file with the same structure.

Any suggestion?

Thanks

[dhermes]

@tswast Is JSON even supported as an upload type? (source_format='NEWLINE_DELIMITED_JSON' seems to indicate yes)

[tswast]

Yes, JSON is supported. Keep in mind that it's "Newline Delimited". That means each row not table is a separate JSON object, each separated by a newline.

[patitonav]

Yes, actually in the JSON file there is just one row.

[tswast]

I recommend using the bq command line tool to experiment with the file format & schema. Once you have a file that BigQuery accepts, then you can automate it with this library. https://cloud.google.com/bigquery/bq-command-line-tool

[dhermes]

@patitonav I am going to pre-emptively close this, since it seems your issue is with your data formatting and this isn't really an issue with this library.

Let me know if you'd like to re-open this issue.

Thanks for the assist @tswast, helpful as always!