Raise insertion errors from BigQuery

[thomasleese]

When inserting data in to BigQuery we need to inspect the response for errors as although it may have a 200 HTTP status code, schema-related errors that occur on insertion are included in the error information of the response.

See https://cloud.google.com/bigquery/docs/error-messages#streaming-success for more information on this.

Trello Card

[duncanjbrown]

🎉 Works very well and causes retries! The log output is still confusingly positive with "Success" and "200" messages — perhaps we can improve that in future.

Re the @insert_errors variable, we're not trapping these anywhere in the gem, so it goes unused — I don't think Sentry can pick it up? Maybe a future improvement to aid quick inspection in error reports could be to munge the message fields from the Google error objects into a string and use that as the exception message?

[duncanjbrown]

great stuff, thanks @thomasleese — approving with one comment about @insert_errors

[thomasleese]

Re the @insert_errors variable, we're not trapping these anywhere in the gem, so it goes unused — I don't think Sentry can pick it up? Maybe a future improvement to aid quick inspection in error reports could be to munge the message fields from the Google error objects into a string and use that as the exception message?

I've amended the commit so it now combines the messages in to a single string. I imagine this string could get quite long and may not be useable as it is, but it's better than nothing for now and we can improve it in the future if we need to.