all: expose the rest api response?

[c0b]

my cloud application was using the NodeJS bindings of Google Cloud API, but for reasons of execution performance and better utilize multi-core CPU, some components have been re-written in Go, but with the Go bindings of Google Cloud API, I feel the major shortage is the access to raw api response from the GCloud's REST API, for example, with Node version's job.get I can always access the apiResponse object

https://googlecloudplatform.github.io/google-cloud-node/#/docs/bigquery/0.8.0/bigquery/job?method=get

job.get(function(err, job, apiResponse) {
  if (!err) {
    // `job.metadata` has been populated.
  }
});

# or with ES6 promise style:

job.get()
.then(( [ job_, apiResponse ] ) => {
   // use the apiResponse, to get the apiResponse.statistics or many other fields
})
.catch(err => ...)

With Go binary of same bigquery, from https://godoc.org/cloud.google.com/go/bigquery I see I can only get this much information from the Job, get the string id, or wait the job is till Done, or read job query results;

I believe both are just wrapping the REST API https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs with Go binding without access to the raw apiResponse, it's hard to get statistics and like many other fields, users have to wait #347 fixed to get the statistics, for information in other fields, may have to wait another PR to get;
Also, if upstream BigQuery REST API keeps changing to add more features in api response, this Go language binding has to to make code changes to adapt; Versus in NodeJS it's just exposing apiResponse for user to interpret whatever it changes

So question to this google-cloud-go repo maintainers, is it possible to expose the raw apiResponse for the better flexibility (as in NodeJS) ?

type Job
  func (j *Job) Cancel(ctx context.Context) error
  func (j *Job) ID() string
  func (j *Job) Read(ctx context.Context) (*RowIterator, error)
  func (j *Job) Status(ctx context.Context) (*JobStatus, error)
  func (j *Job) Wait(ctx context.Context) (*JobStatus, error)
type JobStatus
  func (s *JobStatus) Done() bool
  func (s *JobStatus) Err() error

from https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs

  "statistics": {
    "creationTime": long,
    "startTime": long,
    "endTime": long,
    "totalBytesProcessed": long,
    "query": {
      "queryPlan": [
        {
          "name": string,
          "id": long,
          "waitRatioAvg": double,
          "waitRatioMax": double,
          ...

[jba]

We don't want to expose that type.

It looks like #347 is moribund, so I'll take ownership of it and try to make progress.

[jba]

Clarifying: we don't want to expose it because it's an implementation detail. Our BigQuery client won't always be implemented on top of the HTTP/JSON client.

[c0b]

We don't want to expose that type.
Clarifying: we don't want to expose it because it's an implementation detail.

That can be understood as Go is not as flexible as Javascript (NodeJS) on JSON handling? In fact the NodeJS binding is facing the scenario here, bottom layer implementation detail could be over HTTP, or could be over gRPC, NodeJS is handling both well, and I don't think Go binding have to deal with more details than NodeJS binding when interacting same BigQuery server, over either HTTP/REST API or gRPC;

and the Node binding is intentionally exposing raw api Response to tackle some problems like googleapis/google-cloud-node#1681

I hope in Go can have same level of flexibility.

[jba]

Go is typed, and the gRPC and HTTP/JSON layers use different types. To hide the implementation, we have to wrap or translate the types into a type at the client layer.

[c0b]

I believe the GCloud Nodejs Client library is facing the same BigQuery (and other google cloud services) servers and it's handling both gRPC and HTTP/JSON well, and providing library users a unified apiResponse object, in the doc you can see all bigquery calls have the apiResponse in the callback parameter

So in Go language, I believe there should be some way to represent the raw api response in some type,
for example in the JSON api response how about expose the raw api response in the json.RawMessage type ? https://golang.org/pkg/encoding/json/#RawMessage it's just purely the []byte type to show me the bytes of api response; I'm not knowing enough details about gRPC, but how is google-cloud-node library archived that?

my app is in a state when some parts need better utilizing all available CPU cores, I want to convert the code in Go, for better execution performance because Nodejs is only good for IO-bound applications; but the google-cloud-go library isn't so good so flexible as google-cloud-node, many pending issues (#347 #110 #108 and others) here are road-blockers of turning all code into Go;

this one #554 to expose raw api response is something for users can work around even google-cloud-go library is not doing any translating the type into some specific type; hope it helps to be Future-Proof

the other thing is could you ask Google put more developer resources to build this library better? some issues here pending over 2 years doesn't make sense

[jba]

We'll be focusing on improving the BigQuery client over the next couple of months.