Google Forms API expects JSON, not HTML

[ian-curtis]

I admit this issue may be due to my incompetence at using gargle or a handful of other things but it might also be an issue with how gargle interacts with the Forms API. I also have a StackOverflow issue open but it has not gotten much traffic.

I have been working with googledrive and googlesheets4 to make a Shiny app which serves as a project setup service for Google files. I have everything working locally. However, one of the files is a Google Form and I'd love to use the Forms API to help automate things further. I have been trying to follow along with the request helper functions guide. I, with some slight modifications to the files, successfully downloaded the Discovery Document for the Forms API and set up a call to the API but the call fails each time.

library(googledrive)
library(gargle)

fid <- "1V0ipE3zmRrLyvzmg9UUqtRngpjkWiCCPY8cyaLpjkr4"

# modifying the {googledrive} function
my_request_generate <- function(endpoint = character(),
                             params = list(),
                             key = NULL,
                             token = drive_token()) {
  
  # endpoints derived from Discovery Document which was grabbed from ingest-functions.R within gargle
  # a few modifications had to be made to the ingesting file as the Forms API is a little different?
  .endpoints <- readRDS(here::here("form_endpoints.RData"))
  ept <- .endpoints[[endpoint]]
  if (is.null(ept)) {
    print("Could not find your endpoint!")
  }
  
  req <- gargle::request_develop(endpoint = ept, params = params)
  
  gar_req <- gargle::request_build(
    path = req$path,
    method = req$method,
    params = req$params,
    body = req$body,
    token = token
  )
  gar_req
}

form_data_req <- my_request_generate(
  "forms.forms.get",
  params = list(
    formId = fid
  )
)

form_data_raw <- request_make(form_data_req)
response_process(form_data_raw)

However, it always tells me I need a json and not html. I've tried to do some digging to see what's going on but I'm stuck.

Error:
! Client error: (404) Not Found
✖ Expected content type application/json, not text/html.
ℹ See
  /var/folders/v9/w6xbcfv15tb191g6fft32j1h0000gn/T//RtmpfFbfRL/gargle-unexpected-html-error-136db4a3763f6.html
  for the html error content.
ℹ Or execute
  `browseURL("/var/folders/v9/w6xbcfv15tb191g6fft32j1h0000gn/T//RtmpfFbfRL/gargle-unexpected-html-error-136db4a3763f6.html")`
  to view it in your browser.
Run `rlang::last_trace()` to see where the error occurred.

I think the parameters are correct. Google says the call only needs formId. Is gargle not set up for the Forms API? Am I just missing something crucial? I appreciate any help and/or direction to post somewhere else if needed :).

[jennybc]

Did you do this to see if it reveals more about the error?

browseURL("/var/folders/v9/w6xbcfv15tb191g6fft32j1h0000gn/T//RtmpfFbfRL/gargle-unexpected-html-error-136db4a3763f6.html")

[jennybc]

This is characteristic of sending a request that generates an HTML error from the API, which, if memory serves, is often more about auth than the request itself. But in any case, the error is not about sending JSON vs HTML to the Forms API. This is a 404 for the request and, when we try to handle and display the error, the payload is HTML, not JSON. Which is why the URL to that temp file is provided, so you can perhaps see more about the error.

[ian-curtis]

Hi! I did look at that error file. Gives this:

404. That’s an error.

The requested URL /v1/forms/1V0ipE3zmRrLyvzmg9UUqtRngpjkWiCCPY8cyaLpjkr4 was not found on this server. That’s all we know.

So you're right, it does seem to be an API error. Now it's up to me trying to figure out where :D.

If it is more about the auth side of things, do you think it makes a difference that I am "running" in two different environments? My project is a shiny app so I'm using non-interactive, non-caching auth for that but when testing the API and the functions of googledrive I have been working interactively.

[jennybc]

If the "non-interactive, non-caching auth" is as a service account, that could definitely be the problem. You would need to make sure that the service account has been granted all necessary permissions on the relevant resources (Drive files, Sheets, etc.). The permissions will not be conferred on a service account indirectly, i.e. because you have permission and you control the service account. So that's one possibility, for sure.

[ian-curtis]

It's not a service account, as far as I know (I didn't set one up). The Shiny app uses my own OAuth2 client I created with Google. But my issues are occurring with the interactive case. I wonder if I should try things inside of the Shiny app (which has the OAuth client hooked up) instead. I do have the Sheets API, Drive API, and Forms API turned on with my client but I suppose I'm not sure if the necessary permissions are enabled when I try interactively. Good to know this isn't a gargle issue so I'll work on tackling this from other angles. Thank you so much for your help here!!

[jennybc]

This sentence doesn't fully make sense to me -- i.e. it doesn't tell a whole story, really:

The Shiny app uses my own OAuth2 client I created with Google.

An OAuth client is 1 out of the 3 legs of 3-legged OAuth, but a user still needs to use the client to obtain a token (Google's servers are the last of the 3 legs). And that token is what gets sent with the request. Does your Shiny app facilitate auth by a user? Maybe insert a googledrive::drive_user() call in your app (and reveal that in the app) to see who you are auth'ed as at the moment of this failed request.

[ian-curtis]

My apologies, I should have been clearer there. Still learning how this process works :). You are correct, the client prompts a user to sign in via a browser and creates a token for use. I do have a drive_user() built in to greet the user once they log in and this has worked ok.

However, after some further experimentation with auth and trying things with the client, I did figure out the solution here. The error was that the URL wasn't found and the answer to the problem was that my URL wasn't structured correctly. It looks like the Forms API differs from the Drive/Sheets API in terms of how its API URL is formatted. Where the Drive API has a base URL of https://www.googleapis.com, the Forms API uses https://forms.googleapis.com. Once I manually changed the url of the the call (with code above: form_data_req$url <- gsub("www.", "forms.", form_data_req$url)), I got a 200 response and the data I was expecting.

One note: I was not able to get this to work interactively using the googledrive package. Perhaps this is on purpose but it doesn't seem like the Forms API is enabled on whatever Google API project the package is associated with. The same code that works in my Shiny App (which has the Forms API enabled) gives the following error when used outside of the Shiny App with googledrive:

Error:
! Client error: (403) PERMISSION_DENIED
• Client does not have sufficient permission. This can happen because the OAuth token does not have the
  right scopes, the client doesn't have permission, or the API has not been enabled for the client
  project.
• Google Forms API has not been used in project 603366585132 before or it is disabled. Enable it by
  visiting
  https://console.developers.google.com/apis/api/forms.googleapis.com/overview?project=603366585132 then
  retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems
  and retry.

Error details:
Links
• Description: Google developers console API activation
  URL: https://console.developers.google.com/apis/api/forms.googleapis.com/overview?project=603366585132
• reason: SERVICE_DISABLED
• domain: googleapis.com
• metadata.service: forms.googleapis.com
• metadata.consumer: projects/603366585132

I think my initial problem is solved since I can get things to work in my Shiny app but perhaps it is worth exploring an integration of the Forms API into googledrive? It seems like there are some differences in how the API and is structured so it probably wouldn't be a seamless addition and I'm not sure how many people are trying to use the Forms API at this moment but it could be a future exploration. Just a thought :).

I appreciate your help here! It definitely led me in the right direction to keep testing things. I had tried quite a bit and got stuck but I should be ok for now.

Edit: I realize I can also hook up the client to work interactively outside of Shiny with drive_auth_configure() and test my functions with that. So maybe integrating the Forms API with googledrive is less of a priority.

[jennybc]

It looks like the Forms API differs from the Drive/Sheets API in terms of how its API URL is formatted. Where the Drive API has a base URL of https://www.googleapis.com, the Forms API uses https://forms.googleapis.com.

In googledrive and googlesheets4 this is a piece of info I feel like I learn from the relevant API's Discovery Doc and then somehow bake in at a rather low level. But it's been a long while since I touched any of that.

it doesn't seem like the Forms API is enabled on whatever Google API project the package is associated with

Yeah, that is definitely true. We've had no reason to enable Forms on it. I don't foresee making any changes to googledrive or the associated GCP project related to Forms.

Glad you are sorted out!