Cloud Storage error on local dev server when invoking manage.py runserver #103

jacobg · 2015-05-26T20:49:24Z

Since GAE SDK 1.9.18, I've been getting error using the cloudstorage api on my local dev server when starting server using command line manage.py runserver. It works fine when running via GAE launcher.

This is the error:
ERROR 2015-02-20 09:50:21,729 api_server.py:221] Exception while handling service_name: "app_identity_service"
method: "GetAccessToken"
request: "\n7https://www.googleapis.com/auth/devstorage.full_control"
request_id: "WoMrXkOyfe"

There was known issue in 1.9.18 affecting all GAE users (not just nonrel), but it was since fixed as discussed here:
http://stackoverflow.com/questions/28626222/using-gcs-in-gae-local-devserver

But still in 1.9.20, I'm seeing this error. Since it only occurs when using manage.py runserver, it seems like a bug in setting up stubs. Any ideas?

aburgel · 2015-05-26T20:55:59Z

I don't think the app_identity_service stub is registered by default in djangoappengine.db.stubs. Try adding it.

jacobg · 2015-05-26T20:59:54Z

Thanks for the suggestion. It doesn't seem to fix the error.

aburgel · 2015-05-26T21:01:04Z

Are you running on the code from #78?

jacobg · 2015-05-26T21:08:43Z

Ahh, no. So it looks like the cloudstorage branch is based on the latest master commits including devappserver2. What settings.py config need to be added, as referenced in the commit comments?

aburgel · 2015-05-26T21:11:06Z

I rebased it off master a couple of days ago, and I merged the devappserver2 stuff in to master as well.

You should add these to settings.py:

APPENGINE_STORAGE_SERVICE = 'gs'
CLOUD_STORAGE_DEFAULT_BUCKET = '<bucket name goes here>'

jacobg · 2015-05-26T22:12:16Z

Still not working. Are you setting up your oauth token and project id using instructions here?
https://cloud.google.com/storage/docs/gsutil_install

aburgel · 2015-05-26T22:13:29Z

Are you seeing this running locally only?

I don't think I did any special setup with oauth tokens.

jacobg · 2015-05-26T22:14:58Z

Yes. I thought maybe it was using remote storage. So you're saying it is local storage. But then I don't know what this error is then?

ERROR 2015-02-20 09:50:21,729 api_server.py:221] Exception while handling service_name: "app_identity_service"
method: "GetAccessToken"
request: "\n7https://www.googleapis.com/auth/devstorage.full_control"
request_id: "WoMrXkOyfe"

jacobg · 2015-05-26T22:17:46Z

And also it works totally fine when launching the local dev server using sdk launcher.

aburgel · 2015-05-26T22:20:49Z

is there a stacktrace or something? it sounds like its just in some startup code.

jacobg · 2015-05-26T22:25:44Z

Yes, it turns into an infinite recursion as follows:

ERROR    2015-05-26 22:11:36,604 api_server.py:239] Exception while handling service_name: "app_identity_service"
method: "GetAccessToken"
request: "\n7https://www.googleapis.com/auth/devstorage.full_control"
request_id: "GEDlNLBvSs"

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 214, in _handle_POST
    api_response = _execute_request(request).Encode()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 172, in _execute_request
    make_request()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 167, in make_request
    request_id)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub.py", line 131, in MakeSyncCall
    method(request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity_defaultcredentialsbased_stub.py", line 191, in _Dynamic_GetAccessToken
    token = credentials.get_access_token()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/client.py", line 642, in get_access_token
    self.refresh(http)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/client.py", line 559, in refresh
    self._refresh(http.request)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/appengine.py", line 194, in _refresh
    scopes, service_account_id=self.service_account_id)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 592, in get_access_token
    scopes, service_account_id=service_account_id)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 550, in get_access_token_uncached
    return rpc.get_result()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 522, in get_access_token_result
    rpc.check_success()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
    self.__rpc.CheckSuccess()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
    self.request, self.response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub.py", line 131, in MakeSyncCall
    method(request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity_defaultcredentialsbased_stub.py", line 191, in _Dynamic_GetAccessToken
    token = credentials.get_access_token()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/client.py", line 642, in get_access_token
    self.refresh(http)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/client.py", line 559, in refresh
    self._refresh(http.request)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/appengine.py", line 194, in _refresh
    scopes, service_account_id=self.service_account_id)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 592, in get_access_token
    scopes, service_account_id=service_account_id)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 550, in get_access_token_uncached
    return rpc.get_result()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 522, in get_access_token_result
    rpc.check_success()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 579, in check_success
    self.__rpc.CheckSuccess()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 157, in _WaitImpl
    self.request, self.response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub.py", line 131, in MakeSyncCall
    method(request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity_defaultcredentialsbased_stub.py", line 191, in _Dynamic_GetAccessToken
    token = credentials.get_access_token()
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/client.py", line 642, in get_access_token
    self.refresh(http)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/client.py", line 559, in refresh
    self._refresh(http.request)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/oauth2client/oauth2client/appengine.py", line 194, in _refresh
    scopes, service_account_id=self.service_account_id)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 592, in get_access_token
    scopes, service_account_id=service_account_id)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/app_identity/app_identity.py", line 550, in get_access_token_uncached
    return rpc.get_result()

[.........]

  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 201, in MakeSyncCall
    self._MakeRealSyncCall(service, call, request, response)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 235, in _MakeRealSyncCall
    raise pickle.loads(response_pb.exception())
RuntimeError: RuntimeError('maximum recursion depth exceeded while calling a Python object',)

aburgel · 2015-05-26T22:31:14Z

Did you add that pickle.loads bit? That could be more an issue with the logging than the actual underlying cause.

jacobg · 2015-05-26T22:42:20Z

Sorry, I don't understand what you mean? Isn't it calling remote api and raising an exception?

aburgel · 2015-05-26T22:47:16Z

Nevermind, I sometimes try to do clever things when I'm debugging appengine code, but I usually shoot myself in the foot when I do so. I thought you might be doing the same. 😁

I don't have any good ideas on what's causing this. I don't make use of the app_identity service myself, so I'm not sure how to best debug it. It sounds like you have a good work around tho.

jacobg · 2015-05-26T22:48:43Z

Are you using cloud storage in a different way without app identity?

aburgel · 2015-05-26T22:50:12Z

I don't think so. I'm using pretty much exactly whats in #78. I'm also using the latest djangotoolbox and django-nonrel 1.6.11, maybe that makes a difference?

jacobg · 2015-05-26T23:13:31Z

Yes I'm using those. Ok, so I'll give up on runserver command. Thanks.

mrazzari · 2015-06-10T20:10:26Z

We're not alone?

Despite what the comments in that ticket read, upgrading didn't work for me, up to 1.9.21.
The fix for me was to downgrade to 1.9.17.

jacobg · 2015-06-10T20:15:07Z

Hi @Razzari, The only workaround I have now (on latest SDK version) is to run my development server via GAE SDK Launcher, rather than python manage.py runserver. As I use PyCharm IDE for development, it's baked into it.

mrazzari · 2015-06-10T20:25:59Z

I tried that, from bash (Ubuntu), and was getting errors.
Also the nonrel docs ask us that we "don’t use dev_appserver.py directly".
I've always been a bit behind with SDK versions, and it's never been a problem.
So -at least for me- the path of least resistance seems to be 1.9.17.

speedplane · 2015-11-12T04:30:02Z

Just ran into the same issue. The launcher solution worked for me. Below is the command line that the launcher uses to start up the dev environment.

Running command: "['C:\\Python27\\pythonw.exe', 
'C:\\Program Files (x86)\\Google\\google_appengine\\dev_appserver.py', 
'--skip_sdk_update_check=yes', '--port=8080', 
 '--admin_port=8000', u'c:\\Users\\Michael\\Projects\\Sites\\DocketAlarm']"

Notice that skip_sdk_update_check=yes. I figured maybe that's causing the issue. However, when I pass that flag into manage.py runserver, I get the following error message:

C:\Program Files (x86)\Google\google_appengine\devappserver2.py: 
error: no such option: --skip_sdk_update_check

Does anyone know why --skip_sdk_update_check doesn't work on django nonrel? That may lead us to the root cause.

speedplane · 2015-11-12T15:35:14Z

Okay... the hypothesis led to a dead end. Two things I tried:

Inserting the --skip_sdk_update_check into runserver.py did not help things.
I updated the Cloud Storage Client, no help there either.

alejcas · 2015-11-19T23:48:58Z

I have the same problem

speedplane · 2016-01-08T03:59:27Z

Okay, I think I figured it out. There may be two dev_appserver.pys on your computer. There's one in C:\Program Files (x86)\Google\google_appengine\ and there is another in C:\Users\[USERNAME]\AppData\Local\Google\Cloud SDK\google-cloud-sdk\bin. Make sure all of your scripts point to the latter, not the former. Also, check your PATH and make sure only the latter directory is in there.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud Storage error on local dev server when invoking manage.py runserver #103

Cloud Storage error on local dev server when invoking manage.py runserver #103

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

mrazzari commented Jun 10, 2015

jacobg commented Jun 10, 2015

mrazzari commented Jun 10, 2015

speedplane commented Nov 12, 2015

speedplane commented Nov 12, 2015

alejcas commented Nov 19, 2015

speedplane commented Jan 8, 2016

Cloud Storage error on local dev server when invoking manage.py runserver #103

Cloud Storage error on local dev server when invoking manage.py runserver #103

Comments

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

aburgel commented May 26, 2015

jacobg commented May 26, 2015

mrazzari commented Jun 10, 2015

jacobg commented Jun 10, 2015

mrazzari commented Jun 10, 2015

speedplane commented Nov 12, 2015

speedplane commented Nov 12, 2015

alejcas commented Nov 19, 2015

speedplane commented Jan 8, 2016