generate_signed_url is horrendously slow #183
Comments
|
From @dw on July 28, 2017 11:34 The implementation from https://github.com/GoogleCloudPlatform/storage-signedurls-python looks much more competitive, it manages closer to 18ms per URL, vs. 47ish for this package. I'm still deeply surprised that RSA signing would be so slow, even with the OpenSSL implementation used by storage-signedurls-python |
|
From @arthurdarcet on July 28, 2017 12:4 with the cryptography package: private_key = '-----BEGIN PRIVATE KEY-----\n…'
backend = cryptography.hazmat.backends.default_backend()
key = cryptography.hazmat.primitives.serialization.load_pem_private_key(
private_key.encode('utf-8'),
password=None,
backend=backend,
)
to_sign = '\n'.join((
method.upper(),
content_md5,
content_type,
str(expires),
'/' + path,
))
sign = key.sign(
to_sign.encode('utf-8'),
cryptography.hazmat.primitives.asymmetric.padding.PKCS1v15(),
cryptography.hazmat.primitives.hashes.SHA256(),
)clocks at 1.2ms per signature on my machine |
|
From @dw on July 28, 2017 14:57 Thanks for this! I've borrowed it for our application as a temporary measure. I wonder why PyCrypto is so much slower |
|
From @dhermes on July 28, 2017 15:15
I was going to point this out, but you've already noticed (which is great). We used to support pyOpenSSL and PyCrypto (as well as RE: PyCrypto, that library is no longer maintained, so we won't be adding support for credential signing from it. |
|
From @arthurdarcet on July 28, 2017 15:23 PyCrypto was more or less replaced by pyca/cryptography: https://cryptography.io/en/latest/ |
|
From @dhermes on July 28, 2017 15:28 @arthurdarcet I am currently putting together a "proof-of-concept" signer using pyOpenSSL, so hopefully this will go somewhere. (I've not used I'd like to emphatically point out that |
|
From @dhermes on July 28, 2017 16:12 @dw Thanks for the info. Here is the There isn't a great way in In [1]: import google.auth
In [2]: from openssl_signer import OpenSSLSigner
In [3]:
In [3]: credentials, _ = google.auth.default()
In [4]: credentials
Out[4]: <google.oauth2.service_account.Credentials at 0x7fe1c8296780>
In [5]: old_signer = credentials._signer
In [6]: old_signer
Out[6]: <google.auth.crypt.RSASigner at 0x7fe1c82492e8>
In [7]: key = old_signer._key.save_pkcs1()
In [8]: key[:30]
Out[8]: b'-----BEGIN RSA PRIVATE KEY----'
In [9]: new_signer = OpenSSLSigner.from_string(
...: key, key_id=old_signer.key_id)
...:
In [10]: message = b'sanity check'
In [11]: old_signer.sign(message) == new_signer.sign(message)
Out[11]: True
In [12]:
In [12]: credentials._signer = new_signeractually comparing these two we see about a 46x speedup: In [13]: %timeit old_signer.sign(message)
33.1 ms ± 261 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [14]: %timeit old_signer.sign(message)
33.1 ms ± 139 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [15]: %timeit new_signer.sign(message)
712 µs ± 3.53 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [16]: %timeit new_signer.sign(message)
719 µs ± 1.51 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) |
|
From @dhermes on July 28, 2017 16:12 @jonparrott We definitely need to move this conversation over to |
|
From @jonparrott on July 28, 2017 16:14 Yep, we want to add ECC signers to Google auth so using cryptography for RSA is an option. We'd just need to make it optional. |
|
From @dhermes on July 28, 2017 16:53 I add a In [17]: from cryptography_signer import CryptographySigner
In [18]: cryptography_signer = CryptographySigner.from_string(
...: key, key_id=old_signer.key_id)
...:
In [19]: old_signer.sign(message) == cryptography_signer.sign(message)
Out[19]: True
In [20]: %timeit cryptography_signer.sign(message)
664 µs ± 5.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [21]: %timeit cryptography_signer.sign(message)
669 µs ± 4.98 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)Post-Script: Though it ( In [13]: %timeit old_signer.sign(message)
30.3 ms ± 427 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [14]: %timeit old_signer.sign(message)
30.3 ms ± 323 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [15]: %timeit new_signer.sign(message)
680 µs ± 6.21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [16]: %timeit new_signer.sign(message)
653 µs ± 14.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) |
theacodes
added
enhancement
and removed
api: storage
Fixes googleapis#183. NOTE: This change is incomplete. It needs unit tests and a Verifier implementation.
Fixes googleapis#183. NOTE: This change is incomplete. It needs unit tests and a Verifier implementation.
Fixes googleapis#183. NOTE: This change is incomplete. It needs unit tests and a Verifier implementation.
yoshi-automation
added
triage me
tseaver
removed
triage me
From @dw on July 28, 2017 10:55
Calls to
google.cloud.storage.Blob.generate_signed_url()are too slow.OS type and version
OS X latest / Linux Ubuntu 17.04 / Debian 8
Python version and virtual environment information
python --version2.7.10
google-cloud-python version
pip show google-cloud,pip show google-<service>or `pipStacktrace if available
Steps to reproduce
google.cloud.storage.Client()with an RSA key configuredgoogle.cloud.storage.Bucket()from that Clientgoogle.cloud.storage.Blob()from that Bucketblob.generate_signed_url(86400)in a short (100 iteration) looprsapure-Python RSA implementation is in use, even though OpenSSL and suchlike are availableCode example
Copied from original issue: googleapis/google-cloud-python#3696
The text was updated successfully, but these errors were encountered: