pybase64 returns invalid results on Windows

[GeorgeFischhof]

pybase64 encode created not base64 encoded stream

Hi there,

I wanted to encode an image:

import requests
import pybase64

image_url = 'http://www.123freeicons.com/wp-content/uploads/Images/Nature/003-tick-weather-icons-l.jpg'
image = requests.get(image_url).content
image_base64 = pybase64.b64encode(image)

the returned stream contained non ascii characters. Maybe "just" encode and decode are exchanged (I did not try). But the same behaviour is found when used standrad or url_safe encoding.

used:
windows 7 64 bit, Python 3.6.1 64 bit
4 cores cpu

checked standard lib, it created good base64 encoding

BR,
George

[mayeut]

Thanks for the report @GeorgeFischhof.

It's weird that this issue got past the CI testing setup (which includes comparison of the output between the standard library and pybase64).

I don't have a Windows setup at hand right now to investigate this. I'll check this once I get home tonight, GMT+2.

In the meantime, can you provide me with the CPU name and the version of pybase64 you're using ?

CPU name: from a command prompt, wmic cpu get name

Version of pybase64: how was it installed ? with pip or from sources ?

Again, thanks for the report.

[GeorgeFischhof]

Hi Matthieu, unfortunately i removed the pybase64, but I used the version which was avalable yesterday (29 August 2017, 14:00 UTC 16:00 CEST) via pypi (used pip to install it) cpu version: Intel(R) Core(TM) i5-6440HQ CPU @ 2.60GHz The windows is windows 7 professional, service pack 1, but several patches are installed on it BR, George 2017-08-30 14:24 GMT+02:00 Matthieu Darbois <notifications@github.com>:

…

Thanks for the report @GeorgeFischhof <https://github.com/georgefischhof>. It's weird that this issue got past the CI testing setup (which includes comparison of the output between the standard library and pybase64). I don't have a Windows setup at hand right now to investigate this. I'll check this once I get home tonight, GMT+2. In the meantime, can you provide me with the CPU name and the version of pybase64 you're using ? CPU name: from a command prompt, wmic cpu get name Version of pybase64: how was it installed ? with pip or from sources ? Again, thanks for the report. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AS3JdaTW7QvS3lA35qlO1zCYZdN1wsjQks5sdVRhgaJpZM4PHOEU> .

[mayeut]

CI setup was not testing properly the wheels that were built. Once that was fixed I spent sometime to get it working on AppVeyor.

The 7737893 commit shall fix the issue you're seeing.

There were 2 issues:

1. SIMD does not work correctly under Windows. This was easy to reproduce on my VM and looks like what you're reporting. I disabled SIMD on Windows for now. I will need to open a PR in libbase64 project later this week.

2. I had another test failing only on CI builds. After much more investigation, this seems like a compiler issue. I modified a bit the way the code is written to work around this.

I will wait for the end of the week or for SIMD to be working before releasing to pypi. Whichever comes first.

In the meantime, you can test with this wheel if you're willing to give it a go: https://ci.appveyor.com/api/buildjobs/0x2jn1ktj0vs59ih/artifacts/todeploy%2Fpybase64-0.2.0-cp36-cp36m-win_amd64.whl

[GeorgeFischhof]

Hi Matthieu, I checked the new release, and it seems to be ok ;-) The attached test passed 2017-08-31 2:54 GMT+02:00 Matthieu Darbois <notifications@github.com>:

CI setup was not testing properly the wheels that were built. Once that was fixed I spent sometime to get it working on AppVeyor. The 7737893 <7737893> commit shall fix the issue you're seeing. There were 2 issues: 1. SIMD does not work correctly under Windows. This was easy to reproduce on my VM and looks like what you're reporting. I disabled SIMD on Windows for now. I will need to open a PR in libbase64 project later this week. 2. I had another test failing only on CI builds. After much more investigation, this seems like a compiler issue. I modified a bit the way the code is written to work around this. I will wait for the end of the week or for SIMD to be working before releasing to pypi. Whichever comes first. In the meantime, you can test with this wheel if you're willing to give it a go: https://ci.appveyor.com/api/buildjobs/0x2jn1ktj0vs59ih/ artifacts/todeploy%2Fpybase64-0.2.0-cp36-cp36m-win_amd64.whl — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AS3JdSde6IdGsO9WQGsUCes2q0KOBuW4ks5sdgREgaJpZM4PHOEU> .

import base64 import pybase64 import requests image_url = 'http://www.123freeicons.com/wp-content/uploads/Images/Nature/003-tick-weather-icons-l.jpg' image = requests.get(image_url).content def test_pybase64_b64encode(): image_base64 = (base64.b64encode(image)) image_pybase64 = (pybase64.b64encode(image)) assert image_base64 == image_pybase64 def test_pybase64_standard_b64encode(): image_base64 = (base64.standard_b64decode(image)) image_pybase64 = (pybase64.standard_b64decode(image)) assert image_base64 == image_pybase64 def test_pybase64_urlsafe_b64encode(): image_base64 = (base64.urlsafe_b64decode(image)) image_pybase64 = (pybase64.urlsafe_b64decode(image)) assert image_base64 == image_pybase64

[mayeut]

I just released a new version (0.2.1).
It's available through pip.
I worked around the SIMD issue with libbase64 so that SIMD is enabled on Windows builds as well.
I added tests for all SIMD path in CI (AVX & AVX2 don't always run depending on which hardware is running the tests)