getting 'HTTP Error 400: Bad Request' on submission

[tungol]

This is possibly a problem with musicbrainzngs. Possible cause is that my password uses a few unusual symbols - one or more of these may not be being escaped properly. It's not throwing an authentication error, however, which is what I'd expect looking at the code. On the other hand, I don't get an authentication error even when entering bogus credentials, that generates a 400: Bad Request as well. So I'm not sure what's going on there.

EDIT:
The problem for this 400 error is a proxy.
Another sources of 400 errors could be #67.

This is blocked on https://rt.cpan.org/Public/Bug/Display.html?id=84789 (upstream of MB server)

[JonnyJD]

Hm, non-ascii passwords, that can be a problem in my script, in musicbrainzngs, in the musicbrainz server (web service part) or even at another place (python?)
I am working on a test script with some debug output to find out what actually happens.

What isrcsubmit version and/or package (isrcsubmit-*-mac.zip) are you using?
What python version are you using? (python --version)
Since you are on a Mac, this is probably some Python 2 version, but which?
What version of python-musicbrainzngs are you using? The one shipped with isrcsubmit (see question above)?

I couldn't reproduce your "400: Bad Request", yet, but I get a HTTP Error 401: digest auth failed using python2 (2.7.3)
and an uncaught UnicodeEncodeError: 'ascii' codec can't encode character\xe4in position 26: using python (3.3.1).
(You should remove the actual characters/passwords when posting error message, mine is just a test password anyways)

[JonnyJD]

The test script is available at https://gist.github.com/JonnyJD/5429281.
This just skips all the other steps and tries to submit isrcs (an empty dict actually), but adds a bit of debug output.

Please post the output of that script and python --version so I can see what exact problem you are having.
My 401 problem might have a different cause (I just changed my password on test.musicbrainz.org).

[JonnyJD]

In case this isn't related (as in you also don't get the 400 error using my password_test.py) I need the output of isrcsubmit.py when this 400 error occurs. Inluding the disc ID and the ISRCs.

FYI:
I am not sure the 401 I get with my testcript can be solved. In General HTTP (Digest) Auth (which is what we are using here) is not able to negotiate the encoding used for the username and password. So username and password are expected to be in ASCII.
EDIT: See also http://stackoverflow.com/questions/702629/utf-8-characters-mangled-in-http-basic-auth-username for that (accepted answer)
EDIT:
I got it to work on Python 3 when patching pythons urllib.request with H = lambda x: hashlib.md5(x.encode("latin1")).hexdigest(). I don't think that is a solution that will be accepted by python. It doesn't work pre-encoding it in isrcsubmit.

[JonnyJD]

If possible, running isrcsubmit (current master) with --debug would help.
I added some debug output for submit_isrcs.

[JonnyJD]

I opened a new ticket for the "401: Digest Auth failed" exception: #58 .

There is a ticket to expose the exact 400 error in alastair/python-musicbrainzngs#80.
Without that it is a bit more guessing involved from my side.

[tungol]

The password is all ascii. I use 1password to randomly generate passwords, which includes ascii symbols, since it's a web service I imagined would need URL escaping - I could be wrong about that, I don't know how the MB API works. I'll run the tests you asked for now.

[tungol]

Python 2.7.2
isrcsubmit_version = "2.0.0-beta.1"

python packages installed with pip:
musicbrainzngs==0.3
python-musicbrainz2==0.7.4
discid==0.4.0

libdiscid installed with brew:
libdiscid: stable 0.4.1

discisrc from the isrcsubmit-1.0.1-mac package

it doesn't seem to matter what I put in usename and password:

./password_test.py
ERROR: Entering submit_isrcs
ERROR: Entering authenticate
ERROR: Starting authentication

Please input your MusicBrainz username: [my username]
Please input your MusicBrainz password:

ERROR: Leaving authenticate
ERROR: Starting actual submission
ERROR: Couldn't send ISRCs: caused by: HTTP Error 400: Bad Request

OR just entering user: test123 password: 123

/password_test.py
ERROR: Entering submit_isrcs
ERROR: Entering authenticate
ERROR: Starting authentication

Please input your MusicBrainz username: test123
Please input your MusicBrainz password:

ERROR: Leaving authenticate
ERROR: Starting actual submission
ERROR: Couldn't send ISRCs: caused by: HTTP Error 400: Bad Request

[tungol]

and here's the most recent -dev version:

python isrcsubmit.py -d 1 --debug
isrcsubmit 2.0.0-dev by JonnyJD for MusicBrainz

using discisrc
CD drive #1 corresponds to 1 internally
Warning: Unable to read the disc's media catalog number.

DiscID: kmtZWQ.s_5hxRGCGT4gxzK9MhfU-
Tracks on disc: 19

Artist: Various Artists
Release: The Silver Ticket: Terminal Sales Vol. VI

Warning: Unable to read the disc's media catalog number.
MCN :
Track 1 : USSUB1304801
Track 2 : USSUB1304802
Track 3 : USSUB1304803
Track 4 : USSUB1304804
Track 5 : USSUB1304805
Track 6 : USSUB1304806
Track 7 : USSUB1304807
Track 8 : USSUB1304808
Track 9 : USSUB1304809
Track 10 : USSUB1304810
Track 11 : USSUB1304811
Track 12 : USSUB1304812
Track 13 : USSUB1304813
Track 14 : USSUB1304814
Track 15 : USSUB1304815
Track 16 : USSUB1304816
Track 17 : USSUB1304817
Track 18 : USSUB1304818
Track 19 : USSUB1304819
found new ISRC for track 1: USSUB1304801
found new ISRC for track 2: USSUB1304802
found new ISRC for track 3: USSUB1304803
found new ISRC for track 4: USSUB1304804
found new ISRC for track 5: USSUB1304805
found new ISRC for track 6: USSUB1304806
found new ISRC for track 7: USSUB1304807
found new ISRC for track 8: USSUB1304808
found new ISRC for track 9: USSUB1304809
found new ISRC for track 10: USSUB1304810
found new ISRC for track 11: USSUB1304811
found new ISRC for track 12: USSUB1304812
found new ISRC for track 13: USSUB1304813
found new ISRC for track 14: USSUB1304814
found new ISRC for track 15: USSUB1304815
found new ISRC for track 16: USSUB1304816
found new ISRC for track 17: USSUB1304817
found new ISRC for track 18: USSUB1304818
found new ISRC for track 19: USSUB1304819

Do you want to submit? [y/N] y
tracks2isrcs: {'14e521b8-08c9-4405-ad0a-495282b6cf76': u'USSUB1304803', 'd06b728b-1a7f-4369-bcde-30d189d07884': u'USSUB1304811', 'cba26480-b406-4235-9964-510dd3d3c24b': u'USSUB1304817', '9d70dc5d-640b-4d71-8e78-6441a4adedb7': u'USSUB1304808', 'af4b5fd0-50ff-443d-9e96-a6be19d2f6a6': u'USSUB1304813', '9685f2d2-c484-4a40-99c5-a373fa751f3e': u'USSUB1304809', '66e57e18-384a-4166-9eee-fc2c4479b53a': u'USSUB1304814', 'c8e7dcdd-50e4-457e-bba2-0c293aaa02b6': u'USSUB1304812', '69484878-94ca-4acd-a8d6-d52d535232a4': u'USSUB1304806', '78d439b3-add0-4e41-aa02-30ac38418fb6': u'USSUB1304801', '0388f725-8ba7-484a-a8a6-8c94810f3088': u'USSUB1304804', '78e6f29a-d1c6-4dd2-b073-016593c2faf2': u'USSUB1304810', '5eb809bd-94c0-4cc0-86e0-4526e2420a56': u'USSUB1304807', 'f878c8ac-3fa5-4ded-af02-f7646f89933f': u'USSUB1304818', 'b1d1004f-e0b9-4a58-914a-9f8391dea48b': u'USSUB1304802', '46ba86a8-e2a2-426a-a338-dca05ce08347': u'USSUB1304816', '6d30890d-0412-4211-a070-5311fd2f5d38': u'USSUB1304819', 'f9a8a9be-0780-42e7-8a01-31e8aa41647c': u'USSUB1304805', '8eafe934-a416-4286-b36e-d5618f7f6596': u'USSUB1304815'}

Please input your MusicBrainz username: [my username]
Please input your MusicBrainz password:

ERROR: Couldn't send ISRCs: caused by: HTTP Error 400: Bad Request

[JonnyJD]

Thanks a lot for your tests. I made a couple semi-related changes to the code based on your test data.

However, I could reproduce your problem. The problem you seem to have is probably http://bugs.python.org/issue4140
You have a proxy in between musicbrainz.org and your client?
I still have to figure some things out, but this looks like a musicbrainz server problem then.

When you can get around using a proxy things will work with current master (I added changes after your tests), but if you don't and things need to be fixed on MB server then this takes at least 2 weeks (due to the release cycle for MB server).

EDIT:
This indeed doesn't seem to have anything to do with your unusual ascii characters, since the same error occurs using only alphanumeric characters.

[JonnyJD]

Reported the proxy problem to MusicBrainz-server with http://tickets.musicbrainz.org/browse/MBS-6185.

EDIT:
FYI: If this needs fixes in a library MB-server uses, it can take quite some time until this is fixed.

[JonnyJD]

FYI:
This has gone up all the way to catalyst (library/framework MusicBrainz uses): https://rt.cpan.org/Public/Bug/Display.html?id=84789

[tungol]

That did the trick - the proxy I was behind is just for ad-blocking purposes. I turned that off and it went through no problem. An obnoxious bug, but at least I managed to shake out a bunch of lower-hanging fruit as well.

[Freso]

Is this still an issue?