Fix #1101: Query execution fails if user name has unicode characters

[kitsuyui]

• Encode the query with connection's charset when its type is unicode(not str).
• Ref: Query execution fails if user name has unicode characters #1101

[arikfr]

Thanks! I think a safer solution will be to disable query annotations for MSSQL query runner. All it requires is to add the following to the query runner class:

    @classmethod
    def annotate_query(cls):
        return False

[kitsuyui]

@arikfr
Okay! I will try your suggestion.

    @classmethod
    def annotate_query(cls):
        return False

[kitsuyui]

@arikfr

Okay! I will try your suggestion.

I tried.

But it is incomplete.
When unicodes come from query body(not from query annotations), UnicodeEncodeError still occurs.
Because the query will be unicode string eventually when query body is Unicode.

Example

Query like this:

SELECT 'あいうえお';

Result:

[2016-06-13 10:56:54,335: ERROR/MainProcess] Task redash.tasks.execute_query[3fc57eb9-1fde-4aa3-b176-bdab4e32cbfb] raised unexpected: UnicodeEncodeError('ascii', u"SELECT '\u3042\u3044\u3046\u3048\u304a';
", 8, 13, 'ordinal not in range(128)')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/redash/redash.0.11.0.b1903/redash/tasks/base.py", line 13, in __call__
    return super(BaseTask, self).__call__(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/redash/redash.0.11.0.b1903/redash/tasks/queries.py", line 445, in execute_query
    return QueryExecutor(self, query, data_source_id, metadata).run()
  File "/opt/redash/redash.0.11.0.b1903/redash/tasks/queries.py", line 395, in run
    data, error = query_runner.run_query(annotated_query)
  File "/opt/redash/redash.0.11.0.b1903/redash/query_runner/mssql.py", line 134, in run_query
    cursor.execute(query)
  File "pymssql.pyx", line 445, in pymssql.Cursor.execute (pymssql.c:6242)
  File "_mssql.pyx", line 998, in _mssql.MSSQLConnection.execute_query (_mssql.c:10085)
  File "_mssql.pyx", line 1029, in _mssql.MSSQLConnection.execute_query (_mssql.c:9964)
  File "_mssql.pyx", line 1149, in _mssql.MSSQLConnection.format_and_run_query (_mssql.c:11030)
  File "_mssql.pyx", line 200, in _mssql.ensure_bytes (_mssql.c:2544)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-12: ordinal not in range(128)
[2016-06-13 10:57:29,418: ERROR/MainProcess] Task redash.tasks.execute_query[422329db-3019-46cf-b24e-4e2d5a813e14] raised unexpected: UnicodeEncodeError('ascii', u"SELECT '\u3042\u3044\u3046\u3048\u304a';", 8, 13, 'ordinal not in range(128)')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/redash/redash.0.11.0.b1903/redash/tasks/base.py", line 13, in __call__
    return super(BaseTask, self).__call__(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/redash/redash.0.11.0.b1903/redash/tasks/queries.py", line 445, in execute_query
    return QueryExecutor(self, query, data_source_id, metadata).run()
  File "/opt/redash/redash.0.11.0.b1903/redash/tasks/queries.py", line 395, in run
    data, error = query_runner.run_query(annotated_query)
  File "/opt/redash/redash.0.11.0.b1903/redash/query_runner/mssql.py", line 134, in run_query
    cursor.execute(query)
  File "pymssql.pyx", line 445, in pymssql.Cursor.execute (pymssql.c:6242)
  File "_mssql.pyx", line 998, in _mssql.MSSQLConnection.execute_query (_mssql.c:10085)
  File "_mssql.pyx", line 1029, in _mssql.MSSQLConnection.execute_query (_mssql.c:9964)
  File "_mssql.pyx", line 1149, in _mssql.MSSQLConnection.format_and_run_query (_mssql.c:11030)
  File "_mssql.pyx", line 200, in _mssql.ensure_bytes (_mssql.c:2544)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 8-12: ordinal not in range(128)

[arikfr]

@kitsuyui in the case of the example you gave, what is the value of connection._conn.charset?

[kitsuyui]

@arikfr

In my case, UTF-8.
Because

1. pymssql.connect called at here with default charset parameter.
2. connection has the property _conn. It is an instance of _mssql.MSSQLConnection. It was initialized by pymssql with pymssql.connect.
3. So it is UTF-8.

[kitsuyui]

If you were hesitating to add manipulating private parameters, I can suppose another solution.

To add charset parameters on configuration_schema and pass it to pymssql.connect.

[arikfr]

If you were hesitating to add manipulating private parameters, I can suppose another solution.

I was both hesitant because of the private parameters manipulation, but also because it felt strange that pymssql doesn't support unicode objects...

Let's do something similar to what you suggested -- pass an explicit charset setting to the connection string (which is UTF-8) and use it for encoding. But let's not add another setting.

[arikfr]

Btw, thank you for your effort in investigating this.

[kitsuyui]

@arikfr
Thank you.
I updated my Pull Request to fit with latest commits and the decision!

[arikfr]

Thanks!