Debian builds fail with ascii encoding error on older distros

[callahad]

The Debian builds for debian:stretch, ubuntu:xenial, and ubuntu:bionic fail along these lines:

(Errors taken from an ubuntu:bionic container)

[ERROR]
Traceback (most recent call last):
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/logger/_observer.py", line 131, in __call__
    observer(event)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/logger/_legacy.py", line 93, in __call__
    self.legacyObserver(event)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/python/log.py", line 562, in emit
    util.untilConcludes(self.write, timeStr + " " + msgStr)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/python/util.py", line 815, in untilConcludes
    return f(*a, **kw)
builtins.UnicodeEncodeError: 'ascii' codec can't encode characters in position 771-772: ordinal not in range(128)

tests.handlers.test_oidc.OidcHandlerTestCase.test_map_userinfo_to_invalid_localpart

[ERROR]
Traceback (most recent call last):
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/logger/_observer.py", line 131, in __call__
    observer(event)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/logger/_legacy.py", line 93, in __call__
    self.legacyObserver(event)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/python/log.py", line 562, in emit
    util.untilConcludes(self.write, timeStr + " " + msgStr)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/python/util.py", line 815, in untilConcludes
    return f(*a, **kw)
builtins.UnicodeEncodeError: 'ascii' codec can't encode characters in position 803-804: ordinal not in range(128)

tests.handlers.test_saml.SamlHandlerTestCase.test_map_saml_response_to_invalid_localpart

Will investigate further in the morning.

[callahad]

Something is very strange here.

Changing the test to the following still fails:

def test_map_userinfo_to_invalid_localpart(self):
    """If the mapping provider generates an invalid localpart it should be rejected."""
    print("Hello, föö"))
    assert True

The traceback is:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/__main__.py", line 9, in <module>
    load_entry_point('Twisted', 'console_scripts', 'trial')()
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/scripts/trial.py", line 621, in run
    test_result = trialRunner.run(suite)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/runner.py", line 998, in run
    return self._runWithoutDecoration(test, self._forceGarbageCollection)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/runner.py", line 1044, in _runWithoutDecoration
    result.done()
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/reporter.py", line 737, in done
    self._printErrors()
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/reporter.py", line 690, in _printErrors
    self._formatFailureTraceback)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/reporter.py", line 661, in _printResults
    self._write(reason)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/reporter.py", line 1235, in _write
    super(TreeReporter, self)._write(self.currentLine)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/reporter.py", line 493, in _write
    self._stream.write(s)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/trial/reporter.py", line 76, in write
    return untilConcludes(self.original.write, *a, **kw)
  File "/synapse/build/venv/lib/python3.6/site-packages/twisted/python/util.py", line 815, in untilConcludes
    return f(*a, **kw)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 159-160: ordinal not in range(128)

[callahad]

Maybe we're hitting something like https://twistedmatrix.com/trac/ticket/8864 or https://twistedmatrix.com/trac/ticket/989 where Twisted's default logger doesn't properly handle unicode. Might explain the weird failures in #8993.

It must be something with the build environment, as these tests work fine in CI on older python versions...

[callahad]

A-ha! It's https://bugs.python.org/issue19846, "Python 3 raises Unicode errors with the C locale", which is fixed as of Python 3.7.

Tests pass with LANG="C.UTF-8"

Running a build on all supported distros to make sure we're good, but I expect that solves it.

[richvdh]

these test failures were introduced by #8911 and #8938 respectively.