Skip to content

Commit

Permalink
Make dependency on cchardet optional
Browse files Browse the repository at this point in the history
Ported from #84
  • Loading branch information
horkhe committed Apr 6, 2018
1 parent 8933934 commit 6cf96af
Show file tree
Hide file tree
Showing 4 changed files with 42 additions and 14 deletions.
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ python:
- 2.7
- 3.6
install:
- pip install -e .[validator]
- pip install -e .[cchardet,validator]
- pip install nose mock coverage coveralls
script: ./build.sh
after_success:
Expand Down
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Changelog
All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [Unreleased]
- Dependency on cchardet was made optional. Ported from [PR84](https://github.com/mailgun/flanker/pull/84)

## [0.8.5] - 2018-03-30
### Changed
- MAX_HEADER_LENGTH (8000) and MAX_LINE_LENGTH (10000) parser limits were
removed.
- A bunch of implementation details were "hidden" in underscore prefixed
files/folders. This change is technically breaking, but those files were
never supposed to be used directly.

## [0.8.4] - 2018-02-01
The change log has not been kept up to release v0.8.5. The only reliable
source of information is the commit log on [GitHub](https://github.com/mailgun/flanker/commits/master).
30 changes: 18 additions & 12 deletions flanker/mime/message/utils.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,29 @@
from contextlib import closing
from email.generator import Generator

import cchardet
import chardet
import chardet as fallback_detector
import six

# Made cchardet optional according to https://github.com/mailgun/flanker/pull/84
try:
import cchardet as primary_detector
except ImportError:
primary_detector = fallback_detector

from flanker.mime.message import errors


def python_message_to_string(msg):
"""Converts python message to string in a proper way"""
"""
Converts python message to string in a proper way.
"""
with closing(six.StringIO()) as fp:
g = Generator(fp, mangle_from_=False)
g.flatten(msg, unixfrom=False)
return fp.getvalue()

def _guess_and_convert_with(value, detector=cchardet):

def _guess_and_convert_with(value, detector=primary_detector):
"""
Try to guess the encoding of the passed value with the provided detector
and decode it.
Expand All @@ -34,6 +42,7 @@ def _guess_and_convert_with(value, detector=cchardet):
except (UnicodeError, LookupError) as e:
raise errors.DecodingError(str(e))


def _guess_and_convert(value):
"""
Try to guess the encoding of the passed value and decode it.
Expand All @@ -42,9 +51,9 @@ def _guess_and_convert(value):
back to chardet which is much slower.
"""
try:
return _guess_and_convert_with(value, detector=cchardet)
except:
return _guess_and_convert_with(value, detector=chardet)
return _guess_and_convert_with(value, detector=primary_detector)
except Exception:
return _guess_and_convert_with(value, detector=fallback_detector)


def _make_unicode(value, charset=None):
Expand All @@ -61,12 +70,9 @@ def _make_unicode(value, charset=None):


def to_utf8(value, charset=None):
'''
"""
Safely returns a UTF-8 version of a given string
>>> utils.to_utf8(u'hi')
'hi'
'''

"""
value = _make_unicode(value, charset)

return value.encode("utf-8", "strict")
Expand Down
4 changes: 3 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
install_requires=[
'attrs',
'chardet>=1.0.1',
'cchardet>=0.3.5',
'cryptography>=0.5',
'idna>=2.5',
'ply>=3.10',
Expand All @@ -35,4 +34,7 @@
'dnsq>=1.1.6',
'redis>=2.7.1',
],
'cchardet': [
'cchardet>=0.3.5',
]
})

0 comments on commit 6cf96af

Please sign in to comment.