Bundle emojis in package

[impredicative]

It is difficult for me to provided reusable software if the emjois are not already bundled in the demoji package. I understand and acknowledge that the list of emjois could be routinely updated, but this is not a reason to routinely not bundle the list in the package itself. For instance, I believe Python comes bundled with unicode data. You can still offer users a way to update the emoji list, and otherwise fallback to what's in the package.

[bsolomon1124]

@impredicative Thanks for raising this request. I am open to it and think you make a valid argument. I like the proposed behavior of defaulting to the bundled version that is distributed with the repo, but then offering the ability to live-download and update if needed. I'm going to do some thinking about what would be the best storage format that offers a tradeoff of space-savings and load time.

[bsolomon1124]

Interestingly enough it looks like CPython distributes unicode data as C header files

• https://github.com/python/cpython/blob/master/Modules/unicodename_db.h
• python/cpython@051b9d0

Options I could see here would be:

• plain .py module containing a dict
• JSON
• compressed JSON

[Ronserruya]

unicode.org was down many times in the past few days, which took some of our systems down, I decided to inject the codes to the pkg by myself to solve this

1. I took the file the demoji writes to disk (you can grab it from here, or from ~/.demoji/codes.json), its only 324kb https://gist.github.com/Ronserruya/80b2174957ab95b0f8cd88516037cf44)

2. Before using demoji.download_codes()

def _replace_demoji_method(URL):
    with open('codes.json') as f:
        data = json.load(f)
        yield from data['codes'].items()

# Read from file instead of going to unicode.org
demoji.stream_unicodeorg_emojifile = _replace_demoji_method
demoji.download_codes()

Demoji will not read all the codes from the file instead of going to unicode.org

[impredicative]

@Ronserruya I chose to just go with the emjoi package which doesn't have this issue:

EMOJI_REGEXP = emoji.get_emoji_regexp()
EMOJI_REGEXP.sub("", "your text, optionally having emojis")

[bsolomon1124]

@impredicative

This issue has been tentatively resolved by demoji release 1.0.0, which is in release candidate stage right now.
This release bundles static copy of Unicode emoji data with the package rather than requiring a runtime download.

You can get this pre-release using --pre:

python3 -m pip install --pre -U demoji

I appreciate your feedback prior to releasing 1.0.0!