[Migrated] 502 While Deploying (ModuleNotFound Errors)

[jneves]

Originally from: Miserlou/Zappa#2143 by JordanTreDaniel

Context

I am trying to install a project that uses a package called wordcloud (python). This package has many dependencies, including numpy. I am using Zappa + Flask. (Slim Handler)

Running Python 3.8

Zappa Settings:

{
	"dev": {
		"app_function": "app.app",
		"aws_region": "us-east-1",
		"profile_name": "default",
		"project_name": "myproject",
		"runtime": "python3.8",
		"s3_bucket": "word-cloud-bucket",
		"slim_handler": "true"
	}
}

Requirements.txt:

argcomplete==1.12.0
boto3==1.14.28
botocore==1.17.28
certifi==2020.6.20
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
cycler==0.10.0
docutils==0.15.2
durationpy==0.5
Flask==1.1.2
future==0.18.2
hjson==3.0.1
idna==2.10
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kappa==0.6.0
kiwisolver==1.2.0
MarkupSafe==1.1.1
matplotlib==3.3.0
numpy==1.19.1
Pillow==7.2.0
pip-tools==5.2.1
placebo==0.9.0
pyparsing==2.4.7
python-dateutil==2.6.1
python-slugify==4.0.1
PyYAML==5.3.1
requests==2.24.0
s3transfer==0.3.3
six==1.15.0
text-unidecode==1.3
toml==0.10.1
tqdm==4.48.0
troposphere==2.6.2
urllib3==1.25.10
Werkzeug==1.0.1
wordcloud==1.7.0
wsgi-request-logger==0.4.6
zappa==0.51.0
https://files.pythonhosted.org/packages/ed/1d/c6d942bc569df4ff1574633b159a68c75f79061fe279975d90f9d2180204/wordcloud-1.7.0-cp37-cp37m-manylinux1_x86_64.whl ; sys_platform == "linux"
https://files.pythonhosted.org/packages/3b/84/7d9ce78ddecd970490f29010b23fbfde0adf35d1568b6d1c3fada9dbb7b5/numpy-1.19.1-pp36-pypy36_pp73-manylinux2010_x86_64.whl ; sys_platform == "linux"

I put in the requests for .whl files at the bottom because I thought that may help. But I don't even know if they are triggering a download.

When running zappa deploy dev, everthing goes well up to this point:

Deploying API Gateway..
Scheduling..
Unscheduled wordcloudflask-dev-zappa-keep-warm-handler.keep_warm_callback.
Scheduled wordcloudflask-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

Expected Behavior

Should upload to s3 + Lambda, and use the dependencies correctly.

Actual Behavior

I get the error above, and I can't help but to feel like it has something to do with the way that Zappa is adding/zipping the dependencies. Does it matter that I ran pip install wordcloud from a venv on my Mac? Would that affect the way things are compiled?

Running zappa tail, I get this:

[1595825062669] [ERROR] ModuleNotFoundError: No module named 'wordcloud.query_integral_image'
Traceback (most recent call last):
  File "/var/task/handler.py", line 609, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 240, in lambda_handler
    handler = cls()
  File "/var/task/handler.py", line 134, in __init__
    self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/wordcloudflask/app.py", line 5, in <module>
    from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
  File "/tmp/wordcloudflask/wordcloud/__init__.py", line 1, in <module>
    from .wordcloud import (WordCloud, STOPWORDS, random_color_func,
  File "/tmp/wordcloudflask/wordcloud/wordcloud.py", line 30, in <module>
    from .query_integral_image import query_integral_image

Possible Fix

1. The author of wordCloud, when confronted with the missing module error above, often suggests using conda. I think there is plenty of conversation around that already, but this is another vote for it. Conda support Miserlou/Zappa#108
2. Instead of seeing what we need based on the requirements.txt, why not allow us to package our own dependencies, hand them to you, and have them uploaded to S3?

By conda support, I mean me being able to specify that i need the dependencies installed with conda. I also hope that they are being compiled in the same type of environment as what is on AWS Python Lambdas (some strange form of Linux from what I hear)

Steps to Reproduce

1. Create Virtual env python3 -m venv myEnv
2. Activate env
3. pip install wordcloud
4. pip freeze > requirements.txt
5. Added some .whl files to requirements.txt (first time i didn't do this. Same result)
6. zappa deploy dev (slim handler)

Your Environment

• Zappa version used: 0.51.0
• Operating System and Python version: MacOS Catalina, Python 3.8
• The output of pip freeze: Above
• Link to your project (optional):
• Your zappa_settings.json: Above

[monkut]

This is an older migrated issue, the traceback is incomplete, and appears that it may depend on an unsupported binary.

closing.