502 While Deploying (ModuleNotFound Errors)

[JordanTreDaniel]

Context

I am trying to install a project that uses a package called wordcloud (python). This package has many dependencies, including numpy. I am using Zappa + Flask. (Slim Handler)

Running Python 3.8

Zappa Settings:

{
	"dev": {
		"app_function": "app.app",
		"aws_region": "us-east-1",
		"profile_name": "default",
		"project_name": "myproject",
		"runtime": "python3.8",
		"s3_bucket": "word-cloud-bucket",
		"slim_handler": "true"
	}
}

Requirements.txt:

argcomplete==1.12.0
boto3==1.14.28
botocore==1.17.28
certifi==2020.6.20
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
cycler==0.10.0
docutils==0.15.2
durationpy==0.5
Flask==1.1.2
future==0.18.2
hjson==3.0.1
idna==2.10
itsdangerous==1.1.0
Jinja2==2.11.2
jmespath==0.10.0
kappa==0.6.0
kiwisolver==1.2.0
MarkupSafe==1.1.1
matplotlib==3.3.0
numpy==1.19.1
Pillow==7.2.0
pip-tools==5.2.1
placebo==0.9.0
pyparsing==2.4.7
python-dateutil==2.6.1
python-slugify==4.0.1
PyYAML==5.3.1
requests==2.24.0
s3transfer==0.3.3
six==1.15.0
text-unidecode==1.3
toml==0.10.1
tqdm==4.48.0
troposphere==2.6.2
urllib3==1.25.10
Werkzeug==1.0.1
wordcloud==1.7.0
wsgi-request-logger==0.4.6
zappa==0.51.0
https://files.pythonhosted.org/packages/ed/1d/c6d942bc569df4ff1574633b159a68c75f79061fe279975d90f9d2180204/wordcloud-1.7.0-cp37-cp37m-manylinux1_x86_64.whl ; sys_platform == "linux"
https://files.pythonhosted.org/packages/3b/84/7d9ce78ddecd970490f29010b23fbfde0adf35d1568b6d1c3fada9dbb7b5/numpy-1.19.1-pp36-pypy36_pp73-manylinux2010_x86_64.whl ; sys_platform == "linux"

I put in the requests for .whl files at the bottom because I thought that may help. But I don't even know if they are triggering a download.

When running zappa deploy dev, everthing goes well up to this point:

Deploying API Gateway..
Scheduling..
Unscheduled wordcloudflask-dev-zappa-keep-warm-handler.keep_warm_callback.
Scheduled wordcloudflask-dev-zappa-keep-warm-handler.keep_warm_callback with expression rate(4 minutes)!
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

Expected Behavior

Should upload to s3 + Lambda, and use the dependencies correctly.

Actual Behavior

I get the error above, and I can't help but to feel like it has something to do with the way that Zappa is adding/zipping the dependencies. Does it matter that I ran pip install wordcloud from a venv on my Mac? Would that affect the way things are compiled?

Running zappa tail, I get this:

[1595825062669] [ERROR] ModuleNotFoundError: No module named 'wordcloud.query_integral_image'
Traceback (most recent call last):
  File "/var/task/handler.py", line 609, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 240, in lambda_handler
    handler = cls()
  File "/var/task/handler.py", line 134, in __init__
    self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/wordcloudflask/app.py", line 5, in <module>
    from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
  File "/tmp/wordcloudflask/wordcloud/__init__.py", line 1, in <module>
    from .wordcloud import (WordCloud, STOPWORDS, random_color_func,
  File "/tmp/wordcloudflask/wordcloud/wordcloud.py", line 30, in <module>
    from .query_integral_image import query_integral_image

Possible Fix

1. The author of wordCloud, when confronted with the missing module error above, often suggests using conda. I think there is plenty of conversation around that already, but this is another vote for it. Conda support #108
2. Instead of seeing what we need based on the requirements.txt, why not allow us to package our own dependencies, hand them to you, and have them uploaded to S3?

By conda support, I mean me being able to specify that i need the dependencies installed with conda. I also hope that they are being compiled in the same type of environment as what is on AWS Python Lambdas (some strange form of Linux from what I hear)

Steps to Reproduce

1. Create Virtual env python3 -m venv myEnv
2. Activate env
3. pip install wordcloud
4. pip freeze > requirements.txt
5. Added some .whl files to requirements.txt (first time i didn't do this. Same result)
6. zappa deploy dev (slim handler)

Your Environment

• Zappa version used: 0.51.0
• Operating System and Python version: MacOS Catalina, Python 3.8
• The output of pip freeze: Above
• Link to your project (optional):
• Your zappa_settings.json: Above

[akobig]

Try switching (in zappa_setting of course: "runtime": "python3.6") to python 3.6 and see if its working. (maybe 3.7 will also work)

[JordanTreDaniel]

@akobig That actually helped! Now i get to the second problem I have been seeing, which is No module named 'numpy.core._multiarray_umath', which I'm guessing must be due to the version of NumPy that I'm using (1.19.1) I will try to set up a new env in 3.6 only

[akobig]

yeah its probably due to a mismatch of your local env python version which is I guess 3.8 and the lambda version which is now 3.6. try installing numpy in a 3.6 local python env, zappa update and tell us if worked I'm interested.

[CaseGuide]

EDIT: It's looking like my issue is related to a known issue with slim_handler when deploying from Windows. I opened a new issue at #2145

I'm running into a similar issue/behavior but not sure if it's related.

I'm encountering an issue returning the error errorMessage": "Unable to import module 'index'" when using slim_handler : true. My handler function is a regular python function that is in index.py and was otherwise working fine until the package grew to require slim_handler:true upon installing pandas==1.1.0. I'd expect to find this file in the handler_<project>-<env>-<time>.zip file but it is missing.

Is there a way to force it to put my index.py file in that zip?

zappa_settings.json:

{
    "dev": {
        "aws_region": "us-east-1",
        "profile_name": "default",
        "project_name": "[name]",
        "runtime": "python3.6",
        "s3_bucket": "[bucket]",
        "use_precompiled_packages": true,
        "lambda_handler": "index.handler",
        "memory_size": 2048,
        "timeout_seconds": 300,
        "cloudwatch_log_level": "INFO",
        "slim_handler": true,
        "delete_local_zip": false, // Delete the local zip archive after code updates. Default true.
        "delete_s3_zip": false // Delete the s3 zip archive. Default true.
    }
}

EDIT: Upon including the file in the zip and uploading manually I find that it is also giving me an import error for pandas which is installed in my venv so in fact I may be having the same issue.

The new error is: Unable to import module 'index': No module named 'pandas'

The index.py that contains my handler is essentially:

import os
import subprocess
import shutil
import time
import json
import pandas as pd

def handler(event, context):
    df = pd.DataFrame(columns=['date', 'number', 'url'])

    body = df.to_json()

    lambda_response = {
        "isBase64Encoded": False,
        "statusCode": 200,
        "headers": {},
        "body": json.dumps(body)
    }
    return lambda_response

[JordanTreDaniel]

@akobig Thank you! That worked! I am very confused why i get 3.8 when i run python --version from my env (made with pyenv + virtualenv + pyenv-virtualenv), but zappa_settings is reflecting 3.6 and the deployment worked! I've trying for a week, so thank you very much 👍

[deepansh96]

Hey @akobig Thanks!
I just changed my python version from 3.8 to 3.6 in zappa_settings.json and it worked!

[immuneeb]

Running into this issue, and python3.6 appears to no longer supported by AWS Lambda. Would appreciate alternate troubleshooting ideas.