DOCKER: python script at the end of simulation in BEAM docker image is not working #3119
Conversation
nikolayilyin
changed the title
build.gradle
Outdated
| instruction 'RUN pip3 install --upgrade setuptools' | ||
| instruction 'RUN pip3 install numpy pandas cython sklearn matplotlib mapclassify descartes geopandas contextily plotly collections-extended psutil requests' | ||
|
|
||
| // there is a problem with python in the image: |
There was a problem hiding this comment.
What was the error? This was originally reviewed and working, so I'd be curious what error was being seen?
There was a problem hiding this comment.
the error looks like that:
14:24:26.231 INFO beam.analysis.AnalysisProcessor$ - [1] Running python script: src/main/python/events_analysis/analyze_events.py with args WrappedArray(/app/output/sf-light/sf-light-1k-xml__2021-03-03_14-20-35_qzz/ITERS/it.0/0.events.csv.gz) 14:24:26.375 ERROR beam.analysis.AnalysisProcessor$ - [1] Process Handler Stderr for Python Script: src/main/python/events_analysis/analyze_events.py: Traceback (most recent call last): Process Handler Stdout for Python Script: src/main/python/events_analysis/analyze_events.py: Traceback (most recent call last): 14:24:26.375 ERROR beam.analysis.AnalysisProcessor$ - [2] Process Handler Stderr for Python Script: src/main/python/events_analysis/analyze_events.py: File "src/main/python/events_analysis/analyze_events.py", line 3, in <module> Process Handler Stdout for Python Script: src/main/python/events_analysis/analyze_events.py: File "src/main/python/events_analysis/analyze_events.py", line 3, in <module> Process Handler Stdout for Python Script: src/main/python/events_analysis/analyze_events.py: import plotly.graph_objects as go 14:24:26.375 ERROR beam.analysis.AnalysisProcessor$ - [4] Process Handler Stderr for Python Script: src/main/python/events_analysis/analyze_events.py: ImportError: No module named plotly.graph_objects Process Handler Stdout for Python Script: src/main/python/events_analysis/analyze_events.py: ImportError: No module named plotly.graph_objects
There was a problem hiding this comment.
This is weird - do you have the output from the docker build? If plotly was installed then it should not throw this error. This implies to me that there was an error in install
There was a problem hiding this comment.
here is the log of the docker build:
beam-image-creation.log
There was a problem hiding this comment.
@nikolayilyin I tried to fix this by pulling in python2 also. However there are a lot of hurdles I had to overcome. Right now we need pysal/mapclassify#110 to be merged and released. I mean we can pull the github repo with the PR, but that opens even more cans of worms. Then, beyond that, many libraries are not guaranteeing or even supporting python 2 much any longer. So, my take is that we run everything in python 3. Is there a reason we are using python 2 at all? If not then let's remove that need and simplify things?
There was a problem hiding this comment.
As discussed during daily stand-up, there are no reasons to stay on python 2
|
Superceded by #3148 - shall I close? |
|
there is still an issue with orca |
|
also, there is a new documentation entry and a simple but useful shell script to run the beam docker image |
|
test! |
|
test! |
|
ooks like flaky tests .. everything passing locally |
|
test! |
Currently, python scripts at the end of BEAM simulation are not working because they can not find python modules.
So, I commented they out to reduce image size and image build time.
Investigation of python modules required.
Probably the problem is in using the wrong version of python because for
pythonthere are no installed modules inside the container but forpython3all modules are installed correctly.This change is