Skip to content

Commit

Permalink
Issue629 (#630)
Browse files Browse the repository at this point in the history
* Exceptions do not have a .message attribute.

* Help tests run on Windows - don't assume temp dir or fonts.

* Python already has a feature for finding the temp dir. Changed
test helper to take advantage of it.

* Still outstanding: Several hard-coded references to /tmp appear in
the tests.

* Liberation-Mono is not commonly installed on Windows, and even when
it is, the font has a different name. Provide a fall-back for Windows
fonts. (Considered the use of a 3rd party tool to help select, but
seemed overkill.)

* Help tests run on Windows - allow some flexibility in versions.

Building/finding binaries on Windows is non-trivial. Aallow some
flexibility in the path levels. (I don't want to force existing users
to upgrade, but new users should be allowed the later patches.)

* Issue 596: Add initial support for closing clips.

Doesn't do anything yet. The work is done in the subclasses that need
it.

Also supports context manager, to allow close to be implicitly performed
without being forgotten even if an exception occurs during processes.

* Issue 596: Update doctest examples to call close.

Demonstrate good practice in the examples.

* More exception details for easier debugging of ImageMagick issues.

Especially for Windows.

* Issue #596: Move away from expecting/requiring __del__ to be called.

The work should be done in close(). Deleting can be left for the garbage
collector.

* Issue #596: Move ffmpeg_writer to using close.

Again, avoid depending on __del__. Add a context manager interface.
Use it lower down.

* Issue #596: Update ffmpeg_audiowriter to support close/context manager.

* Issue #596: Move AudioFileClip to use close(), away from __del__.

Was concerned that lambda might include a reference to reader that
wasn't cleaned up by close, so changed it over to an equivalent
self.reader. Probably has no effect, but feels safer.

* Issue #596: Support close() on CompositeVideoClip.

Note: It does NOT close all the subclips, because they may be used
again (by the caller). It is the caller's job to clean them up.

But clips created by this instance are closed by this instance.

* Issue #596: Add tests to see if this issue has been repaired.

test_resourcereleasedemo exercises the path where close is not called
and demonstrates that there is a consistent problem on Windows. Even
after this fix, it remains a problem that if you don't call close,
moviepg will leak locked files and subprocesses. [Because the problem
remains until the process ends, this is included in
a separate test file.]

test_resourcerelease demonstrates that when close() is called, the
problem goes away.

* Issue #596: Update tests to use close().

* Without tests changes, many of these existing tests do not pass on
Windows.

* Further to PR #597: Change to Arial

Helvetica wasn't recognised by ImageMagick. Changing to another
arbitrary font that should be available on all Windows machines.

* Issue #596 and #598: Updated test to support close().

Also changed test to meet Issue #598, but that is also being done in
PR#585, so will require a merge.

* Revert "More exception details for easier debugging of ImageMagick issues."

This reverts commit dc4a16a.

I bundled too much into one commit. Reverting and reapplying as two separate commits for better history.

* Issue #599: test_6 doesn't test anything.

Removed as it was crashing on Windows, achieving nothing on Linux.

* Issue #596: Move comment to avoid incorporate into documents.

* Issue #596: Add usages tips to documentation.

* Clip class missing from reference documents.

Due to failing import.

* Copy-edit: Clumsy sentence in documentation.

* Fix failing doctest.

* Issue 596: Add initial support for closing clips.

* Add key support for close()

   * FFMPEG_VideoWriter and FFMPEG_AudioWriter: Support close() and context managers.
   * Clip: support close() and context manager. Doesn't do anything itself. The work is done in the subclasses that need it.
   * Clip subclasses: Overrride close.
       * Move away from depending on clients calling__del__(). Deleting can be left to Garbage Collector.
   * CompositeVideoClip: Note: Don't close anything that wasn't constructed here. The client needs to be able to control the component clips.
   * AudioFileClip:  Was concerned that lambda might include a reference to reader that wasn't cleaned up by close, so changed it over to an equivalent self.reader. Probably has no effect, but feels safer.

* Update tests to use close().

   * Note: While many tests pass on Linux either way, a large proportion of the existing unit tests fail on Windows without these changes.
   * Include changes to many doctest examples - Demonstrate good practice in the examples.
   * Also, migrate tests to use TEMPDIR where they were not using it.
   * test_duration(): also corrected a bug in the test (described in #598). This bug is also been addressed in #585, so a merge will be required.

* Add two new test files:

   *  test_resourcereleasedemo exercises the path where close is not called and demonstrates that there is a consistent problem on Windows. Even after this fix, it remains a problem that if you don't call close, moviepg will leak locked files and subprocesses. Because the problem remains until the process ends, this is included in a separate test file.]
   * test_resourcerelease demonstrates that when close() is called, the problem goes away.

* Update documentation to include usage tips for close()

Not included:

    *  Example code has not been updated to use close().

* Merge branch 'WindowsSupport' of C:\Users\xboxl\OneDrive\Documents\MyApps\moviepy with conflicts.

* Neaten up output and PEP8 compliance.

Also, make runnable directly (to help debugging)

* Remove references to /tmp to allow to run on Windows.

* Reference to PermissionError failing on Python 2.7.

* Migrate to use requests to avoid certificate problems.

Old versions of urlretrieve have old certificates which means one of the
video downloads was failing.

Also requires changes to setup.py, to come.

* Clean up of dependencies.

Including adding ranges, removing unnecessary entries, adding missing
entries, adding environment markers, changing versions, and updating
pytest parameter handling.

* Simplification of Travis file - letting te setup.py do the heavy lifting

Remove conditional installations repeating the rules in setup.py
Remove some installation of test needs repeating the rules in setup.py
Add testing of installation options.

* Add Appveyor support.

* Solve Issue 629.
  • Loading branch information
Julian-O authored and bearney74 committed Aug 16, 2017
1 parent b2c5909 commit 02fc129
Show file tree
Hide file tree
Showing 32 changed files with 822 additions and 264 deletions.
33 changes: 23 additions & 10 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,23 +8,36 @@ python:
- "3.4"
- "3.5"
- "3.6"
# command to install dependencies

before_install:
- sudo add-apt-repository -y ppa:kirillshkrogalev/ffmpeg-next
- sudo apt-get -y -qq update
- sudo apt-get install -y -qq ffmpeg
- mkdir media

# Ensure PIP is up-to-date to avoid warnings.
- python -m pip install --upgrade pip
# Ensure setuptools is up-to-date to avoid environment_markers bug.
- pip install --upgrade setuptools
# The default py that is installed is too old on some platforms, leading to version conflicts
- pip install --upgrade py pytest

install:
- if [[ $TRAVIS_PYTHON_VERSION == '3.4' || $TRAVIS_PYTHON_VERSION == '3.5' || $TRAVIS_PYTHON_VERSION == '3.6' ]]; then pip install matplotlib; pip install -U scikit-learn; pip install scipy; pip install opencv-python; fi
- if [[ $TRAVIS_PYTHON_VERSION == '2.7' ]]; then pip install scipy; pip install opencv-python; fi
- pip install coveralls
- pip install pytest-cov
- python setup.py install
# command to run tests
before_script:
- py.test tests/ --cov
script: py.test tests/ --doctest-modules -v --cov moviepy --cov-report term-missing
- echo "No install action required. Implicitly performed by the testing."

# before_script:

script:
- python setup.py test --pytest-args "tests/ --doctest-modules -v --cov moviepy --cov-report term-missing"
# Now the *code* is tested, let's check that the setup is compatible with PIP without falling over.
- pip install -e .
- pip install -e .[optional]
- pip install -e .[test]
# Only test doc generation on latest. Doesn't work on some earlier versions (3.3), but doesn't matter.
- if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then pip install -e .[doc]; fi

after_success:
- coveralls

matrix:
fast_finish: true
161 changes: 161 additions & 0 deletions appveyor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# This file is used to configure the AppVeyor CI system, for testing on Windows machines.
#
# Code loosely based on https://github.com/ogrisel/python-appveyor-demo
#
# To test with AppVeyor:
# Register on appveyor.com with your GitHub account.
# Create a new appveyor project, using the GitHub details.
# Ideally, configure notifications to post back to GitHub. (Untested)

environment:
global:
# SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
# /E:ON and /V:ON options are not enabled in the batch script interpreter
# See: http://stackoverflow.com/a/13751649/163740
CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\appveyor\\run_with_env.cmd"

matrix:

# MoviePy supports Python 2.7 and 3.3 onwards.
# Strategy:
# Test the latest known patch in each version
# Test the oldest and the newest 32 bit release. 64-bit otherwise.

- PYTHON: "C:\\Python27-x64"
PYTHON_VERSION: "2.7.13"
PYTHON_ARCH: "64"
MINICONDA: C:\Miniconda
CONDA_INSTALL: "numpy"

- PYTHON: "C:\\Python33-x64"
PYTHON_VERSION: "3.3.5"
PYTHON_ARCH: "64"
MINICONDA: C:\Miniconda3-x64
CONDA_INSTALL: "numpy"

- PYTHON: "C:\\Python34-x64"
PYTHON_VERSION: "3.4.5"
PYTHON_ARCH: "64"
MINICONDA: C:\Miniconda3-x64
CONDA_INSTALL: "numpy"

- PYTHON: "C:\\Python35-x64"
PYTHON_VERSION: "3.5.3"
PYTHON_ARCH: "64"
MINICONDA: C:\Miniconda35-x64
CONDA_INSTALL: "numpy"

- PYTHON: "C:\\Python36-x64"
PYTHON_VERSION: "3.6.2"
PYTHON_ARCH: "64"
MINICONDA: C:\Miniconda36-x64
CONDA_INSTALL: "numpy"

- PYTHON: "C:\\Python27"
PYTHON_VERSION: "2.7.13"
PYTHON_ARCH: "32"
MINICONDA: C:\Miniconda
CONDA_INSTALL: "numpy"

- PYTHON: "C:\\Python34"
PYTHON_VERSION: "3.6.2"
PYTHON_ARCH: "32"
MINICONDA: C:\Miniconda36
CONDA_INSTALL: "numpy"

install:
# If there is a newer build queued for the same PR, cancel this one.
# The AppVeyor 'rollout builds' option is supposed to serve the same
# purpose but it is problematic because it tends to cancel builds pushed
# directly to master instead of just PR builds (or the converse).
# credits: JuliaLang developers.
- ps: if ($env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:APPVEYOR_BUILD_NUMBER -ne ((Invoke-RestMethod `
https://ci.appveyor.com/api/projects/$env:APPVEYOR_ACCOUNT_NAME/$env:APPVEYOR_PROJECT_SLUG/history?recordsNumber=50).builds | `
Where-Object pullRequestId -eq $env:APPVEYOR_PULL_REQUEST_NUMBER)[0].buildNumber) { `
throw "There are newer queued builds for this pull request, failing early." }

# Dump some debugging information about the machine.
# - ECHO "Filesystem root:"
# - ps: "ls \"C:/\""
#
# - ECHO "Installed SDKs:"
# - ps: "ls \"C:/Program Files/Microsoft SDKs/Windows\""
#
# - ECHO "Installed projects:"
# - ps: "ls \"C:\\projects\""
# - ps: "ls \"C:\\projects\\moviepy\""

# - ECHO "Environment Variables"
# - set


# Prepend desired Python to the PATH of this build (this cannot be
# done from inside the powershell script as it would require to restart
# the parent CMD process).
- "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"

# Prepare Miniconda.
- "ECHO Miniconda is installed in %MINICONDA%, and will be used to install %CONDA_INSTALL%"

- "set PATH=%MINICONDA%;%MINICONDA%\\Scripts;%PATH%"
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda

# Avoid warning from conda info.
- conda install -q -n root _license
# Dump the setup for debugging.
- conda info -a

# PIP finds some packages challenging. Let Miniconda install them.
- conda create --verbose -q -n test-environment python=%PYTHON_VERSION% %CONDA_INSTALL%
- activate test-environment

# Upgrade to the latest version of pip to avoid it displaying warnings
# about it being out of date.
- pip install --disable-pip-version-check --user --upgrade pip
- pip install --user --upgrade setuptools


# Install ImageMagick (which also installs ffmpeg.)
# This installation process is a big fragile, as new releases are issued, but no Conda package exists yet.
- "ECHO Downloading ImageMagick"
# Versions >=7.0 have problems - executables changed names.
# Assume 64-bit. Need to change to x86 for 32-bit.
# The available version at this site changes - each time it needs to be corrected in four places
# in the next few lines.
- curl -fskLO ftp://ftp.fifi.org/pub/ImageMagick/binaries/ImageMagick-6.9.9-5-Q16-x64-static.exe
- "ECHO Installing ImageMagick"
- "ImageMagick-6.9.9-5-Q16-x64-static.exe /verySILENT /SP"
- set IMAGEMAGICK_BINARY=c:\\Program Files\\ImageMagick-6.9.9-Q16\\convert.exe
- set FFMPEG_BINARY=c:\\Program Files\\ImageMagick-6.9.9-Q16\\ffmpeg.exe

# Check that we have the expected set-up.
- "ECHO We specified %PYTHON_VERSION% win%PYTHON_ARCH%"
- "python --version"
- "python -c \"import struct; print('Architecture is win'+str(struct.calcsize('P') * 8))\""

build_script:

# Build the compiled extension
- "%CMD_IN_ENV% python c:\\projects\\moviepy\\setup.py build"

test_script:
# Run the project tests
- "%CMD_IN_ENV% python c:\\projects\\moviepy\\setup.py test"

# TODO: Support the post-test generation of binaries - Pending a version number that is supported (e.g. 0.3.0)
#
# after_test:
#
# # If tests are successful, create binary packages for the project.
# - "%CMD_IN_ENV% python c:\\projects\\moviepy\\setup.py bdist_wheel"
# - "%CMD_IN_ENV% python c:\\projects\\moviepy\\setup.py bdist_wininst"
# - "%CMD_IN_ENV% python c:\\projects\\moviepy\\setup.py bdist_msi"
# - ps: "ls dist"
#
# artifacts:
# # Archive the generated packages in the ci.appveyor.com build report.
# - path: dist\*
#
# on_success:
# - TODO: upload the content of dist/*.whl to a public wheelhouse
86 changes: 86 additions & 0 deletions appveyor/run_with_env.cmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
:: To build extensions for 64 bit Python 3, we need to configure environment
:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
::
:: To build extensions for 64 bit Python 2, we need to configure environment
:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
::
:: 32 bit builds, and 64-bit builds for 3.5 and beyond, do not require specific
:: environment configurations.
::
:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
:: cmd interpreter, at least for (SDK v7.0)
::
:: More details at:
:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
:: http://stackoverflow.com/a/13751649/163740
::
:: Author: Olivier Grisel
:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
::
:: Notes about batch files for Python people:
::
:: Quotes in values are literally part of the values:
:: SET FOO="bar"
:: FOO is now five characters long: " b a r "
:: If you don't want quotes, don't include them on the right-hand side.
::
:: The CALL lines at the end of this file look redundant, but if you move them
:: outside of the IF clauses, they do not run properly in the SET_SDK_64==Y
:: case, I don't know why.
@ECHO OFF

SET COMMAND_TO_RUN=%*
SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows
SET WIN_WDK=c:\Program Files (x86)\Windows Kits\10\Include\wdf

:: Extract the major and minor versions, and allow for the minor version to be
:: more than 9. This requires the version number to have two dots in it.
SET MAJOR_PYTHON_VERSION=%PYTHON_VERSION:~0,1%
IF "%PYTHON_VERSION:~3,1%" == "." (
SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1%
) ELSE (
SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,2%
)
:: Based on the Python version, determine what SDK version to use, and whether
:: to set the SDK for 64-bit.
IF %MAJOR_PYTHON_VERSION% == 2 (
SET WINDOWS_SDK_VERSION="v7.0"
SET SET_SDK_64=Y
) ELSE (
IF %MAJOR_PYTHON_VERSION% == 3 (
SET WINDOWS_SDK_VERSION="v7.1"
IF %MINOR_PYTHON_VERSION% LEQ 4 (
SET SET_SDK_64=Y
) ELSE (
SET SET_SDK_64=N
IF EXIST "%WIN_WDK%" (
:: See: https://connect.microsoft.com/VisualStudio/feedback/details/1610302/
REN "%WIN_WDK%" 0wdf
)
)
) ELSE (
ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
EXIT 1
)
)
IF %PYTHON_ARCH% == 64 (
IF %SET_SDK_64% == Y (
ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
SET DISTUTILS_USE_SDK=1
SET MSSdk=1
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
) ELSE (
ECHO Using default MSVC build environment for 64 bit architecture
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
)
) ELSE (
ECHO Using default MSVC build environment for 32 bit architecture
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
)
26 changes: 26 additions & 0 deletions docs/getting_started/efficient_moviepy.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,38 @@ provides all you need to play around and edit your videos but it will take time

.. _previewing:

When to close() a clip
~~~~~~~~~~~~~~~~~~~~~~

When you create some types of clip instances - e.g. ``VideoFileClip`` or ``AudioFileClip`` - MoviePy creates a subprocess and locks the file. In order to release those resources when you are finished you should call the ``close()`` method.

This is more important for more complex applications and it particularly important when running on Windows. While Python's garbage collector should eventually clean it the resources for you, clsing them makes them available earlier.

However, if you close a clip too early, methods on the clip (and any clips derived from it) become unsafe.

So, the rules of thumb are:

* Call ``close()`` on any clip that you **construct** once you have finished using it, and have also finished using any clip that was derived from it.
* Also close any clips you create through ``AudioFileClip.coreader()``.
* Even if you close a ``CompositeVideoClip`` instance, you still need to close the clips it was created from.
* Otherwise, if you have a clip that was created by deriving it from from another clip (e.g. by calling ``set_mask()``), then generally you shouldn't close it. Closing the original clip will also close the copy.

Clips act as `context managers <https://docs.python.org/3/reference/datamodel.html#context-managers>`_. This means you
can use them with a ``with`` statement, and they will automatically be closed at the end of the block, even if there is
an exception. ::

with AudioFileClip("song.wav") as clip:
raise NotImplementedError("I will work out how process this song later")
# clip.close() is implicitly called, so the lock on song.wav file is immediately released.


The many ways of previewing a clip
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


When you are editing a video or trying to achieve an effect with MoviePy through a trial and error process, generating the video at each trial can be very long. This section presents a few tricks to go faster.


clip.save_frame
"""""""""""""""""

Expand Down
Loading

0 comments on commit 02fc129

Please sign in to comment.