Rewrite OpenEXR python bindings using pybind11 and numpy (#1756)

* Rewrite python bindings with pybind11 and numpy This introduces an entirely new python API for reading and writing OpenEXR files that supports all file features (or will soon): scanline, tiled, deep, multi-part, etc. It uses numpy arrays for pixel data, and it supports both separate arrays for each channel and interleaving of RGB data into a single composite channel. It leaves the existing binding API in place for backwards-compatibility; there's no overlap between the two APIs. See src/wrappers/python/README.md for examples of the new API. The API is simple: the ``File`` object holds a py::list of ``Part`` objects, each of which has a py::dict for the header and a py::dict for the channels, and each channels hold a numpy array for the pixel data. There's intentionally no support for selective scanline/time reading; reading a file reads the entire channel data for all parts. There *is*, however, an option to read just the headers and skip the pixel data entirely. A few things don't work yet: - Reading and writing of deep data isn't finished. - ID manfest attributes aren't supported yet. - For mipmaped images, it current only reads the top level. This also does not (yet) properly integrate the real Imath types. It leaves in place for now the internal "Imath.py" module, but defines its own internal set of set of Imath classes. This needs to be resolve once the Imath bindings are distributed via pypi.org The test suite downloads images from openexr-images and runs them through a battery of reading/writing tests. Currently, the download is enabled via the ``OPENEXR_TEST_IMAGE_REPO`` environment variable that is off by default but is on in the python wheel CI workflow. This also adds operator== methods to ``KeyCode`` and ``PreviewImage``, required in order to implement operator== for the ``Part`` and ``File`` objects. Signed-off-by: Cary Phillips <cary@ilm.com> * Fix subsampling Signed-off-by: Cary Phillips <cary@ilm.com> * - Make README.md examples more pythonic - Add __enter__/__exit__ so "with File(name) as f" works - Allow channel dict to reference pixel array directly - Fix subsampling Signed-off-by: Cary Phillips <cary@ilm.com> * Use numpy/tuples in place of Imath types Also: - Use single-element array for DoubleAttribute - Use fractions.Fraction for Rational - Use 8-element tuple for chromaticities - Add __enter__/__exit__ so "with" works with File object - remove operator== entirely, it's way too hard to implement something that's actually useful for testing purposes. The tests do their own validations. Signed-off-by: Cary Phillips <cary@ilm.com> pixel comparison Signed-off-by: Cary Phillips <cary@ilm.com> * Deep reading/writing works for both scanline and tiles Also works properly for separate channels and when coalescing into RGB/RGBA arrays. Signed-off-by: Cary Phillips <cary@ilm.com> * doc strings Signed-off-by: Cary Phillips <cary@ilm.com> * Rename rgba param to separate_channels, off by default. Signed-off-by: Cary Phillips <cary@ilm.com> * remove numpy as build requirement Signed-off-by: Cary Phillips <cary@ilm.com> * Rename rgbaChannel() to channelNameToRGBA; rename is_* to objectTo* Signed-off-by: Cary Phillips <cary@ilm.com> * Add "deprecated" to InputFile/OutputFile/Imath doc strings Signed-off-by: Cary Phillips <cary@ilm.com> * Explicitly set python versions for wheel workflows, and exclude 3.13 Signed-off-by: Cary Phillips <cary@ilm.com> * resolve conflicts Signed-off-by: Cary Phillips <cary@ilm.com> * resolve conflict Signed-off-by: Cary Phillips <cary@ilm.com> --------- Signed-off-by: Cary Phillips <cary@ilm.com>
AcademySoftwareFoundation · Aug 13, 2024 · 84d7d52 · 84d7d52
1 parent be14297
commit 84d7d52
Show file tree

Hide file tree

Showing 23 changed files with 5,327 additions and 410 deletions.
diff --git a/.github/workflows/python-wheels-publish-test.yml b/.github/workflows/python-wheels-publish-test.yml
@@ -54,10 +54,12 @@ jobs:
         env:
           CIBW_ARCHS_LINUX: x86_64 
           CIBW_ARCHS_MACOS: x86_64 arm64 universal2
+          # Build Python 3.7 through 3.11.
           # Skip python 3.6 since scikit-build-core requires 3.7+
           # Skip 32-bit wheels builds on Windows
           # Also skip the PyPy builds, since they fail the unit tests
-          CIBW_SKIP: cp36-* *-win32 *_i686 pp* 
+          CIBW_BUILD: "cp37-* cp38-* cp39-* cp310-* cp311-*"
+          CIBW_SKIP: "*-win32 *_i686"
           CIBW_TEST_SKIP: "*-macosx_universal2:arm64"
           CIBW_ENVIRONMENT: OPENEXR_RELEASE_CANDIDATE_TAG="${{ github.ref_name }}"
 

diff --git a/.github/workflows/python-wheels-publish.yml b/.github/workflows/python-wheels-publish.yml
@@ -46,13 +46,14 @@ jobs:
         with:
           output-dir: wheelhouse
         env:
-          CIBW_BUILD: cp312-*
           CIBW_ARCHS_LINUX: x86_64 
           CIBW_ARCHS_MACOS: x86_64 arm64 universal2
+          # Build Python 3.7 through 3.11.
           # Skip python 3.6 since scikit-build-core requires 3.7+
           # Skip 32-bit wheels builds on Windows
           # Also skip the PyPy builds, since they fail the unit tests
-          CIBW_SKIP: cp36-* *-win32 *_i686 pp* 
+          CIBW_BUILD: "cp37-* cp38-* cp39-* cp310-* cp311-*"
+          CIBW_SKIP: "*-win32 *_i686"
           CIBW_TEST_SKIP: "*arm64"
 
       - name: Upload artifact

diff --git a/.github/workflows/python-wheels.yml b/.github/workflows/python-wheels.yml
@@ -17,13 +17,15 @@ on:
     branches-ignore:
       - RB-*
     paths:
+      - 'src/lib/**'
       - 'src/wrappers/python/**'
       - 'pyproject.toml'
       - '.github/workflows/python-wheels.yml'
   pull_request:
     branches-ignore:
       - RB-*
     paths:
+      - 'src/lib/**'
       - 'src/wrappers/python/**'
       - 'pyproject.toml'
       - '.github/workflows/python-wheels.yml'
@@ -33,7 +35,7 @@ permissions:
 
 jobs:
   build_wheels:
-    name: Python Wheels - ${{ matrix.os }} 
+    name: Python Wheels - ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
@@ -58,11 +60,14 @@ jobs:
         uses: pypa/cibuildwheel@bd033a44476646b606efccdd5eed92d5ea1d77ad # v2.20.0
         env:
           CIBW_ARCHS_MACOS: x86_64 arm64 universal2
+          # Build Python 3.7 through 3.11.
           # Skip python 3.6 since scikit-build-core requires 3.7+
           # Skip 32-bit wheels builds on Windows
           # Also skip the PyPy builds, since they fail the unit tests
-          CIBW_SKIP: cp36-* *-win32 *_i686 pp* 
+          CIBW_BUILD: "cp37-* cp38-* cp39-* cp310-* cp311-*"
+          CIBW_SKIP: "*-win32 *_i686"
           CIBW_TEST_SKIP: "*-macosx*arm64"
+          OPENEXR_TEST_IMAGE_REPO: "https://raw.githubusercontent.com/AcademySoftwareFoundation/openexr-images/main"
 
       - name: Upload artifact
         uses: actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a # v4.3.6
@@ -71,4 +76,3 @@ jobs:
           path: |
             ./wheelhouse/*.whl
             ./wheelhouse/*.tar.gz
-
diff --git a/pyproject.toml b/pyproject.toml
@@ -2,7 +2,7 @@
 # Copyright (c) Contributors to the OpenEXR Project.
 
 [build-system]
-requires = ["scikit-build-core==0.8.1"]
+requires = ["scikit-build-core==0.8.1", "pybind11"]
 build-backend = "scikit_build_core.build"
 
 [project]
@@ -67,6 +67,7 @@ CMAKE_POSITION_INDEPENDENT_CODE = 'ON'
 
 [tool.cibuildwheel]
 test-command = "pytest -s {project}/src/wrappers/python/tests"
+test-requires = ["numpy"]
 test-extras = ["test"]
 test-skip = ["*universal2:arm64"]
 build-verbosity = 1

diff --git a/share/ci/scripts/install_pybind11.sh b/share/ci/scripts/install_pybind11.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright Contributors to the OpenColorIO Project.
+
+set -ex
+
+PYBIND11_VERSION="$1"
+
+if [[ $OSTYPE == "msys" ]]; then
+    SUDO=""
+else
+    SUDO="sudo"
+fi
+
+git clone https://github.com/pybind/pybind11.git
+cd pybind11
+
+if [ "$PYBIND11_VERSION" == "latest" ]; then
+    LATEST_TAG=$(git describe --abbrev=0 --tags)
+    git checkout tags/${LATEST_TAG} -b ${LATEST_TAG}
+else
+    git checkout tags/v${PYBIND11_VERSION} -b v${PYBIND11_VERSION}
+fi
+
+mkdir build
+cd build
+cmake -DCMAKE_BUILD_TYPE=Release \
+      -DPYBIND11_INSTALL=ON \
+      -DPYBIND11_TEST=OFF \
+      ../.
+$SUDO cmake --build . \
+      --target install \
+      --config Release \
+      --parallel 2
+
+cd ../..
+rm -rf pybind11
diff --git a/src/lib/OpenEXR/ImfKeyCode.cpp b/src/lib/OpenEXR/ImfKeyCode.cpp
@@ -61,6 +61,18 @@ KeyCode::operator= (const KeyCode& other)
     return *this;
 }
 
+bool
+KeyCode::operator== (const KeyCode& other) const
+{
+    return (_filmMfcCode   == other._filmMfcCode &&
+            _filmType      == other._filmType &&
+            _prefix        == other._prefix &&
+            _count         == other._count &&
+            _perfOffset    == other._perfOffset &&
+            _perfsPerFrame == other._perfsPerFrame &&
+            _perfsPerCount == other._perfsPerCount);
+}
+
 int
 KeyCode::filmMfcCode () const
 {

diff --git a/src/lib/OpenEXR/ImfKeyCode.h b/src/lib/OpenEXR/ImfKeyCode.h
@@ -93,6 +93,8 @@ class IMF_EXPORT_TYPE KeyCode
     IMF_EXPORT
     KeyCode& operator= (const KeyCode& other);
 
+    bool operator== (const KeyCode& other) const;
+
     //----------------------------
     // Access to individual fields
     //----------------------------

diff --git a/src/lib/OpenEXR/ImfPreviewImage.h b/src/lib/OpenEXR/ImfPreviewImage.h
@@ -36,6 +36,11 @@ struct IMF_EXPORT_TYPE PreviewRgba
         unsigned char a = 255)
         : r (r), g (g), b (b), a (a)
     {}
+
+        bool operator==(const PreviewRgba& other) const
+        {
+            return r == other.r && g == other.g && b == other.b && a == other.a;
+        }
 };
 
 class IMF_EXPORT_TYPE PreviewImage

diff --git a/src/wrappers/python/CMakeLists.txt b/src/wrappers/python/CMakeLists.txt
@@ -10,10 +10,11 @@ if(NOT "${CMAKE_PROJECT_NAME}" STREQUAL "OpenEXR")
 endif()
 
 find_package(Python COMPONENTS Interpreter Development.Module REQUIRED)
+find_package(pybind11 CONFIG REQUIRED)
 
-python_add_library (PyOpenEXR MODULE OpenEXR.cpp)
+python_add_library (PyOpenEXR MODULE PyOpenEXR.cpp PyOpenEXR_old.cpp)
 
-target_link_libraries (PyOpenEXR PRIVATE "${Python_LIBRARIES}" OpenEXR::OpenEXR)
+target_link_libraries (PyOpenEXR PRIVATE "${Python_LIBRARIES}" OpenEXR::OpenEXR pybind11::headers)
 
 # The python module should be called "OpenEXR.so", not "PyOpenEXR.so",
 # but "OpenEXR" is taken as a library name by the main lib, so specify

diff --git a/src/wrappers/python/Imath.py b/src/wrappers/python/Imath.py
@@ -2,12 +2,12 @@
 # Copyright (c) Contributors to the OpenEXR Project.
 
 """
-:mod:`Imath` --- Support types for OpenEXR library
-==================================================
+:mod:`Imath` --- Deprecated component of OpenEXR
+================================================
 """
 
 class chromaticity(object):
-    """Store chromaticity coordinates in *x* and *y*."""
+    """This class is deprecated and will be removed in a future release"""
     def __init__(self, x, y):
         self.x = x
         self.y = y
@@ -17,7 +17,7 @@ def __eq__(self, other):
         return (self.x, self.y) == (other.x, other.y)
 
 class point(object):
-    """Point is a 2D point, with members *x* and *y*."""
+    """This class is deprecated and will be removed in a future release"""
     def __init__(self, x, y):
         self.x = x;
         self.y = y;
@@ -27,15 +27,15 @@ def __eq__(self, other):
         return (self.x, self.y) == (other.x, other.y)
 
 class V2i(point):
-    """V2i is a 2D point, with members *x* and *y*."""
+    """This class is deprecated and will be removed in a future release"""
     pass
 
 class V2f(point):
-    """V2f is a 2D point, with members *x* and *y*."""
+    """This class is deprecated and will be removed in a future release"""
     pass
 
 class Box:
-    """Box is a 2D box, specified by its two corners *min* and *max*, both of which are :class:`point` """
+    """This class is deprecated and will be removed in a future release"""
     def __init__(self, min = None, max = None):
         self.min = min
         self.max = max
@@ -45,18 +45,15 @@ def __eq__(self, other):
         return (self.min, self.max) == (other.min, other.max)
 
 class Box2i(Box):
-    """Box2i is a 2D box, specified by its two corners *min* and *max*."""
+    """This class is deprecated and will be removed in a future release"""
     pass
 
 class Box2f(Box):
-    """Box2f is a 2D box, specified by its two corners *min* and *max*."""
+    """This class is deprecated and will be removed in a future release"""
     pass
 
 class Chromaticities:
-    """
-    Chromaticities holds the set of chromaticity coordinates for *red*, *green*, *blue*, and *white*.
-    Each primary is a :class:`chromaticity`.
-    """
+    """This class is deprecated and will be removed in a future release"""
     def __init__(self, red = None, green = None, blue = None, white = None):
         self.red   = red
         self.green = green
@@ -79,47 +76,14 @@ def __eq__(self, other):
         return self.v == other.v
 
 class LineOrder(Enumerated):
-    """
-    .. index:: INCREASING_Y, DECREASING_Y, RANDOM_Y
-
-    LineOrder can have three possible values:
-    ``INCREASING_Y``,
-    ``DECREASING_Y``,
-    ``RANDOM_Y``.
-
-    .. doctest::
-    
-       >>> import Imath
-       >>> print Imath.LineOrder(Imath.LineOrder.DECREASING_Y)
-       DECREASING_Y
-    """
+    """This class is deprecated and will be removed in a future release"""
     INCREASING_Y = 0
     DECREASING_Y = 1
     RANDOM_Y	 = 2
     names = ["INCREASING_Y", "DECREASING_Y", "RANDOM_Y"]
 
 class Compression(Enumerated):
-    """
-    .. index:: NO_COMPRESSION, RLE_COMPRESSION, ZIPS_COMPRESSION, ZIP_COMPRESSION, PIZ_COMPRESSION, PXR24_COMPRESSION, B44_COMPRESSION, B44A_COMPRESSION, DWAA_COMPRESSION, DWAB_COMPRESSION,
-
-    Compression can have possible values:
-    ``NO_COMPRESSION``,
-    ``RLE_COMPRESSION``,
-    ``ZIPS_COMPRESSION``,
-    ``ZIP_COMPRESSION``,
-    ``PIZ_COMPRESSION``,
-    ``PXR24_COMPRESSION``,
-    ``B44_COMPRESSION``,
-    ``B44A_COMPRESSION``,
-    ``DWAA_COMPRESSION``,
-    ``DWAB_COMPRESSION``.
-
-    .. doctest::
-    
-       >>> import Imath
-       >>> print Imath.Compression(Imath.Compression.RLE_COMPRESSION)
-       RLE_COMPRESSION
-    """
+    """This class is deprecated and will be removed in a future release"""
     NO_COMPRESSION  = 0
     RLE_COMPRESSION = 1
     ZIPS_COMPRESSION = 2
@@ -136,40 +100,14 @@ class Compression(Enumerated):
     ]
 
 class PixelType(Enumerated):
-    """
-    .. index:: UINT, HALF, FLOAT
-
-    PixelType can have possible values ``UINT``, ``HALF``, ``FLOAT``.
-
-    .. doctest::
-    
-       >>> import Imath
-       >>> print Imath.PixelType(Imath.PixelType.HALF)
-       HALF
-    """
+    """This class is deprecated and will be removed in a future release"""
     UINT  = 0
     HALF  = 1
     FLOAT = 2
     names = ["UINT", "HALF", "FLOAT"]
 
 class Channel:
-    """
-    Channel defines the type and spatial layout of a channel.
-    *type* is a :class:`PixelType`.
-    *xSampling* is the number of X-axis pixels between samples.
-    *ySampling* is the number of Y-axis pixels between samples.
-
-    .. doctest::
-    
-       >>> import Imath
-       >>> print Imath.Channel(Imath.PixelType(Imath.PixelType.FLOAT), 4, 4)
-       FLOAT (4, 4)
-       >>> print Imath.Channel(Imath.PixelType.FLOAT, 4, 4)
-       Traceback (most recent call last):
-         ...
-       TypeError: type needs to be a PixelType.
-    """
-
+    """This class is deprecated and will be removed in a future release"""
     def __init__(self, type = PixelType(PixelType.HALF), xSampling = 1, ySampling = 1):
         self.type = type
         self.xSampling = xSampling
@@ -235,30 +173,7 @@ def __eq__(self, other):
         return self.__dict__ == other.__dict__
 
 class PreviewImage:
-    """
-    .. index:: RGBA, thumbnail, preview, JPEG, PIL, Python Imaging Library
-
-    PreviewImage is a small preview image, intended as a thumbnail version of the full image.
-    The image has size (*width*, *height*) and 8-bit pixel values are
-    given by string *pixels* in RGBA order from top-left to bottom-right.
-
-    For example, to create a preview image from a JPEG file using the popular 
-    `Python Imaging Library <http://www.pythonware.com/library/pil/handbook/index.htm>`_:
-
-    .. doctest::
-    
-       >>> import Image
-       >>> import Imath
-       >>> im = Image.open("lena.jpg").resize((100, 100)).convert("RGBA")
-       >>> print Imath.PreviewImage(im.size[0], im.size[1], im.tostring())
-       <Imath.PreviewImage instance 100x100>
-    """
-    def __init__(self, width, height, pixels):
-        self.width = width
-        self.height = height
-        self.pixels = pixels
-    def __repr__(self):
-        return "<Imath.PreviewImage instance %dx%d>" % (self.width, self.height)
+    """This class is deprecated and will be removed in a future release"""
 
 class LevelMode(Enumerated):
     ONE_LEVEL = 0