ulab is a numpy-like array manipulation library for micropython and CircuitPython.
The module is written in C, defines compact containers (ndarrays) for numerical data of one to four
dimensions, and is fast. The library is a software-only standard micropython user module,
i.e., it has no hardware dependencies, and can be compiled for any platform. 8-, and 16-bit signed
and unsigned integer dtypes, as well as float, and, optionally, complex are supported.
The float implementation of micropython (32-bit float, or 64-bit double) is automatically
detected and handled.
- Supported functions and methods
- Usage
- Finding help
- Benchmarks
- Firmware
- Issues, contributing, and testing
ulab implements numpy's ndarray with the ==, !=, <, <=, >, >=, +, -, /, *, **,
+=, -=, *=, /=, **= binary operators, and the len, ~, -, +, abs unary operators that
operate element-wise. Type-aware ndarrays can be initialised from any micropython iterable, lists of
iterables via the array constructor, or by means of the arange, concatenate, diag, eye,
frombuffer, full, linspace, logspace, ones, or zeros functions.
ndarrays can be sliced, and iterated on, and have a number of their own methods, and properties, such as flatten(), itemsize, reshape(),
shape, size, strides, tobytes(), tolist(), and transpose() and T. If the firmware is compiled with complex support,
the imag, and real properties are automatically included.
In addition, ulab includes universal functions, many numpy functions, and functions from the numpy.fft, numpy.linalg, numpy.random, scipy.linalg, scipy.optimize, scipy.signal, and scipy.special modules. A complete list of available routines can be found under micropython-ulab.
The utils module contains functions for
interfacing with peripheral devices supporting the buffer protocol. These functions do not have an obvious
numpy equivalent, but share a similar programming interface, and allow direct data input-output between
numerical arrays and hardware components.
User-defined functions operating on numerical data can easily be added via the user module. This allows for transparent extensions, without having to change anything in the core. Hints as to how to work with ndarrays at the C level can be found in the programming manual.
ulab sports a numpy/scipy-compatible interface, which makes porting of CPython code straightforward. The following
snippet should run equally well in micropython, or on a PC.
try:
from ulab import numpy
from ulab import scipy
except ImportError:
import numpy
import scipy.special
x = numpy.array([1, 2, 3])
scipy.special.erf(x)Documentation can be found on readthedocs under micropython-ulab, as well as at circuitpython-ulab. A number of practical examples are listed in Jeff Epler's excellent circuitpython-ulab overview. The tricks chapter of the user manual discusses methods by which RAM and speed can be leveraged in particular numerical problems.
Representative numbers on performance can be found under ulab samples.
Pre-built, and up-to-date firmware files for select platforms can be downloaded from micropython-builder.
If flash space is a concern, unnecessary functions can be excluded from the compiled firmware with
pre-processor switches. In addition, ulab also has options for trading execution speed for firmware size.
A thorough discussion on how the firmware can be customised can be found in the
corresponding section
of the user manual.
ulab is also included in the following compiled micropython variants and derivatives:
CircuitPythonfor SAMD51 and nRF microcontrollers https://github.com/adafruit/circuitpythonMicroPython for K210https://github.com/loboris/MicroPython_K210_LoBoMaixPyhttps://github.com/sipeed/MaixPyOpenMVhttps://github.com/openmv/openmvpimoroni-picohttps://github.com/pimoroni/pimoroni-pico
If you want to try the latest version of ulab on micropython or one of its forks, the firmware can be compiled
from the source by following these steps:
Simply clone the ulab repository with
git clone https://github.com/v923z/micropython-ulab.git ulaband then run
./build.sh [matrix.dims] # Dimensions is 2 by defaultThis command will clone micropython, and build the unix port automatically, as well as run the test scripts. If you want an interactive unix session, you can launch it in
ulab/micropython/ports/unixFirst, you have to clone the micropython repository by running
git clone https://github.com/micropython/micropython.giton the command line. This will create a new repository with the name micropython. Staying there, clone the ulab repository with
git clone https://github.com/v923z/micropython-ulab.git ulabIf you don't have the cross-compiler installed, your might want to do that now, for instance on Linux by executing
sudo apt-get install gcc-arm-none-eabiIf this step was successful, you can try to run the make command in the port's directory as
make BOARD=PYBV11 USER_C_MODULES=../../../ulab allwhich will prepare the firmware for pyboard.v.11. Similarly,
make BOARD=PYBD_SF6 USER_C_MODULES=../../../ulab allwill compile for the SF6 member of the PYBD series. If your target is unix, you don't need to specify the BOARD parameter.
Provided that you managed to compile the firmware, you would upload that by running either
dfu-util --alt 0 -D firmware.dfuor
python pydfu.py -u firmware.dfuIn case you got stuck somewhere in the process, a bit more detailed instructions can be found under https://github.com/micropython/micropython/wiki/Getting-Started, and https://github.com/micropython/micropython/wiki/Pyboard-Firmware-Update.
ulab can be tested on the ESP32 in wokwi's micropython emulator without having to compile the C code. This utility also offers the possibility to save and share your micropython code.
Firmware for Espressif hardware can be built in two different ways, which are discussed in the next two paragraphs. A solution for issues with the firmware size is outlined in the last paragraph of this section.
Beginning with version 1.15, micropython switched to cmake on the ESP32 port. If your operating system supports CMake > 3.12, you can either simply download, and run the single build script, or follow the step in this section. Otherwise, you should skip to the next one, where the old, make-based approach is discussed.
In case you encounter difficulties during the build process, you can consult the (general instructions for the ESP32)[https://github.com/micropython/micropython/tree/master/ports/esp32#micropython-port-to-the-esp32].
First, clone the ulab, the micropython, as well as the espressif repositories:
export BUILD_DIR=$(pwd)
git clone https://github.com/v923z/micropython-ulab.git ulab
git clone https://github.com/micropython/micropython.git
cd $BUILD_DIR/micropython/
git clone -b v4.0.2 --recursive https://github.com/espressif/esp-idf.git
Also later releases of esp-idf are possible (e.g. v4.2.1).
Then install the ESP-IDF tools:
cd esp-idf
./install.sh
. ./export.shNext, build the micropython cross-compiler, and the ESP sub-modules:
cd $BUILD_DIR/micropython/mpy-cross
make
cd $BUILD_DIR/micropython/ports/esp32
make submodulesAt this point, all requirements are installed and built. We can now compile the firmware with ulab. In $BUILD_DIR/micropython/ports/esp32 create a makefile with the following content:
BOARD = GENERIC
USER_C_MODULES = $(BUILD_DIR)/ulab/code/micropython.cmake
include MakefileYou specify with the BOARD variable, what you want to compile for, a generic board, or TINYPICO (for micropython version >1.1.5, use UM_TINYPICO), etc. Still in $BUILD_DIR/micropython/ports/esp32, you can now run make.
If your operating system does not support a recent enough version of CMake, you have to stay with micropython version 1.14. The firmware can be compiled either by downloading and running the build script, or following the steps below:
First, clone ulab with
git clone https://github.com/v923z/micropython-ulab.git ulaband then, in the same directory, micropython
git clone https://github.com/micropython/micropython.gitAt this point, you should have ulab, and micropython side by side.
With version 1.14, micropython switched to cmake on the ESP32 port, thus breaking compatibility with user modules. ulab can, however, still be compiled with version 1.14. You can check out a particular version by pinning the release tag as
cd ./micropython/
git checkout tags/v1.14
Next, update the submodules,
git submodule update --init
cd ./mpy-cross && make # build cross-compiler (required)and find the ESP commit hash
cd ./micropython/ports/esp32
make ESPIDF= # will display supported ESP-IDF commit hashes
# output should look like: """
# ...
# Supported git hash (v3.3): 9e70825d1e1cbf7988cf36981774300066580ea7
# Supported git hash (v4.0) (experimental): 4c81978a3e2220674a432a588292a4c860eef27bChoose an ESPIDF version from one of the options printed by the previous command:
ESPIDF_VER=9e70825d1e1cbf7988cf36981774300066580ea7In the micropython directory, create a new directory with
mkdir esp32Your micropython directory should now look like
ls
ACKNOWLEDGEMENTS CONTRIBUTING.md esp32 lib mpy-cross README.md
CODECONVENTIONS.md docs examples LICENSE ports tests
CODEOFCONDUCT.md drivers extmod logo py toolsIn ./micropython/esp32, download the software development kit with
git clone https://github.com/espressif/esp-idf.git esp-idf
cd ./esp-idf
git checkout $ESPIDF_VER
git submodule update --init --recursive # get idf submodules
pip install -r ./requirements.txt # install python reqsNext, still staying in ./micropython/eps32/esd-idf/, install the ESP32 compiler. If using an ESP-IDF version >= 4.x (chosen by $ESPIDF_VER above), this can be done by running . $BUILD_DIR/esp-idf/install.sh. Otherwise, for version 3.x, run the following commands in in .micropython/esp32/esp-idf:
# for 64 bit linux
curl https://dl.espressif.com/dl/xtensa-esp32-elf-linux64-1.22.0-80-g6c4433a-5.2.0.tar.gz | tar xvz
# for 32 bit
# curl https://dl.espressif.com/dl/xtensa-esp32-elf-linux32-1.22.0-80-g6c4433a-5.2.0.tar.gz | tar xvz
# don't worry about adding to path; we'll specify that later
# also, see https://docs.espressif.com/projects/esp-idf/en/v3.3.2/get-started for more infoFinally, build the firmware:
cd ./micropython/ports/esp32
# temporarily add esp32 compiler to path
export PATH=../../esp32/esp-idf/xtensa-esp32-elf/bin:$PATH
export ESPIDF=../../esp32/esp-idf # req'd by Makefile
export BOARD=GENERIC # options are dirs in ./boards
export USER_C_MODULES=../../../ulab # include ulab in firmware
make submodules & make allIf it compiles without error, you can plug in your ESP32 via USB and then flash it with:
make erase && make deployWhen selecting BOARD=TINYPICO, the firmware is built but fails to deploy, because it is too large for the standard partitions. We can rectify the problem by creating a new partition table. In order to do so, in $BUILD_DIR/micropython/ports/esp32/, copy the following 8 lines to a file named partitions_ulab.cvs:
# Notes: the offset of the partition table itself is set in
# $ESPIDF/components/partition_table/Kconfig.projbuild and the
# offset of the factory/ota_0 partition is set in makeimg.py
# Name, Type, SubType, Offset, Size, Flags
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 0x200000,
vfs, data, fat, 0x220000, 0x180000,
This expands the factory partition by 128 kB, and reduces the size of vfs by the same amount. Having defined the new partition table, we should extend sdkconfig.board by adding the following two lines:
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_ulab.csv"
This file can be found in $BUILD_DIR/micropython/ports/esp32/boards/TINYPICO/. Finally, run make clean, and make. The new firmware contains the modified partition table, and should fit on the microcontroller.
RP2 firmware can be compiled either by downloading and running the single build script/build script for Pico W, or executing the commands below.
First, clone micropython:
git clone https://github.com/micropython/micropython.gitThen, setup the required submodules:
cd micropython
git submodule update --init lib/tinyusb
git submodule update --init lib/pico-sdk
cd lib/pico-sdk
git submodule update --init lib/tinyusbYou'll also need to compile mpy-cross:
cd ../../mpy-cross
makeThat's all you need to do for the micropython repository. Now, let us clone ulab (in a directory outside the micropython repository):
cd ../../
git clone https://github.com/v923z/micropython-ulab ulabWith this setup, we can now build the firmware. Back in the micropython repository, use these commands:
cd ports/rp2
make USER_C_MODULES=/path/to/ulab/code/micropython.cmakeIf micropython and ulab were in the same folder on the computer, you can set USER_C_MODULES=../../../ulab/code/micropython.cmake. The compiled firmware will be placed in micropython/ports/rp2/build.
Adafruit Industries always include a relatively recent version of ulab in their nightly builds. However, if you really need the bleeding edge, you can easily compile the firmware from the source. Simply clone circuitpython, and move the commit pointer to the latest version of ulab (ulab will automatically be cloned with circuitpython):
git clone https://github.com/adafruit/circuitpython.git
cd circuitpyton/extmod/ulab
# update ulab here
git checkout master
git pullYou might have to check, whether the CIRCUITPY_ULAB variable is set to 1 for the port that you want to compile for. You find this piece of information in the make fragment:
circuitpython/ports/port_of_your_choice/mpconfigport.mkAfter this, you would run make with the single BOARD argument, e.g.:
make BOARD=mini_sam_m4If you find a problem with the code, please, raise an issue! An issue should address a single problem, and should contain a minimal code snippet that demonstrates the difference from the expected behaviour. Reducing a problem to the bare minimum significantly increases the chances of a quick fix.
Feature requests (porting a particular function from numpy or scipy) should also be posted at ulab issue.
Contributions of any kind are always welcome. If you feel like adding to the code, you can simply issue a pull request. If you do so, please, try to adhere to micropython's coding conventions.
However, you can also contribute to the documentation (preferably via the jupyter notebooks, or improve the tests.
If you decide to lend a hand with testing, here are the steps:
- Write a test script that checks a particular function, or a set of related functions!
- Drop this script in one of the folders in ulab tests!
- Run the ./build.sh script in the root directory of
ulab! This will clone the latestmicropython, compile the firmware forunix, execute all scripts in theulab/tests, and compare the results to those in the expected results files, which are also inulab/tests, and have an extension.exp. In case you have a new snippet, i.e., you have no expected results file, or if the results differ from those in the expected file, a new expected file will be generated in the root directory. You should inspect the contents of this file, and if they are satisfactory, then the file can be moved to theulab/testsfolder, alongside your snippet.