arduino-cli should provide "environments"

[s-celles]

Hello,

I wonder if the concept of "environment" (comparable to conda environments or Julia Pkg environments) have been considered (it could help to have several versions of a library installed without odd effects).

Here is a possible workflow for dealing with environment with arduino-cli :

Create an environment named myenv

$ arduino-cli env create myenv

It should create an empty directory name myenv in ~/Documents/Arduino/envs

List all existing environments

$ arduino-cli env list

Add packages to this environment

Latest version of a package

$ arduino-cli lib install WiFi101 --env myenv

A given version

$ arduino-cli lib install "ArduinoJson==5.13.4" --env myenv

(this will need #105 to be fixed)

(An environment could also be created directly from a file but this could probably be implemented later)

I assume core is installed in the global environment (because I think it's important to keep ability to have libraries and core installed in a shared manner)

$ arduino-cli compile --fqbn arduino:samd:mkr1000 Arduino/MyFirstSketch --env myenv

This last command will use ArduinoJson==5.13.4 even if, for example, ArduinoJson==6.x.y is installed in the "global" environment.

What is your opinion of such an idea?

Kind regards

[s-celles]

I assume core is installed in the global environment (because I think it's important to keep ability to have libraries and core installed in a shared manner)

Just an additional idea: libraries installed in an environment should "override" libraries installed in the global (shared) environment

[skandragon]

I faked this up using a Docker image per "environment" I want to use. This lets me be very explicit about what each build platform sees. It's a WIP, and not currently public, but if there is interest I can share.

I know others have similar base images, but for some reason I chose to make one from scratch.

[binaryben]

I'm not personally keen on environments, but having a system like the Node Package Manager (NPM) for managing libraries would be awesome. It could just use Github Repos to start with (no need to set up an official library registry... yet):

arduino-cli install --save @FastLED/FastLED

I would love to have a package.json file (or similar) though which lists required libraries, and then have the cli download them to the sketch directory as needed. You can also allow the user to install them globally like npm does, and that would fulfill @scls19fr's request.

EDIT:

Hold up. Libraries can be saved to the the ./src directory in a sketch directory. I could write a simple package manager cli utility that makes use of that fact. It would be a simple case of checking a package file and downloading the relevant code to ./src before calling arduino-cli separately. Is there a use case that having environments provides additional benefits that this possible solution wouldn't @scls19fr?

[masci]

+1 for using Docker, we're working on official releases for images shipping the CLI that should make this even easier.

You can already have something really close to an env by setting up ARDUINO_DATA_DIR and ARDUINO_SKETCHBOOK_DIR env vars to different values for different envs. You could even use Conda itself to manage such environments but any other dot-env system would work as well.

At this point we've no plan to add a sophisticated env manager with specific commands and stuff like global fallback but let's keep this issue open to collect feedback and alternative approaches.

[matthijskooijman]

#32 was closed, pointing to this issue for further discussion. The core of that issue was to be able to include libraries in the sketch directory, in order to make a sketch directory more self-contained and easier to distribute to others.

In this issue, a broader solution using environments is proposed, which serves a similar goal, but as @masci already says is quite a lot more sophisticated and thus less likely to be implemented.

Also, it seems that such an environment-bases approach needs more setup (e.g. it needs the user to explicitly create an environment and install libraries into it), while the simplier "libraries subdir in the sketch directory" approach is more implicit. The way I envison that, it only requires distributing library files along with the sketch, which will be used automatically, without any specific action required on the part of the user that compiles the sketch. Something similar could be achieved with an environment-based approach, but that does require some additional thought.

[federicobond]

I have been looking into this topic for a while in order to improve the tooling we use and get a nice reproducible environment for an Arduino project I am working on, and I think there is a key insight here:

You can already have something really close to an env by setting up ARDUINO_DATA_DIR and ARDUINO_SKETCHBOOK_DIR env vars to different values for different envs.

arduino-cli is 90% of the way there with regards to having usable separate environments. As long as you prefix each command with ARDUINO_SKETCHBOOK_DIR=., libraries are installed in ./libraries relative to the current directory and sketches are built using those libraries only. What would be really helpful is for the CLI to recognize that we are on a project laid out in such a way and set the sketchbook directory to the project root automatically.

This can be done by placing a special file in the project root directory. We could call this file project.json or similar. The most important thing this file does is to put the CLI into this new "project" mode so the sketchbook directory is configured appropriately and libraries are installed locally by default. We could also list those libraries and their versions in the file (so you can install them all easily by running a single command like arduino-cli install), set a default board fqbn (so you don't need to add the --fqbn arg to each compile program) and other default compile settings.

I think this solves a bunch of common problems previously reported by users and requires little change to the CLI. I would love to hear your opinions on it!

[c4deszes]

I started working on a CI/CD pipeline for embedded systems and Arduino seemed like an easy target to start with. What I'd like is a stateless environment, if the developer has arduino-cli he should be able to build the project with minimal setup required. What npm gets right is that it reduces everything down to 2-3 commands to fully build and deploy the project.

This change should allow at least install, compile and upload to be used without any parameters, all of them relying on the file present in the current workspace (like project.json, arduino-cli.json, etc.).

Install should pull all the cores and libraries necessary to build the project. You shouldn't have to specify which cores, rather potential targets.

Compile should create binaries for all target boards, right now the build output is straight into the project folder and upload depends on the output file name and location which is somewhat annoying.

Upload should try to upload to a currently available board if there's a build for it. If the target is ambiguous it should prompt a port or FQBN specification.

A draft of what the .json might look like.

{
	name: "ProjectName"
	libraries: {
		"FTDebouncer": "1.3.1"
	},
	targets: [
		"arduino:avr:uno",
		"arduino:samd:zero"
	],
	outputDir: "/Project/output"
}

So after these changes in a never before used environment all I need is:

1. pull repository
2. arduino-cli install
3. arduino-cli compile
4. arduino-cli upload

I want to dockerize the build but also allow local builds, if I have to change the scripts I'll have to change the Dockerfile, not to mention that I want to use Github Actions as well. So simplifying these commands would make the process consistent no matter what the build environment is.

[federicobond]

Apparently, there is already a way to attach a board target to a sketch via the arduino-cli board attach command, which generates a sketch.json file inside the sketch directory. This allows for per-sketch target board configuration. The targets defined in a sketch.json file should override any sketchbook global configuration.

[MacroYau]

I have been looking for similar things for a while too. Just implemented a proof-of-concept Dockerized arduino-cli compile that also enables per-project dependency (core and library versioning) management via a YAML file: https://github.com/MacroYau/arduino-cli-compile-docker

[netik]

When people have to resort to docker to solve this problem, we've failed as a community. Why can't the cli just support things the way C has been handling libraries for decades? INCLUDES= and LDFLAGS= would be a nice start, same goes for -I and -L.

The same goes for 'environments'. There is little to no reason to have environments if standard library support is permitted.

It really feels like a lot of the work on the arduino side of things is 'reinventing makefiles'.

[drstevenhale]

My solution to this is to leverage the existing options in arduino-cli.yaml, and the ARDUINO_DATA_DIR environment variable, as mentioned above. I use a makefile where "make env" installs the arduino-cli, and the specified core and libraries. Everything is downloaded and cached within the project, essentially making it an isolated environment. The entire pipeline is reduced to four commands. Clone, make env, make, make upload.

If anyone is interested, I've got a simple example here.

[ianfixes]

@c4deszes if you're looking for CI, check out arduino_ci or the accompanying Arduino CI github action (full disclosure, I maintain them).

@scls19fr could this be solved by placing multiple libraries/ directories in your filesystem? For example, if you did mkdir -p /path/to/environments/myenv/libraries and then put the following in an arduino-cli.yaml:

directories:
  user: /path/to/environments/myenv

I do this in the arduino_ci test suite to enable tests against dummy libraries.

[PhilippeBrissant]

In response to #32...
After a full day of researching and testing, i've got an ugly workaround! My idea was to simply copy my libraries (e.g. my_lib.h and my_lib.cpp files) to arduino-cli default folder (e.g. Arduino15/libraries). Here is my directory tree:

 My-Script:
 	My-Script.ino
 	my-libraries:
 		MyClock:
 			MyClock.cpp
 			MyClock.h
 		ohterlib:
 			ohterlib.cpp
 			ohterlib.h

First I need to get arduino-cli.yaml directories.user setting. To do that, I catched the parse_yaml function from Stefan Farestam's answer and make some changes around the printf() function (and idented the awk script). Then, I created a utils.sh and pasted the code there. Lastly I putted that file in ./utils/shell/utils.sh folder. Here is the full utils.sh file:

get_libraries_path() {
    local s='[[:space:]]*' w='[a-zA-Z0-9_]*' fs=$(echo @|tr @ '\034')
    sed -ne "s|^\($s\):|\1|" \
        -e "s|^\($s\)\($w\)$s:$s[\"']\(.*\)[\"']$s\$|\1$fs\2$fs\3|p" \
        -e "s|^\($s\)\($w\)$s:$s\(.*\)$s\$|\1$fs\2$fs\3|p"  $1 |
    awk -F$fs '{
        indent = length($1)/2;
        vname[indent] = $2;
        for (i in vname) {
            if (i > indent) {
                delete vname[i]
            }
        }
        if (length($3) > 0) {
            vn=""; 
            for (i=0; i<indent; i++) {
                vn=(vn)(vname[i])("_")
            }
            if(vn == "directories_"){
                if($2 == "user"){
                    gsub("\\\\","/",$3)
                    printf("%s/libraries", $3);                    
                }
            }
        }
    }'
}
# Allows to call a function based on arguments passed to the script
$*

This script just get the Arduino15 path and concatenates with "libraries" literal.
In my project, I have a Makefile to automate arduino-cli tasks. In that Makefile I created the following target which sends arduino-cli config dump result to tmp/conf file, applies my bash get_libraries_path() script and stores the result in tmp/lib_path. Lastly it copies my-libraries subfolders to lib_path:

install-my-libs:
	- $(shell if [ ! -d "tmp" ];then mkdir "tmp";fi)
	- $(shell plugins/arduino-cli/bin/arduino-cli config dump > tmp/conf)
	- $(shell ./utils/shell/utils.sh get_libraries_path tmp/conf > tmp/lib_path)
	- @cp -r src/My-Script/my-ibraries/* $(shell cat tmp/lib_path)

The $(shell ) command is necessary in the first 3 commands, due to order of processing of Make. If use @ instead, the first call to make install-my-libs will result in error. That happens because $(shell cat tmp/lib_path) (on 4th command) will be executed before @ commands. My arduino-cli is under plugins/arduino-cli/bin/ folder

With that I can include MyClock lib in My-Script.ino with <> (#include <MyClock.h>) and call arduino-cli compile. I developed it on my windows environment and tested in a debian docker container too. I use espressif boards, such as esp8266 and esp32.

[umbynos]

Actually, we implemented the sketch project files, that should support something really similar to your use case: https://arduino.github.io/arduino-cli/latest/sketch-project-file/

[PhilippeBrissant]

How can we point our own libraries in this yml file? We just point the absolute path to it?

[umbynos]

This is not possible. Because we implemented the sketch project files as a system to have a reproducible environment. Using an unpublished library wouldn't allow such thing.

[ubidefeo]

@PhilippeBrissant
while we digest input and try to slate developments over the next milestones,
I was wondering wether you couldn't simply use the --library /PATH/TO/LIBRARY/ parameter in the arduino-cli command.
I use this all the time to compile against libraries I have in development that I need to test

[PhilippeBrissant]

@ubidefeo
I haven't seen that in the docs hahah omg
After I read your reply, I found it!
The --libraries parameter worked for me! Thanks very much!!!

[ubidefeo]

@PhilippeBrissant
I'm very happy it did the job

[ubidefeo]

@PhilippeBrissant
remember that once you use --libraries you point to a folder containing multiple libraries.
--library allows you to point to a single library and can be used multiple times

arduino-cli compile -b arduino:avr:uno --library /data/GITHUB/ArduinoLibs/FTDebouncer -- library /data/GITHUB/ArduinoLibs/Tweakly

and so on :)

You can decide to have multiple libraries in a folder on your system and put them all there, but if you want to compile against a single library in development you're better off using --library

[ubidefeo]

As we continue grooming through issues, I'm coming back to this one and wonder if you have seen our support for profiles.
It is a really interesting way to support reproducible builds.

Also in the work the option to support locally sourced libraries, but we're still playing around with vendored content :)

https://arduino.github.io/arduino-cli/0.31/sketch-project-file/

[binaryben]

I'm coming back to this one and wonder if you have seen our support for profiles. It is a really interesting way to support reproducible builds.

Haven't really been looking at the Arduino ecosystem for a while but it's a promising development. I still think it needs a way to be able to download the libraries listed in the profiles like $ npm install would do in an NPM package. See my comment 4 years ago.

The ability to do this is arguably the number one reason Node.js took off like it did.

It also appears that there would be a lot of duplicates if multiple boards are being targeted that use the same libraries but have slightly different config options for whatever reason. Maybe include a way to extend a profile if there isn't the option already.

[cmaglie]

Haven't really been looking at the Arduino ecosystem for a while but it's a promising development. I still think it needs a way to be able to download the libraries listed in the profiles like $ npm install would do in an NPM package. See my comment 4 years ago.

The ability to do this is arguably the number one reason Node.js took off like it did.

Actually, the profiles can do all of this already.

It also appears that there would be a lot of duplicates if multiple boards are being targeted that use the same libraries but have slightly different config options for whatever reason. Maybe include a way to extend a profile if there isn't the option already.

This is not possible as of now, the simple workaround is to copy the common libraries in each board's profile. Anyway, if you feel that this feature is compelling, please open a dedicated issue so we can discuss and track it properly.

I'm closing this generic issue as completed since discussing single missing features in this "catch-all" issue may become quickly chaotic and not useful.