[feature] Unlimited Plugins Sets

[soif]

I would like to rewrite the way plugins are compiled or not.

• this would allow MORE (in fact as much as we like) plugins sets, for example a 'micro' one (to still fit on 1M boards)
• you would be able to directly include ALL plugins from the Playground, (more convenient to gather all in one single place, rather than in two repos), while keeping an isolation between stable and unstable plugins
• every one could be able to makes its own plugin set, mixing stable, testing or underdevelopment ones, without having to remove/add files in the repo as of today
• Some plugin specific libraries could be included ONLY whenever the plugin(s) needing it are activated
• Declaring a plugin as stable would be as simple as moving one define from the testing set to the stable set

I will code it , if you agree to add this feature

This could be as simple as a plugins_set.h listing

#ifdef PLUGINS_SET_STABLE
#define USE_P01
#define USE_P02
#define USE_P03
#define USE_P04
 #endif

#ifdef PLUGINS_SET_TESTING
#define USE_P101
#define USE_P102
#define USE_P103
#define USE_P104
 #endif

#ifdef PLUGINS_SET_EXPERIMENTAL
#define USE_P201
#define USE_P202
#define USE_P203
#define USE_P204
 #endif

each plugins would start by #ifdef USE_Pxxx

and espeasy.ino would have a #define PLUGINS_SET_STABLE above #USE_CUSTOM_H 😸

Then everyone could define its own sets of plugins inside custom.h.
And compiling would be as simple as adding -DPLUGINS_SET_MYHOME -DUSE_CUSTOM_H

Simple & convenient, dont break the current design (you just have to change some build flag for the automatic nightie releases.

What do you think ?

[TD-er]

I agree.
Also please check if excluding all modules using an external library also excludes the library itself.
Or else the size will not decrease much

[soif]

Excellent, so I'll certainly code that ... on mega or 2.0 ...whenever a decision is made in #997.

Nevertheless will certainly need some help to figure out the libraries used ONLY by the plugins themselves, vs those needed by the core

[soif]

anyway the goal is to have both micro sets (so remove as much lib as possible), but also huge sets to make tests.

[TD-er]

Maybe also add some predefined ones specific for some devices like Sonoff.
That would give people a start.
And maybe group per function?

[Grovkillen]

Group per function is a nice idea I think. We could try to identify a handful or more "typical use cases"?

[soif]

Maybe also add some predefined ones specific for some devices like Sonoff.

absolutely great suggestions, will do it...

And maybe group per function?

You mean Displays, inputs, etc ????

We could try to identify a handful or more "typical use cases"?

Ie: "sensor device", "relay device" etc... is that what you mean ?

BTW

• are there some plugins (or controller, or notifiers) that are mandatory required by the core?
• are there plugins that depends on each other? I mean : wont compil if one would be removed and not the other?

[Grovkillen]

Group	Name	ESP type	Flash size	Flash mode	Plugins enabled
SONOFF	Basic	8255	1M	DOUT	Switch, MQTT Import
	TH10/TH16	8255	1M	DOUT	Switch, MQTT Import
	POW	8255	1M	DOUT	Switch, MQTT Import
	S20	8255	1M	DOUT	Switch, MQTT Import
	4CH	8255	1M	DOUT	Switch, MQTT Import
	Touch	8255	1M	DOUT	Switch, MQTT Import
Environment	Easy Temperature	8266	4M	DIO	Dallas
	Easy Carbon dioxide	8266	4M	DIO	Senseair
Display	Easy Nextion	8266	4M	DIO	Nextion, Switch, MQTT Import
	Easy OLED	8266	4M	DIO	OLED Framed, Switch, MQTT Import
Relay	Easy Relay	8266	4M	DIO	Switch, MQTT Import
Generic	ESP8266	8266	1M	DOUT	"Minimal"
	ESP8255	8255	1M	DOUT	"Minimal"
	ESP8266	8266	4M	DIO	"All"

[TD-er]

Sonoff should at least have the Switch and perhaps the MQTT importer.
There are also some temp sensors for the Sonoff, sold by them.

@Grovkillen The Nextion already... no pressure then ;)

[Grovkillen]

Great, I update the list as we go on...

[Grovkillen]

@soif Yes, lets list all we can think of (at least commonly used groups of hardware).

[soif]

Excellent!

I can't wait to start coding this... if only #997 was cleared 😸

Also if my split settings idea was implemented, we could automatically include all needed tasks settings and rules as factory settings for each device.

How smart wouln't it be? 🎁 : The flexibility of ESPEASY with the KISS (KeepItStupidSimple) of Espurna!

BTW: As I dropped Arduino IDE developing for a while, and only use pio which is more permissive with file organisation, I have some questions if you want to maintain compatibility with Arduino IDE (?)

• Could plugins / controllers be moved to a separate folder, ie src/plugins + src/controllers
• Could plugins set be organized as follows :

src/
   plugins_sets/
       device_sonoff_pow/
              includes.h
              default_settings.json
              default_rules.txt
       global_stables/
              includes.h
              default_settings.json
              default_rules.txt
       global_testings/
              includes.h
              default_settings.json
              default_rules.txt

[TD-er]

Dropping support for Arduino IDE is a big decision.
That would deserve a separate issue + forum topic I guess.

There are lots of issues reported by users with the Arduino IDE, but I can also imagine people not really willing to be forced to change IDE.

I am not sure about the separation in folders. There is probably a lot overlap and I don't like overlap in code while not sharing it.

[Grovkillen]

Includeifs are the best I think...?

[soif]

I'm not willing to drop Arduino IDE support (except if YOU'd decided to), this is why I asked if better folder organization would keep Arduino IDE compatibility or not (?)

I am not sure about the separation in folders.

This is in the case that we can ALSO add factory settings and rules, else it's not needed at all for sets.

But for the plugins , if ALL (from playground) would be gathered , it would certainly not hurt (if this is Arduino IDE compatible) to move them, as well as controllers, into separated folders.

There is probably a lot overlap and I don't like overlap in code while not sharing it.

Sorry I don't understand what you mean.

[s0170071]

👍 for plugin sets. For the environment I'ld like to see the SHT temperature and humidity sensors added.
I have an additional usage in mind: power management: reading smart meters and SMA inverters. But that may be just me as both plugins are "only" playground so far.

👎 for abandoning the Arduino IDE. As a matter of fact it is not even working right out of the repo because of that src folder. If PIO is so flexible, can we please rename that src folder to ESPEasy ?

[soif]

But that may be just me as both plugins are "only" playground so far.

With Sets correctly implemented, I would hope that playground would be deprecated, and that ALL plugins could be gathered into the main espeasy repo. (this would certainly improve the number of plugins authors/contributors)

BTW if the deeper folder organization would be possible (with Arduino IDE compat) we could think about a bit more organized way of publishing plugins, that would pave the way to fully automate wiki publication of the plugins manual, TOC, etc... ie:

src/
   plugins/
            P01/
               P01.ino
               info.json 
               readme.md
               medias/
            P02/
               P02.ino
               info.json 
               readme.md
               medias/

• info.json would contains : author, version, description, link, etc..... automatically parsed/published into the github wiki TOC
• readme.md would be the full manual, automatically published into the github wiki
• medias/ contains images needed in the readme.md

--> less work for @Grovkillen, better documentation than in the comment of the plugins. The maintainer would just have to request plugins authors to fill the info/readme to get their plugins merged! Then anyone could enhance the documentation with a PR (or a simple edit for a basic user, on GH) . bye bye MediaWiki, at least for plugins, and welcome to community driven documentation. 💝

If PIO is so flexible, can we please rename that src folder to ESPEasy ?

It should be as simple as changing this in platformio.ini

[platformio]
src_dir     = ESPEasy

or may be

[platformio]
src_dir     = ESPEasy/

Flexible, isn't it ? 🎉

[Grovkillen]

Now we're talking! Imagine if I could go for creating a Wiki on Github that could be included in each release!

[Grovkillen]

I work with documentation in my daily job and are using LaTeX as my programming language. It would be awesome if I could use that for this project. I could then make template files that each plugin/controller/core feature etc. uses and we could automatically generate a PDF manual for each release. Just as we do with the bin files today...

Note to self: https://github.com/jackolney/travis-ci-latex-pdf and https://tex.stackexchange.com/questions/398830/how-to-build-my-latex-automatically-using-travis-ci

[TD-er]

@Grovkillen A lot of open source projects now use Sphinx documentation.
See ReadTheDocs for some project examples.

[Grovkillen]

Could that be integrated in our project, I mean, hosted at GitHub?

EDIT: I see that you can.

[soif]

Without directly generating PDF 😜 , the current github wiki offer options, isn't it? and in the meantime making a Readme.md for each developer is absolutely effort-less (no more than writing comments in the code header), just code convention/requests. And this is heavily script-able, hence no more painful work for you.

A lot of open source projects now use Sphinx documentation. See ReadTheDocs for some project examples.

I'm not that used as a developer with ReadTheDocs, but well as a User.
This is certainly a very STRONG option.
I vote for it if, it is not that difficult to impose to plugins developers (easy to do/fast to learn = will be adopted by most).

Even being a bit like a developer 🙄 , I miss a LOT of documentation in the current Wiki (Not your fault of course), while information is spread between the Wiki, the code comments, the forums and some trials and errors)....
Trying to standardize all of this would certainly not solve all the issues, but might really help early adopters as well as some senior users. And this should also in short term LOWER the support enquiries 👓

soif,
(Always tring to find tips to lower the support time in order to improve the true development time....)

[uzi18]

+1 for rename src dir into ESPEasy

[uzi18]

@soif don't You think it will break all repository stuff, cherry-pick between branches will be impossible, etc.

We can think about custom configuration for some targets, eg.: flash 32Mb -> 20tasks -> enable some additional plugins.
Move all globals from EspEasy.ino to header file espeasy_global.h and there conditionally include targets configurations.

Putting all plugins into one repository is a good idea, but first we have some serious issues with memory/webserver/SDK 2.4.0 etc.

[TD-er]

I think we should do these changes in little steps.
Roughly:

• Split used parts in logical blocks to create different builds, just like suggested in the start post of this thread.
• Changing documentation is a separate topic (and an important one too)
• Big files, like all the structs in ESPEasy.ino, all functions in Misc.ino. etc. should be split into separate files. Perhaps even .h/.cpp to be able to use debuggers.
• Split overlap/duplication in code into a separate piece of code.
• Make the ESPeasy core more intuitive to use (again documentation) and more stable.
• Make plugins as basic as possible, just describing the specifics of the device. The rest should be done in 'shared' code for pre- and post-processing.
• Try to remove the strict timing and process things in little steps as they happen. Let the data flow decide when things should happen. Consider each next step for this flow as a "block", which should be small (in duration)
• Remove the web-page generation from the C++ code and let static HTML + CSS + Javascript do it in the browser and the data be managed through REST calls.
• Make the best possible tool using these low-end devices.
• Enter world peace.

[soif]

@TD-er obviously, first of all is :

Enter world peace.

then "do these changes in little steps, like suggested in the start post" to at start improve plugins sets..

Documentation and settings were some digressions, I will just concentrate on implementing Plugins Sets (once #997 is cleared) . sorry about that 😶

We could also discuss/think (in other threads) about

• improving the documentation process
• splitting settings (if you like to), and once JSONized
• rationalize the core code, while lightening the plugins code
• separate the Logic/Controller from the View in the web server, to better conform to the MVC design pattern.

@uzi18

• cherry-pick or gathering all plugins etc.... issues are related to the mess discussed in V2.0 and mega branches too far from each other #997, once cleared I don't see any problem adding "foreign" plugins, as soon as they merge correctly at first
• then Sets would also help to confine plugins into stable , testing, dev, or whatever group is needed. Device sets (Ie Sonoff xxx) would certainly fastly become rock solid because using a very limited set of plugins!!!

[soif]

here is a draft to work on, if you wish

[TD-er]

The bulk of changes is now merged into Mega branch. See #1182
And already this has proven to be useful, since the plugin for the Sonoff Pow can now be built.

This issue still has a lot of useful suggestions to do, so I'll leave it open, while all related PR's now are closed or merged.

[Oxyandy]

@Grovkillen from the table above, you list 8255, all sonoff basics I have seen are 8266
not sure if this is critical or not ? Most sonoffs use 8266, but it is true some sonoff products use 8255..
Do you need me to confirm, cross check & verify each type ?
Cheers
Group | Name | ESP type | Flash size | Flash mode | Plugins enabled
SONOFF | Basic | 8255 | 1M | DOUT | Switch, MQTT Import

[Grovkillen]

The difference between 8266 and 8255 when it comes to the FW way of looking at it is not that big. Essentially it's the DOUT and flash size that needs to be taken care of. Rest is the same.

[Grovkillen]

So you can flash 8255 into a 8266 unit but not the other way around.

[Grovkillen]

I will close this, open if its still a valid issue.