02 How to understand and modify the software

Getting started

OMOTE can be used with almost any device that can be controlled with

• IR signals (infrared)
• MQTT messages
• a BLE keyboard

The software comes with a predefined set of devices that can be taken as starting point.

device	what is it used for	communication	remark
device_samsungTV	TV showing the picture	IR (SAMSUNG)	full set of IR commands for Samsung UE32EH5300
device_yamahaAmp	AV receiver	IR (NEC)	full set of IR commands for Yamaha RX-V359
device_appleTV	Apple TV media player	IR (SONY)	only a small set of IR commands
device_smarthome	control your smart home with MQTT	MQTT	example on how to switch and set brightness of a bulb
device_keyboard_ble	BLE keyboard that can be connected to a media player	BLE	can be used to connect to e.g. a Fire TV or anything else that can be controlled with an BLE keyboard
device_keyboard_mqtt	a special "keyboard" that can send MQTT messages	MQTT	Can be used together with another ESP32 emulating an USB keyboard that can be connected to a media player with an USB port. Use this if you can't or don't want to connect your media player with BLE to the OMOTE (BLE needs lot of resources on the OMOTE)

The devices above are used for the demo application. Commands of all these devices are either used by the gui or by hardware keys.

In folder /src/devices_pool there are more devices. If you want to use them, please read below in section "How To".

Devices included in folder /src/devices_pool are (list is probably not complete):

category	device
TV	LG 42LA6208-ZA
AV receiver	Denon AVR-S660H LG SOUNDBAR Remote AKB73575421
media player	LG BLURAY Remote AKB73896401 SAMSUNG BLURAY AKB73896401 NVIDIA SHIELD TV
misc	Air Conditioner DeLonghi PAC N81

What is the minimum you have to do?

If you simply want to use this setup and to see how it looks like, then

• set the correct hardware revision of the PCB you are using. This is done by setting -D OMOTE_HARDWARE_REV=x in platformio.ini. The hardware revision is printed on your PCB. Hardware rev 4 was published in May 2024.
• copy file secrets_override_example.h to secrets_override.h
• put in this file your WiFi credentials and the address of your MQTT broker
• if you want to use WiFi, check if -D ENABLE_WIFI_AND_MQTT=1 is set in platformio.ini
• if you want to use the BLE keyboard, check if -D ENABLE_BLUETOOTH=1 and -D ENABLE_KEYBOARD_BLE=1 is set in platformio.ini
• compile the firmware and upload it to your OMOTE

If you want to understand how to adjust the software to your needs, please continue reading the next sections.

Basic concepts

Devices

In OMOTE, devices are described with their capabilities they can provide. A device can be anything that can be controlled via IR signals, MQTT messages or the BLE keyboard provided by OMOTE.

• a TV
• an AV receiver
• media players like DVD player, Apple TV, etc.
• smart home devices that can be controlled with MQTT messages
• the BLE keyboard is a special device that can be used to control any device which is paired to the OMOTE, e.g. a Fire TV. Such devices don't need to be defined in OMOTE, they simply have to be paired to the BLE keyboard.
  If you want to connect more than one device via BLE (e.g. a Fire TV and a NVIDIA Shield TV), please read carefully section pairing BLE devices.
  If you have trouble to connect more than one BLE device, you could use the MQTT keyboard instead to connect e.g. to a Fire TV. You can have as many MQTT keyboards as you want.

Each device definition is placed in a separate folder. The definition of a device is simply a list of commands a device can receive.

• the *.h file lists all available commands (e.g. Volume up, Power toggle etc.)
• the *.cpp file needs to provide a function that is called in main.cpp to register this device to be available at runtime.

A lot of devices can be available in your source code, but they are only active at runtime if e.g. register_device_samsung(); is called in function setup() in main.cpp.

All devices in folder /src/devices_pool won't get compiled by default. You first have to move them to folder /src/devices if you want to use them.

Take care: every device you register needs memory. Memory is limited, so don't register all available devices.

Commands

As explained in section "devices", all devices can be registered with the commands they can receive. Wherever you are in code, a command like executeCommand(SAMSUNG_POWER_OFF); can be send. The parameter can be any command that was registered by any device.

Commands can also be used for switching scenes or to do some special purposes. Both will be explained later.

Scenes

Scenes are made up of a set of devices, selecting the correct inputs for each device and other things. In most cases scenes are something like:

• watching TV
• watching DVD
• watching Apple TV, Fire TV or Google TV

For each scene you can define:

• a start sequence to start up all the devices needed for that scene
• a stop sequence that is executed if you leave a scene
• a keymap that defines which hardware keys on the OMOTE will send which commands

The software comes with a predefined set of scenes:

scene	what is it used for
TV	turn on TV and AV receiver, set correct input of TV and receiver
Fire TV	turn on TV and AV receiver, set correct input of TV and receiver, hit HOME on BLE keyboard to bring Fire TV to the start page
Chromecast	turn on TV and AV receiver, set correct input of TV and receiver
Apple TV	turn on TV and AV receiver, set correct input of TV and receiver
Off	turn off all devices

By default, the four scenes are assigned to the four hardware keys at the bottom of the OMOTE. The scene 'off' is selected with the power button in the upper right corner of the OMOTE.

Programmatically, a scene can be selected e.g. with executeCommand(SCENE_FIRETV);

Logitech Harmony users are used to know "Activities", which is something very similar.

User frontends

There are two kind of user frontends available in OMOTE: hardware keys and the touchscreen.

Hardware keys

All hardware keys can be fully customized. They can send any command registered by a device, can switch a scene or can be bind to a special command that can do anything you want to do in code.

For each key you can define a command that is executed if a key is short pressed, and another command (if wanted) that is executed if a key is long pressed. And you can define if the command for a short press is repeated if the key is hold.

enum repeatModes {
  // only as fallback
  REPEAT_MODE_UNKNOWN,
  // if you short press or hold a key on the keypad, only one single command from keyCommands_short is sent
  // -> best used if you do not want a command to be sent more than once, even if you press the key (too) long, e.g. when toggling power
  SHORT,
  // if you hold a key on the keypad, a command from keyCommands_short is sent repeatedly
  // -> best used e.g. for holding the key for "volume up"
  SHORT_REPEATED,
  // if you short press a key, a command from keyCommands_short is sent once.
  // if you hold a key on the keypad, a command from keyCommands_long is sent (no command from keyCommands_short before)
  // -> best used if a long key press should send a different command than a short press
  SHORTorLONG,
};

A default definition for the hardware keys is provdided in scenes/scene__defaultKeys.cpp and looks like this:

  key_repeatModes_default = {
                                                                                                             {KEY_OFF,   SHORT            },
    {KEY_STOP,  SHORT            },    {KEY_REWI,  SHORTorLONG      },    {KEY_PLAY,  SHORT            },    {KEY_FORW,  SHORTorLONG      },
    {KEY_CONF,  SHORT            },                                                                          {KEY_INFO,  SHORT            },
                                                         {KEY_UP,    SHORT            },
                      {KEY_LEFT,  SHORT            },    {KEY_OK,    SHORT            },    {KEY_RIGHT, SHORT            },
                                                         {KEY_DOWN,  SHORT            },
    {KEY_BACK,  SHORT            },                                                                          {KEY_SRC,   SHORT            },
    {KEY_VOLUP, SHORT_REPEATED   },                      {KEY_MUTE,  SHORT            },                     {KEY_CHUP,  SHORT            },
    {KEY_VOLDO, SHORT_REPEATED   },                      {KEY_REC,   SHORT            },                     {KEY_CHDOW, SHORT            },
    {KEY_RED,   SHORT            },    {KEY_GREEN, SHORT            },    {KEY_YELLO, SHORT            },    {KEY_BLUE,  SHORT            },
  };
  
  key_commands_short_default = {
                                                                                                             {KEY_OFF,   SCENE_ALLOFF_FORCE},
  /*{KEY_STOP,  COMMAND_UNKNOWN  },    {KEY_REWI,  COMMAND_UNKNOWN  },    {KEY_PLAY,  COMMAND_UNKNOWN  },    {KEY_FORW,  COMMAND_UNKNOWN  },*/
  /*{KEY_CONF,  COMMAND_UNKNOWN  },                                                                          {KEY_INFO,  COMMAND_UNKNOWN  },*/
                                                     /*  {KEY_UP,    COMMAND_UNKNOWN  },*/
                   /* {KEY_LEFT,  COMMAND_UNKNOWN  },    {KEY_OK,    COMMAND_UNKNOWN  },    {KEY_RIGHT, COMMAND_UNKNOWN  },*/
                                                     /*  {KEY_DOWN,  COMMAND_UNKNOWN  },*/
    {KEY_BACK,  SCENE_SELECTION  },                                                                        /*{KEY_SRC,   COMMAND_UNKNOWN  },*/
    {KEY_VOLUP, YAMAHA_VOL_PLUS  },                      {KEY_MUTE,  YAMAHA_MUTE_TOGGLE},                  /*{KEY_CHUP,  COMMAND_UNKNOWN  },*/
    {KEY_VOLDO, YAMAHA_VOL_MINUS },                   /* {KEY_REC,   COMMAND_UNKNOWN  },*/                 /*{KEY_CHDOW, COMMAND_UNKNOWN  },*/
    {KEY_RED,   SCENE_TV_FORCE   },    {KEY_GREEN, SCENE_FIRETV_FORCE},  {KEY_YELLO, SCENE_CHROMECAST_FORCE},{KEY_BLUE,  SCENE_APPLETV_FORCE},
  };

Remark: you can both use the command COMMAND_UNKNOWN or simply comment out the definition to have no definition for a key.

For each scene, you can override the key definitions. Have a look in files scenes/scene_allOff.cpp, scenes/scene_TV.cpp etc. for an example. In a scene file, you can override none of the keys, only some, or all.

The same is possible for GUIs. They can also override key definitions as long as the GUI is shown. Have a look in file gui_smarthome.cpp how it is done.

At runtime, a GUI specific definition is used first. If not available, the scene specific definition is used. If not available, the default definition is taken. If no default definition is available, then no command is executed.

Touchscreen

The touchscreen can also be used to get status information about devices or to send commands. The touchscreen is organized in so called tabs. You can swipe left or right to go to the next tab.

If you want a specific action to happen, simply call the commandHandler like executeCommand(SAMSUNG_POWER_OFF); in the callback function of a GUI element (button, slider etc.). See one of the examples on how to do it, e.g. devices/misc/device_smarthome/gui_smarthome.cpp.

You can trigger scene specific commands based on the currently active scene from the gui. You know at runtime which scene currently is active, so that you can do context specific actions. But it has to be done in code by yourself. See how it is done in guis/gui_numpad.cpp.

OMOTE is using the Light and Versatile Graphics Library. If you want to learn more about how to use lvgl, this is a good starting point.

The library is very powerful, but also resource demanding, at least for an ESP32. Lack of memory has been an issue in the past. Most of the problems are solved, but you should read section memory.

Important: If you are creating new guis, you may find it difficult and time consuming to write correct lvgl code and to test it on the ESP32. You can significantly speed up the development of guis with use of the software simulator

Scene specific guis

There is a so called "main_gui_list" which consists by default of all guis you have registered. You can swipe horizontally between the guis. By default, the first gui in the "main_gui_list" is the "scene selection gui".

Additionally, you can define scene specific gui_lists which are used whenever a scene is active. When a scene is active, you can switch between the scene specific gui list and the "main_gui_list", e.g. for starting a different scene or changing some settings of the OMOTE.

You can have the same gui in one or more of these lists. E.g. the numpad can be used to send numbers to different devices, based on the scene (see "Scene TV" and "Scene Fire TV").

Switching from the "main_gui_list" go the scene specific list is done simply by selecting a scene on the "scene selection gui".
Remark: if you select a scene on the "scene selection gui", the start sequence of the scene is only sent if the scene was not yet active. If you want to force the start sequence of the scene to be resent, you need to long press the scene on the gui or use the corresponding hardware key.

Switching from the scene specific list back to the scene selection gui in "main_gui_list" can be done in several ways:

• do a "swipe down" from top of the screen
• click on the scene name in the header bar at top of the screen
• click on the gui name in the page indicator at bottom of the screen
• hit the "back" button on the keypad
• programmatically with executeCommand(SCENE_SELECTION);

You can also directly jump from the scene specific list to the "main_gui_list" and back. The last active GUI from each list is automatically activated again. This can be achieved by

• pressing the hardware key "record"
• programmatically by executeCommand(SCENE_BACK_TO_PREVIOUS_GUI_LIST);

This is how the gui lists of the predefined example look like:

If you want to have only one single list of guis (the "main_gui_list"), available in all of the scenes, and no scene specific gui lists, then set -D USE_SCENE_SPECIFIC_GUI_LIST=0 in platformio.ini.
If you also don't want to have the "scene selection gui", comment line register_gui_sceneSelection(); in file main.cpp

How to

Normally, you only have to add your WiFi credentials, devices, guis and scenes. So only these files are of interest:

src/
├── devices/
│   ├── AVreceiver/
│   │   └── device_yamahaAmp/
│   ├── keyboard/
│   │   ├── device_keyboard_ble/
│   │   └── device_keyboard_mqtt/
│   ├── mediaPlayer/
│   │   └── device_appleTV/
│   ├── misc/
│   │   ├── device_smarthome/
│   │   └── device_specialCommands.cpp/.h
│   └── TV/
│       └── device_samsungTV/
├── devices_pool/
│   ├── AVreceiver/
│   │   ├── device_denonAvr/
│   │   └── ...
│   ├── keyboard/
│   │   ├── device_keyboard_ble/
│   │   └── device_keyboard_mqtt/
│   ├── mediaPlayer/
│   │   ├── device_lgbluray/
│   │   └── ...
│   ├── misc/
│   │   ├── device_smarthome/
│   │   ├── ...
│   │   └── device_specialCommands.cpp/.h
│   └── TV/
│       ├── device_lgTV/
│       └── ...
├── guis/
│   ├── gui_numpad.cpp/.h
│   └── ...
├── scenes/
│   ├── scene__defaultKeys.cpp/.h
│   ├── scene_TV.cpp/.h
│   └── ...
├── secrets_override.h
└── secrets.h

The devices have separate folders for TVs, AV receivers, media players, and so on. Try to put your devices into the correct folder. It is not necessary, but helps to keep the structure clean.

add WiFi credentials

Copy file secrets_override_example.h to file secrets_override.h. This file will never by under version control, so you can safely put your secrets in this file.

use one of the devices from the devices_pool

The folder /src/devices_pool is intended for a pool of devices already contributed by the community of OMOTE. Files in this folder won't get compiled by default.

If you want to use one of these devices, then

• copy the corresponding device folder from /src/devices_pool/... to /src/devices/...
• add the corresponding call to register_device_*() into /src/main.cpp (and include the device header file in /src/main.cpp as well)
• start using the commands defined by the device, e.g. by putting them in the hardware key map of scene__default.cpp or of a specific scene

register new device

• create new folder, similar to devices/TV/device_samsungTV/
• take an existing device as template and copy e.g. the two files from device_samsungTV/ to your new folder and rename files
• rename function register_device_samsungTV()
• list all the available commands in the header file
• register the commands in the cpp file
• register device in main.cpp with a call to register_device_*()
• if you need a different kind if IR code to be sent which is currently not supported, add it to enum IRprotocols in file /src/applicationInternal/hardware/hardwarePresenter.h and also add it in /hardware/ESP32/infrared_sender_hal_esp32.cpp or ask for support

Remark: to save memory, remove the devices not needed by you, at least by commenting the call to register_device_*()

register new gui

• take an existing gui as template and copy the two files (.h and .cpp)
• if the new gui is specific for a single device, then place the files in the same folder as the device
• if the new gui is for more than one device or not related at all to a device, place it in folder /src/guis/
• each gui must have a unique name which is defined in the .h file
• rename function register_gui_*()
• register gui in main.cpp with a call to register_gui_*()
• if you want to deactivate a gui, simply comment out the corresponding line in main.cpp
• if you want only specific guis to be available in the "main_gui_list", then adjust the following line in main.cpp:
  main_gui_list = {tabName_sceneSelection, tabName_smarthome, tabName_settings, tabName_irReceiver};

You can define a command to directly activate a GUI. See file gui_smarthome.cpp and command GUI_SMARTHOME_ACTIVATE on how it is done. As an example, the command GUI_SMARTHOME_ACTIVATE is used in the default key map for the hardware key "stop".

Remark: the combination of a specific command to activate a GUI and using the command SCENE_BACK_TO_PREVIOUS_GUI_LIST (by default mapped to hardware key "record") is very powerful to quickly jump to a specific GUI (for finetuning a device, controlling the smart home, ...) and jump back to where you have been before.

Important: If you are creating new guis, you may find it difficult and time consuming to write correct lvgl code and to test it on the ESP32. You can significantly speed up the development of guis with use of the software simulator

register a scene

• take an existing scene as template and copy the two files (.h and .cpp)
• place them in folder scenes/
• each scene must have a unique name which is defined in the .h file
• rename the function register_scene_*()
• define a command with which the scene can be activated
• define scene specific overrides for the hardware keys in scene_setKeys_*()
• define start and end sequences for the scene scene_start_sequence_*() and scene_end_sequence_*()
• define a scene specific gui list, if you want to have one. See "scene_TV" for a scene where such a list is defined, and "scene_chromecast" where no such list is defined
• register scene in main.cpp with a call to register_scene_*()
• if you want to deactivate a scene, simply comment out the corresponding line in main.cpp
• if you want only specific scenes to be available on the "scene selection gui", then adjust the following line in main.cpp:
  set_scenes_on_sceneSelectionGUI({scene_name_TV, scene_name_fireTV, scene_name_chromecast, scene_name_appleTV});

register special commands

• you can define own commands like MY_SPECIAL_COMMAND which is not binded to a single device. Can be used at runtime for whatever you like, e.g. for sending a sequence of commands. Search in the code for MY_SPECIAL_COMMAND to see an example on how to do it.

subscribe to MQTT topics

An example on how to subscribe to MQTT topics and to propagate received values to the gui is available.

Topics you want to subscribe to need to be added in mqtt_hal_esp32.cpp and mqtt_hal_windows_linux.cpp. Search for topic OMOTE/test to see an example.

Received MQTT messages are currently simply displayed on the receiver GUI tab.

Most likely you want to add special handling when a MQTT message has arrived. For this, you can extend function void receiveMQTTmessage_cb(...) in file /src/applicationInternal/commandHandler.cpp

You can send a MQTT message from OMOTE to your home automation software as soon as WiFi is available in OMOTE. As response, your home automation software could send the states of the smart home devices known to OMOTE. See function void receiveWiFiConnected_cb() in file /src/applicationInternal/commandHandler.cpp

Pairing BLE devices

First you have to decide if you want to pair only one single device to OMOTE, or if you need two ore more (e.g. a Fire TV and a NVIDIA Shield TV).

If you need to pair two ore more devices, you have to use an unstable branch of the underlying NimBLE library. The disadvantage of this unstable branch is that it regularly has breaking changes. OMOTE tries to keep up with them, but it is not guaranteed that the OMOTE software can be compiled with the unstable branch at any time.

So stay with the latest stable release of NimBLE if you only need to pair one single device with OMOTE.

One single BLE device

This is the default setting. You don't have to change anything in the code if you want to pair only one single device. Only check if -D ENABLE_BLUETOOTH=1 and -D ENABLE_KEYBOARD_BLE=1 is set in platformio.ini

The latest stable release of NimBLE is automatically used if this line is active in platformio.ini (the second line must be commented)

h2zero/NimBLE-Arduino@^1.4.3
;https://github.com/h2zero/NimBLE-Arduino#master

Simply go to this GUI and press Start advertising. You can watch the pairing process on the GUI.

If the device was successfully paired, it will automatically reconnect after OMOTE reboots.

Two or more BLE devices

As already said, you have to use an unstable branch of the underlying NimBLE library to connect more than one BLE device.

To use this branch, you have to have this setting in platformio.ini (the first line must be commented)

;h2zero/NimBLE-Arduino@^1.4.3
https://github.com/h2zero/NimBLE-Arduino#master

And check if -D ENABLE_BLUETOOTH=1 and -D ENABLE_KEYBOARD_BLE=1 is set in platformio.ini

Only in this unstable branch of the NimBLE library direct advertising works (see below why this is needed). This unstable branch regularly has breaking changes. So if OMOTE does not compile when using the unstable NimBLE branch, please open an issue.

Take care: If you switch between the stable and unstable version of NimBLE, all bonds will automatically been deleted. This is necessary because of a breaking change in the underlying nimble library. So if you switch the NimBLE version, you have to pair again all of your devices!

If for any reason you think the bonds storage is messed up and you are not being able to delete the bonds, you can erase the complete flash. Doing so, all other settings will also be deleted. To erase the flash, use PlatformIO in Visual Studio Code:

Only one connection at the same time

Since the underlying library NimBLE-Arduino supports only one connection at the same time, we have to switch the connection at runtime.

For this to work, you have to

• pair all your devices one by one
• define which command should be sent to which device

Pairing vs. Bonding

Pairing means devices (e.g. OMOTE and a media player like Fire TV) exchange the information necessary to establish an encrypted connection.

Bonding means that the information from the pairing process is stored on the devices, so that the pairing process does not have to be repeated every time the devices reconnect to each other.

OMOTE uses a so called "BLE keyboard" which connects to a media player and behaves like a regular BLE keyboard.

The devices involved in the BLE keyboard connection do not simply use pairing, they also use bonding. The bonded devices are saved on the OMOTE, and you can see which devices are bonded to the OMOTE.

Advertising vs. Direct Advertising

Advertising means that the OMOTE advertises its presence. Every peer (e.g. a media player) who wants to connect can do so. OMOTE does not ask for a PIN to connect. The fastest peer will get the connection.

Direct advertising means that OMOTE sends the advertising message only to a certain already bonded device. Doing so, not the fastest peer responding to the advertising message wins. Only the peer the advertising is directed to can establish the connection.

What do you have to do

For the bonding process you can use the following GUI:

You have to do the following steps:

Bond all your devices

• switch all your media devices (which are using BLE) off
• turn exactly one device on
• check with "print connected" and "print bonds" that no peer is connected to OMOTE and no bonds are stored. Use "disconnect" or "delete bonds" if needed.
• click on "start advertising"
• start the pairing process on your media device
• you should see the connection being established in OMOTE
• check with "print bonds" that a new bond has been added. Write down this address.
• use "disconnect" to disconnect the just bonded media player
• switch that media player off, turn on the next media player you want to bond
• repeat the procedure until all needed media players have been bonded

For testing purposes, you also have stop advertising as a button on the GUI. And you have a dropdown to start a direct advertisement to an already bonded peer.

Remark: the number of devices that can be bonded to OMOTE is limited. The default is three devices. If you need more, please change platformio.ini accordingly.

;-- for BLE Keyboard. Set the maximum number of bonded BLE peers ----------
-D CONFIG_BT_NIMBLE_MAX_BONDS=3
; has to be 4x CONFIG_BT_NIMBLE_MAX_BONDS
-D CONFIG_BT_NIMBLE_MAX_CCCDS=12

Adjust the code and define which command is sent to which device

Normally, BLE keyboard commands are defined without a specific address this command has to be sent to. For example, see this command

  register_command(&KEYBOARD_BLE_RIGHT               , makeCommandData(BLE_KEYBOARD, {}));

If such a command is being used at runtime, either

• the already connected device is being used
• or in case no device is yet connected, a (non direct) advertisement is being started

If you want a BLE keyboard command to be sent to a specific device, you can define it as following

  register_command(&KEYBOARD_BLE_RIGHT_FIRETV        , makeCommandData(BLE_KEYBOARD, {"11:22:33:44:55:66", std::to_string(KEYBOARD_BLE_RIGHT)}));

Search in the code for all appearances of KEYBOARD_BLE_RIGHT_FIRETV to see exactly what has to be done.

Remember to adjust the key map of the device where you want to use this address specific command.

Switching the device at runtime

This is automatically done in file hardware/ESP32/lib/ESP32-BLE-Keyboard/BleKeyboard.cpp in BleKeyboard::forceConnectionToAddress(std::string peerAddress) and BleKeyboard::advertiseAndWaitForConnection(std::string peerAddress), respectively. Here you have two parameters:

  int connectionTimeout   = 10000; // wait until connection is established
  int waitAfterConnection = 2000;  // wait after connection was established, before key is actually being sent

The first is how long OMOTE waits for the connection to be established, if it has to be changed. After that timeout, it will no longer be tried to send the key.

The second is the time OMOTE waits after the connection has been established, before the key is actually being sent. Some devices don't like to receive a command too fast after the connection has been established. Adjust this timeout to your needs.

These timeouts are only relevant if you have to change the connection between devices. They are not relevant if OMOTE is already connected to the correct device.

Memory

The ESP32 has only limited memory. If you are using WiFi, BLE and lvgl at the same time, you quickly run into memory issues.

Please see this very detailed discussion about memory usage and optimization, if you are interested. If you are only interested in the most important results of this analysis, please stay here, and continue reading.

How memory optimization is done

The conclusion of the memory analysis is that you cannot have all the GUIs in memory at the same time. To keep memory usage low, OMOTE always has only three GUIs in memory at the same time. The visible GUI is always the one in the middle. Since during swiping you can already see the previous or next GUI, these two also need to be in memory. As soon as the swiping animation ended, all GUIs are deleted and the next three are created, with the new visible one in the middle.

It has been put a lot of effort into achieving this dynamic recreation of GUIs in a way that it is not recognizable by the user. It is not 100% perfect, but almost 😄 The big advantage of this approach is that in practice you are almost not limited in the number of GUIs you want to have. In tests 100 GUIs worked without any problem. Of course, the size of the flash where the program code is and other things are also limiting, but in reality, you should be able to have all the GUIs you want to have.

Of course, the three GUIs being in memory must not become so big that these three don't fit into memory.

How to see how much memory is used

There are two ways to get memory statstics:

1. on the serial output you get detailed information about memory usage every 5 sec
2. a small status text at top of the screen, updated every second, colored in red if resources are running out

To activate the status text on the screen, simply go the settings GUI and scroll to the bottom.

The first is "freep heap"/"total heap", the second is "free lvgl memory"/"total lvgl memory".

There are two things you must observe:

• lvgl memory must never exceed the fixed memory limit lvgl has (32k by default)
• ESP32 free heap should stay big enough for memory peaks Wifi or BLE need

Whenever lvgl memory is used more than 80%, the corresponding label turns red. Whenever ESP32 free heap will be below a certain threshold, the corresponding label turns red. The thresholds for the ESP32 heap are 15.000 bytes if WiFi and BLE are turned on and 5.000 bytes if both are deactivated. You can change these thresholds in file memoryUsage.cpp

How to optimize the size of lvgl static memory and ESP32 heap

What can you do if lvgl memory is getting low:

• reduce the number of widgets on the GUIs
• increase lvgl static memory (but to the cost of less ESP32 heap)

What can you do if ESP32 heap is getting low:

• reduce lvgl static memory, if possible (-D LV_MEM_SIZE= in platformio.ini)
• deactivate WiFi and MQTT if possible
• deactivate BLE and use the ESP32 emulating a USB keyboard
• use only one draw buffer for lvgl, this saves 15360 bytes RAM. Comment out #define useTwoBuffersForlvgl in file guiBase.h

Architectural overview

This section is only for those who want to understand how everything works behind the scenes. It is not necessary to use and adapt the software to your needs.

The software is divided into components which have only loose coupling, following the MVP pattern.

commandHandler

• devices register at startup at the commandHandler with the commands they can execute
• can be called at runtime to execute a registered command
• uses the hardware which is needed to execute the command (IR sender, MQTT, BLE)

sceneRegistry

• scenes can register at setup to be available at runtime
• each scene must have a unique name
• each scene must provide functions to start and end a scene (which can use the commandHandler to send commands)
• can provide scene specific hardware key definitions

sceneHandler

• will be called by commandHandler to switch a scene. Since switching a scene is a more complex task, this is forwarded by the commandHandler to the sceneHandler

guiRegistry

• every tab available on the touchscreen needs to be registered at the guiRegistry
• each gui must provide functions the create the content of the tab
• guiRegistry will be called by guiBase (which is called by main) to create the content of the touchscreen
• if you want to deactivate a gui, simply comment out the corresponding line in main.cpp

keys

• hardware keys already have been explained in detail above

folder structure for applicationInternal/

Here you can find the corresponding files in the workspace:

src/
└── applicationInternal/
    ├── gui/
    │   ├── guiBase.cpp/.h
    │   ├── guiMemoryOptimizer.cpp/.h
    │   ├── guiRegistry.cpp/.h
    │   └── guiStatusUpdate.cpp/.h
    ├── hardware/
    │   ├── arduinoLayer.cpp/.h
    │   └── hardwarePresenter.cpp/.h
    ├── scenes/
    │   ├── sceneHandler.cpp/.h
    │   └── sceneRegistry.cpp/.h
    ├── commandHandler.cpp/.h
    ├── keys.cpp/.h
    └── memoryUsage.cpp/.h

components involved at setup

components involved for executing a simple command triggered by a hardware key

components involved for executing a scene switch