Merge branch 'doc/stdio_console' into 'master'

docs(console): add an API guides page on standard I/O

Closes IDFGH-12951

See merge request espressif/esp-idf!31343

docs/en/api-guides/index.rst

@@ -35,6 +35,7 @@ API Guides
    performance/index
    reproducible-builds
    :(SOC_WIFI_SUPPORTED or SOC_BT_SUPPORTED or SOC_IEEE802154_SUPPORTED) and not esp32c5 and not esp32c61: RF_calibration
+   stdio
    thread-local-storage
    tools/index
    unit-tests

docs/en/api-guides/stdio.rst

@@ -0,0 +1,151 @@
+Standard I/O and Console Output
+===============================
+
+ESP-IDF provides C standard I/O facilities, such as ``stdin``, ``stdout``, and ``stderr`` streams, as well as C standard library functions such as ``printf()`` which operate on these streams.
+
+As common in POSIX systems, these streams are buffering wrappers around file descriptors:
+
+- ``stdin`` is a buffered stream for reading input from the user, wrapping file descriptor ``STDIN_FILENO`` (0).
+- ``stdout`` is a buffered stream for writing output to the user, wrapping ``STDOUT_FILENO`` (1).
+- ``stderr`` is a buffered stream for writing error messages to the user, wrapping ``STDERR_FILENO`` (2).
+
+In ESP-IDF, there is no practical distinction between ``stdout`` and ``stderr``, as both streams are sent to the same physical interface. Most applications will use only ``stdout``. For example, ESP-IDF logging functions always write to ``stdout`` regardless of the log level.
+
+The underlying stdin, stdout, and stderr file descriptors are implemented based on :doc:`VFS drivers <../api-reference/storage/vfs>`.
+
+On {IDF_TARGET_NAME}, ESP-IDF provides implementations of VFS drivers for I/O over:
+
+.. list::
+
+    - UART
+    :SOC_USB_SERIAL_JTAG_SUPPORTED: - USB Serial/JTAG
+    :esp32s2 or esp32s3: - USB CDC (using USB_OTG peripheral)
+    - "Null" (no output)
+
+Standard I/O is not limited to these options, though. See below on enabling custom destinations for standard I/O.
+
+Configuration
+-------------
+
+Built-in implementations of standard I/O can be selected using several Kconfig options:
+
+.. list::
+
+    - :ref:`CONFIG_ESP_CONSOLE_UART_DEFAULT<CONFIG_ESP_CONSOLE_UART_DEFAULT>` — Enables UART with default options (pin numbers, baud rate) for standard I/O.
+    - :ref:`CONFIG_ESP_CONSOLE_UART_CUSTOM<CONFIG_ESP_CONSOLE_UART_CUSTOM>` — Enables UART for standard I/O, with TX/RX pin numbers and baud rate configurable via Kconfig.
+    :esp32s2 or esp32s3: - :ref:`CONFIG_ESP_CONSOLE_USB_CDC<CONFIG_ESP_CONSOLE_USB_CDC>` — Enables USB CDC (using USB_OTG peripheral) for standard I/O. See :doc:`usb-otg-console` for details about hardware connections required.
+    :SOC_USB_SERIAL_JTAG_SUPPORTED: - :ref:`CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG<CONFIG_ESP_CONSOLE_USB_SERIAL_JTAG>` — Enables USB Serial/JTAG for standard I/O. See :doc:`usb-serial-jtag-console` for details about hardware connections required.
+    - :ref:`CONFIG_ESP_CONSOLE_NONE<CONFIG_ESP_CONSOLE_NONE>` — Disables standard I/O. If this option is selected, ``stdin``, ``stdout``, and ``stderr`` will be mapped to ``/dev/null`` and won't produce any output or generate any input.
+
+Enabling one of these option will cause the corresponding VFS driver to be built into the application and used to open ``stdin``, ``stdout``, and ``stderr`` streams. Data written to ``stdout`` and ``stderr`` will be sent over the selected interface, and input from the selected interface will be available on ``stdin``.
+
+.. only:: SOC_USB_SERIAL_JTAG_SUPPORTED
+
+    Secondary output
+    ^^^^^^^^^^^^^^^^
+
+    ESP-IDF has built-in support for sending standard output to a secondary destination. This option makes the application output visible on two interfaces at once, for example on both UART and USB Serial/JTAG.
+
+    Note that secondary console is output-only:
+
+        - data written to ``stdout`` and ``stderr`` by the application will be sent to both primary and secondary consoles
+        - ``stdin`` will only contain data sent by the host to the primary console.
+
+    The following secondary console options are available:
+
+        - :ref:`CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG<CONFIG_ESP_CONSOLE_SECONDARY_USB_SERIAL_JTAG>`
+
+Standard Streams and FreeRTOS Tasks
+-----------------------------------
+
+In ESP-IDF, to save RAM, ``FILE`` objects for ``stdin``, ``stdout``, and ``stderr`` are shared between all FreeRTOS tasks, but the pointers to these objects are unique for every task. This means that:
+
+- It is possible to change ``stdin``, ``stdout``, and ``stderr`` for any given task without affecting other tasks, e.g., by doing ``stdin = fopen("/dev/uart/1", "r")``.
+- To change the default ``stdin``, ``stdout``, ``stderr`` streams for new tasks, modify ``_GLOBAL_REENT->_stdin`` (``_stdout``, ``_stderr``) before creating the task.
+- Closing default ``stdin``, ``stdout``, or ``stderr`` using ``fclose`` closes the ``FILE`` stream object, which will affect all other tasks.
+
+Each stream (``stdin``, ``stdout``, ``stderr``) has a mutex associated with it. This mutex is used to protect the stream from concurrent access by multiple tasks. For example, if two tasks are writing to ``stdout`` at the same time, the mutex will ensure that the outputs from each task are not mixed together.
+
+Blocking and non-blocking I/O
+-----------------------------
+
+UART
+^^^^
+
+By default, UART VFS uses simplified functions for reading from and writing to UART. Writes busy-wait until all data is put into UART FIFO, and reads are non-blocking, returning only the data present in the FIFO. Due to this non-blocking read behavior, higher level C library calls, such as ``fscanf("%d\n", &var);``, might not have desired results.
+
+Applications which use the UART driver can instruct VFS to use the driver's interrupt driven, blocking read and write functions instead. This can be done using a call to the :cpp:func:`uart_vfs_dev_use_driver` function. It is also possible to revert to the basic non-blocking functions using a call to :cpp:func:`uart_vfs_dev_use_nonblocking`.
+
+When the interrupt-driven driver is installed, it is also possible to enable/disable non-blocking behavior using ``fcntl`` function with ``O_NONBLOCK`` flag.
+
+.. only:: SOC_USB_SERIAL_JTAG_SUPPORTED
+
+    USB Serial/JTAG
+    ^^^^^^^^^^^^^^^
+
+    Similar to UART, the VFS driver for USB Serial/JTAG defaults to a simplified implementation: writes are blocking (busy-wait until all the data has been sent) and reads are non-blocking, returning only the data present in the FIFO. This behavior can be changed to use the interrupt driven, blocking read and write functions of USB Serial/JTAG driver using a call to the :cpp:func:`usb_serial_jtag_vfs_use_nonblocking` function. Note that the USB Serial/JTAG driver has to be initialized using :cpp:func:`usb_serial_jtag_driver_install` beforehand. It is also possible to revert to the basic non-blocking functions using a call to :cpp:func:`usb_serial_jtag_vfs_use_nonblocking`.
+
+    When the interrupt-driven driver is installed, it is also possible to enable/disable non-blocking behavior using ``fcntl`` function with ``O_NONBLOCK`` flag.
+
+.. only:: esp32s2 or esp32s3
+
+    USB CDC (using USB_OTG peripheral)
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    USB CDC VFS driver provides blocking I/O behavior by default. It is possible to enable non-blocking behavior using ``fcntl`` function with ``O_NONBLOCK`` flag.
+
+Newline conversion
+------------------
+
+VFS drivers provide an optional newline conversion feature for input and output. Internally, most applications send and receive lines terminated by the LF (``\n``) character. Different terminal programs may require different line termination, such as CR or CRLF.
+
+Applications can configure this behavior globally using the following Kconfig options:
+
+    - :ref:`CONFIG_NEWLIB_STDOUT_LINE_ENDING_CRLF<CONFIG_NEWLIB_STDOUT_LINE_ENDING_CRLF>`, :ref:`CONFIG_NEWLIB_STDOUT_LINE_ENDING_CR<CONFIG_NEWLIB_STDOUT_LINE_ENDING_CR>`, :ref:`CONFIG_NEWLIB_STDOUT_LINE_ENDING_LF<CONFIG_NEWLIB_STDOUT_LINE_ENDING_LF>` - for output
+    - :ref:`CONFIG_NEWLIB_STDIN_LINE_ENDING_CRLF<CONFIG_NEWLIB_STDIN_LINE_ENDING_CRLF>`, :ref:`CONFIG_NEWLIB_STDIN_LINE_ENDING_CR<CONFIG_NEWLIB_STDIN_LINE_ENDING_CR>`, :ref:`CONFIG_NEWLIB_STDIN_LINE_ENDING_LF<CONFIG_NEWLIB_STDIN_LINE_ENDING_LF>` - for input
+
+
+It is also possible to configure line ending conversion for the specific VFS driver:
+
+.. list::
+
+    - For UART: :cpp:func:`uart_vfs_dev_port_set_rx_line_endings` and :cpp:func:`uart_vfs_dev_port_set_tx_line_endings`
+    :SOC_USB_SERIAL_JTAG_SUPPORTED: - For USB Serial/JTAG: :cpp:func:`usb_serial_jtag_vfs_set_rx_line_endings` and :cpp:func:`usb_serial_jtag_vfs_set_tx_line_endings`
+    :esp32s2 or esp32s3: - For USB CDC (using USB_OTG peripheral): :cpp:func:`esp_vfs_dev_cdcacm_set_rx_line_endings` and :cpp:func:`esp_vfs_dev_cdcacm_set_tx_line_endings`
+
+Buffering
+---------
+
+By default, standard I/O streams are line buffered. This means that data written to the stream is not sent to the underlying device until a newline character is written, or the buffer is full. This means, for example, that if you call ``printf("Hello")``, the text will not be sent to the UART until you call ``printf("\n")`` or the stream buffer fills up due to other prints.
+
+This behavior can be changed using the ``setvbuf()`` function. For example, to disable buffering for ``stdout``:
+
+.. code-block:: c
+
+    setvbuf(stdout, NULL, _IONBF, 0);
+
+You can also use ``setvbuf()`` to increase the buffer size, or switch to fully buffered mode.
+
+Custom channels for standard I/O
+--------------------------------
+
+To send application output to a custom channel (for example, a WebSocket connection), it is possible to create a custom VFS driver. See the :doc:`VFS documentation <../api-reference/storage/vfs>` for details. The VFS driver has to implement at least the following functions:
+
+    - ``open()`` and ``close()``
+    - ``write()``
+    - ``read()`` — only if the custom channel is also used for input
+    - ``fstat()`` — recommended, to provide correct buffering behavior for the I/O streams
+    - ``fcntl()`` — only if non-blocking I/O has to be supported
+
+Once you have created a custom VFS driver, use ``esp_vfs_register()`` to register it with VFS. Then, use ``fopen()`` to redirect ``stdout`` and ``stderr`` to the custom channel. For example:
+
+.. code-block:: c
+
+    FILE *f = fopen("/dev/mychannel", "w");
+    if (f == NULL) {
+        // handle the error here
+    }
+    stdout = f;
+    stderr = f;
+
+Note that logging functions (``ESP_LOGE()``, etc.) write their output to ``stdout``. Keep this in mind when using logging within the implementation of your custom VFS (or any components which it calls). For example, if the custom VFS driver's ``write()`` operation fails and uses ``ESP_LOGE()`` to log the error, this will cause the output to be sent to ``stdout``, which would again call the custom VFS driver's ``write()`` operation. This would result in an infinite loop. It is recommended to keep track of this re-entry condition in the VFS driver's ``write()`` implementation, and return immediately if the write operation is still in progress.

docs/en/api-reference/storage/vfs.rst

@@ -179,42 +179,7 @@ File Descriptors
 
 File descriptors are small positive integers from ``0`` to ``FD_SETSIZE - 1``, where ``FD_SETSIZE`` is defined in ``sys/select.h``. The largest file descriptors (configured by ``CONFIG_LWIP_MAX_SOCKETS``) are reserved for sockets. The VFS component contains a lookup-table called ``s_fd_table`` for mapping global file descriptors to VFS driver indexes registered in the ``s_vfs`` array.
 
-
-Standard IO Streams (``stdin``, ``stdout``, ``stderr``)
--------------------------------------------------------
-
-If the menuconfig option ``UART for console output`` is not set to ``None``, then ``stdin``, ``stdout``, and ``stderr`` are configured to read from, and write to, a UART. It is possible to use UART0 or UART1 for standard IO. By default, UART0 is used with 115200 baud rate; TX pin is GPIO1; RX pin is GPIO3. These parameters can be changed in menuconfig.
-
-Writing to ``stdout`` or ``stderr`` sends characters to the UART transmit FIFO. Reading from ``stdin`` retrieves characters from the UART receive FIFO.
-
-By default, VFS uses simple functions for reading from and writing to UART. Writes busy-wait until all data is put into UART FIFO, and reads are non-blocking, returning only the data present in the FIFO. Due to this non-blocking read behavior, higher level C library calls, such as ``fscanf("%d\n", &var);``, might not have desired results.
-
-Applications which use the UART driver can instruct VFS to use the driver's interrupt driven, blocking read and write functions instead. This can be done using a call to the :cpp:func:`uart_vfs_dev_use_driver` function. It is also possible to revert to the basic non-blocking functions using a call to :cpp:func:`uart_vfs_dev_use_nonblocking`.
-
-VFS also provides an optional newline conversion feature for input and output. Internally, most applications send and receive lines terminated by the LF (''\n'') character. Different terminal programs may require different line termination, such as CR or CRLF. Applications can configure this separately for input and output either via menuconfig, or by calls to the functions :cpp:func:`uart_vfs_dev_port_set_rx_line_endings` and :cpp:func:`uart_vfs_dev_port_set_tx_line_endings`.
-
-
-Standard Streams and FreeRTOS Tasks
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-``FILE`` objects for ``stdin``, ``stdout``, and ``stderr`` are shared between all FreeRTOS tasks, but the pointers to these objects are stored in per-task ``struct _reent``.
-
-The following code is transferred to ``fprintf(__getreent()->_stderr, "42\n");`` by the preprocessor:
-
-.. highlight:: c
-
-::
-
-    fprintf(stderr, "42\n");
-
-
-The ``__getreent()`` function returns a per-task pointer to ``struct _reent`` in newlib libc. This structure is allocated on the TCB of each task. When a task is initialized, ``_stdin``, ``_stdout``, and ``_stderr`` members of ``struct _reent`` are set to the values of ``_stdin``, ``_stdout``, and ``_stderr`` of ``_GLOBAL_REENT`` (i.e., the structure which is used before FreeRTOS is started).
-
-Such a design has the following consequences:
-
-- It is possible to set ``stdin``, ``stdout``, and ``stderr`` for any given task without affecting other tasks, e.g., by doing ``stdin = fopen("/dev/uart/1", "r")``.
-- Closing default ``stdin``, ``stdout``, or ``stderr`` using ``fclose`` closes the ``FILE`` stream object, which will affect all other tasks.
-- To change the default ``stdin``, ``stdout``, ``stderr`` streams for new tasks, modify ``_GLOBAL_REENT->_stdin`` (``_stdout``, ``_stderr``) before creating the task.
+Standard I/O streams (``stdin``, ``stdout``, ``stderr``) are mapped to file descriptors ``0``, ``1``, and ``2`` respectively. For more information on standard I/O, see :doc:`../../api-guides/stdio`.
 
 ``eventfd()``
 -------------

docs/zh_CN/api-guides/index.rst

@@ -35,6 +35,7 @@ API 指南
    performance/index
    reproducible-builds
    :(SOC_WIFI_SUPPORTED or SOC_BT_SUPPORTED or SOC_IEEE802154_SUPPORTED) and not esp32c5 and not esp32c61: RF_calibration
+   stdio
    thread-local-storage
    tools/index
    unit-tests

docs/zh_CN/api-guides/stdio.rst

@@ -0,0 +1 @@
+.. include:: ../../en/api-guides/stdio.rst

docs/zh_CN/api-reference/storage/vfs.rst

@@ -179,42 +179,7 @@ VFS 对文件路径长度没有限制，但文件系统路径前缀受 ``ESP_VFS
 
 文件描述符是一组很小的正整数，从 ``0`` 到 ``FD_SETSIZE - 1``，``FD_SETSIZE`` 定义在 ``sys/select.h``。最大文件描述符由 ``CONFIG_LWIP_MAX_SOCKETS`` 定义，且为套接字保留。VFS 中包含一个名为 ``s_fd_table`` 的查找表，用于将全局文件描述符映射至 ``s_vfs`` 数组中注册的 VFS 驱动索引。
 
-
-标准 IO 流 (``stdin``, ``stdout``, ``stderr``)
-----------------------------------------------------
-
-如果 menuconfig 中 ``UART for console output`` 选项没有设置为 ``None``，则 ``stdin``、 ``stdout`` 和 ``stderr`` 将默认从 UART 读取或写入。UART0 或 UART1 可用作标准 IO。默认情况下，UART0 使用 115200 波特率，TX 管脚为 GPIO1，RX 管脚为 GPIO3。上述参数可以在 menuconfig 中更改。
-
-对 ``stdout`` 或 ``stderr`` 执行写入操作将会向 UART 发送 FIFO 发送字符，对 ``stdin`` 执行读取操作则会从 UART 接收 FIFO 中取出字符。
-
-默认情况下，VFS 使用简单的函数对 UART 进行读写操作。在所有数据放进 UART FIFO 之前，写操作将处于 busy-wait 状态，读操处于非阻塞状态，仅返回 FIFO 中已有数据。由于读操作为非阻塞，高层级 C 库函数调用（如 ``fscanf("%d\n", &var);``）可能获取不到所需结果。
-
-如果应用程序使用 UART 驱动，则可以调用 :cpp:func:`uart_vfs_dev_use_driver` 函数来指导 VFS 使用驱动中断、读写阻塞功能等，也可以调用 :cpp:func:`uart_vfs_dev_use_nonblocking` 来恢复非阻塞函数。
-
-VFS 还为输入和输出提供换行符转换功能（可选）。多数应用程序在程序内部发送或接收以 LF (''\n'') 结尾的行，但不同的终端程序可能需要不同的换行符，比如 CR 或 CRLF。应用程序可以通过 menuconfig 或者调用 :cpp:func:`uart_vfs_dev_port_set_rx_line_endings` 和 :cpp:func:`uart_vfs_dev_port_set_tx_line_endings` 为输入输出配置换行符。
-
-
-标准流和 FreeRTOS 任务
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-``stdin``、``stdout`` 和 ``stderr`` 的 ``FILE`` 对象在所有 FreeRTOS 任务之间共享，指向这些对象的指针分别存储在每个任务的 ``struct _reent`` 中。
-
-预处理器把如下代码解释为 ``fprintf(__getreent()->_stderr, "42\n");``：
-
-.. highlight:: c
-
-::
-
-    fprintf(stderr, "42\n");
-
-
-其中 ``__getreent()`` 函数将为每个任务返回一个指向 newlib libc 中 ``struct _reent`` 的指针。每个任务的 TCB 均拥有一个 ``struct _reent`` 结构体，任务初始化后，``struct _reent`` 结构体中的 ``_stdin``、``_stdout`` 和 ``_stderr`` 将会被赋予 ``_GLOBAL_REENT`` 中 ``_stdin``、 ``_stdout`` 和 ``_stderr`` 的值，``_GLOBAL_REENT`` 即为 FreeRTOS 启动之前所用结构体。
-
-这样设计带来的结果是：
-
-- 允许设置给定任务的 ``stdin``、 ``stdout`` 和 ``stderr``，而不影响其他任务，例如通过 ``stdin = fopen("/dev/uart/1", "r")``；
-- 但使用 ``fclose`` 关闭默认 ``stdin``、 ``stdout`` 或 ``stderr`` 将同时关闭相应的 ``FILE`` 流对象，因此会影响其他任务；
-- 如需更改新任务的默认 ``stdin``、 ``stdout`` 和 ``stderr`` 流，请在创建新任务之前修改 ``_GLOBAL_REENT->_stdin`` (``_stdout``、``_stderr``)。
+标准 I/O 流（stdin、stdout、stderr）分别映射到文件描述符 0、1 和 2。有关标准 I/O 的更多信息，请参见 :doc:`../../api-guides/stdio`。
 
 ``eventfd()``
 -------------