Skip to content

bblanchon/ArduinoStreamUtils

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

StreamUtils: Power-ups for Arduino Streams

Continuous Integration Coverage Status Arduino Library Manager PlatformIO Registry

The stream is an essential abstraction in Arduino; we find it in many places:

This library provides some helper classes and functions for dealing with streams.

For example, with this library, you can:

  • speed up your program by buffering the data it reads from a file
  • reduce the number of packets sent over WiFi by buffering the data you send
  • improve the reliability of a serial connection by adding error correction codes
  • debug your program more easily by logging what it sends to a Web service
  • send large data with the Wire library
  • use a String, EEPROM, or PROGMEM with a stream interface
  • decode HTTP chunks

Read on to see how StreamUtils can help you!

How to add buffering to a Stream?

Buffering read operations

Sometimes, you can significantly improve performance by reading many bytes at once. For example, according to SPIFFS's wiki, reading files in chunks of 64 bytes is much faster than reading them one byte at a time.

ReadBufferingStream

To buffer the input, decorate the original Stream with ReadBufferingStream. For example, suppose your program reads a JSON document from SPIFFS like this:

File file = SPIFFS.open("example.json", "r");
deserializeJson(doc, file);

Then you only need to insert one line to greatly improve the reading speed:

File file = SPIFFS.open("example.json", "r");
ReadBufferingStream bufferedFile{file, 64};  // <- HERE
deserializeJson(doc, bufferedFile);

Unfortunately, this optimization is only possible if:

  1. Stream.readBytes() is declared virtual in your Arduino Code (as it's the case for ESP8266), and
  2. the derived class has an optimized implementation of readBytes() (as it's the case for SPIFFS' File).

When possible, prefer ReadBufferingClient to ReadBufferingStream because Client defines a read() method similar to readBytes(), except this one is virtual on all platforms.

If memory allocation fails, ReadBufferingStream behaves as if no buffer was used: it forwards all calls to the upstream Stream.

Adding a buffer only makes sense for unbuffered streams. For example, there is no benefit to adding a buffer to serial ports because they already include an internal buffer.

Buffering write operations

Similarly, you can improve performance significantly by writing many bytes at once. For example, writing to WiFiClient one byte at a time is very slow; it's much faster if you send large chunks.

WriteBufferingStream

To add a buffer, decorate the original Stream with WriteBufferingStream. For example, if your program sends a JSON document via WiFiClient like this:

serializeJson(doc, wifiClient);

Rewrite it like this:

WriteBufferingStream bufferedWifiClient{wifiClient, 64};
serializeJson(doc, bufferedWifiClient);
bufferedWifiClient.flush();

flush() sends the remaining data; if you forget to call it, the end of the message will be missing. The destructor of WriteBufferingStream calls flush(), so you can remove this line if you destroy the decorator immediately.

If memory allocation fails, WriteBufferingStream behaves as if no buffer was used: it forwards all calls to the upstream Stream.

Adding a buffer only makes sense for unbuffered streams. For example, there is no benefit to adding a buffer to serial ports because they already include an internal buffer.

How to add logging to a stream?

Logging write operations

When debugging a program that makes HTTP requests, you first want to check whether the request is correct. With this library, you can decorate the EthernetClient or the WiFiClient to log everything to the serial.

WriteLoggingStream

For example, if your program is:

client.println("GET / HTTP/1.1");
client.println("User-Agent: Arduino");
// ...

Then, create the decorator and update the calls to println():

WriteLoggingStream loggingClient(client, Serial);
loggingClient.println("GET / HTTP/1.1");
loggingClient.println("User-Agent: Arduino");
// ...

Everything you write to loggingClient is written to client and logged to Serial.

Logging read operations

Similarly, you often want to see what the HTTP server sent back. With this library, you can decorate the EthernetClient or the WiFiClient to log everything to the serial.

ReadLoggingStream

For example, if your program is:

char response[256];
client.readBytes(response, 256);

Then, create the decorator and update the calls to readBytes():

ReadLoggingStream loggingClient(client, Serial);
char response[256];
loggingClient.readBytes(response, 256);
// ...

loggingClient forwards all operations to client and logs read operations to Serial.

âš  WARNING âš 
If your program receives data from one serial port and logs to another, ensure the latter runs at a much higher baud rate. Logging must be at least ten times faster, or it will slow down the receiving port, which may drop incoming bytes.

Logging read and write operations

Of course, you could log read and write operations by combining ReadLoggingStream and WriteLoggingStream, but there is a simpler solution: LoggingStream.

LoggingStream

As usual, if your program is:

client.println("GET / HTTP/1.1");
client.println("User-Agent: Arduino");

char response[256];
client.readBytes(response, 256);

Then, decorate client and replace the calls:

LoggingStream loggingClient(client, Serial);

loggingClient.println("GET / HTTP/1.1");
loggingClient.println("User-Agent: Arduino");

char response[256];
loggingClient.readBytes(response, 256);

How to use error-correction codes (ECC)?

StreamUtils supports the Hamming(7, 4) error-correction code, which encodes 4 bits of data into 7 bits by adding three parity bits. These extra bits increase the amount of traffic but allow correcting any one-bit error within the 7 bits.

If you use this encoding on an 8-bit channel, it effectively doubles the amount of traffic. However, if you use an HardwareSerial instance (like Serial, Serial1...), you can slightly reduce the overhead by configuring the ports as a 7-bit channel, like so:

// Initialize serial port with 9600 bauds, 7 bits of data, no parity, and one stop bit
Serial1.begin(9600, SERIAL_7N1);

Adding parity bits

The class HammingEncodingStream<7, 4> decorates an existing Stream to include parity bits in every write operation.

HammingEncodingStream

You can use this class like so:

HammingEncodingStream<7, 4> eccSerial(Serial1);

eccSerial.println("Hello world!");

Like every Stream decorator in this library, HammingEncodingStream<7, 4> supports all Stream methods (like print(), println(), read(), readBytes(), and available()).

Correcting errors

The class HammingDecodingStream<7, 4> decorates an existing Stream to decode parity bits in every read operation.

HammingDecodingStream

You can use this class like so:

HammingDecodingStream<7, 4> eccSerial(Serial1);

char buffer[256];
size_t n = eccSerial.readBytes(buffer, n);

Like every Stream decorator in this library, HammingDecodingStream<7, 4> supports all Stream methods (like print(), println(), read(), readBytes(), and available()).

Encoding and decoding in both directions

The class HammingStream<7, 4> combines the features of HammingEncodingStream<7, 4> and HammingDecodingStream<7, 4>, which is very useful when you do two-way communication.

HammingStream

You can use this class like so:

HammingStream<7, 4> eccSerial(Serial1);

eccSerial.println("Hello world!");

char buffer[256];
size_t n = eccSerial.readBytes(buffer, n);

Like every Stream decorator in this library, HammingStream<7, 4> supports all Stream methods (like print(), println(), read(), readBytes(), and available()).

How to retry write operations?

Sometimes, a stream is limited to the capacity of its internal buffer. In that case, you must wait before sending more data. To solve this problem, StreamUtils provides the WriteWaitingStream decorator:

WriteWaitingStream

This function repeatedly waits and retries until it times out. You can customize the wait() function; by default, it's yield().

For example, if you want to send more than 32 bytes with the Wire library, you can do the following:

WriteWaitingStream wireStream(Wire, [](){
  Wire.endTransmission(false); // <- don't forget this argument
  Wire.beginTransmission(address);
});

Wire.beginTransmission(address); 
wireStream.print("This is a very very long message that I'm sending!");
Wire.endTransmission();

As you can see, we use the wait() function as a hook to flush the Wire transmission buffer. Notice that we pass false to endTransmission() so that it sends the data but doesn't actually stop the transmission.

How to use a String as a stream?

Writing to a String

Sometimes, you use a piece of code that expects a Print instance (like ReadLoggingStream), but you want the output in a String instead of a regular Stream. In that case, use the StringPrint class. It wraps a String within a Print implementation.

StringPrint

Here is how you can use it:

StringPrint stream;

stream.print("Temperature = ");
stream.print(22.3);
stream.print(" °C");

String result = stream.str();

At the end of this snippet, the string result contains:

Temperature = 22.30 °C

Reading from a String

Similarly, there are cases where you have a String, but you need to pass a Stream to some other piece of code. In that case, use StringStream; it's similar to StrintPrint, except you can also read from it.

StringStream

How to use EEPROM as a stream?

SteamUtils also allows using EEPROM as a stream. Create an instance of EepromStream and specify the start address and the size of the region you want to expose.

EepromStream

For example, it allows you to save a JSON document in EEPROM:

EepromStream eepromStream(0, 128);
serializeJson(doc, eepromStream);
eepromStream.flush();  // <- calls EEPROM.commit() on ESP (optional)

In the same way, you can read a JSON document from EEPROM:

EepromStream eepromStream(0, 128);
deserializeJson(doc, eepromStream);

How to use PROGMEM as a stream?

SteamUtils also allows reading PROGMEM buffers with a Stream interface.

ProgmemStream

Create an instance of ProgmemStream and pass the pointer to the PROGMEM buffer.

const char buffer[] PROGMEM = "This string is in program memory"
ProgmemStream stream{buffer};
Serial.println(stream.readString());

ProgmemStream's constructor also supports const __FlashStringHelper* (the type returned by the F() macro) and an optional second argument to specify the size of the buffer.

How to decode HTTP chunks?

HTTP servers can send their response in multiple parts using Chunked Transfer Encoding. Clients using HTTP 1.1 must support this encoding as it's not optional and is dictated by the server.

ChunkDecodingStream and ChunkDecodingClient are decorators that decode the chunks and make the response available as a regular stream.

ChunkDecodingStream

Here is an example using HTTPClient:

// Initialize HTTPClient
HTTPClient http;
http.begin(client, url);

// Tell HTTPClient to collect the Transfer-Encoding header
// (by default HTTPClient discards the response headers)
const char *keys[] = {"Transfer-Encoding"};
http.collectHeaders(keys, 1);

// Send the request
int status = http.GET();
if (status != 200) return;

// Create the raw and decoded stream
Stream& rawStream = http.getStream();
ChunkDecodingStream decodedStream(http.getStream());

// Choose the stream based on the Transfer-Encoding header
Stream& response = http.header("Transfer-Encoding") == "chunked" ? decodedStream : rawStream;

// Read the response
JsonDocument doc;
deserializeJson(doc, response);

// Close the connection
http.end();

Note that HTTPClient already performs chunk decoding if you use getString(), but you might want to use getStream() to avoid buffering the entire response in memory.

Also, you can avoid chunked transfer encoding by downgrading the HTTP version to 1.0. HTTPClient allows you to do that by calling useHTTP10(true) before sending the request.

Summary

Some of the decorators are also available for the Print and Client classes. See the equivalence table below.

Purpose Client Stream Print
Log write operations WriteLoggingClient WriteLoggingStream LoggingPrint
Log read operations ReadLoggingClient ReadLoggingStream
Log read and write op. LoggingClient LoggingStream
Buffer write operations WriteBufferingClient WriteBufferingStream BufferingPrint
Buffer read operations ReadBufferingClient ReadBufferingStream
Repeat write operations WriteWaitingClient WriteWaitingStream WaitingPrint
Use String as a stream StringStream StringPrint
Use EEPROM as a stream EepromStream
Use PROGMEM as a stream ProgmemStream
Error correction (decode only) HammingDecodingClient HammingDecodingStream
Error correction (encode only) HammingEncodingClient HammingEncodingStream HammingPrint
Error correction (encode & decode) HammingClient HammingStream
Decode HTTP chunks ChunkDecodingClient ChunkDecodingStream

Prefer XxxClient to XxxStream because, unlike Stream::readBytes(), Client::read() is virtual on all cores and therefore allows optimized implementations.

Portability

This library relies on Client, Print, and Stream definitions, which unfortunately differ from one core to another.

It has been tested on the following cores:

If your core is not supported, please open an issue. Thank you for your understanding.