ESP8266HttpClient never seems to finish large files when using lwip2

[schlaegerz]

Basic Infos

Hardware

Hardware: NodeMCU 1.0 (ESP-12E)
Core Version: 2.4

Description

I am trying to download a file that is around 10MB, and on lwip2 the file download seems to get around 1MB done and then just stops. If i go back to 1.4 then it finishes the download, but takes about 4min.

Settings in IDE

Module: NodeMCU 1.0 (ESP-12E)
Flash Size: 4M
CPU Frequency: 160Mhz
Flash Mode:
Upload Using: Serial

Code:

while (success &&
http.connected() &&
(downloadRemaining > 0 || downloadRemaining == -1)
) {
// get available data size

    auto size = stream->available();
    if (size > 0) {
        auto c = stream->readBytes(cur_buffer, ((size > CHUNK_SIZE) ? CHUNK_SIZE : size));
        cur_buffer = cur_buffer + c;
        auto totalSize = cur_buffer - buff;
        if (totalSize + 2* CHUNK_SIZE > buffSize)
        {
            cur_buffer = buff;
        }
         
        hasDownload = true;
        success &= c > 0;
        if (downloadRemaining > 0) {
            downloadRemaining -= c;
        }
        lastDownloadTime = millis();
    }
    yield();
    //delay(1);
    static int i = 0;
    i++;
    if (i % 2000 == 0)
    {
        Serial.println(downloadRemaining);
    }
}

[d-a-v]

Could you please post a full sketch so the issue can be reproduced ?

[schlaegerz]

This sketch downloaded 2Mb for me and then stalled.

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>

#define USE_SERIAL Serial

ESP8266WiFiMulti WiFiMulti;


#define CHUNK_SIZE HTTP_TCP_BUFFER_SIZE/2
uint8_t *buff;
const int buffSize = 4096;
void setup() {

    USE_SERIAL.begin(115200);
    // USE_SERIAL.setDebugOutput(true);

    USE_SERIAL.println();
    USE_SERIAL.println();
    USE_SERIAL.println();

    for (uint8_t t = 4; t > 0; t--) {
        USE_SERIAL.printf("[SETUP] WAIT %d...\n", t);
        USE_SERIAL.flush();
        delay(1000);
    }

    WiFi.mode(WIFI_STA);
    WiFiMulti.addAP("SSID", "password");
    buff = new uint8_t[buffSize];

}



void loop() {
    // wait for WiFi connection
    if ((WiFiMulti.run() == WL_CONNECTED)) {

        HTTPClient http;

        USE_SERIAL.print("[HTTP] begin...\n");

        // configure server and url
        http.begin("http://ipv4.download.thinkbroadband.com/20MB.zip");
        //http.begin("192.168.1.12", 80, "/test.html");

        USE_SERIAL.print("[HTTP] GET...\n");
        // start connection and send HTTP header
        int httpCode = http.GET();
        if (httpCode > 0) {
            // HTTP header has been send and Server response header has been handled
            USE_SERIAL.printf("[HTTP] GET... code: %d\n", httpCode);

            // file found at server
            if (httpCode == HTTP_CODE_OK) {

                // get lenght of document (is -1 when Server sends no Content-Length header)
                int downloadRemaining = http.getSize();
                // create buffer for read
                uint8_t * cur_buffer = buff;
                // get tcp stream
                WiFiClient * stream = http.getStreamPtr();
                bool success = true;
                // read all data from server
                while (success &&
                    http.connected() &&
                    (downloadRemaining > 0 || downloadRemaining == -1)
                    ) {
                    // get available data size

                    auto size = stream->available();
                    if (size > 0) {
                        auto c = stream->readBytes(cur_buffer, ((size > CHUNK_SIZE) ? CHUNK_SIZE : size));
                        cur_buffer = cur_buffer + c;
                        auto totalSize = cur_buffer - buff;
                        if (totalSize + 2 * CHUNK_SIZE > buffSize)
                        {
                            cur_buffer = buff;
                        }

                        success &= c > 0;
                        if (downloadRemaining > 0) {
                            downloadRemaining -= c;
                        }
                    }
                    yield();
                    //delay(1);
                    static int i = 0;
                    i++;
                    if (i % 2000 == 0)
                    {
                        Serial.println(downloadRemaining);
                    }
                }


                USE_SERIAL.println();
                USE_SERIAL.print("[HTTP] connection closed or file end.\n");

            }
        }
        else {
            USE_SERIAL.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
        }

        http.end();
    }
    delay(1000);
}

This stalls with lwip2 but not lwip1.4

[d-a-v]

I have made my tests, no stall.

• lwip2/536: 307s@545kbits/s
• lwip2/1460: 152s@1100kbits/s
• lwip1/1460: 134s@1245kbits/s (<-- you have ~240s).

My downlink BW is ~20Mbits/s.
Did you try lwip2/1460 ?
How long did you wait before giving up the run (there are timers retriggering transfers)?
Could you make a tcp dump of the traffic ?

[schlaegerz]

I just ran it for 20 min until the http connection automatically closed, it downloaded less than half of the file.

When I tried lwip2/1460 it took about 670s

So it seems like the problem is more specifically with lwip2/536

What is the best way to get a tcp dump for this?

Also side note, is there something I can do to speed up the download I get nowhere near your speeds. On the same router my computer gets 30Mb/s

[d-a-v]

The website is 13 hops far from me. Try with

tcptraceroute ipv4.download.thinkbroadband.com 80

The farther peer is, the more chance you get lost packets (among tons of other reasons to get lost packets). TCP has to retransmit them. TCP has its limits, Designers set up tcp stacks with good parameters so it can just work. Under linux and I guess in all regular TCP stacks, the TCP window size (= the receive buffer) is at least 64KB.
With our ESP, it is ~6000 with MSS=1460 and ~2000 with MSS=536. I personally have no doubt about the ESP stability with lwIP-v2 (up to MSS=896) - one always may be wrong - and we cannot dedicate all RAM for tcp.
You still can tweak TCP constants and buffer size in lwipopts.h to try to make it more efficient (TCP_WND, TCP_SND_BUF, ...). Peer's tcp stack also has to be considered: for example a while ago when I was testing lwip2, I realized that there was an issue with linuxkernel-4.4 (I could try and trigger it again) in the fast retransmit algorithm. 2-way traffic just stopped with linux failing to honour the resend request.

In this issue, the slowness could be due to ESP's inability to store many / large holes so lots of retransmissions has to happen on a path with higher probability for lost packets. In these cases, the retransmit-timeout TCP algorithm is more often triggered which dramatically decreases performances.

If we want to precisely understand what is happening in this issue, tcp dumps are needed, but I'm afraid there's nothing more we can do than understand and/or dedicate all the RAM for tcp.

For tcp dumps, a way is to install and run tcpdump on your router (if it can), or wireshark on your PC if you can get the ESP traffic go through it (via an usb-wlan in AP mode).

[schlaegerz]

I'm 15 hops away, do you think that could account for the difference between our downloads? Me getting 5 times slower download seems pretty bad, is there anything else you did differently?

I'll work on getting a tcp dump to see if I can get any more info, but i think I'll just stick lwip1.4 for now since it seems like the faster lwip2 may be unstable?

[d-a-v]

I did nothing different, I can't. Everybody would do if it was possible.
I would consider lwip2/mss=1460 less unstable than lwip1 until I'm proven wrong, the "unstable"ness applies for both.
The average TCP BW can vary from run to run in same conditions.
What you could do is recompile lwip with different values for TCP_WND and report back here.

[schlaegerz]

OK, I can try that, what makes these unstable? Just so I know what to look out for, is it just lower RAM?

[d-a-v]

Can you please retry your sketch with replacing HTTP_TCP_BUFFER_SIZE (always 1460) by TCP_MSS (536 with lower memory or 1460 with higher bandwidth and lwip-1.4 ) ?

[ghost]

One other question about that constants: Is it a bug that HTTP_TCP_BUFFER_SIZE does not always match TCP_MSS?

[d-a-v]

Yes and no. It was initially set to MSS, and was not changed since MSS changed.
This is not required and should work with any large enough value.
What to do has not been discussed yet

• keep TCP_MSS for the buffer length (then add TCP_MSS definition when lwip1.4 is selected) and use it instead of 1460.
• decide what is "large enough" and use it
• use an on-demand buffer length

[d-a-v]

Regarding my previous comment, MSS=1460 is not at all unstable (I was responsible for saying that and I've proven myself wrong).
And after reading ESP8266HTTPClient.cpp again, there is absolutely no need to limit the buffer size to any fixed value. If the required len is mallocable, then we can use it even if it is greater than any MSS. We are dealing here with TCP (unbounded stream) not UDP (packet bounded).

[ghost]

Thank you, the last two comments helped me to understand the meaning of those two constants.

[igrr]

For context, this HTTP_TCP_BUFFER_SIZE constant originates from times when WiFiClient::write would not accept buffers larger than TCP_MSS. Nowadays WiFiClient::write can accept any length of data, and will fragment it as necessary (as window size is increased by LwIP).

[ghost]

There is another related constant WIFICLIENT_MAX_PACKET_SIZE which is always 1460, regardless what lwip option I select in the board configuration. Shouldn't that match TCP_MSS in case of lwip v2?

[d-a-v]

This constant has not real meaning too. It is used nowhere and should be removed. There is no such notion of packet in TCP.
A TCP connection is a stream, and the user needs to see it that way. There is no bound, and WiFiClient, lwIP and any decent TCP implementation work like that.

[ghost]

"A TCP connection is a stream, and the user needs to see it that way. "
Yes of course, but in case of UDP the maximum packet size matters.

[d-a-v]

Very true. UDP with its datagram/packet way of transferring data is ruled by MTU which is not defined as a constant in our core, but is accessible through lwIP's netif->mtu and is 1460 (given by espressif firmware).
Arduino's vision of UDP is wrongly a "class Stream" (check #2928 too).

[ghost]

Thank your for clarification, and for the good work of course.

[schlaegerz]

I tried this again on the newer version and am still getting the same issue whenever I choose v2

[laercionit]

I have the same problem, in this case with 5MB files downloaded from the internet to the SD Card.
It works most of the time, but in some tests the process stops and does not continue.

I could not trace where the problem originated.

[d-a-v]

I sometimes can reproduce this issue, but not always, that makes it difficult to debug.
I will setup a bad network environment using netem for further debugging.
@schlaegerz with v1.4 you never have this issue ?

[laercionit]

@d-a-v

I was able to test with V2 Low, V2 High and V1.4 High.

Unfortunately I could not compile with V1.4 Compile from Source [exec: "make": executable file not found in% PATH% NodeMCU 1.0 (ESP-12E Module).

In all cases I had the same error, unloading instabilities of files. I'm guessing it's something related to Wifi encryption. By connecting through my 4G share through the cell phone, it worked without instabilities.

boolean OTADownloadFile(String URL, String PathToSave) {
  NetworkStopAll();
  if (NetworkInternetStatusLast == true) {
    HTTPClient http;
    http.begin(URL);
    int httpCode = http.GET();
    if ((httpCode > 0)) {
      if (httpCode == HTTP_CODE_OK) {
        unsigned long DownloadSize  = http.getSize();
        unsigned long RemainingSize = DownloadSize;
        if (DebugCheck(Debug)) {
          DebugSerial.println(String(F("OTADownloadFile: Get >> ")) + URL);
          DebugSerial.println(String(F("OTADownloadFile: File Size >> ")) + ConvertFormatBytes(DownloadSize));
          DebugSerial.println(String(F("OTADownloadFile: Save File as  >> ")) + PathToSave);
        }
        uint8_t buff[128] = { 0 };
        WiFiClient * stream = http.getStreamPtr();
        unsigned long DebugTimeCycle = millis();
        if (SD.exists(PathToSave)) SD.remove(PathToSave);
        File DownloadFile = SD.open(PathToSave, FILE_WRITE);
        if (!DownloadFile) DownloadFile = SD.open(PathToSave, FILE_WRITE);
        if (DownloadFile) {
          while (http.connected() && (RemainingSize > 0 || RemainingSize == -1)) {
            size_t size = stream->available();
            if (size) {
              int StreamBytes = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
              DownloadFile.write(buff, StreamBytes);
              //DownloadFile.flush();
              if (RemainingSize > 0) {
                RemainingSize -= StreamBytes;
                if ((DebugCheck(Debug)) && (millis() - DebugTimeCycle >= 1000)) {
                  DebugTimeCycle = millis();
                  float PercentDownloaded = (((DownloadSize - RemainingSize) * 100) / DownloadSize);
                  DebugSerial.print   (String(F("OTADownloadFile: Progress ")) + String(PercentDownloaded) + String(F("%")));
                  DebugSerial.print   (String(F("\tDownloaded ")) + ConvertFormatBytes(DownloadSize - RemainingSize));
                  DebugSerial.println (String(F("\tRemaining ")) + ConvertFormatBytes(RemainingSize));
                }
              }
            }
            delay(1);
          }
          //-----Verify Download  File ------
          if (DownloadFile.size() == DownloadSize) {
            if (DebugCheck(Info)) DebugSerial.println(String(F("OTADownloadFile: Download file DONE")));
            DownloadFile.close();
            return (true);
          } else {
            if (DebugCheck(Error)) DebugSerial.println(String(F("OTADownloadFile: Download file FAIL")));
            DownloadFile.close();
          }
        } else if (DebugCheck(Error)) DebugSerial.println(String(F("OTADownloadFile: Open File Failed")));
      } else if (DebugCheck(Error))DebugSerial.printf("OTADownloadFile: Server response, error: % s\n", http.errorToString(httpCode).c_str());
    } else if (DebugCheck(Error))DebugSerial.printf("OTADownloadFile: Server response, error: % s\n", http.errorToString(httpCode).c_str());
    http.end();
  }
  return (false);

[ghost]

Inever encountered the issue with v1.4

[schlaegerz]

Same it has never been a problem on 1.4

[laercionit]

@d-a-v @schlaegerz
Hello people good night.

I ran several tests and could not determine a setup option that works.

Anyone else with the same problem?

The only test that worked very well was to use my 4G shared internet phone. Is it something related to the type of WIFI encryption?

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <ESP8266HTTPClient.h>

#include <SPI.h>
#include <SD.h>

#define PinSDSS       15
#define DebugSerial Serial

ESP8266WiFiMulti WiFiMulti;


String ConvertFormatBytes(size_t bytes) {
  if (bytes < 1024) {
    return String(bytes) + F("B");
  } else if (bytes < (1024 * 1024)) {
    return String(bytes / 1024.0) + F("KB");
  } else if (bytes < (1024 * 1024 * 1024)) {
    return String(bytes / 1024.0 / 1024.0) + F("MB");
  } else {
    return String(bytes / 1024.0 / 1024.0 / 1024.0) + F("GB");
  }
}

boolean OtaDownloadFile(String URL, String PathToSave) {
  if ((WiFiMulti.run() == WL_CONNECTED)) {
    HTTPClient http;
    http.begin(URL);
    int httpCode = http.GET();
    if ((httpCode > 0)) {
      if (httpCode == HTTP_CODE_OK) {
        int DownloadSize  = http.getSize();
        int RemainingSize = DownloadSize;
        DebugSerial.println(String(F("OTADownloadFile: Get >> ")) + URL);
        DebugSerial.println(String(F("OTADownloadFile: File Size >> ")) + ConvertFormatBytes(DownloadSize));
        DebugSerial.println(String(F("OTADownloadFile: Save File as  >> ")) + PathToSave);
        uint8_t buff[128] = { 0 };
        WiFiClient * stream = http.getStreamPtr();
        unsigned long DebugTimeCycle = millis();

        if (SD.exists(PathToSave)) SD.remove(PathToSave);
        File DownloadFile = SD.open(PathToSave, FILE_WRITE);
        if (!DownloadFile) DownloadFile = SD.open(PathToSave, FILE_WRITE);
        if (DownloadFile) {

          while (http.connected() && (RemainingSize > 0 || RemainingSize == -1)) {
            auto size = stream->available();
            if (size) {

              auto StreamBytes = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
              DownloadFile.write(buff, StreamBytes);
              DownloadFile.flush();

              if (RemainingSize > 0) {
                RemainingSize -= StreamBytes;
                if ((millis() - DebugTimeCycle >= 1000)) {
                  DebugTimeCycle = millis();
                  float PercentDownloaded = (((DownloadSize - RemainingSize) * 100) / DownloadSize);
                  DebugSerial.print   (String(F("OTADownloadFile: Progress ")) + String(PercentDownloaded) + String(F("%")));
                  DebugSerial.print   (String(F("\tDownloaded ")) + ConvertFormatBytes(DownloadSize - RemainingSize));
                  DebugSerial.println (String(F("\tRemaining ")) + ConvertFormatBytes(RemainingSize));
                }
              }
            }
            yield();
          }
          //-----Verify Download  File ------
          if (DownloadFile.size() == DownloadSize) {
            DebugSerial.println(String(F("OTADownloadFile: Download file DONE")));
            DownloadFile.close(); http.end();
            return (true);
          } else {
            DebugSerial.println(String(F("OTADownloadFile: Download file FAIL")));
            DownloadFile.close(); http.end();
          }
        } else DebugSerial.println(String(F("OTADownloadFile: Open File Failed")));
      } else DebugSerial.printf("OTADownloadFile: Server response, error: % s\n", http.errorToString(httpCode).c_str());
    } else DebugSerial.printf("OTADownloadFile: Server response, error: % s\n", http.errorToString(httpCode).c_str());
    http.end();
    return (false);
  }
}



void setup() {

  DebugSerial.begin(115200);
  DebugSerial.setDebugOutput(true);
  for (uint8_t t = 4; t > 0; t--) {
    DebugSerial.printf("[SETUP] WAIT %d...\n", t);
    DebugSerial.flush();
    delay(1000);
  }
  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP("NETWORK", "1a2b3c4d");
  SD.begin(PinSDSS);
}

void loop() {
  if ((WiFiMulti.run() == WL_CONNECTED)) {
    OtaDownloadFile("http://XXXXXXXXXX/Arduino.zip", "/Update/Arduino.zip");
    delay(10000);
  }
}

[d-a-v]

After tcpdump-ing again and again, it seems that those lags are due to SACK (Selective ACKs) missing in lwip.

And what is great, is that lwIP folks implemented partial SACK in their v2.1 pre-release (lwip2 is currently using stable-2.0.3).

Running with this version seems to make large transfers fluent (not hanging anymore for secoooonnds nor stop).

Note: I have no idea why it works with lwip1.4 (I did not test that myself)

[d-a-v]

Update:
With 2.1rc1, I observe no more lag with ipv4.download.thinkbroadband.com/20MB.zip.
I ran 6 times the sketch using lwip2-lower-memory at (GMT+2) 11:49(10mn), 12:00(10mn), 12:11(10mn), 13:40(10mn), 14:23(6mn) and 15:39(6mn).

[laercionit]

@d-a-v

Hello, I'm happy with the news, at least we found the problem. Did the code example I mentioned above help?

Any prediction to launch a new core of esp8266 with this problem solved?

[d-a-v]

@laercionit I did not tried OTA but only this issue's sketch.
I'm not saying the issue is very different though.

Please allow some time to check general side effects of moving from lwIP-2.0.3 to lwIP-2.1.0.

[d-a-v]

Please test #5126

git fetch origin pull/5126/head:5126
git checkout 5126

Revert to master:

git checkout master

edit: in case of updates in the PR, do this before restarting the above.

git checkout master
git branch -D 5126

[teo1978]

Hello everybody.

#5126 seems to work for me too.
I am testing on a Sparkfun Thing Dev and I used to observe anywhere from occasional short lags (i.e. periods with 0 bytes available) to very long ones (minutes), sometimes leading to the connection being closed after several minutes without finishing the download. Now I'm not observing any of that and the downloads are reliable.

I am only downloading files of about 250 kB, though. My test case is similar to the one I posted here.

So far I haven't observed any obvious side effects, but of course I have tested very little, only running a single sketch a handful of times.

[d-a-v]

Thanks for the feedback !

[teo1978]

Even though I have experienced a great improvement since the fix (i.e. it doesn't get stuck "forever"), I still get several-seconds-long intervals with 0 bytes available way too often.

Is there any test you can suggest that would help me figure out whether this is real network slowness (i.e. weak signal) or something fishy that suggests there's still a software bug? I find it hard to believe the connection can be so poor when every other device in the same area gets a very strong signal and the router is just a few meters away (of course it's still possible that the ESP8266 performs more poorly than a smartphone or a laptop).

[teo1978]

Well??

[teo1978]

I have connected an antenna (this one) and the situation hasn't improved the slightest bit.

I am doing my tests few meters away from the router, where every other device (laptops, smartphones) have excellent connection.

I am downloading a file about 250kB in size. As I mentioned, the patch fixed the issue where the download would get stuck forever, but still, I very often observe that there are very long intervals of times (several seconds) with zero bytes received. Then (unlike before the fix), download resumes at a normal speed, but then it gets stuck again for several seconds, and so on. Overall, it often takes several minutes to download the 250kb file, while normally (i.e. when the download goes on at approximately constant speed) it takes about 30 seconds.

The physical reception just can't be that bad.