HTTPS Server works on raspberrypi but not on espressif #8268

ams1 · 2023-08-02T11:55:14Z

Hi,

Thank you for this AWESOME initiative!

Any plan to have an example for SSL http server?

michalpokusa · 2023-08-02T12:17:35Z

Hi, thanks for kind word.

Unfortunately right now implementing SSL is problematic, due to the amount of compute power necessary, it is very slow and in my testing does not work at all on some microcontrollers.

Related issue on CircuitPython itself:
#7657

Attempt to make a HTTPS server:
https://github.com/ide/circuitpython-https-server

If you want to play with it, I would appreciate you sharing your results.

But if you really need a HTTPS, one way is to use e.g. nginx on Raspberry Pi (Zero/4), but at this point you can simply host a website on it too.

zap8600 · 2023-08-07T05:26:17Z

I've tried using the example on CircuitPython 8.2.0, using both version 4.2.0 and 2.3.0, on an Adafruit Feather ESP32-S2. The error I get is the same on both HTTP server versions. The error below is from my attempt to use version 2.3.0, modified to print the socket type on server start. Here is the error I'm getting.

Traceback (most recent call last):
  File "code.py", line 79, in <module>
  File "code.py", line 49, in main
  File "/lib/adafruit_httpserver/server.py", line 137, in poll
RuntimeError: Invalid socket for TLS

Here is the line the error is happening at.

            conn, client_address = self._sock.accept()

Hope this helps!

anecdata · 2023-08-07T06:28:57Z

What are you using for a client? curl -v might yield some clues.

michalpokusa · 2023-08-07T09:26:05Z

During my testing I was getting the exact same error. I was testing using curl, VSCode extension Thunder Client, a script in Python with requests, Google Chrome and even other microcontroller (ESP32-S2 TFT) with adafruit_requests.

I believe it is something in the CP itself. The interesting thing is that adafruit_requests as far as I know can make requests to HTTPS endpoints, so the capability is there.

zap8600 · 2023-08-07T12:57:14Z

What are you using for a client? curl -v might yield some clues.

I'm using curl. curl will stall, then throw an error once the microcontroller resets.

anecdata · 2023-08-07T13:24:07Z

The curl switch -v gives detail about the SSL transaction, I thought it might give a clue where in the process it breaks.

zap8600 · 2023-08-07T13:27:21Z

The curl switch -v gives detail about the SSL transaction, I thought it might give a clue where in the process it breaks.

I haven't looked much into it, but I think it only has to do with CP. Both curl and the browser don't get anything back from the microcontroller. They'll throw a connection reset error when the microcontroller resets, though.

anecdata · 2023-08-07T13:49:04Z

I loaded up a Pico W and a QT Py S2 and a QT Py S3 (all on CP 8.2.0, all freshly reset).

It works on Pico W:

% curl -v --insecure https://192.168.6.198
*   Trying 192.168.6.198:443...
* Connected to 192.168.6.198 (192.168.6.198) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=picow.local
*  start date: Feb 28 05:42:56 2023 GMT
*  expire date: Feb 28 05:42:56 2024 GMT
*  issuer: CN=picow.local
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 192.168.6.198
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Length: 117
< Content-Type: text/html
< Connection: close
< 
<!DOCTYPE html>
<title>Hello world &middot; Raspberry Pi Pico W</title>
<p>Hello world from Raspberry Pi Pico W!</p>
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):

It breaks early in the TLS handshake on S2 and S3:

% curl -v --insecure https://192.168.6.210
*   Trying 192.168.6.210:443...
* Connected to 192.168.6.210 (192.168.6.210) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection 0
curl: (35) Recv failure: Connection reset by peer

Same behavior with CP 8.2.2 on S2.

Probably some difference in common-hal, or esp-idf. Perhaps we should move this issue to the circuitpython repo.

anecdata · 2023-08-07T14:10:55Z

code.py passes in server_side=True in line 74: return self._ssl_context.wrap_socket(socket, server_side=True)
https://docs.circuitpython.org/en/latest/shared-bindings/ssl/index.html#ssl.SSLContext.wrap_socket
espressif does nothing with it:

circuitpython/ports/espressif/common-hal/ssl/SSLContext.c

Line 40 in 6b36bf5

    
           ssl_sslsocket_obj_t *common_hal_ssl_sslcontext_wrap_socket(ssl_sslcontext_obj_t *self,

compared to raspberrypi:

circuitpython/ports/raspberrypi/common-hal/ssl/SSLSocket.c

Line 166 in 6b36bf5

ret = mbedtls_ssl_config_defaults(&o->conf,

zap8600 · 2023-08-07T22:42:43Z

So we just need to modify the espressif port to check for the argument?

anecdata · 2023-08-08T00:52:09Z

There's presumably some ~~lwip~~ logic needed, predicated on server_side=True. But it is odd that the exception is Invalid socket for TLS, which I think indicates the socket isn't SOCK_STREAM, though the library sets it when it creates the socket to listen to.

dhalbert · 2023-08-11T14:11:58Z

We could backport this to 8.2.x if it gets figured out soon.

michalpokusa · 2023-11-05T23:06:53Z

Is there any progress on fixing/updating ESPs to allow HTTPS? I would like to make an update to adafruit_httpserver that allows HTTPS, but currently for all I know only RPi Pico W supports what is needed for it to work.

zap8600 · 2023-11-29T20:19:38Z

Why don't we just add

    ret = mbedtls_ssl_config_defaults(&o->conf,
        server_side ? MBEDTLS_SSL_IS_SERVER : MBEDTLS_SSL_IS_CLIENT,
        MBEDTLS_SSL_TRANSPORT_STREAM,
        MBEDTLS_SSL_PRESET_DEFAULT);
    if (ret != 0) {
        goto cleanup;
    }

to common_hal_ssl_sslcontext_wrap_socket? Is there more that's supposed to be done? I'll make a fork and test this.

tannewt · 2024-02-14T18:07:58Z

@michalpokusa or @anecdata did you try @zap8600's suggestion?

jepler · 2024-02-15T01:19:53Z

I don't think calling mbedtls APIs here will work, the espressif SSLContext is using ep-idf calls like esp_tls_init.

It's tempted to think we could/should use mbedtls API calls (not esp-idf calls) for both espressif & raspberrypi rp2 (I think this is what micropython does!) but that would be a big change.

anecdata · 2024-02-16T02:46:21Z

Does anyone have working test code for HTTPS Server for espressif with this PR? I'm getting RuntimeError: Invalid socket for TLS using code from https://github.com/ide/circuitpython-https-server updated for latest adafruit_httpserver. (I may be doing something wrong)

i.e., how do we know this is completed?

michalpokusa · 2024-02-16T03:36:37Z

Does anyone have working test code for HTTPS Server for espressif with this PR? I'm getting RuntimeError: Invalid socket for TLS using code from https://github.com/ide/circuitpython-https-server updated for latest adafruit_httpserver. (I may be doing something wrong)

i.e., how do we know this is completed?

I also started trying HTTPS with adafruit_httpserver on ESP32-S2 TFT Feather. I get exactly the same error, specifically it gets raised by Server._sock.accept() in Server.poll().

anecdata · 2024-02-16T05:12:39Z

client side (essentially same output for ESP32-S3 and Pico W:

% curl -v --insecure https://192.168.6.243
*   Trying 192.168.6.243:443...
* Connected to 192.168.6.243 (192.168.6.243) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection
curl: (35) Recv failure: Connection reset by peer

jepler · 2024-02-16T16:11:58Z

@anecdata please re-test with #8932, which fixes the "Invalid Socket for TLS" message. I tested on esp32-s3 (matrixportal s3) with ide/circuitpython-https-server#2

(however if it's not working on pico w either there's something else going on as well. That port did already set the socket type on an accepted socket, as far as I could tell from reading the source code)

anecdata · 2024-02-16T16:23:35Z

@jepler works for me now on espressif, and it's plenty fast (QT Py S3)... Thanks!

michalpokusa · 2024-02-16T23:53:16Z

@jepler I also tested on my MatrixPortal S3, works very fast, I also tested with Websockets and wss://.

But I was unable to make it work with ESP32-S2 Feather TFT, I can start the server but when trying to connect I get an empty MemoryError:. Seems strange considering that both have 2MB of RAM, although MatrixPortal has SRAM and Feather has PSRAM, would that make any difference?

@anecdata Could you maybe test on the Feather if you have one on hand?

EDIT1: MemoryError: is present even with .mpy version of the adafruit_httpserver

anecdata · 2024-02-17T00:11:59Z

I get the same on QT Py ESP32-S2 (N4R2), no dual http/https servers (just https):

server:

Traceback (most recent call last):
  File "adafruit_httpserver/server.py", line 404, in poll
MemoryError: 
Traceback (most recent call last):
  File "code.py", line 61, in <module>
  File "adafruit_httpserver/server.py", line 450, in poll
  File "adafruit_httpserver/server.py", line 404, in poll
MemoryError:

client:

$ curl -iL -v --insecure https://192.168.6.180?HTTPS
*   Trying 192.168.6.180:443...
* Connected to 192.168.6.180 (192.168.6.180) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Operation timed out
* LibreSSL/3.3.6: error:02FFF03C:system library:func(4095):Operation timed out
* Closing connection 0
curl: (35) Recv failure: Operation timed out

edit: yes, EESP32-S3 has more internal SRAM than ESP32-S2, but maybe there could be build differences or something else to account for it? I suspect there's some difference of SRAM available to esp-idf.

michalpokusa · 2024-02-17T00:17:25Z

If RPicoW with 264kB can handle HTTPS, maybe there is something else here that needs to be fixed? It is definitely better but still not completely working.

Would it be worth re-opening this issue?

anecdata · 2024-02-17T00:19:06Z

I think it needs an issue, either re-open or new. Some difference in S2 vs. S3, and also "https:// PicoW" breaks with latest everything (unless you've gotten it to work?)

iirc S2 has 320KB SRAM, and S3 has ~~540KB~~ 512KB SRAM, with various claims to that memory from esp-idf/rtos/etc, and maybe some things from CP core too. I don't fully understand the new CP 9 memory model.

michalpokusa · 2024-02-17T00:44:51Z

I double checked and you are right, RPicoW now has the same problem as Feather, which is MemoryError:.

I assumed that considering that only the espressif port changed, code would still work on raspberry.

michalpokusa · 2024-02-18T04:58:15Z

After updating to CP 9.0.0-beta.1 I get a different error:

This does not mean that the MemoryError is gone, as it might appear later, but something else changed that unables even ESP32-S3 boards like MatrixPortal S3 to start a HTTPS server.

anecdata · 2024-02-20T21:57:04Z

Pico W HTTPS Server seems to have broken with the MemoryError somewhere between 8.0.0 and 8.1.0.

tannewt · 2024-02-20T22:54:38Z

iirc S2 has 320KB SRAM, and S3 has ~~540KB~~ 512KB SRAM, with various claims to that memory from esp-idf/rtos/etc, and maybe some things from CP core too. I don't fully understand the new CP 9 memory model.

CP will try to allocate to PSRAM first now in beta.1+. (Not exactly sure when I changed it.) The IDF can allocate to PSRAM now too by default but needs to request it explicitly. Without an explicit request, then internal memory will be used first.

anecdata transferred this issue from adafruit/Adafruit_CircuitPython_HTTPServer Aug 7, 2023

anecdata changed the title ~~Example for SSL~~ HTTPS Server works on raspberrypi but not on espressif Aug 7, 2023

anecdata added bug network espressif applies to multiple Espressif chips labels Aug 7, 2023

dhalbert added this to the 9.0.0 milestone Aug 11, 2023

jepler mentioned this issue Feb 15, 2024

mbedtls: move to shared-module #8926

Merged

dhalbert closed this as completed in #8926 Feb 16, 2024

michalpokusa mentioned this issue Feb 18, 2024

Error: Failure to bind on ESP when using HTTPS with adafruit_httpserver #8947

Closed

anecdata mentioned this issue Feb 23, 2024

HTTPS Server now works on ESP32-S3 but MemoryError on Pico W and ESP32-S2 adafruit/Adafruit_CircuitPython_HTTPServer#87

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPS Server works on raspberrypi but not on espressif #8268

HTTPS Server works on raspberrypi but not on espressif #8268

ams1 commented Aug 2, 2023

michalpokusa commented Aug 2, 2023

zap8600 commented Aug 7, 2023

anecdata commented Aug 7, 2023

michalpokusa commented Aug 7, 2023

zap8600 commented Aug 7, 2023 •

edited

Loading

anecdata commented Aug 7, 2023

zap8600 commented Aug 7, 2023

anecdata commented Aug 7, 2023 •

edited

Loading

anecdata commented Aug 7, 2023

zap8600 commented Aug 7, 2023

anecdata commented Aug 8, 2023 •

edited

Loading

dhalbert commented Aug 11, 2023

michalpokusa commented Nov 5, 2023

zap8600 commented Nov 29, 2023

tannewt commented Feb 14, 2024

jepler commented Feb 15, 2024

anecdata commented Feb 16, 2024 •

edited

Loading

michalpokusa commented Feb 16, 2024

anecdata commented Feb 16, 2024

jepler commented Feb 16, 2024 •

edited

Loading

anecdata commented Feb 16, 2024 •

edited

Loading

michalpokusa commented Feb 16, 2024 •

edited

Loading

anecdata commented Feb 17, 2024 •

edited

Loading

michalpokusa commented Feb 17, 2024

anecdata commented Feb 17, 2024 •

edited

Loading

michalpokusa commented Feb 17, 2024

michalpokusa commented Feb 18, 2024 •

edited

Loading

anecdata commented Feb 20, 2024

tannewt commented Feb 20, 2024

HTTPS Server works on raspberrypi but not on espressif #8268

HTTPS Server works on raspberrypi but not on espressif #8268

Comments

ams1 commented Aug 2, 2023

michalpokusa commented Aug 2, 2023

zap8600 commented Aug 7, 2023

anecdata commented Aug 7, 2023

michalpokusa commented Aug 7, 2023

zap8600 commented Aug 7, 2023 • edited Loading

anecdata commented Aug 7, 2023

zap8600 commented Aug 7, 2023

anecdata commented Aug 7, 2023 • edited Loading

anecdata commented Aug 7, 2023

zap8600 commented Aug 7, 2023

anecdata commented Aug 8, 2023 • edited Loading

dhalbert commented Aug 11, 2023

michalpokusa commented Nov 5, 2023

zap8600 commented Nov 29, 2023

tannewt commented Feb 14, 2024

jepler commented Feb 15, 2024

anecdata commented Feb 16, 2024 • edited Loading

michalpokusa commented Feb 16, 2024

anecdata commented Feb 16, 2024

jepler commented Feb 16, 2024 • edited Loading

anecdata commented Feb 16, 2024 • edited Loading

michalpokusa commented Feb 16, 2024 • edited Loading

anecdata commented Feb 17, 2024 • edited Loading

michalpokusa commented Feb 17, 2024

anecdata commented Feb 17, 2024 • edited Loading

michalpokusa commented Feb 17, 2024

michalpokusa commented Feb 18, 2024 • edited Loading

anecdata commented Feb 20, 2024

tannewt commented Feb 20, 2024

zap8600 commented Aug 7, 2023 •

edited

Loading

anecdata commented Aug 7, 2023 •

edited

Loading

anecdata commented Aug 8, 2023 •

edited

Loading

anecdata commented Feb 16, 2024 •

edited

Loading

jepler commented Feb 16, 2024 •

edited

Loading

anecdata commented Feb 16, 2024 •

edited

Loading

michalpokusa commented Feb 16, 2024 •

edited

Loading

anecdata commented Feb 17, 2024 •

edited

Loading

anecdata commented Feb 17, 2024 •

edited

Loading

michalpokusa commented Feb 18, 2024 •

edited

Loading