sock_dtls: unable to send big messages

[HendrikVE]

Description

When running examples/gcoap_dtls it is not possible to send messages bigger than 99 bytes. When sending >= 100 bytes the application fails with gcoap_cli: msg send failed. The test apps in the original tinydtls repository have no such problem, but apparently the message is split into blocks of 199 bytes. Furthermore, it is most likely a dtls and not a gcoap issue, because the standard examples/gcoap works perfectly fine with bigger messages.

The dtls handshake works fine, but sending the actual message does not.

I have tried to verify the results with tests/pkg_tinydtls_sock_async but when sending more than 91 bytes of payload the test application crashes with *** stack smashing detected ***. However, the server app prints Received 92 bytes -- (echo) before crashing.

Steps to reproduce the issue

• take examples/gcoap_dtls
• increase the shell buffer, e.g. to SHELL_DEFAULT_BUFSIZE * 4
• increase CONFIG_GCOAP_PDU_BUF_SIZE to e.g. 1024 (CFLAGS += -DCONFIG_GCOAP_PDU_BUF_SIZE=1024 in the Makefile)

Expected results

Message exchange should work.

Actual results

> coap post [::1] 5683 / aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
gcoap_cli: sending msg ID 34050, 98 bytes
sock_dtls: decrypted message arrived
gcoap: response Error, code 4.05, empty payload
> coap post [::1] 5683 / aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
gcoap_cli: sending msg ID 34051, 99 bytes
sock_dtls: decrypted message arrived
gcoap: response Error, code 4.05, empty payload
> coap post [::1] 5683 / aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
gcoap_cli: sending msg ID 34052, 100 bytes
gcoap_cli: msg send failed

If you wonder about port 5683: it's proxied ;)

Versions

Latest master

[miri64]

Is the send-buffer maybe too small?

[HendrikVE]

Is the send-buffer maybe too small?

send-buffer? You mean my application-site buffer? It has size 256 and I am trying to send 106 bytes. Or is there an internal buffer I should know about?

[miri64]

gcoap_cli: msg send failed

Could you check, what the return value was?

[HendrikVE]

I went down the rabbit hole:

2022-04-27 00:27:58,600 # gcoap_cli: sending msg ID 63033, 99 bytes
2022-04-27 00:27:58,602 # DTLS_MAX_BUF = 120
2022-04-27 00:27:58,612 # pkg/tinydtls/dtls.c:1667 dtls_prepare_record returned 0
2022-04-27 00:27:58,623 # pkg/tinydtls/dtls.c:312 dsock_dtls: decrypted message arrived
2022-04-27 00:27:58,627 # gcoap: response Error, code 4.05, empty payload
2022-04-27 00:27:58,629 # tls_send returned 99
2022-04-27 00:27:58,638 # pkg/tinydtls/contrib/sock_dtls.c:648 dtls_write returned 99
2022-04-27 00:27:58,649 # sys/net/application_layer/gcoap/gcoap.c:1003 sock_dtls_send returned 99
> coap post ::1 5683 / aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
2022-04-27 00:28:01,712 # coap post ::1 5683 / aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
2022-04-27 00:28:01,716 # gcoap_cli: sending msg ID 63034, 100 bytes
2022-04-27 00:28:01,718 # DTLS_MAX_BUF = 120
2022-04-27 00:28:01,729 # build/pkg/tinydtls/dtls.c:1502 *rlen < res + DTLS_RH_LENGTH + data_len_array[i]
2022-04-27 00:28:01,739 # build/pkg/tinydtls/dtls.c:1667 dtls_prepare_record returned -592
2022-04-27 00:28:01,749 # build/pkg/tinydtls/dtls.c:312 dtls_send returned -592
2022-04-27 00:28:01,759 # pkg/tinydtls/contrib/sock_dtls.c:648 dtls_write returned -592
2022-04-27 00:28:01,770 # sys/net/application_layer/gcoap/gcoap.c:1003 sock_dtls_send returned -592
2022-04-27 00:28:01,772 # gcoap: sock send failed: -592
2022-04-27 00:28:01,775 # gcoap_cli: msg send failed 0

I've increased DTLS_MAX_BUF to 200 and now I can send larger messages 🎉

But the question now is: Is setting DTLS_MAX_BUF to 120 by default really the right thing to do? And if there was documentation about it I totally missed it. I did not even know there was an internal send-buffer :/

I just hit the next issue: sock_dtls: Chunked datagram payload currently not supported yet by tinydtls :(

[miri64]

I just hit the next issue: sock_dtls: Chunked datagram payload currently not supported yet by tinydtls :(

@leandrolanzieri do you have an idea what this is about?

[leandrolanzieri]

I just hit the next issue: sock_dtls: Chunked datagram payload currently not supported yet by tinydtls :(

@leandrolanzieri do you have an idea what this is about?

Hmm not sure, I don't remember bumping into this one. It seems that it was introduced with the addition of sock_async in #12907.

[miri64]

Ah... that's because @HendrikVE is using lwIP and that stack is delivering the UDP packet in chunks, not as a continuous memory area, as GNRC does ... not sure this is fixed quickly :(. @HendrikVE, maybe if you address the TODO this will work out?

[miri64]

Not sure, if TinyDTLS is able to handle chunked messages by now, though :-/

[HendrikVE]

Ah... that's because @HendrikVE is using lwIP and that stack is delivering the UDP packet in chunks, not as a continuous memory area, as GNRC does ... not sure this is fixed quickly :(. @HendrikVE, maybe if you address the TODO this will work out?

Is there a way to increase the chunk size in lwIP as a workaround such that it still delivers chunks, but only a single one? Would that help?

Not sure, if TinyDTLS is able to handle chunked messages by now, though :-/

When I executed the test application by sending like 1000 bytes of data, it was split into chunks of 199 bytes. But I don't know whether these are the chunks we are talking about here. The tinydtls repository does not mention anything about chunks, neither in PRs nor in issues or code in general.

[miri64]

Is there a way to increase the chunk size in lwIP as a workaround such that it still delivers chunks, but only a single one? Would that help?

Mh.. maybe @yarrick has an idea here? By just poking through the lwIP code, I was not able to find a quick way to increase the pbuf chunks.

When I executed the test application by sending like 1000 bytes of data, it was split into chunks of 199 bytes. But I don't know whether these are the chunks we are talking about here. The tinydtls repository does not mention anything about chunks, neither in PRs nor in issues or code in general.

Yeah, that's a thing specific to lwip. Since I was only testing sock_dtls under GNRC back then, I am not sure, I followed up on this.

[yarrick]

I am not sure how this is written but if the pbuf is chunked internally should not matter to any user (copy data using pbuf_take() and similar)

[miri64]

I am not sure how this is written but if the pbuf is chunked internally should not matter to any user (copy data using pbuf_take() and similar)

We wrote sock_dtls specifically to be able to handle fragmented packets, the problem is TinyDTLS does not support it. Mh, maybe we need to rethink that.