Fix bug for upgrading to OpenSSL 3.0. v5.0.189 v6.0.89

[duiniuluantanqin]

The fix is for the DH_set_length error. As shown in lines 2-5, OpenSSL 3.0 added a check for length, which allowed this issue to be exposed.

1 if (dh->params.q == NULL) {
2       /* secret exponent length, must satisfy 2^(l-1) <= p */
3        if (dh->length != 0
4            && dh->length >= BN_num_bits(dh->params.p))
5            goto err;
6        l = dh->length ? dh->length : BN_num_bits(dh->params.p) - 1;
7        if (!BN_priv_rand_ex(priv_key, l, BN_RAND_TOP_ONE,
8                             BN_RAND_BOTTOM_ANY, 0, ctx))
9            goto err;
        ... ...
    }

---

Co-authored-by: john hondaxiao@tencent.com

[winlinvip]

When we used this setting to create a 128-byte key length, it was during the RTMP handshake to generate a 128-byte shared key. But in reality, it's not necessary to call this function.

Also, during the RTMP handshake, we should be getting the shared key, but there seems to be a bit of an issue with the SRS logic, as it directly uses the private key. Of course, since the RTMP handshake doesn't verify the key, it doesn't really matter which key we use.

In addition, RTMP checks the length of the key. If it's not 128, like if it accidentally becomes 127 bytes, it will generate it again. This issue has been noticed in reality, but this logic isn't necessary. As long as the result is a 128-byte key, it's fine.

Basically, RTMP clients don't usually check the 128-byte key data during the handshake process. So, this background might not be clear, hence the special note for clarification.

TRANS_BY_GPT4

[winlinvip]

We need to merge into 5 and 6.

TRANS_BY_GPT4