mbedTLS 3.6.0 breaks curl

[bagder]

I did this

#13646 tried to upgrade the mbedTLS CI jobs to use version 3.6.0

This caused numerous failed CI jobs.

I expected the following

green builds

curl/libcurl version

curl master

operating system

on Linux, but probably a universal problem

[jay]

for the failed tests it looks as though stunnel server is sending a tls alert of bad certificate. (it's not clear to me if it's sending or receiving the alert)

https://github.com/curl/curl/actions/runs/9086442477/job/24972131840#step:35:1170

 2024.05.14 21:33:45 LOG5[ui]: stunnel 5.63 on x86_64-pc-linux-gnu platform
 2024.05.14 21:33:45 LOG5[ui]: Compiled/running with OpenSSL 3.0.2 15 Mar 2022
 2024.05.14 21:33:45 LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,OCSP,PSK,SNI Auth:LIBWRAP
 2024.05.14 21:33:45 LOG5[ui]: Reading configuration from file /home/runner/work/curl/curl/tests/log/server/https_stunnel.conf
 2024.05.14 21:33:45 LOG5[ui]: UTF-8 byte order mark not detected
 2024.05.14 21:33:45 LOG5[ui]: FIPS mode disabled
 2024.05.14 21:33:45 LOG5[ui]: Configuration successful
 2024.05.14 21:33:45 LOG5[ui]: Binding service [curltest] to :::38897: Address already in use (98)
 2024.05.14 21:33:46 LOG5[0]: Service [curltest] accepted connection from 127.0.0.1:36328
 2024.05.14 21:33:46 LOG3[0]: SSL_accept: ../ssl/record/rec_layer_s3.c:1584: error:0A000412:SSL routines::sslv3 alert bad certificate
 2024.05.14 21:33:46 LOG5[0]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket

I can't reproduce in Linux however I did not copy the CI job configuration.

I built mbedtls like this:

curl -OL https://github.com/Mbed-TLS/mbedtls/releases/download/v3.6.0/mbedtls-3.6.0.tar.bz2
tar -xf mbedtls-3.6.0.tar.bz2
cd mbedtls-3.6.0/
LDFLAGS="-Wl,--enable-new-dtags -Wl,-rpath,/usr/local/lib" make SHARED=1
sudo make install

I built curl like LDFLAGS=... ./configure ... --enable-debug --without-openssl --with-mbedtls=/usr/local, make and then ran the full test suite, 100% ok.