future support for mbedTLS' TLS 1.3?

[andreiaugustin]

Hey there, I wanted to leave some notes somewhere for others that might attempt to use TLS 1.3 with mbedTLS as backend as currently it is not supported and I haven't found any notes in the docs. Apologies in advance if this is not the right area - please let me know and I will move this elsewhere.

At the moment, mbedTLS has a partial implementation of TLS 1.3 which started in mbedTLS 3.1.0.

As far as the cURL CLI goes, at the moment TLS 1.3 is not allowed when using mbedTLS as the backend. Furthermore, TLS 1.3, at the time of writing, has to be enabled through MBEDTLS_SSL_PROTO_TLS1_3 when building mbedTLS (could use scripts/config.py set MBEDTLS_SSL_PROTO_TLS1_3).

At first glance, the changes in cURL to have mbedTLS use TLS 1.3 are using MBEDTLS_SSL_MINOR_VERSION_4 for *mbedver if MBEDTLS_VERSION_NUMBER >= 0x03010000 and call psa_crypto_init() before any handshake, I've added those changes to my branch if anybody is interested.

When testing with https://google.com I am assuming the handshake is successful, but it breaks (Failed receiving HTTP2 data: 56(Failure when receiving data from the peer), using nghttp2) after the first HTTP2 GET request over TLS. Last logs from mbedTLS before breaking are:

/* Here I assume the handshake is OK as we're making a GET request */
* <= write
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/8.6.1-DEV
> Accept: */*
> 
* STATE: DO => DID handle 0x144813208; line 2272
* STATE: DID => PERFORMING handle 0x144813208; line 2390
* => read
* => read record
* => fetch input
* in_left: 0, nb_want: 5
* in_left: 0, nb_want: 5
* ssl->f_recv(_timeout)() returned 5 (-0xfffffffb)
* <= fetch input
* dumping 'input record header' (5 bytes)
* 0000:  17 03 03 02 23                                   ....#
* input record: msgtype = 23, version = [0x303], msglen = 547
* => fetch input
* in_left: 5, nb_want: 552
* in_left: 5, nb_want: 552
* ssl->f_recv(_timeout)() returned 547 (-0xfffffddd)
 
...

/* Here is the point of failure */
* dumping 'Ticket-resumed PSK' (32 bytes)
* 0000:  63 f8 5e cf 55 fa 46 c7 cb 40 c0 aa 35 68 58 2d  c.^.U.F..@..5hX-
* 0010:  9a b9 9d 15 ea dc 78 0e a0 39 95 ae 74 75 71 db  ......x..9..tuq.
* print ticket_flags (0x05)
* - ALLOW_PSK_RESUMPTION is set.
* - ALLOW_PSK_EPHEMERAL_RESUMPTION is set.
* <= parse new session ticket
* <= handshake
* mbedtls_ssl_handshake() returned -31488 (-0x7b00)
* Failed receiving HTTP2 data: 56(Failure when receiving data from the peer)
* readwrite_data() -> 56
* Curl_readwrite() -> 56
* multi_done[PERFORMING]: status: 56 prem: 1 done: 0
* Connection #0 to host www.google.com left intact
* Expire cleared
curl: (56) Failed receiving HTTP2 data: 56(Failure when receiving data from the peer)

I am assuming some issues relating to sessions tickets?

I've tested this with mbedTLS 3.5.2 and cURL 8.5.0 and nghttp2 1.52.0 (currently upgrading everything on our end).

Edit:

Sorry if this might become a duplicate of #10587 as I initially searched for "TLS 1.3" but only found it later when searching for "TLS1.3"...

[MAntoniak]

Based on your branch I created my own version. Check if it works, I only tested with HTTP connection version 1.1.

The problem here is returning an "error" that is not an error.... -0x7B00 is MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET, which occurs when curl waits for application data but gets NewSessionTicket instead. Based on the example from ssl_client2.c, this is a signal for the application to save the new ticket and then continue receiving application data.

This is a surprising behavior because curl uses the MBEDTLS_SSL_SESSION_TICKETS_DISABLED option, so why does the NewSessionTicket appear?

I'm not sure yet if this is ready to be published in the official branch. I'll still ask the developers of mbedtls for a comment.

Anyway if you find the time let me know if my changes work in your case.

[andreiaugustin]

It looks promising, I've upgraded to mbedTLS 3.6.0 and compiled with scripts/config.py set MBEDTLS_SSL_PROTO_TLS1_3, but I seem to be getting some errors about the certificate verification:

CURL_SSL_BACKEND=mbedTLS ./curl -Lkv https://www.google.com --tlsv1.3
* Host www.google.com:443 was resolved.
* IPv6: 2a00:1450:4017:805::2004
* IPv4: 216.58.213.100
*   Trying 216.58.213.100:443...
* Connected to www.google.com (216.58.213.100) port 443
* mbedTLS: Connecting to www.google.com:443
* ALPN: curl offers h2,http/1.1
* ssl_handshake returned - mbedTLS: (-0x2700) X509 - Certificate verification failed, e.g. CRL, CA or signature check failed
* Closing connection
curl: (35) ssl_handshake returned - mbedTLS: (-0x2700) X509 - Certificate verification failed, e.g. CRL, CA or signature check failed

I will dig a bit deeper when I have the time!

[MAntoniak]

The mbedtls library requires setting the authorization mode to MBEDTLS_SSL_VERIFY_REQUIRED if a tls 1.3 connection is forced from the client side. Otherwise it returns an MBEDTLS_ERR_SSL_BAD_CONFIG error. They argue that RFC 8446 section 4.4.3

So if you use the --tls13 option then you should also add --cacert with a certificate or --capath to verify the server certificate.

I don't know what the behavior is if it is the server that forces a tls 1.3 connection.

[bagder]

So if you use the --tls13 option then you should also add --cacert with a certificate or --capath to verify the server certificate.

libcurl detects and sets those options by default, so if you need to set them in order to use TLS, something is wrong in the build... and also, all TLS connections need to be verified when you use curl, unless you explicitly switch off verification - which you should only do for testing, never in production.

[jay]

I think it is a mbedtls issue and not our issue. He passed -k to disable peer verification but apparently that is not enough since mbedtls_ssl_handshake fails with MBEDTLS_ERR_X509_CERT_VERIFY_FAILED. And that is apparently happening because as Michal said:

The mbedtls library requires setting the authorization mode to MBEDTLS_SSL_VERIFY_REQUIRED if a tls 1.3 connection is forced from the client side.

I think that's a pretty strange thing for them to require. If I understand correctly that means we cannot disable peer verification with mbedtls and tls 1.3?

curl/lib/vtls/mbedtls.c

Lines 761 to 805 in de7b3e8

	ret = mbedtls_ssl_handshake(&backend->ssl);
	if(ret == MBEDTLS_ERR_SSL_WANT_READ) {
	connssl->connecting_state = ssl_connect_2_reading;
	return CURLE_OK;
	}
	else if(ret == MBEDTLS_ERR_SSL_WANT_WRITE) {
	connssl->connecting_state = ssl_connect_2_writing;
	return CURLE_OK;
	}
	else if(ret) {
	char errorbuf[128];
	mbedtls_strerror(ret, errorbuf, sizeof(errorbuf));
	failf(data, "ssl_handshake returned - mbedTLS: (-0x%04X) %s",
	-ret, errorbuf);
	return CURLE_SSL_CONNECT_ERROR;
	}
	infof(data, "mbedTLS: Handshake complete, cipher is %s",
	mbedtls_ssl_get_ciphersuite(&backend->ssl));
	ret = mbedtls_ssl_get_verify_result(&backend->ssl);
	if(!conn_config->verifyhost)
	/* Ignore hostname errors if verifyhost is disabled */
	ret &= ~MBEDTLS_X509_BADCERT_CN_MISMATCH;
	if(ret && conn_config->verifypeer) {
	if(ret & MBEDTLS_X509_BADCERT_EXPIRED)
	failf(data, "Cert verify failed: BADCERT_EXPIRED");
	else if(ret & MBEDTLS_X509_BADCERT_REVOKED)
	failf(data, "Cert verify failed: BADCERT_REVOKED");
	else if(ret & MBEDTLS_X509_BADCERT_CN_MISMATCH)
	failf(data, "Cert verify failed: BADCERT_CN_MISMATCH");
	else if(ret & MBEDTLS_X509_BADCERT_NOT_TRUSTED)
	failf(data, "Cert verify failed: BADCERT_NOT_TRUSTED");
	else if(ret & MBEDTLS_X509_BADCERT_FUTURE)
	failf(data, "Cert verify failed: BADCERT_FUTURE");
	return CURLE_PEER_FAILED_VERIFICATION;
	}

[andreiaugustin]

@bagder you're right - something was not correct in my build: on Windows and macOS, we use the native TLS libraries and therefore build with --without-ca-bundle and --without-ca-path. When creating these internal scripts to automate the build of curl and its dependencies, I should've removed these parameters for Linux since we actually do rely on it picking up the CA bundle/path automatically. Thanks for pointing this out - it was also the solution to a recent internal bug as well.

I can confirm that now, with the branch from @MAntoniak and mbedTLS 3.6.0, I can successfully use TLS 1.3 with cURL through mbedTLS, even without the -k:

jenkins@284914f2f190:/workspace/omnis/xcomp/ow3/curl/linux/curl/bin$ CURL_SSL_BACKEND=mbedTLS ./curl -Lv https://www.google.com --tlsv1.3
* Host www.google.com:443 was resolved.
* IPv6: 2a00:1450:4017:814::2004
* IPv4: 142.251.140.36
*   Trying 142.251.140.36:443...
* Connected to www.google.com (142.251.140.36) port 443
* mbedTLS: Connecting to www.google.com:443
* ALPN: curl offers h2,http/1.1
* mbedTLS: Handshake complete, cipher is TLS1-3-CHACHA20-POLY1305-SHA256
* Dumping cert info: * cert. version     : 3
* serial number     : 50:70:45:B0:E3:9C:A8:A7:0A:F6:F3:3D:DF:86:31:46
* issuer name       : C=US, O=Google Trust Services LLC, CN=GTS CA 1C3
* subject name      : CN=www.google.com
* issued  on        : 2024-03-18 20:38:49
* expires on        : 2024-06-10 20:38:48
* signed using      : RSA with SHA-256
* EC key size       : 256 bits
* basic constraints : CA=false
* subject alt name  :
*     dNSName : www.google.com
* key usage         : Digital Signature
* ext key usage     : TLS Web Server Authentication
* certificate policies : ???, ???

* ALPN: server accepted h2
* SSL connected
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.google.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.google.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.8.0-DEV]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/8.8.0-DEV
> Accept: */*
> 
* Request completely sent off
< HTTP/2 200 
< date: Thu, 18 Apr 2024 20:35:37 GMT
< expires: -1
< cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
< content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-ZwJSvNLZt4cbfDnfsKB0zw' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< set-cookie: AEC=AQTF6HydD98XI0LXTsSK0y6GqmIA18HXXz9Uk8F09OYN44O6iSJnprJlTg; expires=Tue, 15-Oct-2024 20:35:37 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
< set-cookie: __Secure-ENID=19.SE=TgKWJ0JLvve_pBGjeQdaoyJbBaUb9JlRIHV4a9d-6isfLQY2hsuev7aAK_Kp8PEhhf7MfwDCCElhsdvGtUvLJOzBUE75a6DxNVPsiWXJatWt9lsfxM2HaD4Ot2Qk0Lz51Sdh-JXEXyePC5w2_3Bm-fVzxhu2J2TjTitVUGwb4kfzq2PopA; expires=Mon, 19-May-2025 12:53:55 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< accept-ranges: none
< vary: Accept-Encoding
< 
<!doctype html><html itemscope="" [REMAINING HTML PAGE REDACTED]

[MAntoniak]

I have wrote a message to mbedtls. Let's see what they reply.

[MAntoniak]

The mbedTLS team wrote an answer to my questions. I don't know why the answers are not public, so I'm pasting them here:

Hi, please find some answers below. I hope this will help.

Ronald Cron.

Hello,

I am trying to add TLS 1.3 support to a curl project (https://github.com/MAntoniak/curl). As a result, it has two questions.

The first refers to the error code
MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET. This suggests an error receiving a new session ticket. I couldn't find anything else in the documentation. However, in the ssl_cllient2.c example application (line 2638), this code is used as a message to the application about a new session ticket. This makes me unsure whether it would be correct to continue receiving application data normally.

[RC] It is correct to continue receiving application data normally, MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET should be handled as a non-fatal error code like MBEDTLS_ERR_SSL_WANT_READ for example. We know that the documentation is not as it should be regarding this error code, sorry about that. We have an issue to address this: #8749.

The second question relates to the receipt of tickets. The curl project does not store session tickets, so it uses the MBEDTLS_SSL_SESSION_TICKETS_DISABLED flag. If this is the case, why do we nevertheless receive a new ticket from the server?

[RC] If the TLS 1.3 server is configured to send tickets when the connection is established, the client will receive them and there is nothing it can do about it. However, we have recently merged PR 9014 (it did not make it to 3.6.0 unfortunately but will be in 3.6.1) where when the support for tickets is disabled on client side (MBEDTLS_SSL_SESSION_TICKETS not defined), received tickets are silently ignored in the sense that no error code MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET is returned.

Thank you for your reply.

Michał Antoniak

[bagder]

Note: the disabling of session tickets was probably done as some sort of shortcut at some point. curl users would most likely want tickets support for mbedTLS.

[MAntoniak]

I think this topic can be marked as answered :)