DRIVERS-1541 Retry KMS decrypt requests on transient errors

[adriandole]

Retry GCP, AWS, and Azure requests that encounter transient errors.
Evergreen testing with the C driver implementation: https://spruce.mongodb.com/version/664bc7d2ebaef90007283d77/

Scope doc: https://docs.google.com/document/d/1JUVC2-Pe8mfpgemHyb4ihsIOzIe_QFQPnYDJV2idVY4/edit

[kevinAlbs]

Leaving quick initial feedback. I suggest git merge master to pull in cb4d4fa. That may resolve many Evergreen task failures. The other task failures (e.g. clang-tidy) may have been introduced in this PR.

[kevinAlbs]

Suggested change

	bool mongocrypt_setopt_retry(mongocrypt_t *crypt, bool enable);
	bool mongocrypt_setopt_retry_kms(mongocrypt_t *crypt, bool enable);

[kevinAlbs]

Consider adding separate tests to isolate retry behavior:

void _test_retry_tcp(_mongocrypt_tester_t *tester) {
    mongocrypt_t *crypt;

    crypt = _mongocrypt_tester_mongocrypt(TESTER_MONGOCRYPT_DEFAULT);

    // Test that an HTTP error is retried.
    {
        mongocrypt_ctx_t *ctx = mongocrypt_ctx_new(crypt);
        ASSERT_OK(mongocrypt_ctx_encrypt_init(ctx, "test", -1, TEST_FILE("./test/example/cmd.json")), ctx);
        _mongocrypt_tester_run_ctx_to(tester, ctx, MONGOCRYPT_CTX_NEED_KMS);
        mongocrypt_kms_ctx_t *kms_ctx = mongocrypt_ctx_next_kms_ctx(ctx);
        ASSERT_OK(kms_ctx, ctx);
        // Expect no sleep is requested before any error.
        ASSERT_CMPINT64(mongocrypt_kms_ctx_usleep(kms_ctx), ==, 0);
        // Feed a retryable HTTP error.
        // TODO: use ASSERT_OK once `kms_ctx` is changed not to set error status.
        ASSERT(mongocrypt_kms_ctx_feed(kms_ctx, TEST_FILE("./test/data/rmd/kms-decrypt-reply-429.txt")));
        // Expect KMS request is returned again for a retry.
        kms_ctx = mongocrypt_ctx_next_kms_ctx(ctx);
        ASSERT_OK(kms_ctx, ctx);
        // Feed a successful response.
        ASSERT_OK(mongocrypt_kms_ctx_feed(kms_ctx, TEST_FILE("./test/data/kms-aws/decrypt-response.txt")), kms_ctx);
        ASSERT_OK(mongocrypt_ctx_kms_done(ctx), ctx);
        _mongocrypt_tester_run_ctx_to(tester, ctx, MONGOCRYPT_CTX_DONE);
        mongocrypt_ctx_destroy(ctx);
    }

    mongocrypt_destroy(crypt); /* recreate crypt because of caching. */
}