-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Grpc-proxy cannot handle requests from grpc-gateway correctly #18011
Comments
Here is the related PR #18012. |
It seems like a valid issue. We don't see much use case on grpcProxy, so I don't think this is a major issue.
Can anyone double check this? @ivanvc @siyuanfoundation @chaochn47 @tjungblu
|
I also checked the build-in gateway, it seems OK. I think the implementation is different in etcdserver and grpcproxy server, which leads to the different behavior. |
I think the built-in gateway (if I'm correct, it should be in etcd/server/etcdserver/api/v3rpc/grpc.go Lines 43 to 47 in 97a9af9
And etcd/server/etcdserver/api/v3rpc/interceptor.go Lines 46 to 76 in 97a9af9
The source code for release 3.5 is very similar, it doesn't appear to happen there either: https://github.com/etcd-io/etcd/blob/v3.5.13/server/etcdserver/api/v3rpc/grpc.go#L45-L49 / https://github.com/etcd-io/etcd/blob/v3.5.13/server/etcdserver/api/v3rpc/interceptor.go#L46-L73 Caveats: I'm still not too familiar with |
@ivanvc The source code you mentioned above is where etcdserver receives gRPC requests, this issue is about receiving REST(HTTP) requests. Note that etcdserver receives REST(HTTP) request via the grpc-gateway. I think the reason why the build-in gateway doesn't have such issue is that the gateway carries the http request header over to the following gRPC request, Please read
The existing e2e test cases should have already verified it. |
https://github.com/etcd-io/etcd/blob/main/tests/e2e/v3_curl_auth_test.go |
I am not sure what issue you are talking about here. Please feel free to raise another issue to track it if not present. Please let's only fix one thing/issue in one PR. Please add an e2e test to reproduce the issue firstly, please read the second point in #18011 (comment) |
Thanks for your reply, I'll add e2e tests in a fews days and recommit the PR. The watch context cancled error is described as the first part of this issue, maybe not clearly. I copied the part below. Start etcd(v3.5.13) and grpc-proxy locally by commands below:
Then start grpc-gateway locally(compiled by the code above).
The curl command returns directly instead of waiting for watch response,with the error repsose below:
|
The error is returned by your grpc-gateway, so you need to debug it firstly. Please let me know if you see any issue when you watch a key via etcd or grpc-proxy directly. |
Do you still see the issue after applying the #18012? |
Bug report criteria
What happened?
We use grpc-gateway and grpc-proxy to reduce watch requests from apisix to etcd, as apisix only provides http client for etcd for production use. Here is a simple architecture diagram:
The grpc-gateway is a simple program compiled by the code below:
Start etcd(v3.5.13) and grpc-proxy locally by commands below:
nohup etcd&
nohup etcd grpc-proxy start --endpoints=localhost:2379 &
Then start grpc-gateway locally(compiled by the code above).
Send watch command to grpc-gateway with curl:
curl -X POST -d '{"create_request": {"key":"dGVzdGtleQo=","range_end":"dGVzdGtleTEK"}}' 'localhost:8887/v3/watch'
The curl command returns directly instead of waiting for watch response,with the error repsose below:
{"result":{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"created":true}} {"error":{"grpc_code":1,"http_code":408,"message":"context canceled","http_status":"Request Timeout"}}
We set auth to etcd by command below:
etcdctl user add root
etcdctl auth enable
Then make a kv range call with curl:
curl -X POST -d '{"key": "dGVzdGtleQo=", "range_end": "dGVzdGtleTEK"}' http://localhost:8887/v3/kv/range
The response is as expected:
{"error":"etcdserver: user name is empty","code":2,"message":"etcdserver: user name is empty"}
However, after we get a token with the command below:
curl 'localhost:8887/v3/auth/authenticate' -X POST -d '{"name":"root","password":"123456"}
{"header"{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"token":"ImhAoKYyjbUPSKoN.11"}
We make the same kv range call with auth header:
curl -H 'Authorization: ImhAoKYyjbUPSKoN.11' -X POST -d '{"key": "dGVzdGtleQo=", "range_end": "dGVzdGtleTEK"}' http://localhost:8887/v3/kv/range
We still get the
user name is empty
error.{"error":"etcdserver: user name is empty","code":2,"message":"etcdserver: user name is empty"}
The expected response is
invalid auth token
error:{"error":"etcdserver: invalid auth token","code":2,"message":"etcdserver: invalid auth token"}
After some digging in the source code, we found the grpc server of grpc proxy in etcd did not propagate grpc metadata properly.
And the watch cancled error may caused by the client early disconnection by grpc-gateway.
We propose a PR trying to fix the problem, expecting for reviewing.
What did you expect to happen?
Grpc proxy can handle grpc gateway requests correctly.
How can we reproduce it (as minimally and precisely as possible)?
As describe above.
Anything else we need to know?
No response
Etcd version (please run commands below)
Etcd configuration (command line flags or environment variables)
paste your configuration here
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
Relevant log output
No response
The text was updated successfully, but these errors were encountered: