Evaluate & potentially replace https://github.com/soheilhy/cmux

[ahrtr]

What would you like to be added?

etcdserver depends on https://github.com/soheilhy/cmux to receive both gRPC and REST connections on the same TCP listener/port. But the library isn't well maintained; note it hasn't been updated for more than 3 years at the time of writing this ticket. It also has issue working with HTTP/2, please refer to soheilhy/cmux/issues.

We should evaluate & summarize the problems the library has, and

• either try to find an alternative solution.
• or completely separate gRPC and REST/HTTP connections;

I expect someone with more network expertise could drive this effort.

Why is this needed?

See above

[serathius]

cc @aojea

[fuweid]

The ServeHTTP is still experimental feature in gRPC side https://pkg.go.dev/google.golang.org/grpc#Server.ServeHTTP.
There is different between gRPC and net/http about the underlying implementation of HTTP2 transport.
As far as I know, the gRPC implementation has some auto-tune features on window-update.
If there is good performance provided gRPC, I think we can consider using separate connections.

[serathius]

I think we can consider using separate connections.

Using separate connections is much better as shown by #15402, however removing connection multiplexing would be a breaking change for users. If we look at the list of endpoints in #15402 (comment), the most impacted would be would be:

• Users of /health via health probes, so I assume most of the users.
• Users of /v3/* v3 grpc gateway, which based on the questionnaire we sent are more than we expected as not every programming language supports grpc.
• Users of /metrics that didn't separate the port via --listen-metrics-urls.