[Question]How to implement grpc connection pool via kratos 2.x

[fans3210]

Please see the FAQ in our main README.md before submitting your issue.

In older version of kratos, there was a grpc connection pool implementation. But I'm not able to find such materials in 2.x versions. Appreciate if could help give some examples or introductions.

[shenqidebaozi]

gRPC is based on HTTP2 and has the feature of multiplexing. It does not need to use Connection pool

[fans3210]

gRPC is based on HTTP2 and has the feature of multiplexing. It does not need to use Connection pool

Thanks for the fast response. I'm having one issue with my current app. I have a benchmarking which would fire 10k go routines with a simple grpc request. I noticed that 10k is max number of go-routines I can fire without facing "grpc connection timeout" or "grpc retry xxx" issue. I used a single grpc client though(balancer is kratos's WRR one). May I know what are the possible ways to make my service able to handle more than 10k concurrent requests without error? Buffered channel as semaphore is a possible approach but I prefer to fix it in either system level or in kratos level. Tests are done in aliyun k8s cluster

I'm referring these article too:
https://xiaorui.cc/archives/6001
https://mbd.baidu.com/ma/s/UXQo5q4v

[dosubot]

Hi, @fans3210! I'm Dosu, and I'm here to help the kratos team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding of the issue, you were seeking help with implementing a gRPC connection pool using Kratos 2.x. You mentioned that you couldn't find materials on this topic in the newer versions of Kratos. In the comments, another user suggested that gRPC doesn't need a connection pool due to its multiplexing feature. However, you explained that you are facing issues with your current app and want to handle more than 10k concurrent requests without errors. You also provided links to articles you are referring to.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the kratos repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding, and we look forward to hearing from you soon.

[fans3210]

Hi, @fans3210! I'm Dosu, and I'm here to help the kratos team manage their backlog. I wanted to let you know that we are marking this issue as stale.

Based on my understanding of the issue, you were seeking help with implementing a gRPC connection pool using Kratos 2.x. You mentioned that you couldn't find materials on this topic in the newer versions of Kratos. In the comments, another user suggested that gRPC doesn't need a connection pool due to its multiplexing feature. However, you explained that you are facing issues with your current app and want to handle more than 10k concurrent requests without errors. You also provided links to articles you are referring to.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the kratos repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding, and we look forward to hearing from you soon.

Hi @dosubot. The issue is not resolved yet. Would be appreciated if kratos team can provide more details with explanations on some best practices of implementing grpc pool to achieve higher qps

[dosubot]

@shenqidebaozi Could you please help @fans3210 with their issue regarding implementing a gRPC connection pool in Kratos 2.x? They are seeking best practices to achieve higher QPS and would appreciate more details and explanations. Thank you!

[dosubot]

Hi, @fans3210,

I'm helping the Kratos team manage their backlog and am marking this issue as stale. It seems like you are seeking guidance on implementing a gRPC connection pool using Kratos 2.x, and there was a discussion about the necessity of a connection pool for gRPC due to its multiplexing feature. You mentioned facing issues with handling more than 10k concurrent requests without errors and requested details and explanations on best practices for implementing a gRPC pool to achieve higher QPS.

Could you please confirm if this issue is still relevant to the latest version of the Kratos repository? If it is, please let the Kratos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and cooperation.

[fans3210]

Hi @dosubot. The issue is not resolved yet. Would be appreciated if kratos team can provide more details with explanations on some best practices of implementing grpc pool to achieve higher qps

[shushenghong]

in grpc doc

(Special topic) Each gRPC channel uses 0 or more HTTP/2 connections and each connection usually has a limit on the number of concurrent streams. When the number of active RPCs on the connection reaches this limit, additional RPCs are queued in the client and must wait for active RPCs to finish before they are sent. Applications with high load or long-lived streaming RPCs might see performance issues because of this queueing. There are two possible solutions:

• Create a separate channel for each area of high load in the application.

• Use a pool of gRPC channels to distribute RPCs over multiple connections (channels must have different channel args to prevent re-use so define a use-specific channel arg such as channel number).

Side note: The gRPC team has plans to add a feature to fix these performance issues (see grpc/grpc#21386 for more info), so any solution involving creating multiple channels is a temporary workaround that should eventually not be needed.

and in grpc github, issue 21386 still not resolved.

[dosubot]

To address the challenge of handling more than 10k concurrent gRPC requests in Kratos 2.x, consider implementing a client connection pool and leveraging a custom load balancer. Here's a strategy that aligns with Kratos' capabilities and the nuances of gRPC's performance characteristics:

1. Client Connection Pooling: Extend Kratos' Dial or DialInsecure functions to manage a pool of *grpc.ClientConn objects. This approach helps in reusing connections efficiently, reducing the overhead of establishing new connections for each request.

2. Custom Load Balancer: Utilize Kratos' support for custom balancers. Implement and register a custom balancer that uses health checks and service discovery to distribute requests effectively across your services. This ensures that your load is balanced based on the real-time status of your service instances.

3. Service Discovery Integration: Enhance your load balancing strategy by integrating service discovery. This allows your balancer to adjust dynamically to the pool of available services, maintaining high availability and performance.

4. Optimize gRPC Dial Options: Customize gRPC dial options to adjust keepalive parameters and other settings that help maintain connections and reduce reconnections. This is particularly useful under high load.

5. Middleware and Interceptors: Implement middleware and interceptors for monitoring, logging, and managing request timeouts. This provides you with more control and insights into your application's performance under load.

Here's an example of how you might configure a gRPC client with these optimizations in Kratos:

ctx := context.Background()
conn, err := grpc.DialInsecure(ctx,
    WithEndpoint("your_service_endpoint"),
    WithDiscovery(yourDiscoveryImplementation),
    WithBalancerName("selector"), // Custom balancer
    WithTimeout(2000*time.Millisecond),
    WithMiddleware(yourMiddleware...),
    WithUnaryInterceptor(yourUnaryInterceptors...),
    WithStreamInterceptor(yourStreamInterceptors...),
    WithOptions(yourDialOptions...), // Custom gRPC dial options
)
if err != nil {
    log.Fatalf("did not connect: %v", err)
}
defer conn.Close()

This setup, leveraging Kratos' features and custom balancer, aims to efficiently manage connections and distribute load, helping to mitigate issues related to handling high numbers of concurrent requests.

Details

Sources

• transport/grpc/client.go
• transport/grpc/balancer.go

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.