Timeout Exceptions in .NET 6 (Azure Redis Cache + Kubernetes)

[JoshAnderson4201]

Hello!

We have a .NET Core MVC Application that is deployed to multiple K8s pods, all using the same Azure Redis Cache instance. We are currently experiencing a large number of timeout exceptions, and have been trying to diagnose them for weeks with no luck. A few exception examples:

Timeout performing GET (5000ms), next: GET {KEY}, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: {ENDPOINT NAME}, mc: 1/1/0, mgr: 10 of 10 available, clientName: {CLIENTNAME}, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Timeout performing SETEX (5000ms), next: SETEX {KEY}, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: {ENDPOINT NAME}, mc: 1/1/0, mgr: 10 of 10 available, clientName: {CLIENTNAME}, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

We are using a ConnectionMultiplexer that is registered as a singleton, and is then used in our CacheService class:
public string GetCacheString(string key) { var cache = _connectionMultiplexer.GetDatabase(); return cache.StringGet(key); }

So far we have tried the following with no luck:

• Using the IDistributedCache interface
• Increasing thread count

Our SLOWLOG does report slow commands, but none of them are taking longer than the timeout value we have set (default 5000ms). A note on our apps behavior - there can be thousands of users hitting our app at once, and most requests utilize Redis in some way to see if data exists in Redis before querying our SQL DB. These users can be sending the same exact requests at the same time.

Any help with this is appreciated!!

[KendoSai]

I think you could try to switch to use async methods of IDatabase, for ex StringGetAsync, StringSetAsync,... for all of places where you make calls to Redis cache.

Last time it helped to reduce my application timeout.

[joaooo]

Move all the redis operations to async like @KendoSai said, also if you could group them all into whenAll it would be way more efficient due to redis pipelining and lastly, be careful with using IDistributedCache when adding values to redis, i had some issues with it because it was storing simple values as an hash, which eventually made things much slower (i stopped using IDistributedCache and created my own provider using redis operations only)