Support sharded pubsub commands #2498

vandyvilla · 2023-06-29T19:58:45Z

Support SPUBLISH and SSUBSCRIBE, SUNSUBSCRIBE commands.

We tested this PR internally and will add more tests to this PR.

Would like to gather some early feedback. Thanks!

mgravell · 2023-08-16T15:03:13Z

RedisChannel seems to have a new bool field, increasing the size; to simplify design and avoid additional storage, since SSUBSCRIBE does not support wildcards, I wonder if we should implement this as a new PatternMode value instead of a separate bool - perhaps with a public static RedisChannel Sharded(...) pair to match the Literal and Pattern methods. At the moment there is a private readonly bool _isPatternBased, but ultimately this is padded to 4 bytes, so would be identical to private readonly PatternMode _mode;. As an alternative, we could use bit-bashing to combine the two bools as bits, to avoid increasing the size - but honestly I think the PatternMode approach is simpler.

My main concern, however, is around shard rebalancing; I don't see anything that handles this currently; presumably we'd need to detect shard movement, and validate that subscriptions are still against the correct server - right now I don't see that happening; we may or may not also need to do this during resubscribe against an existing connection?

mgravell · 2023-08-16T15:06:34Z

~~Does this actually work without error? I'm a little surprised because I would have expected CheckReadOnlyOperations to fail, because I don't think you've modified IsPrimaryOnly?~~

My bad; found them

mgravell · 2023-08-16T15:11:07Z

src/StackExchange.Redis/PhysicalConnection.cs

-                    var channel = items[1].AsRedisChannel(ChannelPrefix, RedisChannel.PatternMode.Literal);
-                    Trace("MESSAGE: " + channel);
+                    RedisChannel channel;
+                    if (items[0].IsEqual(message)) {


feels like we're testing this a lot; can we hoist this to a bool somewhere? maybe:

if (items.Length >= 3 && IsSimpleMessage(items[0], out bool isSharded))

then just use isSharded in all the places? for example channel = items[1].AsRedisChannel(ChannelPrefix, RedisChannel.PatternMode.Literal, isSharded);

or alternatively, using the PatternMode approach, maybe we can out var patternMode ?

mgravell · 2023-08-16T15:13:56Z

src/StackExchange.Redis/RedisSubscriber.cs

@@ -485,6 +489,7 @@ private bool UnregisterSubscription(in RedisChannel channel, Action<RedisChannel
            return false;
        }

+        // TODO: We need a new api to support SUNSUBSCRIBE all. Calling this now would unsubscribe both sharded and unsharded channels.


do we actually need such an API?

currently UnsubscribeAll will unsubscribe both unsharded and sharded channels. Alternatively, we need to pass some flag to indicate whether we want to unsubscribe all sharded channels to match the SUNSUBSCRIBE semantics.

I understand that. I just don't know that it is obvious whether that represents a problem. What genuinely likely scenario are we thinking of where this nuance is needed?

mgravell · 2023-08-16T15:17:00Z

src/StackExchange.Redis/ResultProcessor.cs

@@ -1325,7 +1325,7 @@ protected override bool SetResultCore(PhysicalConnection connection, Message mes
                {
                    case ResultType.MultiBulk:
                        var final = result.ToArray(
-                                (in RawResult item, in ChannelState state) => item.AsRedisChannel(state.Prefix, state.Mode),
+                                (in RawResult item, in ChannelState state) => item.AsRedisChannel(state.Prefix, state.Mode, isSharded: false),


do we need to support PUBSUB SHARDCHANNELS (see SubscriptionChannels[Async]) ?

ditto SHARDNUMSUB - see SubscriptionPatternCount / SubscriptionSubscriberCount

ditto ssub in CLIENT - see ClientInfo.TryParse

yes, plan to add it in a separate PR.

ssub handled in my extensions to this PR

vandyvilla · 2023-08-23T20:36:31Z

good idea for leveraging the pattern mode to save one additional bool variable.

Thanks for bringing up the shard movement concern, which is critical. Based on my testing, the subscriber gets a SUNSUBSCRIBE message reply from Redis when the shard is moved, which triggers a re-subscription. The new subscription against the old server will get a MOVED message, which makes it to resubscribe to the right server. Does this make sense to you?

dmitrig89 · 2023-11-29T15:30:02Z

@mgravell No pressure, but any idea when we might expect this feature to be merged?
We're currently experiencing performance bottlenecks when using the pub\sub pattern on a Redis cluster.
I'm hopeful that sharding the pub\sub will help with the issues we're having

Thanks

mgravell · 2024-03-01T14:16:14Z

Sorry, took my eyes off this one; let's get this done! if you're not still eager, I can probably take it over; but!

1: I really don't like these constructors:

+ StackExchange.Redis.RedisChannel.RedisChannel(string! value, bool isSharded) -> void
+ StackExchange.Redis.RedisChannel.RedisChannel(byte[]? value, bool isSharded) -> void

can we instead use (to mirror .Literal and .Pattern:

public static RedisChannel Sharded(string value) => ...
public static RedisChannel Sharded(byte[] value) => ...

? To me that feels more consistent.

2: I still think we should squash the 2 bool, but: let's not worry about those today

caoyang1024 · 2024-03-04T05:54:46Z

isSharded indeed just looks like a hack to me 🤣

mgravell · 2024-03-04T09:26:04Z

I've taken the liberty of pushing the merge fixes; I've also proposed the single-field refactor here: vandyvilla#1 - would love to get your input / thoughts here @vandyvilla , although I guess I can push it through either way; sorry this has been delayed

mgravell · 2024-03-04T09:46:13Z

working on adding some tests; very much needed (this could be from my changes, note):

mgravell · 2024-03-04T10:00:21Z

found and fixed; that was a genuine bug in RedisDatabase

mgravell · 2024-03-04T10:11:22Z

right, I think that's everything - @vandyvilla if you want to merge vandyvilla#1, then I should be able to pull in the combined effort (preserving your credit, etc)

mgravell · 2024-03-04T10:22:47Z

important TODOs before final merge:

investigate break/reconnect logic
investigate cluster shard migration logic
investigate unsubscribe-all logic

caoyang1024 · 2024-04-15T02:16:01Z

hi @mgravell. understand you have a busy schedule, but could you please give a progress update? Thank you!

vandyvilla added 5 commits June 29, 2023 12:53

Support sharded pubsub

6c36478

Support sharded pubsub

248c182

fix api

90e2493

fix enum

ac44789

fix api

fbc8fd5

mgravell reviewed Aug 16, 2023

View reviewed changes

Merge branch 'main' into main

2daf168

mgravell mentioned this pull request Mar 4, 2024

SignalR redis perf : consider sharded pub/sub when available dotnet/aspnetcore#54344

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support sharded pubsub commands #2498

Support sharded pubsub commands #2498

vandyvilla commented Jun 29, 2023 •

edited

mgravell commented Aug 16, 2023

mgravell commented Aug 16, 2023 •

edited

mgravell Aug 16, 2023 •

edited

mgravell Aug 16, 2023

vandyvilla Aug 22, 2023

mgravell Aug 22, 2023

mgravell Aug 16, 2023

mgravell Aug 16, 2023

mgravell Aug 16, 2023

vandyvilla Aug 22, 2023

mgravell Mar 4, 2024

vandyvilla commented Aug 23, 2023 •

edited

dmitrig89 commented Nov 29, 2023

mgravell commented Mar 1, 2024 •

edited

caoyang1024 commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024 •

edited

caoyang1024 commented Apr 15, 2024 •

edited

Support sharded pubsub commands #2498

Are you sure you want to change the base?

Support sharded pubsub commands #2498

Conversation

vandyvilla commented Jun 29, 2023 • edited

mgravell commented Aug 16, 2023

mgravell commented Aug 16, 2023 • edited

mgravell Aug 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vandyvilla commented Aug 23, 2023 • edited

dmitrig89 commented Nov 29, 2023

mgravell commented Mar 1, 2024 • edited

caoyang1024 commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024

mgravell commented Mar 4, 2024 • edited

caoyang1024 commented Apr 15, 2024 • edited

vandyvilla commented Jun 29, 2023 •

edited

mgravell commented Aug 16, 2023 •

edited

mgravell Aug 16, 2023 •

edited

vandyvilla commented Aug 23, 2023 •

edited

mgravell commented Mar 1, 2024 •

edited

mgravell commented Mar 4, 2024 •

edited

caoyang1024 commented Apr 15, 2024 •

edited