Dragonflydb/Keydb Integration

[DanielNetzer]

Having support for DragonflyDB along side Redis and maybe even KeyDB might be a great direction to move towards.

https://github.com/Snapchat/KeyDB
dragonflydb/dragonfly#781

Also here's a working setup with KeyDB and BullMQ
https://github.com/DanielNetzer/bullmq-load/tree/feat/keydb

[manast]

This is not something that we will support, if the maintainers of said projects claim 100% compatibility with Redis then it should work, if it doesn't then they are not 100% compatible.

[manast]

Nevertheless, if all the tests we have for BullMQ pass, we could mention it on the README as these two projects provide crazy performance.

[DanielNetzer]

I know it's not on BullMQ side of things, but it might benefit BullMQ as the first MessageQueue system based on this 2 amazing DB's.
I added a link to an issue open on dragonflydb side to fix this, but they are struggling to find the exact LOC in BullMQ's code that executes this command.
I'm sure that if you will lend a hand and your knowledge, this can be solved pretty fast.

[CanRau]

Just got BullMQ working on a 2 instance multi-leader KeyDB and it seems to work great 🥳

[manast]

This test: "should process delayed jobs with several workers respecting delay" is known to be failing with Dragonfly and there is a theory on why it fails which states that the test itself is not properly written for concurrent systems such as Dragonfly. I have a dialog with the Dragonfly team about a possible change. Did KeyDB fail on the same tests?

[CanRau]

Uh interesting, the logs are from a local KeyDB Docker container.
I'm considering switching to Redis for now, I liked the simple active-active multi-leader setup on fly, for failover etc but in the end I need the queue to work reliably..

Also answered on Slack regarding active-active and potentially processing events more than once, though on this regard I don't fully grasp how a Redis read-replica would handle this?

So my question is how (reliable) does message locking work in a distributed setup?

Update: so in a quick test queueing 10 test messages from 2 producer instances and the 2 instance multi-leader keydb setup I go events being process up to 2 times

when giving each a unique jobId I got less duplications but still 2 or so, which probably happens due to race conditions between synching and locking I guess 🤔

So currently in this setup restricting each worker to only process in its own region seems to be the best guarantee

Or maybe I look into manual processing 🤔

[manast]

Redis guarantees atomicity when using Lua scripts, so that's the premise required for BullMQ, maybe in KeyDB this is not always true.

[CanRau]

Good to know, I'll look into it.

Another thought I just had is let each app instance write to the closest KeyDB, but have all workers only ever connect to one single db then the write load would be distributed, reads not but also guaranteed to not be duplicated and I could benefit from multi-leader for caching and other use-cases 🤓

[CanRau]

So just to follow up, I'm not exactly sure what I did wrong before, maybe had errors in workers so they got re-tried, but now everything is working fine, my app & keydb run both on 2 instances in 2 locations currently and the app doesn't even connect to a location specific keydb instance anymore and everything works perfect 🥳