Kraken: Leaking order cancel updates when using watchOrders()

[mjwvb]

Operating System

Ubuntu

Programming Languages

JavaScript

CCXT Version

4.3.18

Description

When running my trading bot on Kraken I noticed some cancel order updates not getting pushed to watchOrders(). I'm using the watch function without arguments, so watching all symbols (newUpdates turned on). I spent the last couple of hours trying to figure out what is wrong and thankfully found the issue.

When you create two limit orders A and B subsequently, and at a later point in time you can cancel order A first and then cancel order B, watchOrders() will only give you an update on order A.

I looked at the code and it has to do with the way how order updates are filtered from the ArrayCacheBySymbolById array. The call stack for watchOrders() without any parameters (that is: no symbol/since/limit) is as follows:

watchPrivate("openOrders") -> filterBySymbolSinceLimit() -> filterByValueSinceLimit() -> filterByLimit()

filterByLimit() checks whether the ArrayCache is in ascending order by checking the "timestamp" property of the first and last order updates in the cache, which actually is the order creation time and not the update timestamp. I believe it actually doesn't need to do that because we didn't provide the since parameter, so it should simply give me the last X updates from the tail of the cache. Anyway, that's not what it does. I quickly tested it by forcing the "ascending" var to be always true, and the problem was gone.

This is potentially also the case for other exchanges and also other watch functions. It must also result in unexpected results for people that actually do rely on the since parameter, as it's always based on the creation time. I cannot reproduce the problem on e.g. Bybit, which also uses filterBySymbolSinceLimit() under the hood for watchOrders. The only difference there is that it is called with the tail parameter set to true, so there's that.

I think the fix might be a combination of the following:
A) Maybe base the ascending order on the last updated timestamp, and not on the order creation time;
B) When no "since" parameter is given, simply bypass the ascending order check and always return the updates from the tail of the cache. Also good for performance.
C) Pass true to the tail parameter for filterBySymbolSinceLimit() within watchPrivate(). Looks like it works in my test, but unsure about its consequences for other watchPrivate() calls.

Code

Below is some simple code to reproduce the problem:

import * as ccxt from "ccxt";
  
const test = async () => {
  const exchange = new ccxt.pro.kraken({
    apiKey: process.env.EXCHANGE_KRAKEN_API_KEY,
    secret: process.env.EXCHANGE_KRAKEN_API_SECRET,
  });

  const watchAllOrders = async () => {
    while (true) {
      try {
        const orders = await exchange.watchOrders();
        for (let i in orders) {
          if (orders[i].status === "canceled") {
            console.log(new Date(), "Received cancel:", orders[i].id);
          }
        }
      } catch (e) {
        console.log(e);
        if (e instanceof ccxt.BadRequest) throw e;
      }
    }
  };

  watchAllOrders();

  // Place order A and B in that order
  const orderA = await exchange.createLimitBuyOrder("SOL/USD", 1, 2);
  console.log(new Date(), "Created order A:", orderA.id, orderA.symbol);

  const orderB = await exchange.createLimitBuyOrder("SOL/USD", 1, 2);
  console.log(new Date(), "Created order B:", orderB.id, orderB.symbol);

  // Cancel order A and B in that order
  await exchange.cancelOrder(orderA.id);
  console.log(new Date(), "Canceled order A:", orderA.id);

  await exchange.cancelOrder(orderB.id);
  console.log(new Date(), "Canceled order B:", orderB.id);
};

test();

Note the missing received cancel update for order B:
2024-05-09T15:35:38.709Z Created order A: OASIPA-NBQNI-PLIY35
2024-05-09T15:35:38.787Z Created order B: O4LHQA-SYXBR-VSAREX
2024-05-09T15:35:38.838Z Canceled order A: OASIPA-NBQNI-PLIY35
2024-05-09T15:35:38.882Z Received cancel: O4LHQA-SYXBR-VSAREX
2024-05-09T15:35:38.883Z Canceled order B: O4LHQA-SYXBR-VSAREX

Turn around the cancellation of A and B in the code, and both cancel updates are received

[sc0Vu]

@mjwvb thanks, I think it's a known issue when netUpdates is true. We'll fix this asap