Inconsistencies when removing from maps

[sinkingsugar]

We are encountering an issue with state synchronization, where the delete sets are not being applied consistently across peers, leading to state divergence. Specifically, the delete operation seems to partially remove items instead of clearing them completely, resulting in an inconsistent state across different peers.

Log extract:

[26/03 18:16:28.284][T-40951417][fbl][debug][forms.cpp::2571] Integrating remote changes [0 <29 SHType::Bytes>] (PeerID=4894647072)
// THIS CHANGE BASICALLY CLEARS THE SCENE
[26/03 18:16:28.284][T-40951417][shards][debug][runtime.cpp::3209] applying update:  { delete set:  { 15658536145409349539: [0..5993), 3019702115965040724: [0..1796) } }

// Current State before applying
[26/03 18:16:28.284][T-40951417][shards][debug][runtime.cpp::3209] STATE BEFORE: Snapshot { delete_set:  { 15658536145409349539: [0..5976), 3019702115965040724: [ [0..1146) [1168..1169) [1173..1177) [1178..1179) [1180..1183) [1187..1188) [1190..1191) [1192..1204) [1206..1207) [1208..1210) [1212..1214) [1216..1219) [1220..1221) [1222..1223) [1224..1228) [1229..1230) [1232..1235) [1236..1240) [1241..1242) [1243..1244) [1251..1252) [1255..1256) [1263..1264) [1265..1266) [1268..1269) [1275..1277) [1283..1284) [1299..1300) [1311..1312) [1313..1314) [1315..1323) [1324..1325) [1326..1330) [1331..1611) [1620..1622) [1626..1627) [1632..1633) [1634..1637) [1638..1641) [1645..1646) [1648..1650) [1651..1655) [1658..1661) [1664..1665) [1669..1670) [1671..1672) [1675..1676) [1677..1678) [1679..1680) [1683..1686) [1687..1688) [1693..1694) [1695..1697) [1699..1701) [1703..1704) [1705..1706) [1707..1708) [1709..1710) [1712..1713) [1714..1716) [1717..1718) [1719..1721) [1722..1723) [1724..1725) [1726..1727) [1729..1730) [1734..1737) [1739..1740) [1741..1742) [1748..1749) [1765..1766) [1776..1777) [1778..1779) [1780..1789) [1790..1794) ] }, state_map: StateVector({15658536145409349539: 5993, 3019702115965040724: 1796}) }

// apply_update

// Triggered observer events... Just partially removing things... why? It should have removed all blocks?
[26/03 18:16:28.284][T-40951417][shards][debug][runtime.cpp::3209] Changes map keys: {"am9C3Ouum/cards-used-this-round-2": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/hand-2": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/discard-pile-2": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p1-discarded-non-epic-water-cards": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/cards-used-this-round-1": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p2-boons": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p1-boons": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/out-of-play-pile-1": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p2-discarded-non-epic-water-cards": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p1-abilities": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p1-curses": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p2-abilities": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/hand-1": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/p2-curses": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/selected-p2-curse-1": Removed(Any(Buffer([4, 255, 255, 255, 255, 255, 255, 255, 255]))), "am9C3Ouum/out-of-play-pile-2": Removed(Any(Buffer([56, 0, 0, 0, 0]))), "am9C3Ouum/discard-pile-1": Removed(Any(Buffer([56, 0, 0, 0, 0])))}
[26/03 18:16:28.284][T-40951417][shards][debug][runtime.cpp::3209] Changes path: [Key("a41f26d8-cdfc-4802-9c29-3d8734b2e598"), Key("ui")]

// Current State after is not lean (or empty) like remote peer
[26/03 18:16:28.284][T-40951417][shards][debug][runtime.cpp::3209] STATE AFTER: Snapshot { delete_set:  { 15658536145409349539: [0..5993), 3019702115965040724: [ [0..1146) [1168..1169) [1173..1177) [1178..1179) [1180..1183) [1187..1188) [1190..1191) [1192..1204) [1206..1207) [1208..1210) [1212..1214) [1216..1219) [1220..1221) [1222..1223) [1224..1228) [1229..1230) [1232..1235) [1236..1240) [1241..1242) [1243..1244) [1251..1252) [1255..1256) [1263..1264) [1265..1266) [1268..1269) [1275..1277) [1283..1284) [1299..1300) [1311..1312) [1313..1314) [1315..1323) [1324..1325) [1326..1330) [1331..1611) [1620..1622) [1626..1627) [1632..1633) [1634..1637) [1638..1641) [1645..1646) [1648..1650) [1651..1655) [1658..1661) [1664..1665) [1669..1670) [1671..1672) [1675..1676) [1677..1678) [1679..1680) [1683..1686) [1687..1688) [1693..1694) [1695..1697) [1699..1701) [1703..1704) [1705..1706) [1707..1708) [1709..1710) [1712..1713) [1714..1716) [1717..1718) [1719..1721) [1722..1723) [1724..1725) [1726..1727) [1729..1730) [1734..1737) [1739..1740) [1741..1742) [1748..1749) [1765..1766) [1776..1777) [1778..1779) [1780..1789) [1790..1794) ] }, state_map: StateVector({15658536145409349539: 5993, 3019702115965040724: 1796}) }

I noticed recent changes and fixes related to GC and deletions and tried them already btw.
I wish I could give a easy to reproduce example but our stack is complex.
Are we maybe doing something wrong? is the above scenario a possible scenario? Shouldn't delete sets converge eventually?

Thank you 🙏

[Horusiath]

You can try to build against current main: if the source of an issue is the same as in case of #404, this could help until we make the next release.

[sinkingsugar]

You can try to build against current main: if the source of an issue is the same as in case of #404, this could help until we make the next release.

Thank you!
I tried main already as I saw your #404 fix. Didn't help sadly.

[sinkingsugar]

Btw it seems that skip_gc: true fixes the above issue.

[Horusiath]

Are you using UndoManager or Snapshots by any chance?

[sinkingsugar]

Are you using UndoManager or Snapshots by any chance?

UndoManager yes 👍
Not Snapshots (I know docs mention it needs skip_gc: true)