Support `NSUbiquitousKeyValueStore` #136

hank121314 · 2023-05-22T07:05:23Z

Summary

This PR fixes: #75

Create a Defaults.iCloud class which facilitates the synchronization of keys between the local UserDefaults and NSUbiquitousKeyValueStore.
This class utilizes a TaskQueue to execute synchronization tasks sequentially, ensuring proper order and coordination.
Internally, Defaults.iCloud contains an AtomicSet that indicates the synchronization status of each key. This set prevents observation triggers when synchronization is being executed.

And I also create a new protocol Defaults.KeyValueStore which contains essential properties for synchronizing a key value store. It can be useful when mocking a test or even support another key value data source.

Thanks

Please feel free to critique, and thanks for your code review 😄 .
Any feedback on the PR is welcomed and appreciated 🙇 !

sindresorhus · 2023-05-22T09:15:14Z

Wow. This is really awesome! 🎉

Sources/Defaults/Defaults+iCloud.swift

sindresorhus · 2023-05-22T08:33:06Z

Sources/Defaults/Defaults+iCloud.swift

+	private func addObserver(_ key: Defaults.Keys) {
+		backgroundQueue.sync {
+			key.suite.addObserver(self, forKeyPath: key.name, options: [.new], context: nil)
+		}
+	}
+
+	private func removeObserver(_ key: Defaults.Keys) {
+		backgroundQueue.sync {
+			key.suite.removeObserver(self, forKeyPath: key.name, context: nil)
+		}
+	}
+
+	@_documentation(visibility: private)
+	// swiftlint:disable:next block_based_kvo
+	override public func observeValue(
+		forKeyPath keyPath: String?,
+		of object: Any?,
+		change: [NSKeyValueChangeKey: Any]?, // swiftlint:disable:this discouraged_optional_collection
+		context: UnsafeMutableRawPointer?
+	) {
+		guard
+			let keyPath,
+			let object,
+			object is UserDefaults,
+			let key = keys.first(where: { $0.name == keyPath }),
+			!atomicSet.contains(key)
+		else {
+			return
+		}
+
+		backgroundQueue.async {
+			self.recordTimestamp(.local)
+			await self.syncKey(key, .local)
+		}
+	}


It would be nicer to wrap this up into a separate utility that is decoupled from Defaults.iCloud. And also have it return an async stream. I generally prefer to wrap legacy APIs in a nicer interface. This also helps keep code clean.

sindresorhus · 2023-05-22T08:56:08Z

Sources/Defaults/Defaults+iCloud.swift

+
+	- Note: `source` should be specify if `key` has not been added to `Defaults.iCloud`.
+	*/
+	public static func syncKeys(_ keys: Defaults.Keys..., source: DataSource? = nil) {


What's the use-case for this method? Do we really need to expose it?

sindresorhus · 2023-05-22T08:57:46Z

Sources/Defaults/Defaults+iCloud.swift

+	/**
+	Synchronize all of the keys that have been added to Defaults.iCloud.
+	*/
+	public static func syncKeys() {


It's a bit confusing to have both .sync() and .syncKeys() and it's not immediately clear when you would want to use either. Can we just make .sync() do both? Or am I missing something?

syncKeys() will create synchronization task for all keys and push it into backgroundQueue.
await sync() will wait until all synchronization tasks done, maybe we can call it flush()?

flush() or syncWithoutWaiting() would work.

But my question still stands, do we really need to expose this to the user?

I saw some requests in ArtSabintsev/Zephyr#32.
With await flush(), users will know that all pending synchronizations are done.

I think it should be syncWithoutWaiting() and await sync(). That's the most explicit names I can think of.

sindresorhus · 2023-05-22T08:58:44Z

Sources/Defaults/Defaults+iCloud.swift

+				return .remote
+			}
+
+			return localTimestamp.timeIntervalSince1970 > remoteTimestamp.timeIntervalSince1970 ? .local : .remote


Suggested change

return localTimestamp.timeIntervalSince1970 > remoteTimestamp.timeIntervalSince1970 ? .local : .remote

return localTimestamp > remoteTimestamp ? .local : .remote

Sources/Defaults/Utilities.swift

sindresorhus · 2023-05-22T09:13:40Z

Sources/Defaults/Defaults+Protocol.swift

+public protocol _DefaultsKeyValueStore {
+	func object(forKey aKey: String) -> Any?
+	func set(_ anObject: Any?, forKey aKey: String)
+	func removeObject(forKey aKey: String)
+	@discardableResult
+	func synchronize() -> Bool
+}


Suggested change

public protocol _DefaultsKeyValueStore {

func object(forKey aKey: String) -> Any?

func set(_ anObject: Any?, forKey aKey: String)

func removeObject(forKey aKey: String)

@discardableResult

func synchronize() -> Bool

}

public protocol _DefaultsKeyValueStore {

func object(forKey key: String) -> Any?

func set(_ object: Any?, forKey key: String)

func removeObject(forKey key: String)

@discardableResult

func synchronize() -> Bool

}

sindresorhus · 2023-05-22T09:19:04Z

Sources/Defaults/Utilities.swift

+		let semaphore = DispatchSemaphore(value: 0)
+
+		queueContinuation?.yield {
+			await task()
+			semaphore.signal()
+		}
+
+		semaphore.wait()


This will deadlock if the task accidentally calls something that also calls .sync() (nested calls).

According to ChatGPT, you could do something like this:

final class TaskQueue { typealias AsyncTask = @Sendable () async -> Void private var queueContinuation: AsyncStream<AsyncTask>.Continuation? private let queue: DispatchQueue init(priority: TaskPriority? = nil) { queue = DispatchQueue(label: "com.example.taskqueue", attributes: .concurrent) let taskStream = AsyncStream<AsyncTask> { queueContinuation = $0 } Task.detached(priority: priority) { for await task in taskStream { await task() } } } deinit { queueContinuation?.finish() } func async(_ task: @escaping AsyncTask) { queue.async { self.queueContinuation?.yield(task) } } func sync(_ task: @escaping AsyncTask) { let semaphore = DispatchSemaphore(value: 0) queue.async { if DispatchQueue.getSpecific(key: self.queue.key) == self.queue.id { Task { await task() semaphore.signal() } } else { self.queueContinuation?.yield { await task() semaphore.signal() } } } semaphore.wait() } func flush() async { await withCheckedContinuation { continuation in queue.async { self.queueContinuation?.yield { continuation.resume() } } } } }

That's true, I think we should deprecate sync.
Would it be acceptable to use TaskQueue.async for updating remote key changes?

The async queue is concurrent, so it maybe cause ordering problems.

TaskQueue.async is not concurrent; it will run similarly to DispatchQueue.async.
I believe it should not have ordering problems.

Btw, if calling await flush in TaskQueue.async, it will also cause deadlock.
Maybe we should emit a Xcode runtime warning to prevent user doing this.

ex.

let queue = TaskQueue(priority: .background) queue.async { print("1") queue.async { print("2") } await queue.flush() } await queue.flush() // => 1 "2" will never print.

Btw, if calling await flush in TaskQueue.async, it will also cause deadlock.
Maybe we should emit a Xcode runtime warning to prevent user doing this.

Users don't have access to TaskQueue though. It's an internal type.

hank121314 · 2023-05-26T07:29:10Z

It is challenging to change the Key-Value Observing (KVO) mechanism of UserDefaults to asyncstream.

Here is a demonstration of how Defaults.iCloud works.

flowchart TD
    A("UserDefaults \n Changes \n ex.Defaults[key] = 0")
    B("NSUbiquitousKeyValueStore \n didChangeExternallyNotification")
    C("Defaults.iCloud.syncKeys()")
    D[\"Push \n Synchronization \n Task \n Sequentially"/]
    E(["Executing Tasks \n Task.detached"])
    F(["All Tasks Complete"])
    A & B & C --> D --> E -- "await Defaults.iCloud.sync()" --> F

If we change the observation to asyncstream, there is a need to create a task to consume the asyncstream. Since the user's change is not in the same task as the asyncstream, it is difficult to ensure that the synchronization is completed synchronously.

ex.

Task.detached {
	for await change in updates {
		print("\(change)")
	}
}

Defaults[key] = 0
Defaults[key] = 1
await Defaults.iCloud.sync()

// => 0
// await Defaults.iCloud.sync() get call
// => 1

sindresorhus · 2023-05-26T07:36:50Z

Understood. It's fine. We can use KVO then. Just add a todo to look into using async stream when Swift supports custom executors.

hank121314 · 2023-07-12T01:43:10Z

I believe it might be worth considering waiting for the release of Swift 5.9, which will include the implementation of Custom Actor Executors.
Once it's available, we can transition all Key-Value Observing (KVO) and notifications to utilize AsyncStream.

sindresorhus · 2023-08-16T23:18:16Z

Sources/Defaults/Utilities.swift

@@ -327,21 +338,9 @@ final class TaskQueue {
 	Queue a new asynchronous task.
 	*/
 	func async(_ task: @escaping AsyncTask) {
+		lock.lock()


It's generally better to have a .with type method to make sure the locked resource is always unlocked. Like: https://developer.apple.com/documentation/os/osallocatedunfairlock/4061628-withlock

sindresorhus · 2023-08-16T23:29:52Z

Is the custom actor executors feature backdeployed? If not, we would have to target macOS 14, which does not make sense.

sindresorhus · 2023-08-16T23:19:46Z

Sources/Defaults/Defaults+iCloud.swift

+import Combine
+import Foundation
+
+/// Represent  different data sources available for synchronization.


Suggested change

/// Represent different data sources available for synchronization.

/**

Represent different data sources available for synchronization.

*/

sindresorhus · 2023-08-16T23:36:51Z

Sources/Defaults/Defaults+iCloud.swift

+private enum SyncStatus {
+	case start
+	case isSyncing
+	case finish
+}


This would be more idiomatic Swift:

Suggested change

private enum SyncStatus {

case start

case isSyncing

case finish

}

private enum SyncStatus {

case idle

case syncing

case completed

}

Not sure idle is correct though.

sindresorhus · 2023-08-16T23:38:39Z

Sources/Defaults/Defaults+iCloud.swift

+		/**
+		The singleton for Defaults's iCloudSynchronizer.
+		*/
+		static var shared = Defaults.iCloudSynchronizer(remoteStorage: NSUbiquitousKeyValueStore.default)


The name is not correct. It makes it sound like it's the singleton of iCloud.

Suggested change

static var shared = Defaults.iCloudSynchronizer(remoteStorage: NSUbiquitousKeyValueStore.default)

static var synchronizer = Defaults.iCloudSynchronizer(remoteStorage: NSUbiquitousKeyValueStore.default)

hank121314 · 2023-11-14T04:49:39Z

Is the custom actor executors feature backdeployed? If not, we would have to target macOS 14, which does not make sense.

SerialExecutor is back deployed to macOS 10.15+, but some of its methods is deprecated and new methods only target to macOS 14+. And I am also misreading the proposal, we must wait until the Specifying Task executors is completed.

hank121314 · 2023-11-14T05:01:55Z

Btw, my old MacBook can not install macOS 14+ and Xcode 15+, so I can not develop some features which related to swift 5.9 😭 . Will buy the M3 MacBook until it release in Taiwan.

Kinark · 2023-12-12T08:31:10Z

Any updates on this? It's a really awesome PR :/

hank121314 · 2023-12-29T14:12:18Z

I want to check a behavior when initialized with iCloud: true.
Currently, we push synchronization from the local data source triggered by suite.register.
However, I am concerned that this may overwrite remote data when installing on a new device.
Perhaps it would be better to add UserDefaults observation after suite.register and let the user manually call Defaults.syncWithoutWaiting (which compares timestamps between the two data sources).
What do you think?

sindresorhus · 2023-12-30T00:11:08Z

However, I am concerned that this may overwrite remote data when installing on a new device.
Perhaps it would be better to add UserDefaults observation after suite.register and let the user manually call Defaults.syncWithoutWaiting (which compares timestamps between the two data sources).
What do you think?

I would prefer if users don't have to call anything manually. Can we wait until the sync (fetching remote data) is done before pushing the local data to remote?

Maybe also look at how Zephyr handles this.

hank121314 · 2023-12-30T13:53:27Z

Seems Zephyr also facing the same issue. ArtSabintsev/Zephyr#50 (comment)

I have a thought to resolve this issue, but it might involve some trade-offs.
The current implementation only records the latest timestamp when data is set in the data source, using only one key.
If we can record the latest timestamp for all keys, we can ensure whether a specific key needs to be synced or not.
However, the trade-off is a 2x space allocation with both remote and local storage.

sindresorhus · 2023-12-30T13:58:52Z

Local storage is not a problem, but synced storage is a problem as it's limited to 1 Kb in total. Is there a way to use timestamps for each key but not sync them?

hank121314 · 2023-12-30T14:15:53Z

Is there a way to use timestamps for each key but not sync them?

Oh, this is a good idea.
For this use case, where a user has a fresh new device with some data in iCloud Storage, we can consider the iCloud Storage as the latest data since there is no corresponding timestamp key in their local storage.

Will try to implement this in a few days! Thank you 🙌 !

hank121314 · 2024-01-13T12:49:40Z

I tried the method mentioned above but noticed that it may not cover all use cases.

If a user has multiple Defaults.Keys with iCloud set to true, after syncing the first key, the remote timestamp is recorded. When syncing the second key, we cannot be sure about the state of the data source.

For example.

// Choose `local` data source, sync `name` to `remoteStorage`, record `remoteStorage` timestamp.
let name = Defaults.Key<String>("name", default: "0", iCloud: true)
// `remoteStorage` have timestamp record and local timestamp is empty so we will choose `remote` data source, sync `quality` to `localStorage`, but the data source should be `local` instead.
let quality = Defaults.Key<Double>("quality", default: 0.0, iCloud: true)

Considering the scenario, it seems necessary to maintain timestamp records in both remoteStorage and localStorage to accurately determine the appropriate data source 😭 .

And considering the limitation of 1024 keys in remoteStorage, a potential solution could be to consolidate the value and timestamp into a single key?

For example.

class iCloudKey {
	let value: Any?
	let timestamp: Date?
}

sindresorhus · 2024-01-24T13:12:46Z

Considering the scenario, it seems necessary to maintain timestamp records in both remoteStorage and localStorage to accurately determine the appropriate data source 😭 .

You are right. I don't see any way around this either.

hank121314 · 2024-01-26T11:02:59Z

So it is necessary to maintain the timestamp in both storages.

For localStorage, where there is no limitation, we can introduce another key with a prefix (__DEFAULTS__synchronizeTimestamp) to record the timestamp.
As for remoteStorage, considering the 1024 keys limitation, we can combine the value and timestamp into a single key and stored it in remoteStorage.

Is this an acceptable solution?

sindresorhus · 2024-01-26T16:56:12Z

Correct

For the remote storage we should try to make it take the least amount of space, because there is also a total limit of 1MB storage. Maybe we can convert the timestamp to fixed-length data, serialize the value to data too, combine them, and store them both in one Data value? To deserialize, we would just slice of the first part, which would be a fixed length data for the timestamp and then deserialize the rest as data. I'm open to other ideas.

hank121314 · 2024-02-02T10:29:45Z

Maybe we can convert the timestamp to fixed-length data, serialize the value to data too, combine them, and store them both in one Data value?

I believe there might be some effort required in serializing the value into data.
Considering that the value sent to remote storage can be guaranteed as a UserDefaults-acceptable object (retrieved from local storage, which is UserDefaults), we can simplify the process by storing a two-length array.
The first element would be the timestamp, and the second element would be the value (ex. [Date(), 1]).

sindresorhus · 2024-02-03T10:02:02Z

The first element would be the timestamp, and the second element would be the value (ex. [Date(), 1]).

👍

hank121314 · 2024-03-03T08:18:13Z

PR updated!
Do following changes:

The timestamp will now be based on the key to ensure accurate source detection.
To ensure consistency between two storage's timestamp, each synchronization task will be assigned a unique timestamp with @TaskLocal.
Will automatically sync the key after Defaults.iCloud.add

Please feel free to review it again! Many thanks!

Sources/Defaults/Defaults+iCloud.swift

sindresorhus · 2024-04-20T04:00:00Z

I finally had a chance to test this in a real app with syncing between Mac and iPhone and it seems to work fine 👍

sindresorhus · 2024-04-20T04:04:39Z

Superb work @hank121314 🙌

sindresorhus · 2024-04-20T04:29:24Z

I missed that this was not resolved: https://github.com/sindresorhus/Defaults/pull/136/files#r1544553011

sindresorhus · 2024-04-20T04:56:15Z

Also:

sindresorhus · 2024-04-20T07:53:09Z

I decided to make syncOnChange true by default (and internal) and renamed .sync() to .waitForSyncCompletion().

hank121314 · 2024-04-20T08:04:24Z

I decided to make syncOnChange true by default (and internal) and renamed .sync() to .waitForSyncCompletion().

Sure, the new name looks more explicit about the behavior!

sindresorhus requested changes May 22, 2023

View reviewed changes

sindresorhus reviewed May 22, 2023

View reviewed changes

sindresorhus reviewed Aug 16, 2023

View reviewed changes

sindresorhus requested changes Aug 16, 2023

View reviewed changes

frostplexx mentioned this pull request Oct 7, 2023

Add iCloud sync for settings / API key lo-cafe/winston#98

Closed

hank121314 force-pushed the main branch 2 times, most recently from f7d3737 to f38a785 Compare November 14, 2023 04:46

hank121314 force-pushed the main branch 2 times, most recently from fe9a371 to 375ae41 Compare December 29, 2023 13:54

arekf mentioned this pull request Feb 4, 2024

Sync watched status with Invidious yattee/yattee#611

Open

1 task

sindresorhus reviewed Mar 29, 2024

View reviewed changes

Sources/Defaults/Defaults+iCloud.swift Outdated Show resolved Hide resolved

sindresorhus reviewed Mar 29, 2024

View reviewed changes

Sources/Defaults/Defaults+iCloud.swift Show resolved Hide resolved

sindresorhus reviewed Mar 29, 2024

View reviewed changes

Sources/Defaults/Defaults+iCloud.swift Outdated Show resolved Hide resolved

sindresorhus reviewed Mar 29, 2024

View reviewed changes

Sources/Defaults/Defaults+iCloud.swift Outdated Show resolved Hide resolved

sindresorhus reviewed Mar 29, 2024

View reviewed changes

Sources/Defaults/Defaults+iCloud.swift Show resolved Hide resolved

hank121314 force-pushed the main branch 2 times, most recently from ccea22a to 082144e Compare April 4, 2024 16:00

hank121314 and others added 11 commits April 5, 2024 00:03

Support NSUbiquitousKeyValueStore

c82dd99

Refactor some code and fix some race conditions

c30d0c2

resolve comments and fix ambiguous issue when targeting swift 5.9

bdc9ab4

fix linter warnings

e79ec46

try to fix github action failure

3c27bb0

feat: minor improvement

3bd78e2

Update Defaults+iCloud.swift

91e5879

Update Defaults+iCloud.swift

67e4b4d

feat: apply comments

d87391a

Update readme.md

bb65888

feat: apply comments

90e3060

hank121314 force-pushed the main branch from 082144e to 90e3060 Compare April 4, 2024 16:04

feat: add visionOS availability

b4f92e0

hank121314 force-pushed the main branch from 1d15246 to b4f92e0 Compare April 4, 2024 16:21

fix: lintint violation

cd6153c

sindresorhus mentioned this pull request Apr 20, 2024

Improve UserDefaultsKeyObservation #170

Open

sindresorhus merged commit e837052 into sindresorhus:main Apr 20, 2024
2 checks passed

	return localTimestamp.timeIntervalSince1970 > remoteTimestamp.timeIntervalSince1970 ? .local : .remote
	return localTimestamp > remoteTimestamp ? .local : .remote

	static var shared = Defaults.iCloudSynchronizer(remoteStorage: NSUbiquitousKeyValueStore.default)
	static var synchronizer = Defaults.iCloudSynchronizer(remoteStorage: NSUbiquitousKeyValueStore.default)

Support NSUbiquitousKeyValueStore #136

Support NSUbiquitousKeyValueStore #136

Conversation

hank121314 commented May 22, 2023

Summary

Thanks

sindresorhus commented May 22, 2023

sindresorhus May 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hank121314 May 29, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hank121314 commented May 26, 2023 • edited

sindresorhus commented May 26, 2023

hank121314 commented Jul 12, 2023

Choose a reason for hiding this comment

sindresorhus commented Aug 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hank121314 commented Nov 14, 2023

hank121314 commented Nov 14, 2023

Kinark commented Dec 12, 2023

hank121314 commented Dec 29, 2023 • edited

sindresorhus commented Dec 30, 2023 • edited

hank121314 commented Dec 30, 2023

sindresorhus commented Dec 30, 2023

hank121314 commented Dec 30, 2023

hank121314 commented Jan 13, 2024 • edited

sindresorhus commented Jan 24, 2024

hank121314 commented Jan 26, 2024

sindresorhus commented Jan 26, 2024 • edited

hank121314 commented Feb 2, 2024 • edited

sindresorhus commented Feb 3, 2024

hank121314 commented Mar 3, 2024 • edited

sindresorhus commented Apr 20, 2024

sindresorhus commented Apr 20, 2024

sindresorhus commented Apr 20, 2024

sindresorhus commented Apr 20, 2024

sindresorhus commented Apr 20, 2024 • edited

hank121314 commented Apr 20, 2024

Support `NSUbiquitousKeyValueStore` #136

Support `NSUbiquitousKeyValueStore` #136

sindresorhus May 22, 2023 •

edited

hank121314 May 29, 2023 •

edited

hank121314 commented May 26, 2023 •

edited

hank121314 commented Dec 29, 2023 •

edited

sindresorhus commented Dec 30, 2023 •

edited

hank121314 commented Jan 13, 2024 •

edited

sindresorhus commented Jan 26, 2024 •

edited

hank121314 commented Feb 2, 2024 •

edited

hank121314 commented Mar 3, 2024 •

edited

sindresorhus commented Apr 20, 2024 •

edited