Implement pairing with btstack.

[felixdoerre]

Btstack offers two abstraction layers for secret storage, one
called "device db" and another called "tlv". Pairing information
is stored in the "device db", additional secrets (like own keys)
are stored directly in the tlv. Luckily there is a "device db"
implementation using tlv, so we only need to provide one interface.

Additionally, I've removed some btstack files from compilation that
were not referenced.

This PR builds upon #10838 where, from what I can tell, only the signed-off-header is missing. If that's a problem I could re-implement those changes.

With these changes I've tested bonding (with mp/btstack as peripheral), with random, public and RPA address.

[github-actions]

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS

[andrewleech]

@brianreinhold you might like to compare notes here!

[brianreinhold]

@felixdoerre We have been trying to get pairing to work on the PICO W which uses the btstack. We have a working implementation (except for interactive passkeys) but it is sharply geared toward our needs. We do not have python expertise and are new to Micropython and the btstack (I have done a lot with other BTLE C stacks designed for specific platforms such as Nordic and Renesas).

Do you have an 'official' version supported through aioble?

We found that some of the trickiest parts were devices that sent security requests to invoke the client to begin either pairing or encryption or both and sometimes as a replacement to an insufficient authentication error.

The difficulty was that the btstack handles the security request and aioble and the app know nothing about it BUT the app (via aioble) has to handle the insufficient encryption/authentication errors. To solve this problem I had to add a method to aioble that would register for an encrytion changed event but do nothing else. In the modbluetooth_btstack.c file I had to add more events to those that would be fed to aioble's encryption changed event. I would run this listener as a task in the background after connection. In this manner I could inform aioble and my application of a pairing or re-encryption event due to security request. It works but it is not nice. Ideally a security request event should be passed up to aioble and the application so the application could invoke pairing/encryption as it does with an insufficient authentication/encryption error.

I suppose I should add to this discussion on pairing the problem of bonding. Many devices (at least health devices that we are interested in) assume that on a bonded reconnect that they can send their data as soon as encryption is established and disconnect (saves battery). aioble does not save service discovery results so these devices often disconnect before aioble completes re-doing service discovery. aioble even invokes descriptor discovery when one tries to 'subscribe' which is slow exasperating the problem. I tried to see if it was possible to save the results of service discovery but it is not as they are objects and not just data and on the next connection (or after a power cycle) this information is no longer valid and one is forced to redo these transactions.

[felixdoerre]

Do you have an 'official' version supported through aioble?

I aim that this will be the 'official' version, but that depends on maintainers picking it up. I have not tested it with aioble, but in the scenario that I tested this with, I don't see anything that would prevent it from working with aioble.

We found that some of the trickiest parts were devices that sent security requests to invoke the client to begin either pairing or encryption or both and sometimes as a replacement to an insufficient authentication error.

I did have an issue, where btstack would not answer a pairing request (bluekitchen/btstack#584), but that was a mistake in btstack and it got resolved. The scenario that I have working is micropython/btstack as a ble peripheral (ble hid keyboard) and an android device as ble central. The android device initiates pairing, and micropython/btstack correctly responds and stores bonding information. Reconnection across micropython restarts work flawlessly. This works now both with public and with rpa addresses.

The difficulty was that the btstack handles the security request and aioble and the app know nothing about it BUT the app (via aioble) has to handle the insufficient encryption/authentication errors

So you have micropython acting as ble central/gatt client? I have not tested that.

In the modbluetooth_btstack.c file I had to add more events to those that would be fed to aioble's encryption changed event.

I don't get which problem you are trying to avoid here. I have not tested btstack as ble central, so I am not sure how pairing in response to a insufficient authentication/encryption error would work, but for BLE peripheral/gatt server, I just flag the attribute as authentication/encryption required and btstack handles the corresponding responses.

aioble even invokes descriptor discovery when one tries to 'subscribe' which is slow exasperating the problem

I thought that you only need to "subscribe" once per device/bond and that this information is supposed to be saved persistently.

[brianreinhold]

@felixdoerre our application is a BTLE central running on the PICO W. The central/client is responsible for initiating pairing (first time) encryption (reconnects). An insufficient authentication/encryption happens when the peripheral has secured an action on a characteristic/descriptor and the client tries to perform that action pre-pairing or pre-encryption. Peripherals do not have to deal with that error. A client is supposed to either pair or encrypt and then redo the action that caused the error.

The btstack does not automatically handle this error (the Android client does) so my application has to catch the GattError exception raised by aioble and use aioble to initiate pairing (same method works for encryption since btstack knows if it is paired).

However, a peripheral can also invoke a security request. Many health devices do this. This tells the client to get off its butt and initiate pairing/encryption. The btstack handles this automatically (as does the Android client) and aioble (and my application) has no idea that it ever happened.

Ideally the btstack would handle BOTH insufficient authentication/encryption errors and the security requests automatically (as on Android), or pass both situations up to the application and let the application do it. I would prefer the latter as ongoing pairing the application may not know about could lead to timeouts on operations the application is trying to do. This is a big pain in Android as I have to check every write and read operation to see if pairing is ongoing and wait until it is finished.

As far as bonding is concerned you are correct - in theory you should only need to subscribe once and if you are bonded that should not have to be repeated. We have not tested that by itself, however, service discovery results are not persisted and that is part of bonding. We need to read the Current Time Service and other characteristics like battery level every connection which requires that we redo service discovery. Also, many BTLE devices require that one re-subscribe event though they say they support bonding. That is just a bad implementation by the peripheral but such devices are on the market and we need to work with them. Resubscribing is fast if the results of service discovery are persisted so it has not been a problem (on Android).

IN any case, that is where we are at. We have it working including some passkey support and unpairing/bonding support at least for a PICO W client. We have no need at this time for peripheral support.

[felixdoerre]

However, a peripheral can also invoke a security request. Many health devices do this. This tells the client to get off its butt and initiate pairing/encryption. The btstack handles this automatically (as does the Android client) and aioble (and my application) has no idea that it ever happened.

I would assume that this is what the _IRQ_ENCRYPTION_UPDATE-event is for. I am not sure if this PR already ensures it is triggered correctly, but I imagine this event should be triggered, when pairing and/or encryption was established. aioble does seem to update the DeviceConnection-object (https://github.com/micropython/micropython-lib/blob/master/micropython/bluetooth/aioble/aioble/security.py#L82), but as far as I can see, you cannot subscribe to this event, if this is was you need.

We need to read the Current Time Service and other characteristics like battery level every connection which requires that we redo service discovery.

To me, this feels like a limitation of aioble and not of a limitation of the low-level bluetooth API. And from what I see, there is nothing stopping you from storing the integer-handles from the individual characteristics, including CCCD, and re-creating the objects yourself. However the subscribe method, as implemented in aioble, always does one descriptor lookup, so you would need to provide your own.

[brianreinhold]

I fully agree that the btstack has no limitations. Once can do all Bluetooth transactions in that library.

The problem with the _IRQ_ENCRYPTION_UPDATE event is that you have to invoke some method that anticipates it, for example, the pairing method in aioble's security module. In order to detect the pairing caused by a security request I added a method to aioble which did nothing but register for that _IRQ_ENCRYPTION_UPDATE event. I invoke a background task on connection which calls that aioble method and when a security request happens, that task gets notified (and I also notify the DeviceConnection object of aioble). It's not the most elegant approach but it does work.