Add entities to support per-device and per-scanner options for ref_power and to exclude proxies from calculations

[agittins]

TL/DR:

• Allow certain proxies (or local bluetooth adaptors) to be excluded from Bermuda's calculations
• Provide per-receiver calibration via a ref_power_offset setting to account for varying receiver sensitivity
• Provide per-device ref_power setting to allow overriding the default ref_power value.

This will allow us to account for varying beacon and receiver sensitivities caused by radio design, antenna and enclosure variations, while still allowing use of defaults for easier basic setup.

Original description:

Currently all devices use the global ref_power setting, and device names only come from local_name advertisement data, which seems unreliable but is also limiting for users, especially when doing initial setup.

Effective ref_power is a function of these variables

• device transmitter power (as well as "reported" device tx power)
• device antenna design, placement and enclosure
• receiver sensitivity
• receiver antenna design, placement and enclosure

It makes sense to be able to specify a ref power @1m value for each transmitter (device), and also an offset value for each receiver (or receiver type).

Devices:

• Allow the user to specify a name for the device (helpful if local_name isn't known, and save renaming sensors later) (no, just let them rename in the UI. Naming seems stable now).
• Global ref_power should be set to match one's most common transmitter/beacon.
• Each device can optionally have a custom ref-power defined.
• Unsure if better to define per-device setting as an offset from global ref_power or as an absolute. The latter might be less confusing.

Scanners:

• Will eventually need a setting for co-ords anyway
• A ref_power_offset makes more sense for scanners, since one is likely to have multiple identical scanners, but a wide range of devices. Also an absolute value wouldn't make sense as it's a function of the combination of tx/rx pair.
• max_radius as a per-area setting makes sense, so you don't need to enter an adjacent area in order to no longer be in "this" area.

[agittins]

Issue #164 is dependent on this being resolved.

[jeremysherriff]

The stronger BLE signal reception available via a quality Bluetooth USB dongle (on a good USB extension cable with ferrite interference suppressor bead) in comparison to the repeated signals presented via the common ESP32 BLE proxies (usually with internal antenna) is the main drawback for me for this integration right now.

Is it viable to introduce an interim workaround of the environmental attenuation factor and rssi at 1m settings specific to the hci devices, as a 2nd set of variables? That would allow a method of tuning down (or up, if appropriate) the factor of the directly-connected USB devices, without all the work required to refactor code to allow for per-device equivalents.

My thinking is;

• this would immediately solve my problem 😃
• it would begin the process of scaling the signal values from different receivers, without significant refactoring
• it would likely expose other challenges with scaling per-receiver signals and lead to better outcomes when Add entities to support per-device and per-scanner options for ref_power and to exclude proxies from calculations #45 is fully implemented with per-receiver settings

Downsides;

• because the Config Flow setup would still be global, this would be a single value to apply to all hci devices so it does not solve advanced use-cases such as people using multiple dongles
• it may detract from efforts to implement the full end-state
• it may complicate configuration migration to the full end-state, forcing the release of the end-state to be a breaking change

Thanks for listening to my brain-dump.
- Jeremy

[agittins]

Is it viable to introduce an interim workaround of the environmental attenuation factor and rssi at 1m settings specific to the hci devices, as a 2nd set of variables?

I think we're pretty close to having the per-device settings sorted out and it's currently being worked on. We need to have per-scanner adjustments for trilateration (a key goal) and modifying it for a class of scanners (hci) would be fairly equivalent to the work required to do it for all - and I think it would require more effort to do both in sequence than just go straight to per-scanner.

• this would immediately solve my problem 😃

😆 this is indeed a noble motivation!

Thanks for listening to my brain-dump. - Jeremy

It was good quality and worth listening to!

[Cenedd]

Commenting partly to follow the discussion but also (in case I'm not the only one that didn't know this!) that the signal strength received by ESP32-S3 units is significantly higher than those of a ESP32. Presumably this is either a stronger antenna or a result of the bump from Bluetooth 4 to 5. With both types of unit in play, I'm sometimes being reported as being two floors higher (in the loft) than the unit I'm literally sitting next to.

[Edivion]

Any news on this feature? I'm eagerly waiting for this to be released so that I can finally re-enable my hci bluetooth devices. They are messing up my otherwise nicely working ESP32 fleet.

[adidoes]

@Edivion I'm on the beta and it's been working pretty well, I recommend giving it a try 👍

[lazyboy0172]

@adidoes dumb question but I’m on the beta as well and don’t notice a difference/config option- could you quickly explain how to change a nodes strength or ignore it? I have m5 atoms around the house and a Pi4 in a utility closet and consistently see jumps between wherever the device actually is and the utility room.

[klim123123]

@Edivion I'm on the beta and it's been working pretty well, I recommend giving it a try 👍

how to install beta version if you already have 0.6.8 stable? i searched all over Home Assistant, can't find this option. and nobody talked about this anywhere.

[lazyboy0172]

@klim123123 I'm assuming you're using HACS- Go to HACS menu and find Bermuda. Click the 3 dots and select 'Redownload'. A pop up will come up asking which version to download. Select the drop-down and choose the latest (v0.6.9rc2 as of time of this comment) and select Download. It should install and require a restart.

Once restarted, go to Integrations, find Bermuda and select it. select 'Configure'. You can then get to the Calibration options. I went to Calibration 2 RSSI offsets, and for my one 'problem' node that I wanted to eliminate entirely I just changed that value from 0 to -500 and it worked.

[Edivion]

Thanks @adidoes, switched to the rc2 and I am finally able to have all my bluetooth devices enabled again without interfering with the BLE triangulation.

@agittins thank you for your dedication and amazing work on this project!

[evertide]

Thanks, I have a main bluetooth adapter as well picking everything up.

The calibration in rc2 worked by setting rssi offset to -500. However I had to select a device as well to submit the value. I thought it was a global setting for all devices including future paired ones

regards

[jeremysherriff]

The device selection is only for showing you the outcome, so that you can experiment with different values before submitting. The value is applied to all devices.

…

On Fri, Oct 25, 2024, 4:11 PM evertide ***@***.***> wrote: Thanks, I have a main bluetooth adapter as well picking everything up. The calibration in rc2 worked by setting rssi offset to -500. However I had to select a device as well to submit the value. I thought it was a global setting for all devices including future paired ones regards — Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIIGZPYTSAF3PN6SOXGRWDZ5GZHZAVCNFSM6AAAAAA6AYKEU2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZWG4YTSOBVHE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[marcuscps]

RC2 is a great progress.
I believe the ideal solution would be allow at least specifying the following configurations per Base Station:

• Max Radius [m]
• RSSI Offset [db]
• Attenuation Factor [1]

RSSI @ 1m is advertised on the iBeacon Descriptor, but the receiver (base station) could offset it to make sure readings are correct. Antena design can influence readings dramatically.
In the end, most of the global configurations could be configurable for individual Base Stations.

A very useful feature could have an "Observer" tool that shows Distance and RSSI readings for all devices detected by a given Base Station. It would make it a lot easier to fine tune configurations.

I've done a lot of Bluetooth, iBeacon and Eddystone development in the past. If you want some help, I could join the fun.
Just would need a quick intro to become productive quickly.
Let me know if my help is welcome.

Thanks a lot for the great project. Amazing work!

[marcuscps]

RC2 is a great progress. I believe the ideal solution would be allow at least specifying the following configurations per Base Station:

• Max Radius [m]
• RSSI Offset [db]
• Attenuation Factor [1]

RSSI @ 1m is advertised on the iBeacon Descriptor, but the receiver (base station) could offset it to make sure readings are correct. Antena design can influence readings dramatically. In the end, most of the global configurations could be configurable for individual Base Stations.

A very useful feature could have an "Observer" tool that shows Distance and RSSI readings for all devices detected by a given Base Station. It would make it a lot easier to fine tune configurations.

I've done a lot of Bluetooth, iBeacon and Eddystone development in the past. If you want some help, I could join the fun. Just would need a quick intro to become productive quickly. Let me know if my help is welcome.

Thanks a lot for the great project. Amazing work!

Just adding a bit more detials:

I found the "Calibration 1: Global" and "Calibration 2: Scanner RSSI Offsets" a bit confusing.
It wasn't clear to me if I'd be applying those values for a (Scanner, Device) only or for (Scanner, All devices).
@jeremysherriff you clarified above that it is for All devices, but it is not clear to me where one can see the outcomes of such configuration changes. Could you clarify? Is it by observing the behaviour of sensor.<sensor name>_area and sensor.<sensor name>_distance in Dev Tools?
This is where my suggestion for an "Observer" tool comes from.

[jeremysherriff]

@marcuscps Yes, you can see the outcomes in Dev Tools or in Settings -> Integrations -> Bermuda. As per the docs, it is recommended you use one specific representative device to help calibrate the distance and that would be true for the per-scanner rssi offsets too.

Distance is a server-side estimate based on received signal strength, so if you have lots of different BLE devices with differing antenna designs/battery voltages/signal strengths then the distance will not be consistent or accurate.

@agittins While I was typing this it dawned on me that perhaps a per-device offset (as well as a per-scanner offset) would help - but starting to get very complex to configure (and presumably to code)..

[Cenedd]

Depends what the priority is. If you're trying to do room-level location, the per-device variation wouldn't matter as you can determine it's location based on the relative strengths to each scanner.
The only thing that might make per-device differences relevant is positioning within a room - the absolute distance of two different devices from a particular sensor...and even that could be avoided if the triangulation/trilateration feature can be made to work.

[agittins]

@Cenedd
Depends what the priority is. If you're trying to do room-level location, the per-device variation wouldn't matter as you can determine it's location based on the relative strengths to each scanner.

I think you are the first person (other than myself, a few hundred times) to explicitly point this out. The actual distances don't really matter, it's their relative relationships between proxies where we get the real power.

The absolute distance is super noisy and unreliable - so the plan has always been to leverage the relative "distances" rather than depend on absolutes. The current closest-area thing sort of does this, but full trilateration will "really" do it.

That said, most people won't believe it, plus I haven't grokked the math enough to fully appreciate the non-linearities, so that's partially why per-device calibration offsets has always been 1/3 of this year-old planned feature :-) That, plus, it turns out that the distances can be moderately useful, given the right sort of filtering and assumptions - within limits.

@marcuscps
I believe the ideal solution would be allow at least specifying the following configurations per Base Station:

Max Radius [m]
RSSI Offset [db]
Attenuation Factor [1]

Max_radius is the bane of my existence. I should never have made it a configurable setting as it is constantly abused by people. The correct setting for every single proxy is 1 million kilometres. But.... sometimes being able to use incorrect settings is helpful for working around a shortcoming elsewhere. I have no current intention of making max_radius a per-scanner setting though, and it would take some convincing to bring me around on that.

In theory, Attenuation should be the same for all proxies that share the same atmosphere. Differences in antenna, enclosure and immediate surroundings should be accounted for by offsetting the rssi, but (waves hands around) shouldn't affect attenuation. The attenuation is meant to be a constant for the air - it might vary inside vs outside but the research I've read shows very little affect from humidity and temp (if you look this up: They all see a consistent affect. But it is stupidly small). For indoors, I suspect it might be more affected by reflections than anything. Of course, this is RF propagation at 2.4GHz we're talking about - so well into Dark Arts and Magiks territories.

As for whether the model is broken, that's entirely possible. If someone did some solid research showing if different receivers benefited from using differing attenuation factors for the same environment, I'd be all over that. If anyone knows someone at LTT or anywhere else with an anechoic RF chamber and too much spare time, hook me up! 😄

A very useful feature could have an "Observer" tool that shows Distance and RSSI readings for all devices detected by a given Base Station. It would make it a lot easier to fine tune configurations.

This is what the calibration UI is meant to be. But not for "all devices" because I don't have room on my screen for hundreds of entities! The idea is to focus on one device to use as a "reference" during calibration.

Ultimately HA is really limited in what an integration can do with the UI, so being able to jam tables and stuff into the strings shown in the config ui is a dirty hack, but it works.

There's always been a service (aka "action") called bermuda.dump_devices which responds with a json blob giving Bermuda's entire internal state. From this one could create all sorts of views or reports, if so inclined. But as for building things into Bermuda I think I need to be fairly careful to keep it focused, because too much info can also cause extra support load - so it's about finding a good balance. We're not there yet, but making progress, I think.

RSSI @ 1m is advertised on the iBeacon Descriptor

Yes, but it's mostly useless for us, because it assumes a given receiver setup which is 0% likely to match ours - which is why we don't use it. The place where it can be useful is if the user calibrates it themselves, or if the beacon changes its TX power and adjusts its advertised tx_power to match, in which case we could track and offset against that. But I'm a long way from wanting to implement that currently, since in the first case the user could just calibrate bermuda instead, and in the second it is a messy config, requiring us to store the tx_power at the time the device was calibrated in order to know when to apply an offset to it. It may end up worthwhile, because I see iPhones in particular seem to vary their tx_power, but then - maybe with trilat those sorts of "absolute" measurements won't matter very much. Also, many (most?) beacons seem to use fairly bogus values in their tx_power.

I've done a lot of Bluetooth, iBeacon and Eddystone development in the past. If you want some help, I could join the fun.
Just would need a quick intro to become productive quickly.
Let me know if my help is welcome.

Absolutely! I'm a bit swamped at the moment, and until I can bed down a release and then get initial trilateration support out, I think I'm ill-equipped to make use of extra hands. I have raging ADHD and while I've worked in dev shops as sysadmin/devops, I've always coded fairly solo, so don't have good workflows for playing with others yet. Once I get trilat out I think we'll then have a basic platform that has most of the architecture sorted, and we can make better use of feature contributions.

Most of the work is python / HA related though, and less so on the bluetooth-specific sides of things, but all help is welcomed. Hmmm - have you done any stuff with nrf51822? I've been prototyping a cheapest-possible BLE BTHome remote button, ideally to snap over my existing light switches - but it's been months since I touched it!

I have sort of been trying to focus on bedding down a stable Bermuda v1 that's easy to support, before leaping into trilat. This is because as the integration has become more popular the support load has increased, and I know it's going to explode once trilat is out - so building some infra around that like the calibration and diagnostics has been a focus for making support less frequent / easier. With varying levels of success :-)

That said, there's at least 1700 active Bermuda installs out there currently, averaging 16 new installs a day! Figures might be 3 to 4 times that depending on how many people enable HA's telemetry - so I guess the support needs are reasonable given those figures. 🤷🏼

I found the "Calibration 1: Global" and "Calibration 2: Scanner RSSI Offsets" a bit confusing.

If I added to this something like "Although you will choose a device and a scanner for measuring your calibration, the resulting changes will apply to all scanner / device combinations" do you think that would help?

Does this header need re-writing? Or maybe just add at the end something similar to above, eg: "Although you will choose a device below for measuring the calibration, the resulting configuration is per-scanner, and will apply to any device."

It's a bit of an awkward thing to describe, really. Frankly I just need to make some videos, I think.

but it is not clear to me where one can see the outcomes [...] Is it by observing the behaviour of sensor._area and sensor._distance in Dev Tools?

You don't need dev tools, you just look at the sensor values for distance, at a time when it is closest to that particular scanner. But you can also just select another device while in the calibration, and see how the figures look for that device.