python: look into using asyncio

[mpvader]

Copy paste from Ivo's info in Slack:

I think asyncio is a good match for velib. It's perfect for IO bound applications.
I guess velib applications probably spend more than 90% waiting for external events. network, dbus, serial, filesystem, ...
A lot of performance can be gained if we can make those calls non-blocking:

a)
    send request A
    wait x time
    handle response A
    send request B
    wait x time
    handle response B
    send request C
    wait x time
    handle response C

=>

b)
    send request A
    send request B
    send request C
    wait x time
    handle response A
    handle response B
    handle response C

# almost 3x faster

The beauty with async/await is that you can still write your code in the style of a), but get the runtime behavior of b).

asyncio advantages
- no blocking IO calls
- code can be written (and looks) as if it was synchronous
- easier, more maintainable than other async approaches like Twisted, etc.

asyncio drawbacks
- code looks synchronous, but isn't
- when stepping through code with a debugger, it can jump all over the place
- care must be taken with exception handling
- better not mix it with threads
- can seem "magic" if you don't know what happens underneath

I have a lot of experience with async/await in c#. AFAICT, in Python it works almost the same.

Dbussy

I have looked around for alternative async dbus implementations for python,
but did not find anything noteworthy. As for Dbussy, someone has put a lot of thought into this, as evident from the readme

• it has a low level layer with async calls to dbus
• it has high level layer called 'ravel' which does basically what vedbus.py does but more general

One thing that worries me is that some users report segfaults on ARM, although someone seems to have found a workaround. ldo/dbussy#15
I think it would be instructive to implement vedbus.py with dbussy, and I'd be happy to do it if you want me to 🙂

PyModbus has asyncio bindings too:
https://pymodbus.readthedocs.io/en/latest/source/example/async_asyncio_serial_client.html

Another thing I'd like to bring forward, because I think it might be useful:
RX, aka Reactive Extensions. https://github.com/ReactiveX/RxPY/tree/release/v1.6.x

It's a library to work with observables. These are streams of values, which can be thought of as lists in time.

Below some examples how this might be used e.g. in vrmlogger: (It's ~ 80% real code + 20% pseudo)

def average(values):
    return sum(values) / len(values)

# dbus_msgs is an observable stream of dbus messages, think 'add_signal_receiver'

dbus_msgs.filter(lambda b: b.service_name == 'com.victronenergy.grid')  
         .filter(lambda b: b.object_path == '/Ac/Energy/Forward')       
         .buffer_with_time(5 * 60 * 1000)  # creates a list every 5 minutes containing the values of the last 5m
         .map(average)                     # take the average for each of these lists
         .subscribe(send_to_vrm)           # send to server

more advanced example:

buffered = dbus_msgs
          .observe_on(AsyncIOScheduler)      # offload onto AsyncIOScheduler
          .filter(lambda b: b.service_name == 'com.victronenergy.grid') 
          .filter(lambda b: b.object_path == '/Ac/Energy/Forward')      
          .buffer_with_time(15 * 60 * 1000)            

# split the stream in 3 to calculate min/max/avg

avg = buffered.map(average)
min = buffered.map(min)
max = buffered.map(max)

# combine again and push to vrm in one go

zip(avg,min,max).subscribe(send_to_vrm)

It has also the potential to simplify DBusMonitor enormously.

[jhofstee]

@izak can't this be closed by now?

[izak]

I made aiovelib for this. I opted for dbus-next rather than dbussy (it looks way more mature and better documented).

Performance-wise, I cannot say it is significantly better. No in-depth testing was done, I only checked that it isn't worse.

Since we now have dbus-next in Venus, there is really no obstacle against writing python asyncio code anymore and we can close this.