Merge pull request #223 from t3r/shjq

New plugin **jsonread**

a generic json plugin based on 'jq'

jsonread/README.md

@@ -0,0 +1,221 @@
+# The jsonread plugin
+
+This is a generic JSON to smarthome plugin. Fetch any JSON encoded
+data via http(s) or from a file, extract the interesting data and feed
+the values to smarthome items.
+
+It uses the lightweight and flexible JSON processor jq [https://stedolan.github.io/jq/]
+
+# Dependencies
+
+This plugin requires
+- requests
+- requests-file
+- pyjq
+
+# Configuration
+
+## Enable the plugin in etc/plugin.yaml
+
+You can add as many instances as you have JSON sources to process. Make sure to add a
+unique instance name to each instance if you configure more than on instance. That
+instance name will serve as a key to the item configuration further down.
+
+### Plugin specific attributes
+#### url
+
+The address of the json data. Currently http://, https:// and file:// schemes are supported.
+Note: using absolute file paths for the file:// schema yields in tripple forward slashes. This
+might look funny but it is not an error.
+
+#### cycle
+
+The repetitive polling interval in seconds. Defaults to 30 seconds.
+
+### Example
+
+The following examples uses openweathermap and it's sample API Key. Don't use this key for anything else
+but testing.  Examples were taken from
+[https://openweathermap.org/current].
+
+### http client
+
+    jsonread:
+      class_name: JSONREAD
+      class_path: plugins.jsonread
+      url: https://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b6907d289e10d714a6e88b30761fae22
+      cycle: 30
+
+### file client
+
+    jsonread:
+      class_name: JSONREAD
+      class_path: plugins.jsonread
+      url: file:///path/to/data.json
+      cycle: 30
+
+### multi instance
+    jsonreadlon:
+      class_name: JSONREAD
+      class_path: plugins.jsonread
+      url: https://samples.openweathermap.org/data/2.5/weather?q=London,uk&appid=b6907d289e10d714a6e88b30761fae22
+      instance: london
+
+    jsonreadcair:
+      class_name: JSONREAD
+      class_path: plugins.jsonread
+      url: https://samples.openweathermap.org/data/2.5/weather?id=2172797&appid=b6907d289e10d714a6e88b30761fae22
+      instance: cairns
+
+
+## item configuration in items/myitem.yaml
+This is the fun part. This is a sample item. Notice the special attribute jsonread_filter.
+
+### Single instance
+just use the jsonread_filter attribute
+
+    temperature:
+      type: num
+      jsonread_filter: .main.temp
+
+### Multi instance
+Use the jsonread_filter@instance attribute syntax
+
+    temperature:
+      london:
+        type: num
+        jsonread_filter@london: .main.temp
+      cairns:
+        type: num
+        jsonread_filter@cairns: .main.temp
+
+The value for the jsonread_filter attribute is a jq filter, passed directly to jq itself
+Use any kind of jq filter that suites your needs. Make sure your filter returns a single value.
+Jq filters can be tricky to develop for complex json structures. Getting them straight might be easier
+outside of smarthome by using the commandline version of jq and curl like this
+
+    curl https://json.server.org/data.json | jq '.object'
+
+Look at [https://stedolan.github.io/jq/tutorial/] to get startet with jq filters.
+
+# Example
+## Fetch data from a SolarWatt Energy Manager and MyReserve
+Note: this example uses the mutli-instance feature as it fetches data from two
+sources.
+
+### Sample input data
+#### BMSData.shtml
+This file gets generated by SolarWatt's receiveBLE.py process on their raspi. The interesting
+part looks like this:
+
+    {
+      "FData": {
+        "IPV": 5.17,
+        "VBat": 170.1,
+        "VPV": 418.5,
+        "PGrid": 18,
+        "IBat": -9.91
+      },
+      "SData": {
+        "ACS": {
+          "U_L2": 239,
+          "f": 49.98
+        },
+        "SoC": 10
+      }
+    }
+
+#### energy-manager json data
+The energy manager produces a fairly huge JSON dataset, more than 4500 lines on my system.
+This is an example, stripped down to what we use in this document
+
+    {
+      "result": {
+        "items": [
+          {
+            "guid": "urn:your-inverter-guid",
+            "tagValues": {
+              "PowerACOut": {
+                "value": 2419,
+                "tagName": "PowerACOut"
+              }
+            }
+          }
+        ]
+      }
+    }
+
+### etc/plugin.yaml
+
+    swem:
+      class_name: JSONREAD
+      class_path: plugins.jsonread
+      # Replace with the ip address or hostname of your energy-manager
+      url: http://192.168.x.y/rest/kiwigrid/wizard/devices
+      instance: swem
+      cycle: 30
+
+    myreserve:
+      class_name: JSONREAD
+      class_path: plugins.jsonread
+      # Replace with the path to your BMSData-file
+      url: file:///tmp/BMSData.shtml
+      instance: myreserve
+      cycle: 10
+
+
+### items/myenergy.yaml
+
+Get the batteries voltage, charge current and charge power.
+
+    battery:
+      u:
+        # note: we are fetching from the 'myreserve' plugin instance
+        # easy: get attribute VBat of the FData object
+        type: num
+        jsonread_filter@myreserve: .FData.VBat
+      i:
+        # easy: get attribute IBat of the FData object
+        type: num
+        jsonread_filter@myreserve: .FData.IBat
+      power:
+        # doing simple math is also straightforward
+        type: num
+        jsonread_filter@myreserve: (.FData.VBat * .FData.IBat * -1)
+
+Get the current inverter AC out power
+
+    inverter:
+      type: num
+      # note: we are fetching from the 'swem' plugin instance
+      # from the items array in the result object, fetch the object where the guid matches our given guid
+      # from the resulting object, walk down the tree and fetch the "value" attribute
+      jsonread_filter@swem: (.result.items[] | select(.guid == "urn:your-inverter-guid").tagValues.PowerACOut.value)
+      # all standard attributes work as expected
+      visu_acl: r
+      sqlite: yes
+
+Compute the grid power, use yaml's block style feature (https://yaml-multiline.info/) to wrap long lines
+
+    grid:
+      type: num
+      jsonread_filter@swem: >
+        (.result.items[] |
+          select(.deviceModel[].deviceClass == "com.kiwigrid.devices.solarwatt.MyReservePowermeter").tagValues.PowerOut.value) -
+        (.result.items[] |
+          select(.deviceModel[].deviceClass == "com.kiwigrid.devices.solarwatt.MyReservePowermeter").tagValues.PowerIn.value)
+
+# Disclaimer and License
+This document and the jsonread plugin for smarthome.py is free software:
+you can redistribute it and/or modify it under the terms of the
+GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+smjq is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with jsonread. If not, see <http://www.gnu.org/licenses/>.

jsonread/__init__.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+#
+#########################################################################
+#  Copyright 2019 Torsten Dreyer torsten (at) t3r (dot) de
+#  Version 1.0.0
+#########################################################################
+#
+#  This file is part of SmartHomeNG.
+#
+#  SmartHomeNG is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  SmartHomeNG is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with SmartHomeNG. If not, see <http://www.gnu.org/licenses/>.
+#
+#########################################################################
+
+import logging
+import requests
+from requests_file import FileAdapter
+import pyjq
+from lib.model.smartplugin import SmartPlugin
+
+
+class JSONREAD(SmartPlugin):
+    ALLOW_MULTIINSTANCE = True
+    PLUGIN_VERSION = "1.0.0"
+
+    def __init__(self, sh, *args, **kwargs):
+        """
+        Initializes the plugin
+        @param url: URL of the json data to fetch
+        @param cycle: the polling interval in seconds
+        """
+        self.logger = logging.getLogger(__name__)
+        self._url = self.get_parameter_value('url')
+        self._cycle = self.get_parameter_value('cycle')
+        self._session = requests.Session()
+        self._session.mount('file://', FileAdapter())
+        self._items = {}
+
+    def run(self):
+        self.alive = True
+        self.scheduler_add(__name__, self.poll_device, cycle=self._cycle)
+
+    def stop(self):
+        self.scheduler_remove(__name__ )
+        self.alive = False
+
+    def parse_item(self, item):
+        if self.has_iattr(item.conf, 'jsonread_filter'):
+            self._items[item] = self.get_iattr_value(item.conf, 'jsonread_filter')
+
+    def poll_device(self):
+        try:
+            response = self._session.get(self._url)
+
+        except Exception as ex:
+            self.logger.error("Exception when sending GET request for {}: {}".format(self._url,str(ex)))
+            return
+
+        if response.status_code != 200:
+            self.logger.error("Bad response code from GET '{}': {}".format(self._url, response.status_code))
+            return
+
+        try:
+            json_obj = response.json()
+        except Exception as ex:
+            self.logger.error("Response from '{}' doesn't look like json '{}'".format(self._url, str(response.content)[:30]))
+            return
+
+        for k in self._items.keys():
+            try:
+                jqres = pyjq.first(self._items[k], json_obj)
+
+            except Exception as ex:
+                self.logger.error("jq filter failed: {}'".format(str(ex)))
+                continue
+
+            k(jqres)
+

jsonread/plugin.yaml

@@ -0,0 +1,42 @@
+# Metadata for the Smart-Plugin
+plugin:
+    # Global plugin attributes
+    type: interface               # plugin type (gateway, interface, protocol, system, web)
+    description:
+        de: 'json parser plugin basierend auf jq'
+        en: 'json parser plugin based on jq'
+    maintainer: Torsten Dreyer
+    tester: none (yet)
+    keywords: json
+    documentation: http://smarthomeng.de/user/plugins_doc/config/not-yet.html
+    support: https://knx-user-forum.de/forum/supportforen/smarthome-py/not-yet
+
+    version: 1.0.0                 # Plugin version
+    sh_minversion: 1.4             # minimum shNG version to use this plugin
+#    sh_maxversion:                 # maximum shNG version to use this plugin (leave empty if latest)
+    multi_instance: True          # plugin supports multi instance
+    classname: JSONREAD             # class containing the plugin
+
+parameters:
+    # Definition of parameters to be configured in etc/plugin.yaml
+    url:
+        type: str
+        mandatory: True
+        description:
+            de: 'URL der json Eingabe-Daten (z.B. http://foo.bar/input.json oder file:///pfad/zu/data.json)'
+            en: 'URL of the json input (e.g. http://foo.bar/input.json or file:///path/to/data.json)'
+    cycle:
+        type: int
+        mandatory: False
+        default: 30
+        valid_min: 0
+        description:
+            de: 'Das Abfrage-Intervall für die gegebene URL in Sekunden'
+            en: 'The polling interval for the given url in seconds'
+
+
+item_attributes:
+    # Definition of item attributes defined by this plugin
+    jsonread_filter:
+        type: str
+

jsonread/requirements.txt

@@ -0,0 +1,2 @@
+requests-file
+pyjq