From 606c84bace96d04db325b8c7bbc10a3265810c99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 11 Jan 2022 14:45:10 +0100 Subject: [PATCH 01/46] merge for release 0.9.4 (#621) (#622) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * include LICENSE in source distribution (#365) * arxml: rx/tx typo fix fix for #366 * annotate and fix codestyle dbc (#369) * Always use Signal.is_little_endian as bool #326 (#371) * Add coding, remove #!python where inappropriate (#370) * annotation cleanup ticket #323 (#372) * Update cli doc (inspired by #361) (#373) * fix: xls is using the wrong number for arbitration id (#377) * add outputformat for Scapy (#378) * add output format for scapy * fix special char handling in scapy exporter * add unit test simplify get_fmt * Dump factor, offset, min, max as float. Load as numeric (#380) * Dump factor, offset, min, max as float. Load as numeric * fixed tests * fixed tests * added --jsonNativeTypes as cli option * review remarks * ARXML: merge container PDU and Secured PDU support (#384) * #381 support for containers * #381 create signalgroups for containers * #382 support secured pdus) (#383) * add unittests for #363 and #382 * Iss381 - fix for #381 - wrong default ArbitrationId handling (#386) * fix for #381 wrong default ArbitrationId handling * support for Scapy input file (#387) * add output format for scapy * optional ignoring failures durig character encoding #375 (#388) * fix for #238 (j1939 Frame setters might raise TypeError) (#389) move j1939 (pgn, prio, source) handling direct to arbitration-id class (fixes #238) * Iss385 // Arxml Container support and ARXML refactoring (#390) ARXML updates: * fix for #385 * json fix multiplex reading issue (#136) * ARXML refactoring, DLC for canFD support canmatrix.Frame.fit_dlc now fits to next correct dlc value, DBC export reniced for candb++ (#385) * add test for fit_dlc * arxml refactoring and better type handling #368 * improvement for #242 * DBC: fix #242. J1939 and FD Frames in dbc candb seems to be queasy about the order of the VFrameFormat attribute definition. This Patch seems to generate a dbc, which can contain can_fd AND J1939-frames. * XLSX fixes (#392) XLSX: fixes #240 (signal init value in excel template) XLSX: fixes #367 (2.) * fix for #288 (#393) * switch from optparse to click (#236) (#394) * [WIP] add Basic cli tests (#395) * add tests for global, manipulation and filter swithces of cli.convert * add basic cli test for cli.compare * fix some bugs, which were found by these tests * Annotation cleanup #323 (#396) * Remove cm prefix (#236) (#399) Thanks * add muxing support to scapy output; merge PR #398 also (#400) * [try to] Unify imports (#402) * unify imports (#236) * [WIP] dump/export wireshark lua can subdissector (#404) * add basic wireshark dissector creator creastes lua script for dissecting can messages * add cannelloni dissector for wireshark * add some doc for scapy and wireshark usage * [WIP] cycle_time as generic signal and frame attribute (#405) * implement cycle_time as generic signal and frame attribute / remove Gen[Msg/Sig]cycle from attributes #146 * use effective frame cycle time for export use gcd of singnal-cycletimes for calculating effective cycle time This removes one (of many) dbc specific parts from canmatrix and other formats * Iss407: [dbc] Signal Comments with space bevore semicolon broken import (#409) * fix for issue #407 [dbc] Signal Comments with space bevore semicolon broken import * make initial value to a native attribute of Signal class (#408) * make initial value to a native attribute of Signal class * [dbf] exended ids corrected * optionally allow multiple singals with same name in frame #411 (#412) * Fix setup.py for console_scripts (#417) * [WIP] [dbc] Iss413: fix comment reading with whitespaces in front of ; (#416) * optionally allow multiple singals with same name in frame #411 * fixes #413 (space before semicolon mulit line comment) * fix for iss #414, keeps defines in dbc: this keeps defines from dbc in matrix object, even if native attribute exists. * convert: selective rx/tx ecu extraction issue #421 (#422) * [WIP] [ARXML] can_fd info (potentional fix for #410) (#418) * potentional fix for #410 * [ARXML] can-fd recogintion * add baudrate attributes * [WIP] Auto deploy (#420) * add auto deploy for tags on master * add submodules doc * update doc to make start_bit more clear in documentation #424 * [dbc] fix dbc issues (#431) * fix dbc export for empty matrix * copy_frame handles default values now (fix #430) * fix copy_frame * fix for issue #436 (#439) * [convert] add delete_obsolete_ecus (fix for #434) (#438) * [copy] Iss430 better attribute handling (#437) * handling missing default values for defines * add ecu attribute handling #430 * fix for #441 (#442) * fix for #441 jsonAll export option working again * make old test less loud * Fix some bugs in FIBEX export (#445) * Fix FIBEX convertion, Handle duplicate signal name * Repair more click convert parameters (#451) * Repair command line param names for csv, json, xls(x) * update tests * use last supported colorama for pypy3.4 * Fixing issue where we see dictionary attributes change during iteration (#455) * Arxml limits number formats (#457) * add number decoding in arxml for different basis * fix #454 * fix for #460 * fix #460 * Update cli.rst add arxmlExportVersion documentation (#461) * Check for EOF when parsing enums in a .sym file. Fixes #465 (#468) * Check for EOF when parsing enums in a .sym file. Fixes #465 * Fix test to check for different exception in the python2.7 case * Log the exception as well when importing a format failed (#467) * Process the .sym file "Title" field. fixes #469 And Fix Enum dictionary comma splitting (#470) * Implement processing of the Title property of .sym files. * Fix CanMatrix.attribute to return default as documented * Set the default for the "Title" attribute properly * Reimplement the quote_aware_comma_split function to handle spaces between fields * Change the sym parser to use quote_aware_comma_split to split enum value tables. Fixes #462. Add test cases to show that enums are read in correctly from .sym files * fix for #471 (#473) * Fix J1939 PGN getter/setter to use full 18bits. Fixes #474 (#475) * Fix J1939 PGN getter/setter to use full 18bits. Fixes #474 Remove the criteria that PF must be >= 240 for PS field to be set as this seems to be incorrect. (It may be that for some specific case the PS need to not be used, but it isn't documented here and in general the PGN should be all 18 bits) Modifed Tests: test_canmatrix_get_frame_by_pgn() seemed to be using made up data and didn't accound for the DP and EDP fields in expected PGN test_canid_repr() Modified the expected repr value to include the PS field in the PGN. test_canid_parse_values() Modified expected PGN to include the PS field, Moved the list of test_data into parametrized cases on a new test. New Tests: test_frame_j1939_id_from_components() Tests constructing a full 29-bit CAN ID from the Priority, Source, and PGN test_frame_decode_j1939_id() Tests extracting Priority Source and PGN from an Arbiration ID * Add the global desistantion Address 0xFF to the BAM arbitration IDs * Improve the fix, to more explicitly handle the difference between PDU-Format 1 and PDU-Format 2, (where the PF >= 240 comes in). The PGN setter now will now set the arbitration ID to contain the destination address, but when the PGN getter is used it will strip the destination field off. I think that it is unlikely for the PGN setter to be used with PDU-F1 PGN containing a destination. But in the case that it is I think it makes more sense to not throw the destination address field away. * Add enumberations as a field to the json file to store the value tables (#476) Add a test to check that if a json file is read containing enums and then saved and re-read it get the same enum table * Adding changes to allow ArbitrationId classes to be sorted (#466) * Adding comparators to ArbitrationId * Changing default Arbitration ID extended to False * Review based changes Co-authored-by: Syed Raza * Parse sym types (#479) * Allow parsing of the main .sym signal types : string, raw, char. Add a type_label attribute so that symbol files can be saved back to .sym format without loss of information. Add an is_ascii attribute for symbols Add a check to see if Signal names need Quoting when saving to a .sym file (if they start with a number or contain a space etc.) Add a test which reads in a .sym file containing all signal types and checks that they are all correctly imported and exported. * Allow parsing of the main .sym signal types : string, raw, char. Add a type_label attribute so that symbol files can be saved back to .sym format without loss of information. Add an is_ascii attribute for symbols Add a check to see if Signal names need Quoting when saving to a .sym file (if they start with a number or contain a space etc.) Add a test which reads in a .sym file containing all signal types and checks that they are all correctly imported and exported. * Add fix for "isidentifier" fucntion on python 2.7 * Fix logic to match variable name. (Functionality unchanged) Co-authored-by: Eduard Bröcker * Fix sym.dump to export the value_table into the {ENUMS} field of the (#481) .sym file. Refactor to remove the use of global variables "enum_dict" and "enums" Add a test which checks that enums in the original .sym file and enums which appear as values on a signal are exported to the .sym file. * Accept types which are in the value table as enum sgnals (#482) * Fix additional case in sym enum export (#486) * Accept types which are in the value table as enum sgnals * Fix previous change to also generate enums from the signal values field when signal is in a multiplexed frame. * [DBF] support j1939 read (#501) Co-authored-by: Broecker * possible fix for #490 (#503) * add tests for issue #424 (#500) add tests for Motorola forward MSB and Motorola forward LSB start_bit * Iss484 (#502) * [arxml] add some basic support for flexray and ethernet data * extract most of relevant flexray info (#432) not yet integraded - only extracted by now * prove of concept for flexray dump (#432) * starting rework ARXML * add xlsxwriter to test deps * add pyyaml to test-reqs * Update requirements.test.py3.txt * fix so that tests work again * remove py3.4 add py3.8 * disable py34 test - enable py38 test * disable py34 * remove py3.4 * remove py8 warnings * fix dbc for py8 * fix for #484 buggy ARXMLs with no System-Signals referenced * fix ci * fix ci (#504) * fix ci * Iss492 (#505) * integrate fix from #492 * Arxml read baudrate (#452) * Read Baudrate from arxml * Enable reading hex values with prefix 0x * Adaption to pass test with string speed values * Update to read in hex values define with 0x and integers e.g. 0 * Removal of commented line * Explicit conversion to string for logger output Co-authored-by: ForestRupicolous <> Co-authored-by: Eduard Bröcker * Iss496 (#506) * #496 fix ignore encoding switch * Iss499 (#507) * fixes #499 decode_number now supports floating point value * [WIP] [arxml] add some basic support for flexray and ethernet data (#426) * [arxml] add some basic support for flexray and ethernet data * extract most of relevant flexray info (#432) * starting rework ARXML * fix dbc for py8 * support for container-pdus with "none" type * fix for container id byteorder * fix for secured-ipdus in container * fix for byteorder big endian header_ids of container-i-pdus * once again header_id * Header_id again * fix for missing compu method * make SOME/IP at least work a bixt #283 * interprete OPAQUE as intel * dbc allow '"' in Values * fix container-pdu without header but with offset * arxml: fix for container-ipdus, containing same pdu multiple times * fix enum support for py2.7 * Iss509 (#510) dbf fix for exporting empty matrix * dbf fix for exporting empty matrix * Iss496 (#511) * #496 fix ignore encoding switch * clean encoding error implementation * Master (#514) * remegerge release (#512) * Fix sym mux decode, by always treating mux field as unsigned (#517) * Add a test to show that sym file mux values arent working * Add a fix to always treat MUX values as unsigned signals * Log as warning instead of exception (#519) Logging as exception prints a stack backtrace. Since the modules are optional and listed as extra instead of a required dependency, this should probably be a warning instead of an error/exception. Co-authored-by: Lennart Moltrecht * src/canmatrix.py: fix nested mux with no values (#527) If a complexly multiplexed signal has a sub-multiplexer with no values, a TypeError is thrown. * add is_fd flag for json output (#532) Co-authored-by: Dennis Röck * Iss526 (#536) * fix ci * fix ci * fixes #526 * fix for #526escaped quotes * #526 use raw strings for test * Bump lxml from 4.5.2 to 4.6.2 (#539) Bumps [lxml](https://github.com/lxml/lxml) from 4.5.2 to 4.6.2. - [Release notes](https://github.com/lxml/lxml/releases) - [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt) - [Commits](https://github.com/lxml/lxml/compare/lxml-4.5.2...lxml-4.6.2) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Iss535 (#537) * fix ci * possible fix for issue #535 * Iss522 (#530) * arxml: add reciever information from signals to frames (issue #522) * Fix ci (#542) * fix dbc testcase * remove xlsx read from tests (#541) * basic ldf import * update requirements for ldfparser * first implementation for odx import#544 * start diag frame at byte 1 * odx service muxer * odx zwischenstand * issue #531 Lin Converters add support for LIN-Converters * odx zwischnstand * Revert "odx zwischnstand" This reverts commit f4476de70ab4b51726b240414a23a4e31b498442. * Update xls.py (#549) Thanks! * add direct decode function for python can frames (#548) * better mux export hanling in sym * Correctly the result if compare two frame with different ID (#552) When comparing two dbc, the the frame has different ID it should be marked as "changed" frame. * Overall code improvements (#555) * fix: use six.moves instead of future.moves to fix annoying import of top level modules * fix: missing future import + clean up dependencies * fix: __package__ does not exist in Python 2.7 * fix: string decoding issue in Python 2.7 * update tox.ini * fix: install all extras in tox.init * update travis and appveyor config * remove unused requirements files * update tox.ini * remove unwanted line in test.sh * fix: Python version 3.9 not available in AppVeyor yet * fix: only keep minimum requirements for test * improvement dbc to dbf (#556) temporary hot fix for canconvert dbc to dbf * fix issue #547 (#560) DeprecationWarning: The usage of `cmp` is deprecated and will be removed on or after 2021-06-01. Please use `eq` and `order` instead. @attr.s(cmp=False) Signed-off-by: An Nguyen * fix issue #559 (#561) Signed-off-by: An Nguyen * fix initial value setting (#566) Thanks Co-authored-by: JAZI Nadhmi * Set initial_value to physical value (#567) Currently the initial_value is incorrectly set to the signal raw value. * arxml fix (#564) Co-authored-by: Eduard Bröcker * arxml.py: (#569) optimize find/findall fix ECU transmitter/receivers for clusters rework signals receiers settings Co-authored-by: JAZI Nadhmi * [arxml]: Use ISignal Name for canmatrix.Signal (#570) Add attributes to Signal for other names Co-authored-by: JAZI Nadhmi * ARXML Replace ArTree with more simple cache add option to rename Signal from Signal-Attribute * add/update doc * Arxml refactoring (#571) * ARXML Replace ArTree with more simple cache * add option to rename Signal from Signal-Attribute * add/update doc * rework arxml parsing and use Pdu for PDU contained frames (#576) Add a converter from PDU contained to multiplexed frame Add option to handle it in the CLI Update decoder function Co-authored-by: JAZI Nadhmi * fix convert pdu container (#577) Co-authored-by: JAZI Nadhmi * Remove prints from the code and add the message to the exception (#578) * remove duplicated pdus (#583) Co-authored-by: JAZI Nadhmi * Improve multiplexer JSON output (#585) Co-authored-by: Edward Pierzchalski * Fix typo (#586) * bugfix: to get the signals for a pdu you have to use follow_all_ref instead of follow_ref (#591) Co-authored-by: Markus Konrad * improve logging output to be able to identify problems faster/better (#592) Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker * fix for issue #596 (#597) cope with signals without receiver Co-authored-by: Eduard Bröcker * fix for #403 * arxml init value may not decoded just ignore #550 * fixes #572 * arxml helpers * Added option calcSignalMax & recalcSignalMax to calculate max value during converting dbc * Fixed a wrong keypress mistake * Removed float conversion before checking if the signal max is zero * fix PDU container decoding (#605) Co-authored-by: JAZI Nadhmi * Kankan patch1 (#604) * Added option calcSignalMax & recalcSignalMax to calculate max value during converting dbc * Fixed a wrong keypress mistake * Removed float conversion before checking if the signal max is zero * fix tox and continuous integration (#609) * Fixing Issue#610 :: Custom Signal Attributes are removed due to wrong indent * Kankan patch :: Custom Signal Attributes are removed (#611) * Fix #610 :: Custom Signal Attributes are removed due to wrong indent * #612 Bugfix formats.xls and formats.xlsx missing method parameters (#613) * Update guess_value method in utils.py (#615) add a conversion of 0b and 0x string values into int string representation in the guess_value method initial motivation was to get valid input values for the decimal.Decimal() class since it crashed in the arxml.py file line 1234: 1232 if initvalue is not None and initvalue.text is not None: 1233 initvalue.text = canmatrix.utils.guess_value(initvalue.text) 1234 new_signal.initial_value = float_factory(initvalue.text) * Added deleteFloatingSignals , checkFloatingFrames, checkSignalRange, checkSignalUnit, checkSignalReceiver & checkFloatingSignals * Added J1939 to Extended dbc and vice versa * feat: add support for Python 3.10 (#619) * feat: add support for Python 3.10 * chore: fix appveyor configuration * chore: remove useless commands in appveyor configuration * chore: Python 3.9 and 3.10 only available for Visual Studio 2019 and up in appveyor * arxml: read data related to E2E-protection (#594) * read informations about E2E-Profiles (of I-Signalgroup) into the canmatrix-object * also read informations about E2E-Profiles (of I-Signalgroup) into the canmatrix-object if a Container-PDU is processed Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker Co-authored-by: Daniel Hrisca Co-authored-by: Funth0mas <43621609+Funth0mas@users.noreply.github.com> Co-authored-by: Thomas Fritzsche Co-authored-by: chrisoro <4160557+chrisoro@users.noreply.github.com> Co-authored-by: Bröcker Co-authored-by: Kyle Altendorf Co-authored-by: akaanich-technica <58173657+akaanich-technica@users.noreply.github.com> Co-authored-by: Syed Co-authored-by: Seneda Co-authored-by: Ulf Rüegg Co-authored-by: Broecker Co-authored-by: Martin Korinek Co-authored-by: Leo <4062160+leosh64@users.noreply.github.com> Co-authored-by: Lennart Moltrecht Co-authored-by: Ryan Rowe Co-authored-by: Dennis <39051193+DennisRoeck@users.noreply.github.com> Co-authored-by: Dennis Röck Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cesare12 <591576872@qq.com> Co-authored-by: Moris Yun Co-authored-by: Darkin Co-authored-by: jazi007 <38762095+jazi007@users.noreply.github.com> Co-authored-by: pierreluctg Co-authored-by: L X <59323065+leixao@users.noreply.github.com> Co-authored-by: JAZI Nadhmi Co-authored-by: Edward Pierzchalski Co-authored-by: Edward Pierzchalski Co-authored-by: Ajinkya Pasalkar <43365292+AjinkyaPasalkar@users.noreply.github.com> Co-authored-by: tainnok <16083651+tainnok@users.noreply.github.com> Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker Co-authored-by: Kankan Sarkar Co-authored-by: Arcola-Kankan <89925850+Arcola-Kankan@users.noreply.github.com> Co-authored-by: AlexDLSy <75117170+AlexDLSy@users.noreply.github.com> Co-authored-by: tobiasandorfer <64536839+tobiasandorfer@users.noreply.github.com> Co-authored-by: Daniel Hrisca Co-authored-by: Funth0mas <43621609+Funth0mas@users.noreply.github.com> Co-authored-by: Thomas Fritzsche Co-authored-by: chrisoro <4160557+chrisoro@users.noreply.github.com> Co-authored-by: Bröcker Co-authored-by: Kyle Altendorf Co-authored-by: akaanich-technica <58173657+akaanich-technica@users.noreply.github.com> Co-authored-by: Syed Co-authored-by: Seneda Co-authored-by: Ulf Rüegg Co-authored-by: Broecker Co-authored-by: Martin Korinek Co-authored-by: Leo <4062160+leosh64@users.noreply.github.com> Co-authored-by: Lennart Moltrecht Co-authored-by: Ryan Rowe Co-authored-by: Dennis <39051193+DennisRoeck@users.noreply.github.com> Co-authored-by: Dennis Röck Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cesare12 <591576872@qq.com> Co-authored-by: Moris Yun Co-authored-by: Darkin Co-authored-by: jazi007 <38762095+jazi007@users.noreply.github.com> Co-authored-by: pierreluctg Co-authored-by: L X <59323065+leixao@users.noreply.github.com> Co-authored-by: JAZI Nadhmi Co-authored-by: Edward Pierzchalski Co-authored-by: Edward Pierzchalski Co-authored-by: Ajinkya Pasalkar <43365292+AjinkyaPasalkar@users.noreply.github.com> Co-authored-by: tainnok <16083651+tainnok@users.noreply.github.com> Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker Co-authored-by: Kankan Sarkar Co-authored-by: Arcola-Kankan <89925850+Arcola-Kankan@users.noreply.github.com> Co-authored-by: AlexDLSy <75117170+AlexDLSy@users.noreply.github.com> Co-authored-by: tobiasandorfer <64536839+tobiasandorfer@users.noreply.github.com> From f56652f6c6d7ab9761e7657acec82addea15cbb4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Fri, 11 Feb 2022 14:40:52 +0100 Subject: [PATCH 02/46] [dbc] allow multiple space after : in Frame definition (#624) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Ádám KINYIK --- src/canmatrix/formats/dbc.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/formats/dbc.py b/src/canmatrix/formats/dbc.py index 3845326a..246ff9a6 100644 --- a/src/canmatrix/formats/dbc.py +++ b/src/canmatrix/formats/dbc.py @@ -543,7 +543,7 @@ def add_frame_by_id(new_frame): # type: (canmatrix.Frame) -> None continue decoded = l.decode(dbc_import_encoding).strip() if decoded.startswith("BO_ "): - regexp = re.compile(r"^BO_ ([^\ ]+) ([^\ ]+) *: ([^\ ]+) ([^\ ]+)") + regexp = re.compile(r"^BO_ ([^\ ]+) ([^\ ]+) *: *([^\ ]+) ([^\ ]+)") temp = regexp.match(decoded) # db.frames.addFrame(Frame(temp.group(1), temp.group(2), temp.group(3), temp.group(4))) frame = canmatrix.Frame(temp.group(2), arbitration_id=int(temp.group(1)), From f7dbf41e5f196f98ac85b3ffb3dcb16cbcca2f81 Mon Sep 17 00:00:00 2001 From: wksentini <101653610+wksentini@users.noreply.github.com> Date: Wed, 23 Mar 2022 10:33:00 +0100 Subject: [PATCH 03/46] support cycle (#629) Co-authored-by: Wael --- src/canmatrix/formats/fibex.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/src/canmatrix/formats/fibex.py b/src/canmatrix/formats/fibex.py index aba8ce35..7f9c6437 100644 --- a/src/canmatrix/formats/fibex.py +++ b/src/canmatrix/formats/fibex.py @@ -92,7 +92,7 @@ def dump(db, f, **options): # ELEMENTS # elements = create_sub_element_fx(root, "ELEMENTS") - + # # CLUSTERS # @@ -118,6 +118,23 @@ def dump(db, f, **options): channel = create_sub_element_fx(channels, "CHANNEL") # for each channel create_short_name_desc(channel, "CANCHANNEL01", "Can Channel Description") + + # for pdu triggerings + pdu_triggerings = create_sub_element_fx(channel, "PDU-TRIGGERINGS") + for pdu in db.frames: + pdu_triggering = create_sub_element_fx( + pdu_triggerings, "PDU-TRIGGERING") + pdu_triggering.set("ID", "PDU_" + pdu.name) + pdu_timings = create_sub_element_fx(pdu_triggering, "TIMINGS") + if pdu.cycle_time > 0: + cyclic_timing = create_sub_element_fx(pdu_timings, "CYCLIC-TIMING") + repeating_time_range = create_sub_element_fx(cyclic_timing, "REPEATING-TIME-RANGE") + create_sub_element_fx(repeating_time_range, "VALUE", "PT" + str(pdu.cycle_time/1000.0) + "S") + + pdu_ref = create_sub_element_fx(pdu_triggering, "PDU-REF") + pdu_ref.set("ID-REF", "PDU_" + pdu.name) + + frame_triggerings = create_sub_element_fx(channel, "FRAME-TRIGGERINGS") for frame in db.frames: frame_triggering = create_sub_element_fx( @@ -149,6 +166,12 @@ def dump(db, f, **options): input_port = create_sub_element_fx(inputs, "INPUT-PORT") frame_triggering_ref = create_sub_element_fx(input_port, "FRAME-TRIGGERING-REF") frame_triggering_ref.set("ID-REF", "FT_" + frame.name) + # Reference to PDUs + included_pdus = create_sub_element_fx(input_port, "INCLUDED-PDUS") + included_pdu = create_sub_element_fx(included_pdus, "INCLUDED-PDU") + pdu_triggering_ref = create_sub_element_fx(included_pdu, "PDU-TRIGGERING-REF") + pdu_triggering_ref.set("ID-REF", "PDU_" + frame.name) + outputs = create_sub_element_fx(connector, "OUTPUTS") for frame in db.frames: @@ -156,6 +179,11 @@ def dump(db, f, **options): input_port = create_sub_element_fx(outputs, "OUTPUT-PORT") frame_triggering_ref = create_sub_element_fx(input_port, "FRAME-TRIGGERING-REF") frame_triggering_ref.set("ID-REF", "FT_" + frame.name) + # Reference to PDUs + included_pdus = create_sub_element_fx(input_port, "INCLUDED-PDUS") + included_pdu = create_sub_element_fx(included_pdus, "INCLUDED-PDU") + pdu_triggering_ref = create_sub_element_fx(included_pdu, "PDU-TRIGGERING-REF") + pdu_triggering_ref.set("ID-REF", "PDU_" + frame.name) # ignore CONTROLERS/CONTROLER From 57897b00dec6a1e29a2e03f94baebd825476f026 Mon Sep 17 00:00:00 2001 From: Nadhmi JAZI <38762095+jazi007@users.noreply.github.com> Date: Wed, 23 Mar 2022 12:34:37 +0100 Subject: [PATCH 04/46] [LDF]: use parse_ldf and make signals unsigned (#623) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Nadhmi JAZI Co-authored-by: Eduard Bröcker --- src/canmatrix/formats/ldf.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/canmatrix/formats/ldf.py b/src/canmatrix/formats/ldf.py index e3932196..c239724e 100644 --- a/src/canmatrix/formats/ldf.py +++ b/src/canmatrix/formats/ldf.py @@ -6,9 +6,10 @@ def load(f, **options): # type: (typing.IO, **typing.Any) -> canmatrix.CanMatrix - ldf = ldfparser.parseLDF(path=f.name) # using f.name is not nice, but works + ldf = ldfparser.parse_ldf(path=f.name) # using f.name is not nice, but works db = canmatrix.CanMatrix() + db.baudrate = ldf.get_baudrate() for lin_frame in ldf.frames: cm_frame = canmatrix.Frame() @@ -20,6 +21,7 @@ def load(f, **options): # type: (typing.IO, **typing.Any) -> canmatrix.CanMatri for mapping in lin_frame.signal_map: lin_signal = mapping[1] cm_signal = canmatrix.Signal() + cm_signal.is_signed = False if lin_signal.name in ldf.converters: for converter in ldf.converters[lin_signal.name]._converters: if isinstance(converter, ldfparser.encoding.LogicalValue): From 116f7e78d6fbba07d7f4e8818d72b20f9965e06b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Wed, 23 Mar 2022 14:03:18 +0100 Subject: [PATCH 05/46] Iss 625 (#631) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * merge for release 0.9.4 (#621) * include LICENSE in source distribution (#365) * arxml: rx/tx typo fix fix for #366 * annotate and fix codestyle dbc (#369) * Always use Signal.is_little_endian as bool #326 (#371) * Add coding, remove #!python where inappropriate (#370) * annotation cleanup ticket #323 (#372) * Update cli doc (inspired by #361) (#373) * fix: xls is using the wrong number for arbitration id (#377) * add outputformat for Scapy (#378) * add output format for scapy * fix special char handling in scapy exporter * add unit test simplify get_fmt * Dump factor, offset, min, max as float. Load as numeric (#380) * Dump factor, offset, min, max as float. Load as numeric * fixed tests * fixed tests * added --jsonNativeTypes as cli option * review remarks * ARXML: merge container PDU and Secured PDU support (#384) * #381 support for containers * #381 create signalgroups for containers * #382 support secured pdus) (#383) * add unittests for #363 and #382 * Iss381 - fix for #381 - wrong default ArbitrationId handling (#386) * fix for #381 wrong default ArbitrationId handling * support for Scapy input file (#387) * add output format for scapy * optional ignoring failures durig character encoding #375 (#388) * fix for #238 (j1939 Frame setters might raise TypeError) (#389) move j1939 (pgn, prio, source) handling direct to arbitration-id class (fixes #238) * Iss385 // Arxml Container support and ARXML refactoring (#390) ARXML updates: * fix for #385 * json fix multiplex reading issue (#136) * ARXML refactoring, DLC for canFD support canmatrix.Frame.fit_dlc now fits to next correct dlc value, DBC export reniced for candb++ (#385) * add test for fit_dlc * arxml refactoring and better type handling #368 * improvement for #242 * DBC: fix #242. J1939 and FD Frames in dbc candb seems to be queasy about the order of the VFrameFormat attribute definition. This Patch seems to generate a dbc, which can contain can_fd AND J1939-frames. * XLSX fixes (#392) XLSX: fixes #240 (signal init value in excel template) XLSX: fixes #367 (2.) * fix for #288 (#393) * switch from optparse to click (#236) (#394) * [WIP] add Basic cli tests (#395) * add tests for global, manipulation and filter swithces of cli.convert * add basic cli test for cli.compare * fix some bugs, which were found by these tests * Annotation cleanup #323 (#396) * Remove cm prefix (#236) (#399) Thanks * add muxing support to scapy output; merge PR #398 also (#400) * [try to] Unify imports (#402) * unify imports (#236) * [WIP] dump/export wireshark lua can subdissector (#404) * add basic wireshark dissector creator creastes lua script for dissecting can messages * add cannelloni dissector for wireshark * add some doc for scapy and wireshark usage * [WIP] cycle_time as generic signal and frame attribute (#405) * implement cycle_time as generic signal and frame attribute / remove Gen[Msg/Sig]cycle from attributes #146 * use effective frame cycle time for export use gcd of singnal-cycletimes for calculating effective cycle time This removes one (of many) dbc specific parts from canmatrix and other formats * Iss407: [dbc] Signal Comments with space bevore semicolon broken import (#409) * fix for issue #407 [dbc] Signal Comments with space bevore semicolon broken import * make initial value to a native attribute of Signal class (#408) * make initial value to a native attribute of Signal class * [dbf] exended ids corrected * optionally allow multiple singals with same name in frame #411 (#412) * Fix setup.py for console_scripts (#417) * [WIP] [dbc] Iss413: fix comment reading with whitespaces in front of ; (#416) * optionally allow multiple singals with same name in frame #411 * fixes #413 (space before semicolon mulit line comment) * fix for iss #414, keeps defines in dbc: this keeps defines from dbc in matrix object, even if native attribute exists. * convert: selective rx/tx ecu extraction issue #421 (#422) * [WIP] [ARXML] can_fd info (potentional fix for #410) (#418) * potentional fix for #410 * [ARXML] can-fd recogintion * add baudrate attributes * [WIP] Auto deploy (#420) * add auto deploy for tags on master * add submodules doc * update doc to make start_bit more clear in documentation #424 * [dbc] fix dbc issues (#431) * fix dbc export for empty matrix * copy_frame handles default values now (fix #430) * fix copy_frame * fix for issue #436 (#439) * [convert] add delete_obsolete_ecus (fix for #434) (#438) * [copy] Iss430 better attribute handling (#437) * handling missing default values for defines * add ecu attribute handling #430 * fix for #441 (#442) * fix for #441 jsonAll export option working again * make old test less loud * Fix some bugs in FIBEX export (#445) * Fix FIBEX convertion, Handle duplicate signal name * Repair more click convert parameters (#451) * Repair command line param names for csv, json, xls(x) * update tests * use last supported colorama for pypy3.4 * Fixing issue where we see dictionary attributes change during iteration (#455) * Arxml limits number formats (#457) * add number decoding in arxml for different basis * fix #454 * fix for #460 * fix #460 * Update cli.rst add arxmlExportVersion documentation (#461) * Check for EOF when parsing enums in a .sym file. Fixes #465 (#468) * Check for EOF when parsing enums in a .sym file. Fixes #465 * Fix test to check for different exception in the python2.7 case * Log the exception as well when importing a format failed (#467) * Process the .sym file "Title" field. fixes #469 And Fix Enum dictionary comma splitting (#470) * Implement processing of the Title property of .sym files. * Fix CanMatrix.attribute to return default as documented * Set the default for the "Title" attribute properly * Reimplement the quote_aware_comma_split function to handle spaces between fields * Change the sym parser to use quote_aware_comma_split to split enum value tables. Fixes #462. Add test cases to show that enums are read in correctly from .sym files * fix for #471 (#473) * Fix J1939 PGN getter/setter to use full 18bits. Fixes #474 (#475) * Fix J1939 PGN getter/setter to use full 18bits. Fixes #474 Remove the criteria that PF must be >= 240 for PS field to be set as this seems to be incorrect. (It may be that for some specific case the PS need to not be used, but it isn't documented here and in general the PGN should be all 18 bits) Modifed Tests: test_canmatrix_get_frame_by_pgn() seemed to be using made up data and didn't accound for the DP and EDP fields in expected PGN test_canid_repr() Modified the expected repr value to include the PS field in the PGN. test_canid_parse_values() Modified expected PGN to include the PS field, Moved the list of test_data into parametrized cases on a new test. New Tests: test_frame_j1939_id_from_components() Tests constructing a full 29-bit CAN ID from the Priority, Source, and PGN test_frame_decode_j1939_id() Tests extracting Priority Source and PGN from an Arbiration ID * Add the global desistantion Address 0xFF to the BAM arbitration IDs * Improve the fix, to more explicitly handle the difference between PDU-Format 1 and PDU-Format 2, (where the PF >= 240 comes in). The PGN setter now will now set the arbitration ID to contain the destination address, but when the PGN getter is used it will strip the destination field off. I think that it is unlikely for the PGN setter to be used with PDU-F1 PGN containing a destination. But in the case that it is I think it makes more sense to not throw the destination address field away. * Add enumberations as a field to the json file to store the value tables (#476) Add a test to check that if a json file is read containing enums and then saved and re-read it get the same enum table * Adding changes to allow ArbitrationId classes to be sorted (#466) * Adding comparators to ArbitrationId * Changing default Arbitration ID extended to False * Review based changes Co-authored-by: Syed Raza * Parse sym types (#479) * Allow parsing of the main .sym signal types : string, raw, char. Add a type_label attribute so that symbol files can be saved back to .sym format without loss of information. Add an is_ascii attribute for symbols Add a check to see if Signal names need Quoting when saving to a .sym file (if they start with a number or contain a space etc.) Add a test which reads in a .sym file containing all signal types and checks that they are all correctly imported and exported. * Allow parsing of the main .sym signal types : string, raw, char. Add a type_label attribute so that symbol files can be saved back to .sym format without loss of information. Add an is_ascii attribute for symbols Add a check to see if Signal names need Quoting when saving to a .sym file (if they start with a number or contain a space etc.) Add a test which reads in a .sym file containing all signal types and checks that they are all correctly imported and exported. * Add fix for "isidentifier" fucntion on python 2.7 * Fix logic to match variable name. (Functionality unchanged) Co-authored-by: Eduard Bröcker * Fix sym.dump to export the value_table into the {ENUMS} field of the (#481) .sym file. Refactor to remove the use of global variables "enum_dict" and "enums" Add a test which checks that enums in the original .sym file and enums which appear as values on a signal are exported to the .sym file. * Accept types which are in the value table as enum sgnals (#482) * Fix additional case in sym enum export (#486) * Accept types which are in the value table as enum sgnals * Fix previous change to also generate enums from the signal values field when signal is in a multiplexed frame. * [DBF] support j1939 read (#501) Co-authored-by: Broecker * possible fix for #490 (#503) * add tests for issue #424 (#500) add tests for Motorola forward MSB and Motorola forward LSB start_bit * Iss484 (#502) * [arxml] add some basic support for flexray and ethernet data * extract most of relevant flexray info (#432) not yet integraded - only extracted by now * prove of concept for flexray dump (#432) * starting rework ARXML * add xlsxwriter to test deps * add pyyaml to test-reqs * Update requirements.test.py3.txt * fix so that tests work again * remove py3.4 add py3.8 * disable py34 test - enable py38 test * disable py34 * remove py3.4 * remove py8 warnings * fix dbc for py8 * fix for #484 buggy ARXMLs with no System-Signals referenced * fix ci * fix ci (#504) * fix ci * Iss492 (#505) * integrate fix from #492 * Arxml read baudrate (#452) * Read Baudrate from arxml * Enable reading hex values with prefix 0x * Adaption to pass test with string speed values * Update to read in hex values define with 0x and integers e.g. 0 * Removal of commented line * Explicit conversion to string for logger output Co-authored-by: ForestRupicolous <> Co-authored-by: Eduard Bröcker * Iss496 (#506) * #496 fix ignore encoding switch * Iss499 (#507) * fixes #499 decode_number now supports floating point value * [WIP] [arxml] add some basic support for flexray and ethernet data (#426) * [arxml] add some basic support for flexray and ethernet data * extract most of relevant flexray info (#432) * starting rework ARXML * fix dbc for py8 * support for container-pdus with "none" type * fix for container id byteorder * fix for secured-ipdus in container * fix for byteorder big endian header_ids of container-i-pdus * once again header_id * Header_id again * fix for missing compu method * make SOME/IP at least work a bixt #283 * interprete OPAQUE as intel * dbc allow '"' in Values * fix container-pdu without header but with offset * arxml: fix for container-ipdus, containing same pdu multiple times * fix enum support for py2.7 * Iss509 (#510) dbf fix for exporting empty matrix * dbf fix for exporting empty matrix * Iss496 (#511) * #496 fix ignore encoding switch * clean encoding error implementation * Master (#514) * remegerge release (#512) * Fix sym mux decode, by always treating mux field as unsigned (#517) * Add a test to show that sym file mux values arent working * Add a fix to always treat MUX values as unsigned signals * Log as warning instead of exception (#519) Logging as exception prints a stack backtrace. Since the modules are optional and listed as extra instead of a required dependency, this should probably be a warning instead of an error/exception. Co-authored-by: Lennart Moltrecht * src/canmatrix.py: fix nested mux with no values (#527) If a complexly multiplexed signal has a sub-multiplexer with no values, a TypeError is thrown. * add is_fd flag for json output (#532) Co-authored-by: Dennis Röck * Iss526 (#536) * fix ci * fix ci * fixes #526 * fix for #526escaped quotes * #526 use raw strings for test * Bump lxml from 4.5.2 to 4.6.2 (#539) Bumps [lxml](https://github.com/lxml/lxml) from 4.5.2 to 4.6.2. - [Release notes](https://github.com/lxml/lxml/releases) - [Changelog](https://github.com/lxml/lxml/blob/master/CHANGES.txt) - [Commits](https://github.com/lxml/lxml/compare/lxml-4.5.2...lxml-4.6.2) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Iss535 (#537) * fix ci * possible fix for issue #535 * Iss522 (#530) * arxml: add reciever information from signals to frames (issue #522) * Fix ci (#542) * fix dbc testcase * remove xlsx read from tests (#541) * basic ldf import * update requirements for ldfparser * first implementation for odx import#544 * start diag frame at byte 1 * odx service muxer * odx zwischenstand * issue #531 Lin Converters add support for LIN-Converters * odx zwischnstand * Revert "odx zwischnstand" This reverts commit f4476de70ab4b51726b240414a23a4e31b498442. * Update xls.py (#549) Thanks! * add direct decode function for python can frames (#548) * better mux export hanling in sym * Correctly the result if compare two frame with different ID (#552) When comparing two dbc, the the frame has different ID it should be marked as "changed" frame. * Overall code improvements (#555) * fix: use six.moves instead of future.moves to fix annoying import of top level modules * fix: missing future import + clean up dependencies * fix: __package__ does not exist in Python 2.7 * fix: string decoding issue in Python 2.7 * update tox.ini * fix: install all extras in tox.init * update travis and appveyor config * remove unused requirements files * update tox.ini * remove unwanted line in test.sh * fix: Python version 3.9 not available in AppVeyor yet * fix: only keep minimum requirements for test * improvement dbc to dbf (#556) temporary hot fix for canconvert dbc to dbf * fix issue #547 (#560) DeprecationWarning: The usage of `cmp` is deprecated and will be removed on or after 2021-06-01. Please use `eq` and `order` instead. @attr.s(cmp=False) Signed-off-by: An Nguyen * fix issue #559 (#561) Signed-off-by: An Nguyen * fix initial value setting (#566) Thanks Co-authored-by: JAZI Nadhmi * Set initial_value to physical value (#567) Currently the initial_value is incorrectly set to the signal raw value. * arxml fix (#564) Co-authored-by: Eduard Bröcker * arxml.py: (#569) optimize find/findall fix ECU transmitter/receivers for clusters rework signals receiers settings Co-authored-by: JAZI Nadhmi * [arxml]: Use ISignal Name for canmatrix.Signal (#570) Add attributes to Signal for other names Co-authored-by: JAZI Nadhmi * ARXML Replace ArTree with more simple cache add option to rename Signal from Signal-Attribute * add/update doc * Arxml refactoring (#571) * ARXML Replace ArTree with more simple cache * add option to rename Signal from Signal-Attribute * add/update doc * rework arxml parsing and use Pdu for PDU contained frames (#576) Add a converter from PDU contained to multiplexed frame Add option to handle it in the CLI Update decoder function Co-authored-by: JAZI Nadhmi * fix convert pdu container (#577) Co-authored-by: JAZI Nadhmi * Remove prints from the code and add the message to the exception (#578) * remove duplicated pdus (#583) Co-authored-by: JAZI Nadhmi * Improve multiplexer JSON output (#585) Co-authored-by: Edward Pierzchalski * Fix typo (#586) * bugfix: to get the signals for a pdu you have to use follow_all_ref instead of follow_ref (#591) Co-authored-by: Markus Konrad * improve logging output to be able to identify problems faster/better (#592) Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker * fix for issue #596 (#597) cope with signals without receiver Co-authored-by: Eduard Bröcker * fix for #403 * arxml init value may not decoded just ignore #550 * fixes #572 * arxml helpers * Added option calcSignalMax & recalcSignalMax to calculate max value during converting dbc * Fixed a wrong keypress mistake * Removed float conversion before checking if the signal max is zero * fix PDU container decoding (#605) Co-authored-by: JAZI Nadhmi * Kankan patch1 (#604) * Added option calcSignalMax & recalcSignalMax to calculate max value during converting dbc * Fixed a wrong keypress mistake * Removed float conversion before checking if the signal max is zero * fix tox and continuous integration (#609) * Fixing Issue#610 :: Custom Signal Attributes are removed due to wrong indent * Kankan patch :: Custom Signal Attributes are removed (#611) * Fix #610 :: Custom Signal Attributes are removed due to wrong indent * #612 Bugfix formats.xls and formats.xlsx missing method parameters (#613) * Update guess_value method in utils.py (#615) add a conversion of 0b and 0x string values into int string representation in the guess_value method initial motivation was to get valid input values for the decimal.Decimal() class since it crashed in the arxml.py file line 1234: 1232 if initvalue is not None and initvalue.text is not None: 1233 initvalue.text = canmatrix.utils.guess_value(initvalue.text) 1234 new_signal.initial_value = float_factory(initvalue.text) * Added deleteFloatingSignals , checkFloatingFrames, checkSignalRange, checkSignalUnit, checkSignalReceiver & checkFloatingSignals * Added J1939 to Extended dbc and vice versa * feat: add support for Python 3.10 (#619) * feat: add support for Python 3.10 * chore: fix appveyor configuration * chore: remove useless commands in appveyor configuration * chore: Python 3.9 and 3.10 only available for Visual Studio 2019 and up in appveyor * arxml: read data related to E2E-protection (#594) * read informations about E2E-Profiles (of I-Signalgroup) into the canmatrix-object * also read informations about E2E-Profiles (of I-Signalgroup) into the canmatrix-object if a Container-PDU is processed Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker Co-authored-by: Daniel Hrisca Co-authored-by: Funth0mas <43621609+Funth0mas@users.noreply.github.com> Co-authored-by: Thomas Fritzsche Co-authored-by: chrisoro <4160557+chrisoro@users.noreply.github.com> Co-authored-by: Bröcker Co-authored-by: Kyle Altendorf Co-authored-by: akaanich-technica <58173657+akaanich-technica@users.noreply.github.com> Co-authored-by: Syed Co-authored-by: Seneda Co-authored-by: Ulf Rüegg Co-authored-by: Broecker Co-authored-by: Martin Korinek Co-authored-by: Leo <4062160+leosh64@users.noreply.github.com> Co-authored-by: Lennart Moltrecht Co-authored-by: Ryan Rowe Co-authored-by: Dennis <39051193+DennisRoeck@users.noreply.github.com> Co-authored-by: Dennis Röck Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cesare12 <591576872@qq.com> Co-authored-by: Moris Yun Co-authored-by: Darkin Co-authored-by: jazi007 <38762095+jazi007@users.noreply.github.com> Co-authored-by: pierreluctg Co-authored-by: L X <59323065+leixao@users.noreply.github.com> Co-authored-by: JAZI Nadhmi Co-authored-by: Edward Pierzchalski Co-authored-by: Edward Pierzchalski Co-authored-by: Ajinkya Pasalkar <43365292+AjinkyaPasalkar@users.noreply.github.com> Co-authored-by: tainnok <16083651+tainnok@users.noreply.github.com> Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker Co-authored-by: Kankan Sarkar Co-authored-by: Arcola-Kankan <89925850+Arcola-Kankan@users.noreply.github.com> Co-authored-by: AlexDLSy <75117170+AlexDLSy@users.noreply.github.com> Co-authored-by: tobiasandorfer <64536839+tobiasandorfer@users.noreply.github.com> * add testcase for Issue #625 * bugfix for #625 * fix is vs. == * attributes are more often strings now Co-authored-by: Daniel Hrisca Co-authored-by: Funth0mas <43621609+Funth0mas@users.noreply.github.com> Co-authored-by: Thomas Fritzsche Co-authored-by: chrisoro <4160557+chrisoro@users.noreply.github.com> Co-authored-by: Bröcker Co-authored-by: Kyle Altendorf Co-authored-by: akaanich-technica <58173657+akaanich-technica@users.noreply.github.com> Co-authored-by: Syed Co-authored-by: Seneda Co-authored-by: Ulf Rüegg Co-authored-by: Broecker Co-authored-by: Martin Korinek Co-authored-by: Leo <4062160+leosh64@users.noreply.github.com> Co-authored-by: Lennart Moltrecht Co-authored-by: Ryan Rowe Co-authored-by: Dennis <39051193+DennisRoeck@users.noreply.github.com> Co-authored-by: Dennis Röck Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Cesare12 <591576872@qq.com> Co-authored-by: Moris Yun Co-authored-by: Darkin Co-authored-by: jazi007 <38762095+jazi007@users.noreply.github.com> Co-authored-by: pierreluctg Co-authored-by: L X <59323065+leixao@users.noreply.github.com> Co-authored-by: JAZI Nadhmi Co-authored-by: Edward Pierzchalski Co-authored-by: Edward Pierzchalski Co-authored-by: Ajinkya Pasalkar <43365292+AjinkyaPasalkar@users.noreply.github.com> Co-authored-by: tainnok <16083651+tainnok@users.noreply.github.com> Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker Co-authored-by: Kankan Sarkar Co-authored-by: Arcola-Kankan <89925850+Arcola-Kankan@users.noreply.github.com> Co-authored-by: AlexDLSy <75117170+AlexDLSy@users.noreply.github.com> Co-authored-by: tobiasandorfer <64536839+tobiasandorfer@users.noreply.github.com> --- src/canmatrix/canmatrix.py | 21 +++++++++++++++++--- src/canmatrix/convert.py | 6 +++--- src/canmatrix/formats/dbc.py | 2 +- src/canmatrix/tests/test_canmatrix.py | 6 +++--- src/canmatrix/tests/test_copy.py | 2 +- src/canmatrix/tests/test_dbc.py | 28 +++++++++++++++++++++++++++ 6 files changed, 54 insertions(+), 11 deletions(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 138dab07..e218dd86 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -106,7 +106,12 @@ def add_attribute(self, attribute, value): # type (attribute: str, value: typin :param str attribute: Attribute name :param any value: Attribute value """ - self.attributes[attribute] = value + try: + self.attributes[attribute] = str(value) + except UnicodeDecodeError: + self.attributes[attribute] = value + if type(self.attributes[attribute]) == str: + self.attributes[attribute] = self.attributes[attribute].strip() def del_attribute(self, attribute): if attribute in self.attributes: @@ -282,7 +287,12 @@ def add_attribute(self, attribute, value): :param str attribute: attribute name :param value: attribute value """ - self.attributes[attribute] = value + try: + self.attributes[attribute] = str(value) + except UnicodeDecodeError: + self.attributes[attribute] = value + if type(self.attributes[attribute]) == str: + self.attributes[attribute] = self.attributes[attribute].strip() def del_attribute(self, attribute): """ @@ -1715,7 +1725,12 @@ def add_attribute(self, attribute, value): # type: (str, typing.Any) -> None :param str attribute: attribute name :param value: attribute value """ - self.attributes[attribute] = value + try: + self.attributes[attribute] = str(value) + except UnicodeDecodeError: + self.attributes[attribute] = value + if type(self.attributes[attribute]) == str: + self.attributes[attribute] = self.attributes[attribute].strip() def add_signal_defines(self, type, definition): """ diff --git a/src/canmatrix/convert.py b/src/canmatrix/convert.py index b7ae9015..d6429cb8 100644 --- a/src/canmatrix/convert.py +++ b/src/canmatrix/convert.py @@ -297,14 +297,14 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non if options.get('checkSignalRange') is not None and options['checkSignalRange']: for frame in db.frames: for signal in frame.signals: - if (signal.phys2raw(signal.max) - signal.phys2raw(signal.min)) is 0: + if (signal.phys2raw(signal.max) - signal.phys2raw(signal.min)) == 0: logger.warning("Invalid Min , Max value of %s", (frame.name+"::"+signal.name)) # Check for Signals without unit and Value table , the idea is to improve signal readability if options.get('checkSignalUnit') is not None and options['checkSignalUnit']: for frame in db.frames: for signal in frame: - if signal.unit is "" and len(signal.values) == 0: + if signal.unit == "" and len(signal.values) == 0: logger.warning("Please add value table for the signal %s or add appropriate Unit", (frame.name+"::"+signal.name)) # Convert dbc from J1939 to Extended format @@ -317,7 +317,7 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non if options.get('convertToJ1939') is not None and options['convertToJ1939']: for frame in db.frames: frame.is_j1939=True - db.add_attribute("ProtocolType","J1939") + db.add_attribute("ProtocolType", "J1939") logger.info(name) logger.info("%d Frames found" % (db.frames.__len__())) diff --git a/src/canmatrix/formats/dbc.py b/src/canmatrix/formats/dbc.py index 246ff9a6..4ecfa7a6 100644 --- a/src/canmatrix/formats/dbc.py +++ b/src/canmatrix/formats/dbc.py @@ -91,7 +91,7 @@ def create_attribute_string(attribute, attribute_class, name, value, is_string): # type: (str, str, str, typing.Any, bool) -> str if is_string: value = '"' + value + '"' - elif not value: + elif value is None: value = '""' attribute_string = 'BA_ "' + attribute + '" ' + attribute_class + ' ' + name + ' ' + str(value) + ';\n' diff --git a/src/canmatrix/tests/test_canmatrix.py b/src/canmatrix/tests/test_canmatrix.py index d15433e5..84571265 100644 --- a/src/canmatrix/tests/test_canmatrix.py +++ b/src/canmatrix/tests/test_canmatrix.py @@ -122,7 +122,7 @@ def test_decode_signal(): def test_ecu_find_attribute(): ecu = canmatrix.canmatrix.Ecu(name="Gateway") ecu.add_attribute("attr1", 255) - assert ecu.attribute("attr1") == 255 + assert ecu.attribute("attr1") == '255' def test_ecu_no_attribute(): @@ -159,7 +159,7 @@ def test_signal_find_mandatory_attribute(some_signal): def test_signal_find_optional_attribute(some_signal): some_signal.add_attribute("attr1", 255) - assert some_signal.attribute("attr1") == 255 + assert some_signal.attribute("attr1") == '255' def test_signal_no_attribute(some_signal): @@ -213,7 +213,7 @@ def test_signal_delete_wrong_attribute_doesnt_raise(some_signal): def test_signal_spn(some_signal): assert some_signal.spn is None some_signal.add_attribute("SPN", 10) - assert some_signal.spn == 10 + assert some_signal.spn == '10' def test_signal_set_startbit(): diff --git a/src/canmatrix/tests/test_copy.py b/src/canmatrix/tests/test_copy.py index 80bbfb1b..6340a770 100644 --- a/src/canmatrix/tests/test_copy.py +++ b/src/canmatrix/tests/test_copy.py @@ -77,7 +77,7 @@ def test_copy_ecu_with_attributes(): assert len(matrix1.frames) == 1 assert len(matrix1.ecus) == 1 assert matrix1.ecu_by_name("ECU") is not None - assert matrix1.ecu_by_name("ECU").attribute("Node Address") == 42 + assert matrix1.ecu_by_name("ECU").attribute("Node Address") == '42' assert matrix1.ecu_by_name("ECU").attribute("some_ecu_define", matrix1) == "default_value" def test_copy_frame_default_attributes(): diff --git a/src/canmatrix/tests/test_dbc.py b/src/canmatrix/tests/test_dbc.py index aababd88..67b70cd6 100644 --- a/src/canmatrix/tests/test_dbc.py +++ b/src/canmatrix/tests/test_dbc.py @@ -516,3 +516,31 @@ def test_without_ecu(): matrix = canmatrix.formats.dbc.load(dbc, dbcImportEncoding="utf8") matrix.frames[0].signals[0].name == "A_B_C_D_E" + + +def test_int_attribute_zero(): + db = canmatrix.CanMatrix() + frame = canmatrix.Frame("some Frame") + frame.add_signal(canmatrix.Signal("signal_name", size=1, start_bit=1)) + db.add_frame(frame) + db.add_ecu_defines("ecu_define", "INT 0 10") + db.add_ecu_defines("ecu_define2", "INT 0 10") + db.add_ecu_defines("ecu_define3", "INT 0 10") + + db.add_frame_defines("test", "INT 0 10") + db.add_frame_defines("test2", "INT 0 10") + frame.add_attribute("test", 7) + frame.add_attribute("test2", 0) + + ecu = canmatrix.Ecu('TestEcu') + ecu.add_attribute('ecu_define', 1) + ecu.add_attribute('ecu_define2', 0) + + db.add_ecu(ecu) + + outdbc = io.BytesIO() + canmatrix.formats.dump(db, outdbc, "dbc") + assert 'BO_ 0 7' in outdbc.getvalue().decode('utf8') + assert 'BO_ 0 0' in outdbc.getvalue().decode('utf8') + assert 'TestEcu 1' in outdbc.getvalue().decode('utf8') + assert 'TestEcu 0' in outdbc.getvalue().decode('utf8') From 6ed291b73a5824e367615c99ee1b4e6084eb026e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 29 Mar 2022 10:06:27 +0200 Subject: [PATCH 06/46] possible bugfix for #632 (#633) --- src/canmatrix/formats/arxml.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index 3ceafc93..f659ab43 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -1132,7 +1132,7 @@ def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset if system_signal is not None and "SYSTEM-SIGNAL-GROUP" in system_signal.tag: system_signals = ea.selector(system_signal, "SYSTEM-SIGNAL-REFS>>SYSTEM-SIGNAL-REF") - get_sys_signals(system_signal, system_signals, frame, group_id, ea) + get_signalgrp_and_signals(system_signal, system_signals, frame, group_id, ea) group_id = group_id + 1 continue From 97ebe53c66556410d9484f7335c6702f24c19b5f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Mon, 18 Apr 2022 21:25:36 +0200 Subject: [PATCH 07/46] allow hex values for most int() conversations (#639) --- src/canmatrix/formats/arxml.py | 52 +++++++++++++++++----------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index f659ab43..ea962610 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -782,7 +782,7 @@ def dump(dbs, f, **options): compu_int_to_phys = create_sub_element( compu_method, 'COMPU-INTERNAL-TO-PHYS') compu_scales = create_sub_element(compu_int_to_phys, 'COMPU-SCALES') - for value in sorted(signal.values, key=lambda x: int(x)): + for value in sorted(signal.values, key=lambda x: int(x, 0)): compu_scale = create_sub_element(compu_scales, 'COMPU-SCALE') desc = create_sub_element(compu_scale, 'DESC') l2 = create_sub_element(desc, 'L-2') @@ -986,7 +986,7 @@ def get_signalgrp_and_signals(sys_signal, sys_signal_array, frame, group_id, ea) } data_id_elems = ea.get_children(ea.get_child(sys_signal, "TRANSFORMATION-I-SIGNAL-PROPSS"), "DATA-ID") if data_id_elems is not None: - e2e_transform['data_ids'] = [int(x.text) for x in data_id_elems] + e2e_transform['data_ids'] = [int(x.text, 0) for x in data_id_elems] frame.add_signal_group(ea.get_element_name(sys_signal), group_id, members, e2e_transform) @@ -1269,8 +1269,8 @@ def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset if start_bit is not None: new_signal = canmatrix.Signal( name, - start_bit=int(start_bit.text) + bit_offset, - size=int(length.text) if length is not None else 0, + start_bit=int(start_bit.text, 0) + bit_offset, + size=int(length.text, 0) if length is not None else 0, is_little_endian=is_little_endian, is_signed=is_signed, factor=factor, @@ -1288,7 +1288,7 @@ def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset if not new_signal.is_little_endian: # startbit of motorola coded signals are MSB in arxml - new_signal.set_startbit(int(start_bit.text) + bit_offset, bitNumbering=1) + new_signal.set_startbit(int(start_bit.text, 0) + bit_offset, bitNumbering=1) communication_direction = ea.selector(isignal, "I-SIGNAL-PORT-REF/COMMUNICATION-DIRECTION") if len(communication_direction) > 0: @@ -1336,8 +1336,8 @@ def get_frame_from_multiplexed_ipdu(pdu, target_frame, multiplex_translation, ea is_signed = False # unsigned multiplexor = canmatrix.Signal( "Multiplexor", - start_bit=int(selector_start.text), - size=int(selector_len.text), + start_bit=int(selector_start.text, 0), + size=int(selector_len.text, 0), is_little_endian=is_little_endian, multiplex="Multiplexor") @@ -1414,12 +1414,12 @@ def get_frame_from_container_ipdu(pdu, target_frame, ea, float_factory, headers_ cycle_time = 0 value = ea.get_child(repeating_time, "VALUE") if value is not None: - cycle_time = int(float_factory(value.text) * 1000) + cycle_time = int(float_factory(value.text, 0) * 1000) else: time_period = ea.get_child(cyclic_timing, "TIME-PERIOD") value = ea.get_child(time_period, "VALUE") if value is not None: - cycle_time = int(float_factory(value.text) * 1000) + cycle_time = int(float_factory(value.text, 0) * 1000) try: if header_type == "SHORT-HEADER": header_id = ea.get_child(ipdu, "HEADER-ID-SHORT-HEADER").text @@ -1439,7 +1439,7 @@ def get_frame_from_container_ipdu(pdu, target_frame, ea, float_factory, headers_ ipdu = ea.follow_ref(payload, "I-PDU-REF") logger.info("found secured pdu '%s', dissolved to '%s'", secured_i_pdu_name, ea.get_element_name(ipdu)) try: - offset = int(ea.get_child(ipdu, "OFFSET").text) * 8 + offset = int(ea.get_child(ipdu, "OFFSET").text, 0) * 8 except: offset = 0 @@ -1451,7 +1451,7 @@ def get_frame_from_container_ipdu(pdu, target_frame, ea, float_factory, headers_ pdu_port_type = ea.get_child(cpdu, "I-PDU-PORT-REF").attrib["DEST"] except (AttributeError, KeyError): pdu_port_type = "" - ipdu_length = int(ea.get_child(ipdu, "LENGTH").text) + ipdu_length = int(ea.get_child(ipdu, "LENGTH").text, 0) ipdu_name = ea.get_element_name(ipdu) ipdu_triggering_name = ea.get_element_name(cpdu) target_pdu = canmatrix.Pdu(name=ipdu_name, size=ipdu_length, id=header_id, @@ -1511,7 +1511,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header if arb_id is None: logger.info("found Frame %s without arbitration id", frame_name_elem.text) return None - arbitration_id = int(arb_id.text) + arbitration_id = int(arb_id.text, 0) if frame_elem is not None: if frame_elem in frames_cache: @@ -1525,7 +1525,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header pdu = ea.selector(pdu, ">PAYLOAD-REF>I-PDU-REF")[0] # logger.info("found secured pdu - no signal extraction possible: %s", get_element_name(pdu, ns)) - new_frame = canmatrix.Frame(ea.get_element_name(frame_elem), size=int(dlc_elem.text)) + new_frame = canmatrix.Frame(ea.get_element_name(frame_elem), size=int(dlc_elem.text, 0)) comment = ea.get_element_desc(frame_elem) if comment is not None: new_frame.add_comment(comment) @@ -1541,7 +1541,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header pdu = None dlc_elem = ea.get_child(pdu, "LENGTH") new_frame = canmatrix.Frame(frame_name_elem.text, arbitration_id=arbitration_id, - size=int(int(dlc_elem.text) / 8)) + size=int(int(dlc_elem.text, 0) / 8)) if pdu is None: logger.error("pdu is None") @@ -1664,18 +1664,18 @@ def ecuc_extract_signal(signal_node, ea): # timeout = 0 for attribute in attributes: if attribute.text.endswith("ComBitPosition"): - start_bit = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text) + start_bit = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text, 0) if attribute.text.endswith("ComBitSize"): - size = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text) + size = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text, 0) if attribute.text.endswith("ComSignalEndianness"): endianness = attribute.getparent().find(".//" + ea.ns + "VALUE").text is_little = "LITTLE_ENDIAN" in endianness if attribute.text.endswith("ComSignalInitValue"): - init_value = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text) + init_value = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text, 0) if attribute.text.endswith("ComSignalType"): signal_type = attribute.getparent().find(".//" + ea.ns + "VALUE").text if attribute.text.endswith("ComTimeout"): - timeout = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text) + timeout = int(attribute.getparent().find(".//" + ea.ns + "VALUE").text, 0) return canmatrix.Signal(ea.get_element_name(signal_node), start_bit=start_bit, size=size, is_little_endian=is_little) @@ -1741,10 +1741,10 @@ def decode_ethernet_helper(ea, float_factory): logger.info("ETH PDU " + ipdu_name + " found") target_frame = canmatrix.Frame(name=ipdu_name) try: - target_frame.header_id = int(header_id.text) + target_frame.header_id = int(header_id.text, 0) except: try: - target_frame.header_id = int(pdu_triggering_header_id_map[ipdu_triggering]) + target_frame.header_id = int(pdu_triggering_header_id_map[ipdu_triggering], 0) except: target_frame.header_id = 0 # continue @@ -1773,13 +1773,13 @@ def decode_flexray_helper(ea, float_factory): frames = ea.findall("FLEXRAY-FRAME-TRIGGERING", pc) for frame_element in frames: frame_counter += 1 - slot_id = int(ea.get_child(frame_element, "SLOT-ID").text) + slot_id = int(ea.get_child(frame_element, "SLOT-ID").text, 0) base_cycle = ea.get_child(frame_element, "BASE-CYCLE").text ipdu_triggerings = ea.get_children(frame_element, "I-PDU-TRIGGERING") frame_repetition_cycle = ea.find_children_by_path(frame_element, "CYCLE-REPETITION/CYCLE-REPETITION")[ 0].text network_endpoints = pc.findall('.//' + ea.ns + "NETWORK-ENDPOINT") - frame_size = int(ea.find_children_by_path(frame_element, "FRAME/FRAME-LENGTH")[0].text) + frame_size = int(ea.find_children_by_path(frame_element, "FRAME/FRAME-LENGTH")[0].text, 0) frame = canmatrix.Frame(size=frame_size, arbitration_id=frame_counter) frame.slot_id = slot_id frame.base_cycle = base_cycle @@ -1789,7 +1789,7 @@ def decode_flexray_helper(ea, float_factory): ipdu_triggering_name = ea.get_element_name(ipdu_triggering) ipdu = ea.get_child(ipdu_triggering, "I-PDU") pdu_type = ea.get_child(ipdu_triggering, "I-PDU-REF").attrib["DEST"] - ipdu_length = int(ea.get_child(ipdu, "LENGTH").text) + ipdu_length = int(ea.get_child(ipdu, "LENGTH").text,0) pdu_port_type = ea.get_child(ipdu_triggering, "I-PDU-PORT-REF").attrib["DEST"] ipdu_name = ea.get_element_name(ipdu) target_pdu = canmatrix.Pdu(name=ipdu_name, size=ipdu_length, @@ -1837,13 +1837,13 @@ def decode_can_helper(ea, float_factory, ignore_cluster_info): bus_name = ea.get_element_name(cc) if speed is not None: - db.baudrate = int(speed.text) + db.baudrate = int(speed.text, 0) elif baudrate_elem is not None: - db.baudrate = int(baudrate_elem.text) + db.baudrate = int(baudrate_elem.text, 0) logger.debug("Baudrate: " + str(db.baudrate)) if fd_baudrate_elem is not None: - db.fd_baudrate = int(fd_baudrate_elem.text) + db.fd_baudrate = int(fd_baudrate_elem.text, 0) can_frame_trig = ea.selector(cc, "/CAN-PHYSICAL-CHANNEL//CAN-FRAME-TRIGGERING") From d30945f88b8e8244eba7ad2ae4e57d2575d1aa4d Mon Sep 17 00:00:00 2001 From: Arcola-Kankan <89925850+Arcola-Kankan@users.noreply.github.com> Date: Tue, 17 May 2022 21:41:04 +0100 Subject: [PATCH 08/46] Added Switches checkFrameNameLength, checkSignalNameLength (#637) * Updated Switch names and Added recalcSignalMinimums switch to update any signal with min physical value set to 0 * Added frame and signal name length check switches * Warn only when the frame or signal name is greater than 32 characters * Removed Signal/Frame name checks and added warning when the Frame/Signal name exceeds 32 characters * Adding a new line to resolve merge conflict --- src/canmatrix/cli/convert.py | 7 ++++--- src/canmatrix/convert.py | 14 +++++++++++--- src/canmatrix/formats/dbc.py | 8 ++++++-- 3 files changed, 21 insertions(+), 8 deletions(-) diff --git a/src/canmatrix/cli/convert.py b/src/canmatrix/cli/convert.py index ac226cfc..3878b485 100644 --- a/src/canmatrix/cli/convert.py +++ b/src/canmatrix/cli/convert.py @@ -78,13 +78,14 @@ def get_formats(): @click.option('--signals', help="Copy only given Signals (comma separated list) to target matrix just as 'free' signals without containing frame") @click.option('--merge', help="merge additional can databases.\nSyntax: --merge filename[:ecu=SOMEECU][:frame=FRAME1][:frame=FRAME2],filename2") @click.option('--ignorePduContainer/--no-ignorePduContainer', 'ignorePduContainer', default = False, help="Ignore any Frame with PDU container; if no export as multiplexed Frames\ndefault False") -@click.option('--calcSignalMax/--no-calcSignalMax', 'calcSignalMax', default = False, help="Calculate Signals Maximum Physical Value; If maximum value is set to 0\ndefault False") -@click.option('--recalcSignalMax/--no-recalcSignalMax', 'recalcSignalMax', default = False, help="Recalculate Signals Maximum Physical Value for the entire database\ndefault False") +@click.option('--calcSignalMaximumsWhereZero/--no-calcSignalMaximumsWhereZero', 'calcSignalMaximumsWhereZero', default = False, help="Calculate Signals Maximum Physical Value; If maximum value is set to 0\ndefault False") +@click.option('--recalcSignalMaximums/--no-recalcSignalMaximums', 'recalcSignalMaximums', default = False, help="Recalculate Signals Maximum Physical Value for the entire database\ndefault False") @click.option('--deleteFloatingSignals/--no-deleteFloatingSignals', 'deleteFloatingSignals', default = False, help="if deleteFloatingSignals is set , then unassigned signals to a frame/message will be deleted \tdefault: False") +@click.option('--recalcSignalMinimums/--no-recalcSignalMinimums', 'recalcSignalMinimums', default = False, help="Recalculate Signals Minimum Physical Value for the entire database\ndefault False") #Frame/Signal Check switches @click.option('--checkFloatingFrames/--no-checkFloatingFrames', 'checkFloatingFrames', default = False, help="if checkFloatingFrames is set, CAN message/frame without sender node will be warned .\tdefault: False") -@click.option('--checkSignalRange/--no-checkSignalRange', 'checkSignalRange', default = False, help="if checkSignalRange is set, then signals consisting raw min/max value set to 0 will be warned. \tdefault: False ") +@click.option('--warnSignalMinMaxSame/--no-warnSignalMinMaxSame', 'warnSignalMinMaxSame', default = False, help="if warnSignalMinMaxSame is set, then signals consisting same raw min & max value will be warned. \tdefault: False ") @click.option('--checkSignalUnit/--no-checkSignalUnit', 'checkSignalUnit', default = False, help="if checkSignalUnit is set , then signals without units and value table will be warned. \tdefault: False") @click.option('--checkSignalReceiver/--no-checkSignalReceiver', 'checkSignalReceiver', default = False, help="if checkSignalReceiver is set, then signals without an assigned Receiver will be warned \tdefault: False") @click.option('--checkFloatingSignals/--no-checkFloatingSignals', 'checkFloatingSignals', default = False, help="if checkFloatingSignals is set, then unassigned signals to a frame/message will be warned \tdefault: False") diff --git a/src/canmatrix/convert.py b/src/canmatrix/convert.py index d6429cb8..193973b5 100644 --- a/src/canmatrix/convert.py +++ b/src/canmatrix/convert.py @@ -252,18 +252,24 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non signal.name = signal.attributes.get(options.get('signalNameFromAttrib'), signal.name) # Max Signal Value Calculation , if max value is 0 - if options.get('calcSignalMax') is not None and options['calcSignalMax']: + if options.get('calcSignalMaximumsWhereZero') is not None and options['calcSignalMaximumsWhereZero']: for signal in [b for a in db for b in a.signals]: if signal.max == 0 or signal.max is None: signal.calc_max_for_none = True signal.set_max(None) # Max Signal Value Calculation - if options.get('recalcSignalMax') is not None and options['recalcSignalMax']: + if options.get('recalcSignalMaximums') is not None and options['recalcSignalMaximums']: for signal in [b for a in db for b in a.signals]: signal.calc_max_for_none = True signal.set_max(None) + # Min Signal Value Calculation + if options.get('recalcSignalMinimums') is not None and options['recalcSignalMinimums']: + for signal in [b for a in db for b in a.signals]: + signal.calc_min_for_none = True + signal.set_min(None) + # Delete Unassigned Signals to a Valid Frame/Message if options.get('deleteFloatingSignals') is not None and options['deleteFloatingSignals']: for frame in db.frames: @@ -294,7 +300,7 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non logger.warning("No Transmitter Node Found for Frame %s", frame.name) # Check & Warn for Signals with Min/Max set to 0 - if options.get('checkSignalRange') is not None and options['checkSignalRange']: + if options.get('warnSignalMinMaxSame') is not None and options['warnSignalMinMaxSame']: for frame in db.frames: for signal in frame.signals: if (signal.phys2raw(signal.max) - signal.phys2raw(signal.min)) == 0: @@ -319,6 +325,8 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non frame.is_j1939=True db.add_attribute("ProtocolType", "J1939") + + logger.info(name) logger.info("%d Frames found" % (db.frames.__len__())) diff --git a/src/canmatrix/formats/dbc.py b/src/canmatrix/formats/dbc.py index 4ecfa7a6..43295669 100644 --- a/src/canmatrix/formats/dbc.py +++ b/src/canmatrix/formats/dbc.py @@ -200,16 +200,20 @@ def dump(in_db, f, **options): output_names = collections.defaultdict(dict) # type: typing.Dict[canmatrix.Frame, typing.Dict[canmatrix.Signal, str]] for frame in db.frames: - # fix long frame names + # fix long frame names , warn if the frame name exceeds 32 characters if len(frame.name) > 32: frame.add_attribute("SystemMessageLongSymbol", frame.name) + logger.warning("Frame %s name exceeds 32 characters, consider updating the frame name within" + " character limit(Max 32 characters) ", frame.name) frame.name = frame.name[0:32] db.add_frame_defines("SystemMessageLongSymbol", "STRING") - # fix long signal names + # fix long signal names, warn if the signal name exceeds 32 characters for s in frame.signals: if len(s.name) > 32: s.add_attribute("SystemSignalLongSymbol", s.name) + logger.warning("Signal %s::%s name exceeds 32 characters, consider updating the signal name " + "within the character limit(Max 32 characters)", frame.name, s.name) s.name = s.name[0:32] db.add_signal_defines("SystemSignalLongSymbol", "STRING") From 4c8e054d8d8a33c069158b8e023359d039fa4077 Mon Sep 17 00:00:00 2001 From: tainnok <16083651+tainnok@users.noreply.github.com> Date: Tue, 17 May 2022 22:42:58 +0200 Subject: [PATCH 09/46] the float_factory - does not matter if Decimal() or float() - does not have the base-argument as int() (#641) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Markus Konrad Co-authored-by: Eduard Bröcker --- src/canmatrix/formats/arxml.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index ea962610..07e0b8fc 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -1414,12 +1414,12 @@ def get_frame_from_container_ipdu(pdu, target_frame, ea, float_factory, headers_ cycle_time = 0 value = ea.get_child(repeating_time, "VALUE") if value is not None: - cycle_time = int(float_factory(value.text, 0) * 1000) + cycle_time = int(float_factory(value.text) * 1000) else: time_period = ea.get_child(cyclic_timing, "TIME-PERIOD") value = ea.get_child(time_period, "VALUE") if value is not None: - cycle_time = int(float_factory(value.text, 0) * 1000) + cycle_time = int(float_factory(value.text) * 1000) try: if header_type == "SHORT-HEADER": header_id = ea.get_child(ipdu, "HEADER-ID-SHORT-HEADER").text From 6e6da2372ab002aa70971df2f596d1c7cec17cfb Mon Sep 17 00:00:00 2001 From: tainnok <16083651+tainnok@users.noreply.github.com> Date: Tue, 17 May 2022 22:46:40 +0200 Subject: [PATCH 10/46] bugfix: get all receivers/transmitters and not only one (#642) Co-authored-by: Markus Konrad --- src/canmatrix/formats/arxml.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index 07e0b8fc..1ba10f08 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -1851,16 +1851,16 @@ def decode_can_helper(ea, float_factory, ignore_cluster_info): for frameTrig in can_frame_trig: # type: _Element frame = get_frame(frameTrig, ea, multiplex_translation, float_factory, headers_are_littleendian) if frame is not None: - comm_direction = ea.selector(frameTrig, ">>FRAME-PORT-REF/COMMUNICATION-DIRECTION") - if len(comm_direction) > 0: - ecu_elem = ea.get_ecu_instance(element=comm_direction[0]) + comm_directions = ea.selector(frameTrig, ">>FRAME-PORT-REF/COMMUNICATION-DIRECTION") + for comm_direction in comm_directions: + ecu_elem = ea.get_ecu_instance(element=comm_direction) if ecu_elem is not None: if ecu_elem in nodes: ecu = nodes[ecu_elem] else: ecu = process_ecu(ecu_elem, ea) nodes[ecu_elem] = ecu - if comm_direction[0].text == "OUT": + if comm_direction.text == "OUT": frame.add_transmitter(ecu.name) else: frame.add_receiver(ecu.name) From a7e6f91d8edcb635e520ecfa23b6ee12d26b81e2 Mon Sep 17 00:00:00 2001 From: ebroecker Date: Sun, 17 Jul 2022 21:27:16 +0200 Subject: [PATCH 11/46] fix for #641 --- src/canmatrix/canmatrix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index e218dd86..208797e6 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -439,7 +439,7 @@ def phys2raw(self, value=None): "Value {} is not valid for {}. Min={} and Max={}".format( value, self, self.min, self.max) ) - raw_value = (value - self.offset) / self.factor + raw_value = (value - self.float_factory(self.offset)) / self.float_factory(self.factor) if not self.is_float: raw_value = int(raw_value) From a20442def32f168a2933282599f66ab34364d347 Mon Sep 17 00:00:00 2001 From: Ayoub Kaanich Date: Wed, 10 Aug 2022 11:30:35 +0200 Subject: [PATCH 12/46] Support signal values in FIBEX (#649) * [ANDI-8717/AnAm]: Implemented export prototype. * [ANDI-8717/AnAm]: Implemented signal interpretation. * [ANDI-8717/AnAm]: Adapted signal interpretation structure.] * [ANDI-8717/AnAm]: Removed a typing error caused by window focus problems. * [ANDI-8717/AnAm]: Corrected FiBEx exporter to generate a correct FiBEx file with signal interpretations. * [ANDI-8717/AnAm]: Corrected structural errors in the FiBEx exporter. * [ANDI-8717/AnAm]: Added an Internal-To-Phys node into the signal interpretation structure. * ]ANDI-8717/AnAm}: Corrected the unpacking bug. * Small fixes Co-authored-by: Andreas Amereller --- src/canmatrix/formats/fibex.py | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/src/canmatrix/formats/fibex.py b/src/canmatrix/formats/fibex.py index 7f9c6437..22069010 100644 --- a/src/canmatrix/formats/fibex.py +++ b/src/canmatrix/formats/fibex.py @@ -339,6 +339,26 @@ def dump(db, f, **options): create_sub_element_ho(compu_denominator, "V", "1") # nenner # defaultValue = create_sub_element_ho(compuInternalToPhys,"COMPU-DEFAULT-VALUE") + # CAN signal interpretation + if signal.values: + compu_method = create_sub_element_ho(compu_methods, "COMPU-METHOD") + create_sub_element_ho(compu_method, "SHORT-NAME", + "SIGNAL_INTERPRETATION_" + signal.name) + create_sub_element_ho(compu_method, "CATEGORY", "TEXTTABLE") + compu_int_to_phys = create_sub_element_ho(compu_method, "COMPU-INTERNAL-TO-PHYS") + compu_scales = create_sub_element_ho(compu_int_to_phys, "COMPU-SCALES") + + for value, text in signal.values.items(): + compu_scale = create_sub_element_ho(compu_scales, "COMPU-SCALE") + + lower_limit = create_sub_element_ho(compu_scale, "LOWER-LIMIT", str(value)) + lower_limit.set("INTERVAL-TYPE", "CLOSED") + upper_limit = create_sub_element_ho(compu_scale, "UPPER-LIMIT", str(value)) + upper_limit.set("INTERVAL-TYPE", "CLOSED") + + compu_const = create_sub_element_ho(compu_scale, "COMPU-CONST") + create_sub_element_ho(compu_const, "VT", text) + # # REQUIREMENTS # From 4f4a30a665fdc585211100b9b35dd36a88838402 Mon Sep 17 00:00:00 2001 From: Yeternity <980888907@qq.com> Date: Wed, 10 Aug 2022 17:31:01 +0800 Subject: [PATCH 13/46] fix signals sort (#648) startbit will be up to 512 in canfd, and sorting signals should in right bit numbering condition. --- src/canmatrix/formats/xlsx.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/canmatrix/formats/xlsx.py b/src/canmatrix/formats/xlsx.py index c5d1f33a..7fa8d890 100644 --- a/src/canmatrix/formats/xlsx.py +++ b/src/canmatrix/formats/xlsx.py @@ -217,7 +217,12 @@ def dump(db, filename, **options): # sort signals: sig_hash = {} for sig in frame.signals: - sig_hash["%02d" % int(sig.get_startbit()) + sig.name] = sig + if motorola_bit_format == "msb": + sig_hash["%03d" % int(sig.get_startbit(bit_numbering=1)) + sig.name] = sig + elif motorola_bit_format == "msbreverse": + sig_hash["%03d" % int(sig.get_startbit()) + sig.name] = sig + else: # motorolaBitFormat == "lsb" + sig_hash["%03d" % int(sig.get_startbit(bit_numbering=1, start_little=True)) + sig.name] = sig # set style for first line with border signal_style = sty_first_frame From 360dcd11cdd79ef3fb5b0565066fe2b351977212 Mon Sep 17 00:00:00 2001 From: Grayson Nocera Date: Wed, 28 Sep 2022 16:38:09 -0400 Subject: [PATCH 14/46] fix: fix rounding error (#655) --- src/canmatrix/canmatrix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 208797e6..10dd8a10 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -442,7 +442,7 @@ def phys2raw(self, value=None): raw_value = (value - self.float_factory(self.offset)) / self.float_factory(self.factor) if not self.is_float: - raw_value = int(raw_value) + raw_value = int(round(raw_value)) return raw_value def raw2phys(self, value, decode_to_str=False): From 973be3226423d53ab37630fd70e1005147cd10a9 Mon Sep 17 00:00:00 2001 From: tainnok <16083651+tainnok@users.noreply.github.com> Date: Fri, 14 Oct 2022 08:53:27 +0200 Subject: [PATCH 15/46] add pdu_name to frame which represents the pdu-name of the pdu-frame-mapping (#658) Co-authored-by: Markus Konrad --- src/canmatrix/canmatrix.py | 1 + src/canmatrix/formats/arxml.py | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 10dd8a10..3850ca68 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -883,6 +883,7 @@ class Frame(object): pdus = attr.ib(factory=list) # type: typing.MutableSequence[Pdu] header_id = attr.ib(default=None) #type: int # header_id + pdu_name = attr.ib(default="") # type: str @property def is_multiplexed(self): # type: () -> bool diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index 1ba10f08..4bea21c5 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -1546,7 +1546,8 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header if pdu is None: logger.error("pdu is None") else: - logger.debug("PDU: " + ea.get_element_name(pdu)) + new_frame.pdu_name = ea.get_element_name(pdu) + logger.debug("PDU: " + new_frame.pdu_name) if new_frame.comment is None: new_frame.add_comment(ea.get_element_desc(pdu)) From 14217318dd0f242eca8074fb8612f2fb7af4a730 Mon Sep 17 00:00:00 2001 From: tainnok <16083651+tainnok@users.noreply.github.com> Date: Fri, 14 Oct 2022 08:54:25 +0200 Subject: [PATCH 16/46] precise logging related to Frame/Frametrigger to avoid confusions (#659) Co-authored-by: Markus Konrad --- src/canmatrix/formats/arxml.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index 4bea21c5..9d2600c2 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -1506,14 +1506,15 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header arb_id = ea.get_child(frame_triggering, "IDENTIFIER") frame_elem = ea.follow_ref(frame_triggering, "FRAME-REF") - frame_name_elem = ea.get_child(frame_triggering, "SHORT-NAME") - logger.debug("processing Frame: %s", frame_name_elem.text) + frame_trig_name_elem = ea.get_child(frame_triggering, "SHORT-NAME") + logger.debug("processing Frame-Trigger: %s", frame_trig_name_elem.text) if arb_id is None: - logger.info("found Frame %s without arbitration id", frame_name_elem.text) + logger.info("found Frame-Trigger %s without arbitration id", frame_trig_name_elem.text) return None arbitration_id = int(arb_id.text, 0) if frame_elem is not None: + logger.debug("Frame: %s", ea.get_element_name(frame_elem)) if frame_elem in frames_cache: return copy.deepcopy(frames_cache[frame_elem]) dlc_elem = ea.get_child(frame_elem, "FRAME-LENGTH") @@ -1531,7 +1532,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header new_frame.add_comment(comment) else: # without frameinfo take short-name of frametriggering and dlc = 8 - logger.debug("Frame %s has no FRAME-REF", frame_name_elem.text) + logger.debug("Frame-Trigger %s has no FRAME-REF", frame_trig_name_elem.text) pdu = ea.selector(frame_triggering, ">I-PDU-TRIGGERING-REF>I-PDU-REF") # AR4.2 if len(pdu) == 0: pdu = ea.selector(frame_triggering, ">PDU-TRIGGERING-REF>I-PDU-REF") @@ -1540,7 +1541,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header else: pdu = None dlc_elem = ea.get_child(pdu, "LENGTH") - new_frame = canmatrix.Frame(frame_name_elem.text, arbitration_id=arbitration_id, + new_frame = canmatrix.Frame(frame_trig_name_elem.text, arbitration_id=arbitration_id, size=int(int(dlc_elem.text, 0) / 8)) if pdu is None: From 34a2b1bfa4c62bbe5bbc7ee37adc62acfd747536 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Sun, 16 Oct 2022 20:53:33 +0200 Subject: [PATCH 17/46] arxml - pdu name (#660) add pdu-name and can-frame-triggering-name to frame attributes during arxml read add option for convert to rename frame form frame.attribute --- src/canmatrix/convert.py | 4 ++++ src/canmatrix/formats/arxml.py | 18 +++++++++++++++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/src/canmatrix/convert.py b/src/canmatrix/convert.py index 193973b5..cdc3f014 100644 --- a/src/canmatrix/convert.py +++ b/src/canmatrix/convert.py @@ -251,6 +251,10 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non for signal in [b for a in db for b in a.signals]: signal.name = signal.attributes.get(options.get('signalNameFromAttrib'), signal.name) + if options.get('frameNameFromAttrib') is not None: + for frame in db: + frame.name = frame.attributes.get(options.get('frameNameFromAttrib'), frame.name) + # Max Signal Value Calculation , if max value is 0 if options.get('calcSignalMaximumsWhereZero') is not None and options['calcSignalMaximumsWhereZero']: for signal in [b for a in db for b in a.signals]: diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index 9d2600c2..91c82cb7 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -51,6 +51,7 @@ _MultiplexId = typing.Union[str, int, None] _FloatFactory = typing.Callable[[typing.Any], typing.Any] + class Earxml: def __init__(self): self.xml_element_cache = dict() # type: typing.Dict[str, _Element] @@ -1528,6 +1529,10 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header new_frame = canmatrix.Frame(ea.get_element_name(frame_elem), size=int(dlc_elem.text, 0)) comment = ea.get_element_desc(frame_elem) + if pdu is not None: + new_frame.add_attribute("PduName", ea.get_short_name(pdu)) + new_frame.add_attribute("FrameTriggeringName", ea.get_short_name(frame_triggering)) + if comment is not None: new_frame.add_comment(comment) else: @@ -1543,6 +1548,9 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header dlc_elem = ea.get_child(pdu, "LENGTH") new_frame = canmatrix.Frame(frame_trig_name_elem.text, arbitration_id=arbitration_id, size=int(int(dlc_elem.text, 0) / 8)) + if pdu is not None: + new_frame.add_attribute("PduName", ea.get_short_name(pdu)) + new_frame.add_attribute("FrameTriggeringName", ea.get_short_name(frame_triggering)) if pdu is None: logger.error("pdu is None") @@ -1819,10 +1827,17 @@ def decode_can_helper(ea, float_factory, ignore_cluster_info): db.add_ecu_defines("NWM-Stationsadresse", 'HEX 0 63') db.add_ecu_defines("NWM-Knoten", 'ENUM "nein","ja"') db.add_signal_defines("LongName", 'STRING') + db.add_signal_defines("CompuMethodName", 'STRING') + db.add_signal_defines("ISignalName", 'STRING') + db.add_signal_defines("SysSignalName", 'STRING') db.add_frame_defines("GenMsgDelayTime", 'INT 0 65535') db.add_frame_defines("GenMsgNrOfRepetitions", 'INT 0 65535') db.add_frame_defines("GenMsgStartValue", 'STRING') + db.add_frame_defines("FrameTriggeringName", 'STRING') + db.add_frame_defines("PduName", 'STRING') db.add_frame_defines("GenMsgStartDelayTime", 'INT 0 65535') + + db.add_frame_defines( "GenMsgSendType", 'ENUM "cyclicX","spontanX","cyclicIfActiveX","spontanWithDelay","cyclicAndSpontanX","cyclicAndSpontanWithDelay","spontanWithRepitition","cyclicIfActiveAndSpontanWD","cyclicIfActiveFast","cyclicWithRepeatOnDemand","none"') @@ -1938,7 +1953,6 @@ def load(file, **options): ea = Earxml() ea.open(file) - com_module = ea.get_short_name_path("/ActiveEcuC/Com") if com_module is not None and len(com_module) > 0: @@ -1960,6 +1974,4 @@ def load(file, **options): result.update(decode_can_helper(ea, float_factory, ignore_cluster_info)) - - return result From 390354c91138e5d2476d294ef9e10e55c67d76a5 Mon Sep 17 00:00:00 2001 From: rawiamoalla <90904507+rawiamoalla@users.noreply.github.com> Date: Wed, 19 Oct 2022 13:52:10 +0100 Subject: [PATCH 18/46] support missing information like channel_id, included-pdu_id and connector_id (#662) Co-authored-by: rawia.moalla --- src/canmatrix/formats/fibex.py | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/src/canmatrix/formats/fibex.py b/src/canmatrix/formats/fibex.py index 22069010..2ddc6b9b 100644 --- a/src/canmatrix/formats/fibex.py +++ b/src/canmatrix/formats/fibex.py @@ -101,7 +101,7 @@ def dump(db, f, **options): cluster.set('ID', 'canCluster1') create_short_name_desc(cluster, "clusterShort", "clusterDesc") create_sub_element_fx(cluster, "SPEED", "500") - create_sub_element_fx(cluster, "IS-HIGH-LOW-BIT-ORDER", "false") + create_sub_element_fx(cluster, "IS-HIGH-LOW-BIT-ORDER", "true") create_sub_element_fx(cluster, "BIT-COUNTING-POLICY", "MONOTONE") protocol = create_sub_element_fx(cluster, "PROTOCOL", "CAN") protocol.attrib['{{{pre}}}type'.format(pre=xsi)] = "can:PROTOCOL-TYPE" @@ -117,6 +117,7 @@ def dump(db, f, **options): channels = create_sub_element_fx(elements, "CHANNELS") channel = create_sub_element_fx(channels, "CHANNEL") # for each channel + channel.set('ID', 'CANCHANNEL01') create_short_name_desc(channel, "CANCHANNEL01", "Can Channel Description") # for pdu triggerings @@ -156,10 +157,19 @@ def dump(db, f, **options): function_refs = create_sub_element_fx(ecu, "FUNCTION-REFS") func_ref = create_sub_element_fx(function_refs, "FUNCTION-REF") func_ref.set("ID-REF", "FCT_" + bu.name) - + + controllers = create_sub_element_fx(ecu, "CONTROLLERS") + controller = create_sub_element_fx(controllers, "CONTROLLER") + create_short_name_desc(controller, bu.name, bu.comment) + controller.set('ID', 'Controller_' + bu.name) + connectors = create_sub_element_fx(ecu, "CONNECTORS") connector = create_sub_element_fx(connectors, "CONNECTOR") - + connector.set('ID', 'Connector' + bu.name) + channel_ref = create_sub_element_fx(connector, "CHANNEL-REF") + channel_ref.set("ID-REF", "CANCHANNEL01") + controller_ref = create_sub_element_fx(connector, "CONTROLLER-REF") + controller_ref.set("ID-REF", 'Controller_' + bu.name) inputs = create_sub_element_fx(connector, "INPUTS") for frame in db.frames: if bu.name in frame.receivers: @@ -169,6 +179,7 @@ def dump(db, f, **options): # Reference to PDUs included_pdus = create_sub_element_fx(input_port, "INCLUDED-PDUS") included_pdu = create_sub_element_fx(included_pdus, "INCLUDED-PDU") + included_pdu.set('ID', 'input_included_pdu_' + frame.name) pdu_triggering_ref = create_sub_element_fx(included_pdu, "PDU-TRIGGERING-REF") pdu_triggering_ref.set("ID-REF", "PDU_" + frame.name) @@ -182,6 +193,7 @@ def dump(db, f, **options): # Reference to PDUs included_pdus = create_sub_element_fx(input_port, "INCLUDED-PDUS") included_pdu = create_sub_element_fx(included_pdus, "INCLUDED-PDU") + included_pdu.set('ID', 'output_included_pdu_' + frame.name) pdu_triggering_ref = create_sub_element_fx(included_pdu, "PDU-TRIGGERING-REF") pdu_triggering_ref.set("ID-REF", "PDU_" + frame.name) From 92d8c38b03ad7a50cd7e5c0e303c83ca6844cd24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Thu, 20 Oct 2022 21:46:51 +0200 Subject: [PATCH 19/46] Update yaml.py (#664) switch to yaml.safe_load() --- src/canmatrix/formats/yaml.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/formats/yaml.py b/src/canmatrix/formats/yaml.py index c62b005b..972c7f6a 100644 --- a/src/canmatrix/formats/yaml.py +++ b/src/canmatrix/formats/yaml.py @@ -74,7 +74,7 @@ def dump(db, f, **options): # type: (canmatrix.CanMatrix, typing.IO, **typing.A def load(f, **options): # type: (typing.IO, **typing.Any) -> canmatrix.CanMatrix __init_yaml() - db = yaml.load(f) + db = yaml.safe_load(f) return db From 6f0042f89c66dfb9d2ad09435442b69903912edf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Mon, 21 Nov 2022 21:28:04 +0100 Subject: [PATCH 20/46] maybe fix for issue 661 (#666) * maybe fix for issue 661 * create safe start values again --- src/canmatrix/formats/dbc.py | 16 ++++++++++--- src/canmatrix/tests/test_dbc.py | 42 +++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 3 deletions(-) diff --git a/src/canmatrix/formats/dbc.py b/src/canmatrix/formats/dbc.py index 43295669..4bcc243b 100644 --- a/src/canmatrix/formats/dbc.py +++ b/src/canmatrix/formats/dbc.py @@ -237,8 +237,13 @@ def dump(in_db, f, **options): for signal in frame.signals: if signal.cycle_time != 0: signal.add_attribute("GenSigCycleTime", signal.cycle_time) - if signal.phys2raw(None) != 0: - signal.add_attribute("GenSigStartValue", signal.phys2raw(None)) + if "GenSigStartValue" in db.signal_defines: + if signal.phys2raw(None) != 0: + if db.signal_defines["GenSigStartValue"].defaultValue is not None and \ + float(signal.initial_value) != float(db.signal_defines["GenSigStartValue"].defaultValue): + signal.add_attribute("GenSigStartValue", signal.phys2raw(float(db.signal_defines["GenSigStartValue"].defaultValue))) + elif db.signal_defines["GenSigStartValue"].defaultValue is None: + signal.add_attribute("GenSigStartValue", signal.phys2raw(None)) name = normalized_names[signal] if compatibility: @@ -956,7 +961,12 @@ def add_frame_by_id(new_frame): # type: (canmatrix.Frame) -> None # frame.extended = 1 for signal in frame.signals: - gen_sig_start_value = float_factory(signal.attributes.get("GenSigStartValue", "0")) + if "GenSigStartValue" in db.signal_defines \ + and db.signal_defines["GenSigStartValue"].defaultValue is not None: + default_value = signal.phys2raw(float_factory(db.signal_defines["GenSigStartValue"].defaultValue)) + else: + default_value = signal.phys2raw(None) + gen_sig_start_value = float_factory(signal.attributes.get("GenSigStartValue", default_value)) signal.initial_value = (gen_sig_start_value * signal.factor) + signal.offset signal.cycle_time = int(signal.attributes.get("GenSigCycleTime", 0)) if signal.attribute("SystemSignalLongSymbol") is not None: diff --git a/src/canmatrix/tests/test_dbc.py b/src/canmatrix/tests/test_dbc.py index 67b70cd6..bf60deda 100644 --- a/src/canmatrix/tests/test_dbc.py +++ b/src/canmatrix/tests/test_dbc.py @@ -53,6 +53,7 @@ def test_create_comment_string(): test_string = canmatrix.formats.dbc.create_comment_string("BO_", "ident", "some comment", "utf8", "utf8", "") assert test_string == b'CM_ BO_ ident "some comment";\n' + def test_parse_comment_from_dbc(): dbc = io.BytesIO(textwrap.dedent(u'''\ BO_ 1 someFrame: 1 someEcu @@ -64,6 +65,7 @@ def test_parse_comment_from_dbc(): matrix = canmatrix.formats.dbc.load(dbc) assert matrix.frames[0].signals[0].comment == "resistance setting (0-100%)" + def test_parse_multi_line_comment(): dbc = io.BytesIO(textwrap.dedent(u'''\ BO_ 1 someFrame: 1 someEcu @@ -240,6 +242,7 @@ def test_export_of_unknown_defines(): if line.startswith("BA_ "): assert line.endswith('";') + def test_braces_in_attributes(): dbc = io.BytesIO(textwrap.dedent(u'''\ BO_ 20 frameName: 1 someEcu @@ -249,6 +252,7 @@ def test_braces_in_attributes(): ''').encode('utf-8')) matrix = canmatrix.formats.dbc.load(dbc, dbcImportEncoding="utf8") + def test_defines_with_spaces(): dbc = io.BytesIO(textwrap.dedent(u'''\ BU_: someOtherEcu @@ -276,6 +280,7 @@ def test_defines_with_spaces(): assert matrix.env_vars["someEnvVar"]["attributes"]["some attrib"] == '"some space"' assert matrix.ecus[0].attributes["Description X"] == "Some Some Text" + def test_writing_complex_multiplex(): db = canmatrix.CanMatrix() frame = canmatrix.Frame("someFrame") @@ -307,6 +312,7 @@ def test_defines_with_special_cars(): matrix = canmatrix.formats.dbc.load(dbc, dbcImportEncoding="utf8") assert matrix.frames[0].signals[0].attributes["Accuracy"] == "+/- 10.2 at 55.1%" + def test_j1939_frametype(): dbc = io.BytesIO(textwrap.dedent(u'''\ BU_: someOtherEcu @@ -346,6 +352,7 @@ def test_attributes_with_spaces_before_semicolumn(): assert matrix.frames[0].attributes["someAttribute"] == 'str' assert matrix.frames[1].attribute("someAttribute", matrix) == 'asd' + def test_cycle_time_handling(): dbc = io.BytesIO(textwrap.dedent(u'''\ BO_ 17 Frame_1: 8 Vector__XXX @@ -381,6 +388,7 @@ def test_cycle_time_handling(): outdbc = io.BytesIO() canmatrix.formats.dump({"aa":matrix}, outdbc, "kcd") + def test_keep_cycle_time_defines(): dbc = io.BytesIO(textwrap.dedent(u'''\ BO_ 17 Frame_1: 8 Vector__XXX @@ -396,6 +404,7 @@ def test_keep_cycle_time_defines(): assert 'BA_DEF_ BO_ "GenMsgCycleTime" INT 0 50000' in outdbc.getvalue().decode('utf8') assert 'BA_DEF_DEF_ "GenMsgCycleTime" 0' in outdbc.getvalue().decode('utf8') + def test_unique_signal_names(): db = canmatrix.CanMatrix() frame = canmatrix.Frame("some Frame") @@ -413,6 +422,7 @@ def test_unique_signal_names(): assert "signal_name1" not in outdbc.getvalue().decode('utf8') assert "signal_name" in outdbc.getvalue().decode('utf8') + def test_signal_inital_value(): dbc = io.BytesIO(textwrap.dedent(u'''\ BO_ 17 Frame_1: 8 Vector__XXX @@ -480,6 +490,7 @@ def test_missing_space(): matrix = canmatrix.formats.dbc.load(dbc, dbcImportEncoding="utf8") assert matrix.frames[0].signals[0].name == "sig1" + def test_escaped_quotes(): dbc = io.BytesIO(textwrap.dedent(r''' BO_ 17 Frame_1: 8 Vector__XXX @@ -518,6 +529,37 @@ def test_without_ecu(): matrix.frames[0].signals[0].name == "A_B_C_D_E" +def test_default_initial_value(): + dbc = io.BytesIO(textwrap.dedent(u'''\ + BO_ 560 ECU1_Message: 1 ECU1 + SG_ ECU2_Signal : 0|8@0+ (1,-5) [-2|250] "g" ECU2 + + BA_DEF_ SG_ "GenSigStartValue" FLOAT 0.0 100.0; + + BA_DEF_DEF_ "GenSigStartValue" 10.0; + ''').encode('utf-8')) + + matrix = canmatrix.formats.dbc.load(dbc, dbcImportEncoding="utf8") + assert matrix.frames[0].signals[0].initial_value == 10 +# outdbc = io.BytesIO() +# canmatrix.formats.dump(matrix, outdbc, "dbc") + + +def test_no_initial_value(): + dbc = io.BytesIO(textwrap.dedent(u'''\ + BO_ 560 ECU1_Message: 1 ECU1 + SG_ ECU2_Signal : 0|8@0+ (1,-5) [-2|250] "g" ECU2 + + BA_DEF_ SG_ "GenSigStartValue" FLOAT 0.0 100.0; + ''').encode('utf-8')) + + matrix = canmatrix.formats.dbc.load(dbc, dbcImportEncoding="utf8") + outdbc = io.BytesIO() + canmatrix.formats.dump(matrix, outdbc, "dbc") + assert 'BA_ "GenSigStartValue" SG_ 560 ECU2_Signal 5;' in outdbc.getvalue().decode('utf8') +# assert matrix.frames[0].signals[0].initial_value == 10 + + def test_int_attribute_zero(): db = canmatrix.CanMatrix() frame = canmatrix.Frame("some Frame") From 6f95b7a212eb07b5d7fef06d6e3e9374e4c83f33 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Mon, 21 Nov 2022 21:30:11 +0100 Subject: [PATCH 21/46] fix first issue #667 (not second one!) (#668) --- src/canmatrix/formats/arxml.py | 11 ++++-- src/canmatrix/tests/ARXMLCompuMethod1.arxml | 42 +++++++++++++++++++++ src/canmatrix/tests/test_arxml.py | 15 +++++++- 3 files changed, 63 insertions(+), 5 deletions(-) create mode 100644 src/canmatrix/tests/ARXMLCompuMethod1.arxml diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index 91c82cb7..a0764fc0 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -77,7 +77,6 @@ def fill_caches(self, start_element=None, ar_path=""): def open(self, filename): self.tree = lxml.etree.parse(filename) - self.root = self.tree.getroot() # type: _Element self.ns = "{" + self.tree.xpath('namespace-uri(.)') + "}" # type: str @@ -1017,7 +1016,14 @@ def decode_compu_method(compu_method, ea, float_factory): # keyword definition. 06Jun16 ##################################################################################################### - if ll is not None and desc is not None and canmatrix.utils.decode_number(ul.text, + if len(desc) == 0: + vt = ea.get_sub_by_name(compu_scale, 'VT') + if vt is not None: + desc = vt.text + + rational = ea.get_child(compu_scale, "COMPU-RATIONAL-COEFFS") + + if rational is None and ll is not None and desc is not None and canmatrix.utils.decode_number(ul.text, float_factory) == canmatrix.utils.decode_number( ll.text, float_factory): ##################################################################################################### @@ -1025,7 +1031,6 @@ def decode_compu_method(compu_method, ea, float_factory): values[ll.text] = desc # scale_desc = ea.get_element_desc(compu_scale) - rational = ea.get_child(compu_scale, "COMPU-RATIONAL-COEFFS") if rational is not None: numerator_parent = ea.get_child(rational, "COMPU-NUMERATOR") numerator = ea.get_children(numerator_parent, "V") diff --git a/src/canmatrix/tests/ARXMLCompuMethod1.arxml b/src/canmatrix/tests/ARXMLCompuMethod1.arxml new file mode 100644 index 00000000..dab5874e --- /dev/null +++ b/src/canmatrix/tests/ARXMLCompuMethod1.arxml @@ -0,0 +1,42 @@ + + + + + New_Frame_NewPDU_NewSignal_Encoding + SCALE_LINEAR_AND_TEXTTABLE + + + + + 0 + 0 + + no trailer detected + + + + 1 + 1 + + trailer detected + + + + TrailerPresence + 0 + 0 + + + 17 + 42 + + + 1 + + + + + + + + diff --git a/src/canmatrix/tests/test_arxml.py b/src/canmatrix/tests/test_arxml.py index 9ac336e7..b39c027b 100644 --- a/src/canmatrix/tests/test_arxml.py +++ b/src/canmatrix/tests/test_arxml.py @@ -27,14 +27,25 @@ def test_get_signals_from_container_i_pdu(): assert matrix["New_CanCluster"].frames[0].pdus[0].signals[0].attributes["SysSignalName"] == 'PDU_Contained_1_Signal1_905db81da40081cb' - def test_get_signals_from_secured_pdu(): here = Path(__file__).parent matrix = canmatrix.formats.arxml.load(str(here / "ARXMLSecuredPDUTest.arxml")) assert matrix["CAN"].frames[0].signals[0].name == 'someTestSignal' assert matrix["CAN"].frames[0].signals[1].name == 'Signal' + def test_min_max(): here = Path(__file__).parent matrix = canmatrix.formats.arxml.load(str(here / "ARXML_min_max.arxml")) - assert matrix["New_CanCluster"].frames[0].signals[0].is_signed == False + assert matrix["New_CanCluster"].frames[0].signals[0].is_signed is False + + +def test_decode_compu_method_1(): + here = Path(__file__).parent + ea = canmatrix.formats.arxml.Earxml() + ea.open(str(here / "ARXMLCompuMethod1.arxml")) + compu_method = ea.find("COMPU-METHOD") + values, factor, offset, unit, const = canmatrix.formats.arxml.decode_compu_method(compu_method, ea, float) + assert values == {'0': 'no trailer detected', '1': 'trailer detected'} + assert factor == 42 + assert offset == 17 From 4f8a94d8c31da3c14c5588298104dfc8d1070600 Mon Sep 17 00:00:00 2001 From: Fahrj Date: Thu, 15 Dec 2022 21:15:22 +0100 Subject: [PATCH 22/46] bugfix: set flags as string in scapy export (#678) This commit fixes #677 --- src/canmatrix/formats/scapy.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/canmatrix/formats/scapy.py b/src/canmatrix/formats/scapy.py index 3124abad..7a5c47d4 100644 --- a/src/canmatrix/formats/scapy.py +++ b/src/canmatrix/formats/scapy.py @@ -64,7 +64,7 @@ def dump(db, f, **options): # type: (canmatrix.CanMatrix, typing.IO, **typing.A for frame in db.frames: scapy_decoder += "class " + frame.name + "(SignalPacket):\n" - scapy_decoder += " fields_desc = [ \n" + scapy_decoder += " fields_desc = [\n" if frame.is_multiplexed and not frame.is_complex_multiplexed: multiplexer = frame.get_multiplexer @@ -83,7 +83,7 @@ def dump(db, f, **options): # type: (canmatrix.CanMatrix, typing.IO, **typing.A for frame in db.frames: if frame.arbitration_id.extended: scapy_decoder += "bind_layers(SignalHeader, " + frame.name + ", identifier = " + hex( - frame.arbitration_id.id) + ", flags = extended)\n" + frame.arbitration_id.id) + ", flags = \"extended\")\n" else: scapy_decoder += "bind_layers(SignalHeader, " + frame.name + ", identifier = " + hex( frame.arbitration_id.id) + ")\n" From 7332f914e4683ea3d1f4630f26b892ef775799f2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Fri, 16 Dec 2022 09:51:40 +0100 Subject: [PATCH 23/46] fix for #671 (#673) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit value-tables contain raw values fixes #671 Co-authored-by: Eduard Bröcker --- src/canmatrix/canmatrix.py | 10 ++++++---- src/canmatrix/tests/test_canmatrix.py | 6 +++--- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 3850ca68..f4a18b3e 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -428,7 +428,7 @@ def phys2raw(self, value=None): for value_key, value_string in self.values.items(): if value_string == value: value = value_key - break + return value else: raise ValueError( "{} is invalid value choice for {}".format(value, self) @@ -455,12 +455,14 @@ def raw2phys(self, value, decode_to_str=False): """ if self.is_float: value = self.float_factory(value) - result = value * self.factor + self.offset # type: typing.Union[canmatrix.types.PhysicalValue, str] if decode_to_str: for value_key, value_string in self.values.items(): - if value_key == result: - result = value_string + if value_key == value: + return value_string break + + result = value * self.factor + self.offset # type: typing.Union[canmatrix.types.PhysicalValue, str] + return result def __str__(self): # type: () -> str diff --git a/src/canmatrix/tests/test_canmatrix.py b/src/canmatrix/tests/test_canmatrix.py index 84571265..49b954e4 100644 --- a/src/canmatrix/tests/test_canmatrix.py +++ b/src/canmatrix/tests/test_canmatrix.py @@ -276,7 +276,7 @@ def test_signal_decode_named_value(some_signal): some_signal.add_values(255, "Init") some_signal.add_values(254, "Error") assert some_signal.raw2phys(254, decode_to_str=True) == "Error" - assert some_signal.raw2phys(200, decode_to_str=True) == 200 + assert some_signal.raw2phys(300, decode_to_str=True) == 450 def test_signal_encode_named_value(some_signal): @@ -383,7 +383,7 @@ def the_group(): @pytest.fixture def some_signal(): - return canmatrix.canmatrix.Signal(name="speed", size=8) + return canmatrix.canmatrix.Signal(name="speed", size=8, factor=1.5) def test_signalgroup_empty(the_group): @@ -854,7 +854,7 @@ def test_decoded_signal_phys_value(some_signal): def test_decoded_signal_named_value(): signal = canmatrix.canmatrix.Signal(factor="0.1", values={10: "Init"}) - decoded = canmatrix.canmatrix.DecodedSignal(100, signal) + decoded = canmatrix.canmatrix.DecodedSignal(10, signal) assert decoded.named_value == "Init" From 7f4b2825c3114da21f25345c39cc0f3bda0ed541 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Fri, 20 Jan 2023 14:26:01 +0100 Subject: [PATCH 24/46] quick-fix for issue/pr #680 (#681) --- src/canmatrix/formats/json.py | 3 ++- src/canmatrix/tests/test_json.py | 15 +++++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/src/canmatrix/formats/json.py b/src/canmatrix/formats/json.py index e32c8662..676947a3 100644 --- a/src/canmatrix/formats/json.py +++ b/src/canmatrix/formats/json.py @@ -195,7 +195,8 @@ def load(f, **_options): if "messages" in json_data: for frame in json_data["messages"]: # new_frame = Frame(frame["id"],frame["name"],8,None) - new_frame = canmatrix.Frame(frame["name"], arbitration_id=frame["id"], size=8) + arb_id = canmatrix.canmatrix.ArbitrationId(id=frame["id"], extended=frame.get("is_extended_frame", "False")) + new_frame = canmatrix.Frame(frame["name"], arbitration_id=arb_id, size=8) if "length" in frame: new_frame.size = frame["length"] diff --git a/src/canmatrix/tests/test_json.py b/src/canmatrix/tests/test_json.py index 7345d5b0..8b43db92 100644 --- a/src/canmatrix/tests/test_json.py +++ b/src/canmatrix/tests/test_json.py @@ -115,6 +115,7 @@ def test_import_min_max(): assert matrix.frames[0].signals[0].min == -5 assert matrix.frames[0].signals[0].max == 42 + def test_import_native(): json_input = """{ "messages": [ @@ -208,6 +209,7 @@ def test_export_native(): assert (data['messages'][0]['signals'][0]['factor'] == 0.123) assert (data['messages'][0]['signals'][0]['offset'] == 1) + def test_export_all_native(): matrix = canmatrix.canmatrix.CanMatrix() frame = canmatrix.canmatrix.Frame(name="test_frame", size=6, arbitration_id=10) @@ -222,3 +224,16 @@ def test_export_all_native(): assert (data['messages'][0]['signals'][0]['factor'] == 0.123) assert (data['messages'][0]['signals'][0]['offset'] == 1) assert (data['messages'][0]['is_fd'] is False) + + +def test_export_extended(): + matrix = canmatrix.canmatrix.CanMatrix() + frame = canmatrix.canmatrix.Frame(name="test_frame", size=6, arbitration_id=canmatrix.canmatrix.ArbitrationId(extended=True, id=10)) + frame.pgn=22 + signal = canmatrix.Signal(name="someSigName", size=40) + frame.add_signal(signal) + matrix.add_frame(frame) + out_file = io.BytesIO() + canmatrix.formats.dump(matrix, out_file, "json") +# data = json.loads(out_file.getvalue().decode("utf-8")) + matrix = canmatrix.formats.loads_flat(out_file.getvalue().decode("utf-8"), "json", jsonExportAll=True) From 9dee68a0b315278d7293a860aa44e7e683136070 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Mon, 23 Jan 2023 13:22:42 +0100 Subject: [PATCH 25/46] Fix for 661 (#675) * maybe fix for issue 661 * create safe start values again From d892f966942d4cc2eb00ab923f26dca6810762fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Mon, 23 Jan 2023 13:23:14 +0100 Subject: [PATCH 26/46] fix first issue #667 (not second one!) (#674) From a0034851ae6b15103204a51cc1930a79b29ad598 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 24 Jan 2023 20:48:21 +0100 Subject: [PATCH 27/46] possible fix for #672 (#679) --- src/canmatrix/formats/arxml.py | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index a0764fc0..546ceba5 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -248,7 +248,7 @@ def selector(self, start_element, selector): result_list = [start_element] last_found_token = 0 while start_pos < len(selector): - token_match = re.search(r'//|/|>>|>|<<|<|#|$', selector[start_pos:]) + token_match = re.search(r'//|/|>>|>|<<|<|#|:|$', selector[start_pos:]) found_token = token_match.span() if start_pos > 0: # at least one Token found... @@ -273,6 +273,13 @@ def selector(self, start_element, selector): result_list = [self.find_references_of_type(a, value, referencable_parent=True)[0] for a in result_list if len(self.find_references_of_type(a, value, referencable_parent=True)) > 0] + elif token == ":": + filtered_results = [] + for item in result_list: + if value == item.text: + filtered_results.append(item) + result_list = filtered_results + elif token == "#": sn_snippets = value.split("|") filtered_results = [] @@ -1085,7 +1092,7 @@ def ar_byteorder_is_little(in_string): return False -def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset=0): +def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset=0, signal_triggerings=[]): # type: (typing.Sequence[_Element], typing.Union[canmatrix.Frame, canmatrix.Pdu], Earxml, int, typing.Callable, int) -> None """Add signals from xml to the Frame.""" @@ -1115,6 +1122,16 @@ def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset 'Frame %s, no isignal for %s found', frame.name, ea.get_child(signal, "SHORT-NAME").text) + receiver = [] # type: typing.List[str] + + for triggering in signal_triggerings: + try: + if ea.selector(triggering, ">I-SIGNAL-REF")[0] == isignal: + reciving_ecu_instances = ea.selector(triggering, ">>I-SIGNAL-PORT-REF//COMMUNICATION-DIRECTION:IN/../../..") + receiver = [ea.get_short_name(a) for a in reciving_ecu_instances] + except IndexError: + pass + base_type = ea.follow_ref(isignal, "BASE-TYPE-REF") # AR4 if base_type is None: a = ea.selector(isignal, ">SYSTEM-SIGNAL-REF>DATA-TYPE-REF>BASE-TYPE-REF") @@ -1149,7 +1166,6 @@ def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset signal_min = None # type: canmatrix.types.OptionalPhysicalValue signal_max = None # type: canmatrix.types.OptionalPhysicalValue - receiver = [] # type: typing.List[str] signal_description = ea.get_element_desc(system_signal) @@ -1518,6 +1534,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header logger.info("found Frame-Trigger %s without arbitration id", frame_trig_name_elem.text) return None arbitration_id = int(arb_id.text, 0) + isignaltriggerings_of_current_cluster = ea.selector(frame_triggering, "/..//I-SIGNAL-TRIGGERING") if frame_elem is not None: logger.debug("Frame: %s", ea.get_element_name(frame_elem)) @@ -1607,7 +1624,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header else: pdu_sig_mapping = ea.selector(pdu, "//I-SIGNAL-TO-I-PDU-MAPPING") if pdu_sig_mapping: - get_signals(pdu_sig_mapping, new_frame, ea, None, float_factory) + get_signals(pdu_sig_mapping, new_frame, ea, None, float_factory, signal_triggerings=isignaltriggerings_of_current_cluster) # Seen some pdu_sig_mapping being [] and not None with some arxml 4.2 else: # AR 4.2 pdu_trigs = ea.follow_all_ref(frame_triggering, "PDU-TRIGGERINGS-REF") @@ -1626,7 +1643,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header ea.get_element_name(ipdus)) # signal_to_pdu_map = get_children(signal_to_pdu_maps, "I-SIGNAL-TO-I-PDU-MAPPING", arDict, ns) get_signals(signal_to_pdu_maps, new_frame, ea, None, - float_factory) # todo BUG expects list, not item + float_factory, signal_triggerings=isignaltriggerings_of_current_cluster) # todo BUG expects list, not item else: logger.debug("Frame %s (assuming AR4.2) no PDU-TRIGGERINGS found", new_frame.name) if new_frame.is_pdu_container and new_frame.cycle_time == 0: @@ -1638,6 +1655,7 @@ def get_frame(frame_triggering, ea, multiplex_translation, float_factory, header new_frame.fit_dlc() if frame_elem is not None: frames_cache[frame_elem] = new_frame + return copy.deepcopy(new_frame) From 32fb4932b6d9124932ca8fed7c0b2c7caea6edf0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Wed, 25 Jan 2023 17:21:40 +0100 Subject: [PATCH 28/46] add basic ecu info to json format (#683) * add basic ecu info to json format * add more details to json --- src/canmatrix/formats/json.py | 92 +++++++++++++++++++++++++++++++++-- 1 file changed, 87 insertions(+), 5 deletions(-) diff --git a/src/canmatrix/formats/json.py b/src/canmatrix/formats/json.py index 676947a3..3afe17a4 100644 --- a/src/canmatrix/formats/json.py +++ b/src/canmatrix/formats/json.py @@ -30,7 +30,7 @@ import sys import typing from builtins import * - +import decimal import canmatrix @@ -104,6 +104,20 @@ def dump(db, f, **options): symbolic_frame["attributes"] = frame_attributes export_dict['messages'].append(symbolic_frame) else: # export_all + _define_mapping = {"signal_defines": db.signal_defines, "frame_defines": db.frame_defines, + "global_defines": db.global_defines, "env_defines": db.env_defines, "ecu_defines": db.ecu_defines} + for define_type in _define_mapping: + export_dict[define_type] = [{"name": a, + "define": _define_mapping[define_type][a].definition, + "default": _define_mapping[define_type][a].defaultValue, + "type": _define_mapping[define_type][a].type} for a in _define_mapping[define_type]] + export_dict['ecus'] = {ecu.name: ecu.comment for ecu in db.ecus} + export_dict['attributes'] = db.attributes + export_dict['value_tables'] = db.value_tables + export_dict['env_vars'] = db.env_vars + export_dict['baudrate'] = db.baudrate + export_dict['fd_baudrate'] = db.fd_baudrate + for frame in db.frames: frame_attributes = {attribute: frame.attribute(attribute, db=db) for attribute in db.frame_defines} symbolic_signals = [] @@ -132,10 +146,13 @@ def dump(db, f, **options): "is_signed": signal.is_signed, "is_float": signal.is_float, "comment": signal.comment, + "comments": signal.comments, "attributes": attributes, + "initial_value": number_converter(signal.initial_value), "values": values, - "is_multiplexer" : signal.is_multiplexer, - "mux_value" : signal.mux_val + "is_multiplexer": signal.is_multiplexer, + "mux_value": signal.mux_val, + "receivers": signal.receivers, } if signal.multiplex is not None: symbolic_signal["multiplex"] = signal.multiplex @@ -156,7 +173,14 @@ def dump(db, f, **options): "signals": symbolic_signals, "attributes": frame_attributes, "comment": frame.comment, - "length": frame.size}) + "length": frame.size, + "is_complex_multiplexed": frame.is_complex_multiplexed, + "mux_names": frame.mux_names, + "cycle_time": frame.cycle_time, + "is_j1939": frame.is_j1939, + "header_id": frame.header_id, + "pdu_name": frame.pdu_name, + "transmitters": frame.transmitters}) if sys.version_info > (3, 0): import io temp = io.TextIOWrapper(f, encoding='UTF-8') @@ -192,6 +216,10 @@ def load(f, **_options): key = int(key) db.value_tables.setdefault(val_tab_name, {})[key] = val + if "ecus" in json_data: + for ecu in json_data["ecus"]: + new_ecu = canmatrix.Ecu(name=ecu, comment=json_data["ecus"][ecu]) + db.add_ecu(new_ecu) if "messages" in json_data: for frame in json_data["messages"]: # new_frame = Frame(frame["id"],frame["name"],8,None) @@ -199,9 +227,25 @@ def load(f, **_options): new_frame = canmatrix.Frame(frame["name"], arbitration_id=arb_id, size=8) if "length" in frame: new_frame.size = frame["length"] + simple_mapping = ["is_complex_multiplexed", "mux_names", "cycle_time", "is_j1939", "header_id", "pdu_name"] + for key in simple_mapping: + if key in frame: + if key == "is_complex_multiplexed": + new_frame.is_complex_multiplexed = frame[key] + elif key == "mux_names": + new_frame.mux_names = frame[key] + elif key == "cycle_time": + new_frame.cycle_time = frame[key] + elif key == "is_j1939": + new_frame.is_j1939 = frame[key] + elif key == "header_id": + new_frame.header_id = frame[key] + elif key == "pdu_name": + new_frame.pdu_name = frame[key] new_frame.arbitration_id.extended = frame.get("is_extended_frame", False) - + if "transmitters" in frame: + new_frame.transmitters = frame["transmitters"] for signal in frame["signals"]: is_little_endian = not signal.get("is_big_endian", False) is_float = signal.get("is_float", False) @@ -233,10 +277,48 @@ def load(f, **_options): if signal.get("values", False): for key in signal["values"]: new_signal.add_values(key, signal["values"][key]) + + if "comment" in signal: + new_signal.comment = signal["comment"] + + if "comments" in signal: + new_signal.comments = signal["comments"] + + if "initial_value" in signal: + new_signal.initial_value = decimal.Decimal(signal["initial_value"]) + + if signal.get("receivers", False): + for ecu in signal["receivers"]: + new_signal.add_receiver(ecu) if new_signal.is_little_endian is False: new_signal.set_startbit( new_signal.start_bit, bitNumbering=1, startLittle=True) new_frame.add_signal(new_signal) db.add_frame(new_frame) + + _define_list = {"signal_defines": db.add_signal_defines, "frame_defines": db.add_frame_defines, + "global_defines": db.add_global_defines, "env_defines": db.add_env_defines, + "ecu_defines": db.add_ecu_defines} + for define_type, fptr in _define_list.items(): + if define_type in json_data: + for define in json_data[define_type]: + fptr(define['name'], define['define']) + + cm_import_list_dict = {'attributes': db.attributes, 'value_tables': db.value_tables, 'env_vars': db.env_vars} + + cm_import_list_val = ['baudrate', 'fd_baudrate'] + + for key in cm_import_list_dict: + if key in json_data: + cm_import_list_dict[key].update(json_data[key]) + + for key in cm_import_list_val: + if key in json_data: + if key == 'baudrate': + db.baudrate = json_data[key] + elif key == 'fd_baudrate': + db.fd_baudrate = json_data[key] + f.close() + db.update_ecu_list() return db From cd100323ca1da9f9fca85218f2f163592b45fc76 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Mon, 13 Feb 2023 13:39:21 +0100 Subject: [PATCH 29/46] add feature to compress frame (fill gaps between signals) --- docs/cli.rst | 16 +++++++++ src/canmatrix/canmatrix.py | 48 +++++++++++++++++++++++++++ src/canmatrix/cli/convert.py | 1 + src/canmatrix/convert.py | 8 +++++ src/canmatrix/tests/test_canmatrix.py | 24 ++++++++++++++ 5 files changed, 97 insertions(+) diff --git a/docs/cli.rst b/docs/cli.rst index aa67c528..27952d46 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -193,6 +193,18 @@ this will remove frames ``myFrame`` and ``myFrame2`` in ``source.dbc`` and store this will load ``source.dbc`` and rename frames ``myFrame`` in ``myNewFrame`` and ``myFrame2`` in ``myNewFrame2``. The result is stored in ``target.dlc``. +**compress Frame:** + +:: + + $ canconvert --compressFrame=myFrame,myFrame2,someFrames* source.dbc target.dbc + +this will load ``source.dbc`` and compress frames ``myFrame`` in ``myFrame2`` and all frames starting with ``someFrames``. +compress means, it tries to fill gaps between signals. +Works only for frames which have only big_endian signals or frames which have only little_endian singals +Frame name could be * which will compress all frames +The result is stored in ``target.dlc``. + **delete Signal:** @@ -365,6 +377,10 @@ ____________________ rename Frame form databases. (comma separated list) Syntax: --renameFrame=myOldFrame:myNewFrame,mySecondFrame:mySecondNewFrame + --compressFrame=FRAME + + compress Frame form databases. Syntax: --renameFrame=frame1,frame2,* + --deleteSignal=DELETESIGNAL delete Signal form databases. (comma separated list) Syntax: --deleteSignal=mySignal1,mySecondSignal diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index f4a18b3e..049e3dbe 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -1546,6 +1546,54 @@ def decode(self, data): else: return decoded + + def _compress_little(self): + for signal in self.signals: + if not signal.is_little_endian: + return + gap_found = True + while gap_found: + gap_found = False + layout = self.get_frame_layout() + gap_len = None + for byte in range(len(layout)//8): + for bit in range(7,-1,-1): + bit_nr = byte*8+bit + signal_list = layout[bit_nr] + if signal_list == []: + if gap_len is None: + gap_len = 1 + else: + gap_len += 1 + else: + if gap_len is not None: + signal = layout[bit_nr][0] + signal.start_bit -= gap_len + gap_found = True + break + if gap_found: + break + + def compress(self): + for signal in self.signals: + if signal.is_little_endian: + return self._compress_little() + gap_found = True + while gap_found: + gap_found = False + layout = self.get_frame_layout() + free_start = None + for bit_nr, signal_list in enumerate(layout): + if signal_list == []: + if free_start is None: + free_start = bit_nr + else: + if free_start is not None: + signal = layout[bit_nr][0] + signal.start_bit = free_start + gap_found = True + break + def __str__(self): # type: () -> str """Represent the frame by its name only.""" diff --git a/src/canmatrix/cli/convert.py b/src/canmatrix/cli/convert.py index 3878b485..e8caa851 100644 --- a/src/canmatrix/cli/convert.py +++ b/src/canmatrix/cli/convert.py @@ -65,6 +65,7 @@ def get_formats(): @click.option('--deleteSignalAttributes', 'deleteSignalAttributes', help="delete attributes from all signals\nExample --deleteSignalAttributes GenMsgSomeVar,CycleTime") @click.option('--deleteFrame', 'deleteFrame', help="delete Frame form databases. (comma separated list)\nSyntax: --deleteFrame=myFrame1,mySecondFrame") @click.option('--renameFrame', 'renameFrame', help="increment each frame.id in database by increment\nSyntax: --frameIdIncrement=increment") +@click.option('--compressFrame', 'compressFrame', help="remove gaps between signals in Frame. (comma separated list of Frames)\nSyntax: --compressFrame=myFrame1,mySecondFrame") @click.option('--addFrameReceiver', 'addFrameReceiver', help="add receiver Ecu to frame(s) (comma separated list)\nSyntax: --addFrameReceiver=framename:myNewEcu,mySecondEcu:myNEWEcu") @click.option('--changeFrameId', 'changeFrameId', help="change frame.id in database\nSyntax: --changeFrameId=oldId:newId") @click.option('--setFrameFd', 'setFrameFd', help="set Frame from database to canfd. (comma separated list)\nSyntax: --setFrameFd=myFrame1,mySecondFrame") diff --git a/src/canmatrix/convert.py b/src/canmatrix/convert.py index cdc3f014..f265e99d 100644 --- a/src/canmatrix/convert.py +++ b/src/canmatrix/convert.py @@ -132,6 +132,14 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non for renameTuple in rename_tuples: old, new = renameTuple.split(':') db.rename_frame(old, new) + + if 'compressFrame' in options and options['compressFrame'] is not None: + frames_cmdline = options['compressFrame'].split(',') + for frame_name in frames_cmdline: + frames = db.glob_frames(frame_name) + for frame in frames: + frame.compress() + if 'deleteFrame' in options and options['deleteFrame'] is not None: delete_frame_names = options['deleteFrame'].split(',') for frame_name in delete_frame_names: diff --git a/src/canmatrix/tests/test_canmatrix.py b/src/canmatrix/tests/test_canmatrix.py index 49b954e4..3c1e6330 100644 --- a/src/canmatrix/tests/test_canmatrix.py +++ b/src/canmatrix/tests/test_canmatrix.py @@ -1092,3 +1092,27 @@ def test_baudrate(): assert cm.baudrate == 500000 cm.fd_baudrate = 1000000 assert cm.fd_baudrate == 1000000 + +def test_frame_compress(): + frame = canmatrix.Frame("my_frame", size=8) + frame.add_signal(canmatrix.Signal(name = "Sig1", start_bit = 2, size = 13, is_little_endian=False )) + frame.add_signal(canmatrix.Signal(name = "Sig2", start_bit = 17, size = 14, is_little_endian=False)) + frame.add_signal(canmatrix.Signal(name = "Sig3", start_bit = 35, size = 6, is_little_endian=False)) + frame.add_signal(canmatrix.Signal(name = "Sig4", start_bit = 49, size = 8, is_little_endian=False)) + frame.compress() + assert frame.signal_by_name("Sig1").start_bit == 0 + assert frame.signal_by_name("Sig2").start_bit == 13 + assert frame.signal_by_name("Sig3").start_bit == 27 + assert frame.signal_by_name("Sig4").start_bit == 33 + + frame = canmatrix.Frame("my_frame", size=8) + # some signals overlap! + frame.add_signal(canmatrix.Signal(name = "Sig1", start_bit = 12, size = 12, is_little_endian=True)) + frame.add_signal(canmatrix.Signal(name = "Sig2", start_bit = 17, size = 9, is_little_endian=True)) + frame.add_signal(canmatrix.Signal(name = "Sig3", start_bit = 33, size = 5, is_little_endian=True)) + frame.add_signal(canmatrix.Signal(name = "Sig4", start_bit = 48, size = 9, is_little_endian=True)) + frame.compress() + assert frame.signal_by_name("Sig1").start_bit == 0 + assert frame.signal_by_name("Sig2").start_bit == 12 + assert frame.signal_by_name("Sig3").start_bit == 21 + assert frame.signal_by_name("Sig4").start_bit == 26 From 58c45c2053475854a8f9adce15aefb32b0a6bf22 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 14 Feb 2023 18:53:56 +0100 Subject: [PATCH 30/46] do frame-compressing after deleting and before recalc-dlc --- src/canmatrix/convert.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/canmatrix/convert.py b/src/canmatrix/convert.py index f265e99d..07966775 100644 --- a/src/canmatrix/convert.py +++ b/src/canmatrix/convert.py @@ -132,13 +132,6 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non for renameTuple in rename_tuples: old, new = renameTuple.split(':') db.rename_frame(old, new) - - if 'compressFrame' in options and options['compressFrame'] is not None: - frames_cmdline = options['compressFrame'].split(',') - for frame_name in frames_cmdline: - frames = db.glob_frames(frame_name) - for frame in frames: - frame.compress() if 'deleteFrame' in options and options['deleteFrame'] is not None: delete_frame_names = options['deleteFrame'].split(',') @@ -235,6 +228,13 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non 'deleteObsoleteEcus']: db.delete_obsolete_ecus() + if 'compressFrame' in options and options['compressFrame'] is not None: + frames_cmdline = options['compressFrame'].split(',') + for frame_name in frames_cmdline: + frames = db.glob_frames(frame_name) + for frame in frames: + frame.compress() + if 'recalcDLC' in options and options['recalcDLC']: db.recalc_dlc(options['recalcDLC']) From 4266de8a226bf04d0d70642ebd030e5166491ee7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Sun, 26 Feb 2023 20:27:09 +0100 Subject: [PATCH 31/46] add feature to compress frame (#687) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * add feature to compress frame (fill gaps between signals) * do frame-compressing after deleting and before recalc-dlc --------- Co-authored-by: Eduard Bröcker --- docs/cli.rst | 16 +++++++++ src/canmatrix/canmatrix.py | 48 +++++++++++++++++++++++++++ src/canmatrix/cli/convert.py | 1 + src/canmatrix/convert.py | 8 +++++ src/canmatrix/tests/test_canmatrix.py | 24 ++++++++++++++ 5 files changed, 97 insertions(+) diff --git a/docs/cli.rst b/docs/cli.rst index aa67c528..27952d46 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -193,6 +193,18 @@ this will remove frames ``myFrame`` and ``myFrame2`` in ``source.dbc`` and store this will load ``source.dbc`` and rename frames ``myFrame`` in ``myNewFrame`` and ``myFrame2`` in ``myNewFrame2``. The result is stored in ``target.dlc``. +**compress Frame:** + +:: + + $ canconvert --compressFrame=myFrame,myFrame2,someFrames* source.dbc target.dbc + +this will load ``source.dbc`` and compress frames ``myFrame`` in ``myFrame2`` and all frames starting with ``someFrames``. +compress means, it tries to fill gaps between signals. +Works only for frames which have only big_endian signals or frames which have only little_endian singals +Frame name could be * which will compress all frames +The result is stored in ``target.dlc``. + **delete Signal:** @@ -365,6 +377,10 @@ ____________________ rename Frame form databases. (comma separated list) Syntax: --renameFrame=myOldFrame:myNewFrame,mySecondFrame:mySecondNewFrame + --compressFrame=FRAME + + compress Frame form databases. Syntax: --renameFrame=frame1,frame2,* + --deleteSignal=DELETESIGNAL delete Signal form databases. (comma separated list) Syntax: --deleteSignal=mySignal1,mySecondSignal diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index f4a18b3e..049e3dbe 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -1546,6 +1546,54 @@ def decode(self, data): else: return decoded + + def _compress_little(self): + for signal in self.signals: + if not signal.is_little_endian: + return + gap_found = True + while gap_found: + gap_found = False + layout = self.get_frame_layout() + gap_len = None + for byte in range(len(layout)//8): + for bit in range(7,-1,-1): + bit_nr = byte*8+bit + signal_list = layout[bit_nr] + if signal_list == []: + if gap_len is None: + gap_len = 1 + else: + gap_len += 1 + else: + if gap_len is not None: + signal = layout[bit_nr][0] + signal.start_bit -= gap_len + gap_found = True + break + if gap_found: + break + + def compress(self): + for signal in self.signals: + if signal.is_little_endian: + return self._compress_little() + gap_found = True + while gap_found: + gap_found = False + layout = self.get_frame_layout() + free_start = None + for bit_nr, signal_list in enumerate(layout): + if signal_list == []: + if free_start is None: + free_start = bit_nr + else: + if free_start is not None: + signal = layout[bit_nr][0] + signal.start_bit = free_start + gap_found = True + break + def __str__(self): # type: () -> str """Represent the frame by its name only.""" diff --git a/src/canmatrix/cli/convert.py b/src/canmatrix/cli/convert.py index 3878b485..e8caa851 100644 --- a/src/canmatrix/cli/convert.py +++ b/src/canmatrix/cli/convert.py @@ -65,6 +65,7 @@ def get_formats(): @click.option('--deleteSignalAttributes', 'deleteSignalAttributes', help="delete attributes from all signals\nExample --deleteSignalAttributes GenMsgSomeVar,CycleTime") @click.option('--deleteFrame', 'deleteFrame', help="delete Frame form databases. (comma separated list)\nSyntax: --deleteFrame=myFrame1,mySecondFrame") @click.option('--renameFrame', 'renameFrame', help="increment each frame.id in database by increment\nSyntax: --frameIdIncrement=increment") +@click.option('--compressFrame', 'compressFrame', help="remove gaps between signals in Frame. (comma separated list of Frames)\nSyntax: --compressFrame=myFrame1,mySecondFrame") @click.option('--addFrameReceiver', 'addFrameReceiver', help="add receiver Ecu to frame(s) (comma separated list)\nSyntax: --addFrameReceiver=framename:myNewEcu,mySecondEcu:myNEWEcu") @click.option('--changeFrameId', 'changeFrameId', help="change frame.id in database\nSyntax: --changeFrameId=oldId:newId") @click.option('--setFrameFd', 'setFrameFd', help="set Frame from database to canfd. (comma separated list)\nSyntax: --setFrameFd=myFrame1,mySecondFrame") diff --git a/src/canmatrix/convert.py b/src/canmatrix/convert.py index cdc3f014..07966775 100644 --- a/src/canmatrix/convert.py +++ b/src/canmatrix/convert.py @@ -132,6 +132,7 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non for renameTuple in rename_tuples: old, new = renameTuple.split(':') db.rename_frame(old, new) + if 'deleteFrame' in options and options['deleteFrame'] is not None: delete_frame_names = options['deleteFrame'].split(',') for frame_name in delete_frame_names: @@ -227,6 +228,13 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non 'deleteObsoleteEcus']: db.delete_obsolete_ecus() + if 'compressFrame' in options and options['compressFrame'] is not None: + frames_cmdline = options['compressFrame'].split(',') + for frame_name in frames_cmdline: + frames = db.glob_frames(frame_name) + for frame in frames: + frame.compress() + if 'recalcDLC' in options and options['recalcDLC']: db.recalc_dlc(options['recalcDLC']) diff --git a/src/canmatrix/tests/test_canmatrix.py b/src/canmatrix/tests/test_canmatrix.py index 49b954e4..3c1e6330 100644 --- a/src/canmatrix/tests/test_canmatrix.py +++ b/src/canmatrix/tests/test_canmatrix.py @@ -1092,3 +1092,27 @@ def test_baudrate(): assert cm.baudrate == 500000 cm.fd_baudrate = 1000000 assert cm.fd_baudrate == 1000000 + +def test_frame_compress(): + frame = canmatrix.Frame("my_frame", size=8) + frame.add_signal(canmatrix.Signal(name = "Sig1", start_bit = 2, size = 13, is_little_endian=False )) + frame.add_signal(canmatrix.Signal(name = "Sig2", start_bit = 17, size = 14, is_little_endian=False)) + frame.add_signal(canmatrix.Signal(name = "Sig3", start_bit = 35, size = 6, is_little_endian=False)) + frame.add_signal(canmatrix.Signal(name = "Sig4", start_bit = 49, size = 8, is_little_endian=False)) + frame.compress() + assert frame.signal_by_name("Sig1").start_bit == 0 + assert frame.signal_by_name("Sig2").start_bit == 13 + assert frame.signal_by_name("Sig3").start_bit == 27 + assert frame.signal_by_name("Sig4").start_bit == 33 + + frame = canmatrix.Frame("my_frame", size=8) + # some signals overlap! + frame.add_signal(canmatrix.Signal(name = "Sig1", start_bit = 12, size = 12, is_little_endian=True)) + frame.add_signal(canmatrix.Signal(name = "Sig2", start_bit = 17, size = 9, is_little_endian=True)) + frame.add_signal(canmatrix.Signal(name = "Sig3", start_bit = 33, size = 5, is_little_endian=True)) + frame.add_signal(canmatrix.Signal(name = "Sig4", start_bit = 48, size = 9, is_little_endian=True)) + frame.compress() + assert frame.signal_by_name("Sig1").start_bit == 0 + assert frame.signal_by_name("Sig2").start_bit == 12 + assert frame.signal_by_name("Sig3").start_bit == 21 + assert frame.signal_by_name("Sig4").start_bit == 26 From f441aed2affa021092dd5bcc66344f1e1a25c521 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Sun, 26 Feb 2023 20:49:46 +0100 Subject: [PATCH 32/46] fix docu --- docs/cli.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli.rst b/docs/cli.rst index 27952d46..9b547e63 100644 --- a/docs/cli.rst +++ b/docs/cli.rst @@ -379,7 +379,7 @@ ____________________ --compressFrame=FRAME - compress Frame form databases. Syntax: --renameFrame=frame1,frame2,* + compress Frame form databases. Syntax: --compressFrame=frame1,frame2,* --deleteSignal=DELETESIGNAL From 6bda8ab7c1d9bec0ef528d9b7fbdca0240e5ee17 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Sun, 26 Feb 2023 21:17:17 +0100 Subject: [PATCH 33/46] fix json test --- src/canmatrix/tests/test_json.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/tests/test_json.py b/src/canmatrix/tests/test_json.py index 8b43db92..7a4e7a1b 100644 --- a/src/canmatrix/tests/test_json.py +++ b/src/canmatrix/tests/test_json.py @@ -194,7 +194,7 @@ def test_import_export_enums(): f_in = io.BytesIO(f_out.getvalue()) new_matrix = canmatrix.formats.json.load(f_in) - assert new_matrix.value_tables == {"Options": {0: "North", 1: "East", 2: "South", 3: "West"}} + assert new_matrix.value_tables == {"Options": {"0": "North", "1": "East", "2": "South", "3": "West"}} def test_export_native(): From 6b2ec9f1aa2ea32c7499fdceb607cd9251bb3f70 Mon Sep 17 00:00:00 2001 From: Russ Date: Thu, 2 Mar 2023 21:32:45 +1100 Subject: [PATCH 34/46] fix: load json attributes from each message (#691) * fix: load attributes from each message * fix: spelling error in exception --- src/canmatrix/canmatrix.py | 18 +++++++++--------- src/canmatrix/formats/json.py | 4 ++++ 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 049e3dbe..5daeb19d 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -63,7 +63,7 @@ class MissingMuxSignal(ExceptionTemplate): pass class DecodingComplexMultiplexed(ExceptionTemplate): pass class DecodingFrameLength(ExceptionTemplate): pass class ArbitrationIdOutOfRange(ExceptionTemplate): pass -class J1939needsExtendedIdetifier(ExceptionTemplate): pass +class J1939NeedsExtendedIdentifier(ExceptionTemplate): pass class DecodingConatainerPdu(ExceptionTemplate): pass class EncodingConatainerPdu(ExceptionTemplate): pass @@ -643,7 +643,7 @@ def j1939_pgn(self): @property def pgn(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier # PGN is bits 8-25 of the 29-Bit Extended CAN-ID # Made up of PDU-S (8-15), PDU-F (16-23), Data Page (24) & Extended Data Page (25) # If PDU-F >= 240 the PDU-S is interpreted as Group Extension @@ -677,7 +677,7 @@ def j1939_tuple(self): # type: () -> typing.Tuple[int, int, int] @property def j1939_destination(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier if self.j1939_pdu_format == 1: destination = self.j1939_ps else: @@ -687,7 +687,7 @@ def j1939_destination(self): @property def j1939_source(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier return self.id & 0xFF @j1939_source.setter @@ -698,13 +698,13 @@ def j1939_source(self, value): # type: (int) -> None @property def j1939_ps(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier return (self.id >> 8) & 0xFF @property def j1939_pf(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier return (self.id >> 16) & 0xFF @property @@ -714,19 +714,19 @@ def j1939_pdu_format(self): @property def j1939_dp(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier return (self.id >> 24) & 0x1 @property def j1939_edp(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier return (self.id >> 25) & 0x1 @property def j1939_priority(self): if not self.extended: - raise J1939needsExtendedIdetifier + raise J1939NeedsExtendedIdentifier return (self.id >> 26) & 0x7 @j1939_priority.setter diff --git a/src/canmatrix/formats/json.py b/src/canmatrix/formats/json.py index 3afe17a4..bffc4330 100644 --- a/src/canmatrix/formats/json.py +++ b/src/canmatrix/formats/json.py @@ -278,6 +278,10 @@ def load(f, **_options): for key in signal["values"]: new_signal.add_values(key, signal["values"][key]) + if signal.get("attributes", False): + for key in signal["attributes"]: + new_signal.add_attribute(key, signal["attributes"][key]) + if "comment" in signal: new_signal.comment = signal["comment"] From 441019a9494ea5664ae49d89c7d82d6ab2f20dc9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Sat, 4 Mar 2023 17:47:44 +0100 Subject: [PATCH 35/46] quick and dirty fibex read implementation (#682) * quick and dirty fibex read implementation * fix startbit handling * recognize FD Frames --- src/canmatrix/formats/fibex.py | 230 ++++++++++++++++++++++++++++++++- 1 file changed, 227 insertions(+), 3 deletions(-) diff --git a/src/canmatrix/formats/fibex.py b/src/canmatrix/formats/fibex.py index 2ddc6b9b..56c6cae3 100644 --- a/src/canmatrix/formats/fibex.py +++ b/src/canmatrix/formats/fibex.py @@ -29,10 +29,15 @@ import typing from builtins import * - import lxml.etree - +import logging import canmatrix +import re +import decimal + +clusterImporter = 1 + +logger = logging.getLogger(__name__) fx = "http://www.asam.net/xml/fbx" ho = "http://www.asam.net/xml" @@ -73,6 +78,226 @@ def create_sub_element_ho(parent, element_name, element_text=None): return new +class Fe: + def __init__(self, filename): + self.tree = lxml.etree.parse(filename) + self.root = self.tree.getroot() # type: _Element + + self.ns = "{" + self.tree.xpath('namespace-uri(.)') + "}" # type: str + self.nsp = self.tree.xpath('namespace-uri(.)') + self.ans = "{http://www.asam.net/xml}" + self._id_cache = {a.attrib["ID"]:a for a in self.root.xpath(".//*[@ID]")} + self._id_rev_cache = {} + for referencer in self.root.xpath(".//*[@ID-REF]"): + ref_id = referencer.attrib["ID-REF"] + if ref_id not in self._id_rev_cache: + self._id_rev_cache[ref_id] = [referencer] + else: + self._id_rev_cache[ref_id].append(referencer) + + def sn(self, tag): + sn = tag.find("./" + self.ans + "SHORT-NAME").text + return sn + + def get_referencable_parent(self, xml_element): + path = "" + while xml_element != self.root: + try: + current_short_name = self.sn(xml_element) + return xml_element + except AttributeError: + pass + xml_element = xml_element.getparent() + return xml_element + + def find_parent(self, start_element, parent_tag): + while start_element != self.root: + if start_element.tag == self.ns + parent_tag or start_element == self.ans + parent_tag: + return start_element + start_element = start_element.getparent() + return None + + def get_desc_or_longname(self, element): + long_name_elements = self.selector(element, "./!LONG-NAME") + if len(long_name_elements) > 0: + return long_name_elements[0].text + + desc_elements = self.selector(element, "./!DESC") + if len(desc_elements) > 0: + return desc_elements[0].text + + return "" + + def selector(self, start_element, selector): + start_pos = 0 + token = "" + result_list = [start_element] + last_found_token = 0 + while start_pos < len(selector): + token_match = re.search(r'//|/!|/|!|>>|>!|<|$', selector[start_pos:]) + found_token = token_match.span() + if start_pos > 0: # at least one Token found... + value = selector[last_found_token:start_pos + found_token[0]] + if token == "//": + result_list = [c for a in result_list for c in a.findall(".//" + self.ns + value)] + elif token == "/!": + result_list = [c for a in result_list for c in a.findall(".//" + self.ans + value)] + elif token == "/": + if value == "..": + result_list = [a.getparent() for a in result_list] + else: + result_list = [a.find("./" + self.ns + value) for a in result_list] + elif token == "!": + result_list = [a.find("./" + self.ans + value) for a in result_list] + elif token == ">": + start_points = [a.find("./" + self.ns + value).attrib["ID-REF"] for a in result_list] + result_list = [self._id_cache[a] for a in start_points] + elif token == "<<": + id_list = [a.attrib["ID"] for a in result_list] + result_list = [b for a in id_list for b in self._id_rev_cache[a]] + result_list = [a for a in result_list if self.get_referencable_parent(a) is not None and self.get_referencable_parent(a).tag.endswith(value)] + elif token == "<>": + start_points = [c.attrib["ID-REF"] for a in result_list for c in a.findall(".//" + self.ns + value)] + result_list = [self._id_cache[a] for a in start_points] + elif token == ">!": + start_points = [c.attrib["ID-REF"] for a in result_list for c in a.findall(".//" + self.ans + value)] + result_list = [self._id_cache[a] for a in start_points] + elif token == ">>": + start_points = [c.attrib["ID-REF"] for a in result_list for c in a.findall(".//" + self.ns + value)] + result_list = [self._id_cache[a] for a in start_points] + elif token == "^": + result_list = [self.find_parent(a, value) for a in result_list if a is not None and self.find_parent(a, value) is not None] + + result_list = [a for a in result_list if a is not None] + last_found_token = found_token[1] + start_pos + token = selector[start_pos + found_token[0]:start_pos + found_token[1]] + start_pos += found_token[1] + return sorted(result_list, key=lambda element: element.sourceline) + + +def load(f, **_options): + fe = Fe(f) + result = {} + + clusters = fe.selector(fe.root, "//CLUSTER") + names = [fe.sn(a) for a in clusters] + logger.info("Found clusters: " + ",".join(names)) + + for cluster in clusters: + if "CAN" not in fe.selector(cluster, "//PROTOCOL")[0].text: + logger.info(fe.sn(cluster) + " seems not to be a CAN cluster - ignoring") + continue + + db = canmatrix.CanMatrix() + result[fe.sn(cluster)] = db + channels = fe.selector(cluster, ">>CHANNEL-REF") + for channel in channels: + for ft in fe.selector(channel, "//FRAME-TRIGGERING"): + ports = fe.selector(ft, "<FRAME-REF")[0] + frame_size = int(fe.selector(frame_element, "/BYTE-LENGTH")[0].text) + pdu = fe.selector(frame_element, ">>PDU-REF")[0] + + frame_name = fe.sn(pdu) +# fe.selector(pdu, "< 0: + pdu_triggerings = fe.selector(output_ports[0], ">>PDU-TRIGGERING-REF") + if len(pdu_triggerings) > 0: + cyclic_timing_element = fe.selector(pdu_triggerings[0], + "/TIMINGS/CYCLIC-TIMING/REPEATING-TIME-RANGE/VALUE") + if len(cyclic_timing_element) > 0: + time_value_string = cyclic_timing_element[0].text + if time_value_string.startswith("PT") and time_value_string.endswith("S"): + frame.cycle_time = decimal.Decimal(time_value_string[2:-1])*1000 + frame.transmitters = [fe.sn(a) for a in sending_ecus] + for ecu_element in sending_ecus: + ecu_name = fe.sn(ecu_element) + cm_ecu = canmatrix.Ecu(ecu_name) + cm_ecu.add_comment(fe.get_desc_or_longname(ecu_element)) + db.add_ecu(cm_ecu) + frame.arbitration_id = canmatrix.ArbitrationId(extended=extended, id=arbitration_id) + + frame.add_comment(fe.get_desc_or_longname(pdu)) + if "CAN-FD" in [a.text for a in + fe.selector(ft, "//CAN-FRAME-TX-BEHAVIOR") + fe.selector(ft, "//CAN-FRAME-RX-BEHAVIOR")]: + frame.is_fd = True + for signal_instance in fe.selector(pdu, "//SIGNAL-INSTANCE"): + byte_order_element = fe.selector(signal_instance, "/IS-HIGH-LOW-BYTE-ORDER") + if byte_order_element[0].text == "false": + is_little_endian = True + else: + is_little_endian = False + + start_bit = int(fe.selector(signal_instance, "/BIT-POSITION")[0].text, 0) + signal = fe.selector(signal_instance, ">SIGNAL-REF")[0] + ecu_instance_refs = fe.selector(signal_instance, "< 0: + ecu_name = fe.sn(fe.get_referencable_parent(ecu_instance_ref)) + receiver_ecus.append(ecu_name) + db.add_ecu(canmatrix.Ecu(ecu_name)) + + signal_name = fe.sn(signal) + coding = fe.selector(signal, ">CODING-REF")[0] + bit_length = int(fe.selector(coding, "/!BIT-LENGTH")[0].text) + compu_methods = fe.selector(coding, "/!COMPU-METHOD") + sig = canmatrix.Signal(name=signal_name) + + for compu_method in compu_methods: + category = fe.selector(compu_method, "/!CATEGORY") + if len(category) > 0 and category[0].text == "LINEAR": + numerator = fe.selector(compu_method, "/!COMPU-NUMERATOR")[0] + denominator = fe.selector(compu_method, "/!COMPU-DENOMINATOR")[0] + teiler = decimal.Decimal(fe.selector(denominator, "/!V")[0].text) + [offset, factor] = [decimal.Decimal(a.text) for a in fe.selector(numerator, "/!V")] + [offset, factor] = [a / teiler for a in [offset, factor]] + sig.offset = offset + sig.factor = factor + try: + sig.min=decimal.Decimal(fe.selector(compu_method, "!PHYS-CONSTRS!LOWER-LIMIT")[0].text) + sig.max = decimal.Decimal(fe.selector(compu_method, "!PHYS-CONSTRS!UPPER-LIMIT")[0].text) + except: + pass + unit = fe.selector(compu_method, ">!UNIT-REF") + if len(unit) > 0: + try: + sig.unit = fe.selector(unit[0],"!DISPLAY-NAME")[0].text + except: + pass + elif len(category) > 0 and category[0].text == "TEXTTABLE": + for compu_scale in fe.selector(compu_method, "/!COMPU-SCALE"): + try: + value_name = fe.selector(compu_scale, "!COMPU-CONST!VT")[0].text + except IndexError: + value_name = fe.get_desc_or_longname(compu_scale) + value_value = fe.selector(compu_scale, "!LOWER-LIMIT")[0].text + sig.add_values(value_value, value_name) + sig.is_little_endian = is_little_endian + if not sig.is_little_endian: + sig.set_startbit(start_bit, bitNumbering=1) + else: + sig.start_bit = start_bit + sig.size = bit_length + sig.receivers = list(set(receiver_ecus)) + + sig.add_comment(fe.get_desc_or_longname(signal)) + + frame.add_signal(sig) + db.add_frame(frame) + return result + + def dump(db, f, **options): # type: (canmatrix.CanMatrix, typing.IO, **typing.Any) -> None ns_map = {"fx": fx, "ho": ho, "can": can, "xsi": xsi} @@ -135,7 +360,6 @@ def dump(db, f, **options): pdu_ref = create_sub_element_fx(pdu_triggering, "PDU-REF") pdu_ref.set("ID-REF", "PDU_" + pdu.name) - frame_triggerings = create_sub_element_fx(channel, "FRAME-TRIGGERINGS") for frame in db.frames: frame_triggering = create_sub_element_fx( From b5abc6982ee1b1e52b395b5d93039df73c379ac2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Wed, 12 Apr 2023 22:21:38 +0200 Subject: [PATCH 36/46] correct is to == --- src/canmatrix/convert.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/convert.py b/src/canmatrix/convert.py index 07966775..7e329b87 100644 --- a/src/canmatrix/convert.py +++ b/src/canmatrix/convert.py @@ -308,7 +308,7 @@ def convert(infile, out_file_name, **options): # type: (str, str, **str) -> Non # Check & Warn for Frame/Messages without Transmitter Node if options.get('checkFloatingFrames') is not None and options['checkFloatingFrames']: for frame in db.frames: - if len(frame.transmitters) is 0: + if len(frame.transmitters) == 0: logger.warning("No Transmitter Node Found for Frame %s", frame.name) # Check & Warn for Signals with Min/Max set to 0 From 698c36be9c62a68d3062daa2d44369199a46e50c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 25 Apr 2023 16:36:40 +0200 Subject: [PATCH 37/46] implement pdu routing info for arxml-clusters (#692) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * implement pdu routing info for arxml-clusters Co-authored-by: Eduard Bröcker --- src/canmatrix/cancluster.py | 44 ++++++++++ src/canmatrix/formats/arxml.py | 93 ++++++++++++++++++++- src/canmatrix/tests/test_arxml_gw.py | 116 +++++++++++++++++++++++++++ 3 files changed, 252 insertions(+), 1 deletion(-) create mode 100644 src/canmatrix/tests/test_arxml_gw.py diff --git a/src/canmatrix/cancluster.py b/src/canmatrix/cancluster.py index df5e517b..54155bff 100644 --- a/src/canmatrix/cancluster.py +++ b/src/canmatrix/cancluster.py @@ -14,6 +14,8 @@ def __init__(self, *arg, **kw): self._frames = [] # type: typing.List[canmatrix.Frame] self._signals = [] # type: typing.List[canmatrix.Signal] self._ecus = [] # type: typing.List[canmatrix.Ecu] + self._pdu_gateway_list = [] # type: typing.List[dict[str, str]] + self._signal_gateway_list = [] # type: typing.List[dict[str, str]] self.update() def update_frames(self): # type: () -> typing.MutableSequence[canmatrix.Frame] @@ -82,3 +84,45 @@ def signals(self): # type: () -> typing.MutableSequence[canmatrix.Signal] if not self._signals: self.update_signals() return self._signals + + def pdu_gateway(self, pdu_gateway_list=[]): + self._pdu_gateway_list += pdu_gateway_list + return self._pdu_gateway_list + + def signal_gateway(self, signal_gateway_list=[]): + self._signal_gateway_list += signal_gateway_list + return self._signal_gateway_list + + def get_pdu_routing_info(self, pdu_name, strict_search=False): + routing_source = [] + routing_target = [] + if strict_search: + for pdu in self._pdu_gateway_list: + if pdu_name == pdu["source"]: + routing_source.append({"pdu": pdu["target"], "cluster": pdu["target_cluster"], "ecu": pdu["ecu"], "type": pdu["target_type"]}) + if pdu_name == pdu["target"]: + routing_target.append({"pdu": pdu["source"], "cluster": pdu["source_cluster"], "ecu": pdu["ecu"], "type": pdu["source_type"]}) + else: + for pdu in self._pdu_gateway_list: + if pdu_name in pdu["source"]: + routing_source.append({"pdu": pdu["target"], "cluster": pdu["target_cluster"], "ecu": pdu["ecu"], "type": pdu["target_type"]}) + if pdu_name in pdu["target"]: + routing_target.append({"pdu": pdu["source"], "cluster": pdu["source_cluster"], "ecu": pdu["ecu"], "type": pdu["source_type"]}) + return {"source": routing_source, "target": routing_target} + + def get_signal_routing_info(self, signal_name, strict_search=False): + routing_source = [] + routing_target = [] + if strict_search: + for signal_gw in self._signal_gateway_list: + if signal_name == signal_gw["source"]: + routing_source.append({"signal": signal_gw["target"], "cluster": signal_gw["target_cluster"], "ecu": signal_gw["ecu"], "type": signal_gw["target_type"]}) + if signal_name == signal_gw["target"]: + routing_target.append({"signal": signal_gw["source"], "cluster": signal_gw["source_cluster"], "ecu": signal_gw["ecu"], "type": signal_gw["source_type"]}) + else: + for signal_gw in self._signal_gateway_list: + if signal_name in signal_gw["source"]: + routing_source.append({"signal": signal_gw["target"], "cluster": signal_gw["target_cluster"], "ecu": signal_gw["ecu"], "type": signal_gw["target_type"]}) + if signal_name in signal_gw["target"]: + routing_target.append({"signal": signal_gw["source"], "cluster": signal_gw["source_cluster"], "ecu": signal_gw["ecu"], "type": signal_gw["source_type"]}) + return {"source": routing_source, "target": routing_target} diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index 546ceba5..fe666abf 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -35,6 +35,7 @@ import lxml.etree import canmatrix +import canmatrix.cancluster import canmatrix.types import canmatrix.utils import re @@ -127,6 +128,13 @@ def get_referencable_parent(self, xml_element): xml_element = xml_element.getparent() return xml_element + def get_parent_with_name(self, xml_element, tag_name): + while xml_element != self.root: + if xml_element.tag == self.ns + tag_name: + return xml_element + xml_element = xml_element.getparent() + return None + def find_references_of_type(self, element, xml_tag, referencable_parent=False): referencing_elements = [] current_ar_path = self.get_short_name_path_of_element(element) @@ -1860,7 +1868,6 @@ def decode_can_helper(ea, float_factory, ignore_cluster_info): db.add_frame_defines("PduName", 'STRING') db.add_frame_defines("GenMsgStartDelayTime", 'INT 0 65535') - db.add_frame_defines( "GenMsgSendType", 'ENUM "cyclicX","spontanX","cyclicIfActiveX","spontanWithDelay","cyclicAndSpontanX","cyclicAndSpontanWithDelay","spontanWithRepitition","cyclicIfActiveAndSpontanWD","cyclicIfActiveFast","cyclicWithRepeatOnDemand","none"') @@ -1997,4 +2004,88 @@ def load(file, **options): result.update(decode_can_helper(ea, float_factory, ignore_cluster_info)) + result = canmatrix.cancluster.CanCluster(result) + + def get_cluster_for_triggering(triggering): + while triggering != ea.root: + if triggering.tag.endswith("-CLUSTER"): + return triggering + triggering = triggering.getparent() + + def get_ecu_for_triggering(triggering): + phys_channel = None + while triggering != ea.root: + if triggering.tag.endswith("-CHANNEL"): + phys_channel = triggering + break + triggering = triggering.getparent() + + if phys_channel is not None: + ecu_name = "" + channel = ea.selector(phys_channel, ">COMMUNICATION-CONNECTOR-REF") + if len(channel) == 0: + return "" + ecu_instance = ea.get_parent_with_name(channel[0], "ECU-INSTANCE") + if ecu_instance is not None: + ecu_name = ea.get_short_name(ecu_instance) + return ecu_name + return "" + + + pdu_gateway_mappings = [] + for pdu_mapping in ea.selector(ea.root, "//I-PDU-MAPPING"): + source_triggering = ea.selector(pdu_mapping, ">SOURCE-I-PDU-REF") + target_triggering = ea.selector(pdu_mapping, ">TARGET-I-PDU-REF") + if len(source_triggering) == 0 or len(target_triggering) == 0: + continue + source_pdu = ea.selector(source_triggering[0], "/I-PDU-REF") + target_pdu = ea.selector(target_triggering[0], "/I-PDU-REF") + + if len(source_pdu) == 0 or len(target_pdu) == 0: + continue + source_cluster_name = ea.get_short_name_path_of_element(get_cluster_for_triggering(source_triggering[0])) + target_cluster_name = ea.get_short_name_path_of_element(get_cluster_for_triggering(target_triggering[0])) + ecu = get_ecu_for_triggering(source_triggering[0]) + source_type = "" + target_type = "" + if "DEST" in source_pdu[0].attrib: + source_type = source_pdu[0].attrib["DEST"] + if "DEST" in target_pdu[0].attrib: + target_type = target_pdu[0].attrib["DEST"] + + mapping_info = {"ecu": ecu, "source": source_pdu[0].text, "source_cluster": source_cluster_name, + "target": target_pdu[0].text, "target_cluster": target_cluster_name, + "target_type": target_type, "source_type": source_type} + + pdu_gateway_mappings.append(mapping_info) + + result.pdu_gateway(pdu_gateway_mappings) + + signal_gateway_mappings = [] + for signal_mapping in ea.selector(ea.root, "//I-SIGNAL-MAPPING"): + source_triggering = ea.selector(signal_mapping, ">SOURCE-SIGNAL-REF") + target_triggering = ea.selector(signal_mapping, ">TARGET-SIGNAL-REF") + if len(source_triggering) == 0 or len(target_triggering) == 0: + continue + source_signal = ea.selector(source_triggering[0], "/I-SIGNAL-REF") + target_signal = ea.selector(target_triggering[0], "/I-SIGNAL-REF") + if len(source_signal) == 0 or len(target_signal) == 0: + continue + + source_cluster_name = ea.get_short_name_path_of_element(get_cluster_for_triggering(source_triggering[0])) + target_cluster_name = ea.get_short_name_path_of_element(get_cluster_for_triggering(target_triggering[0])) + ecu = get_ecu_for_triggering(source_triggering[0]) + source_type = "" + target_type = "" + if "DEST" in source_signal[0].attrib: + source_type = source_signal[0].attrib["DEST"] + if "DEST" in target_signal[0].attrib: + target_type = source_signal[0].attrib["DEST"] + mapping_info = {"ecu": ecu, "source": source_signal[0].text, "source_cluster": source_cluster_name, + "target": target_signal[0].text, "target_cluster": target_cluster_name, + "source_type": source_type, "target_type": target_type} + + signal_gateway_mappings.append(mapping_info) + result.signal_gateway(signal_gateway_mappings) + return result diff --git a/src/canmatrix/tests/test_arxml_gw.py b/src/canmatrix/tests/test_arxml_gw.py new file mode 100644 index 00000000..8f3ee7af --- /dev/null +++ b/src/canmatrix/tests/test_arxml_gw.py @@ -0,0 +1,116 @@ +import canmatrix.formats.arxml +import textwrap +import io + + +def test_pdu_gateway(): + arxml = io.BytesIO(textwrap.dedent(u'''\ + + + + Cluster + + + someCluster + + + + + CHNL + + + SomeFrameTriggering + + + /Cluster/someCluster/CHNL/somePDUTriggering + + + + + + + someSignalTriggering + /ISignal/someSignal + + + + + somePduTriggering + /PDU/somePdu + + + + + + + + + + someOtherCluster + + + + + CHNL + + + SomeOtherFrameTriggering + + + /Cluster/someCluster/CHNL/someOtherPDUTrigering + + + + + + + someOtherSignalTriggering + /ISignal/someOtherSignal + + + + + someOtherPduTriggering + /PDU/someOtherPdu + + + + + + + + + + + + + Gateway + + + someECU + /ECU/someECU + + + /Cluster/someCluster/CHNL/somePduTriggering + + /Cluster/someOtherCluster/CHNL/someOtherPduTriggering + + + + + + /Cluster/someCluster/CHNL/someSignalTriggering + /Cluster/someOtherCluster/CHNL/someOtherSignalTriggering + + + + + + ''').encode('utf-8')) + + cluster = canmatrix.formats.arxml.load(arxml) + + assert cluster.get_pdu_routing_info("someOtherPdu")["target"][0]["pdu"] == "/PDU/somePdu" + assert cluster.get_pdu_routing_info("/PDU/somePdu", strict_search=True)["source"][0]["pdu"] == "/PDU/someOtherPdu" + assert cluster.get_signal_routing_info("someSignal")["source"][0]["signal"] == '/ISignal/someOtherSignal' + assert cluster.get_signal_routing_info("/ISignal/someOtherSignal")["target"][0]["signal"] == '/ISignal/someSignal' From 46d4de4841c306c936afc802f190040f619ea845 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 20 Jun 2023 15:24:02 +0200 Subject: [PATCH 38/46] fix issue #699 (#700) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Eduard Bröcker --- src/canmatrix/canmatrix.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 5daeb19d..e834dd83 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -37,6 +37,7 @@ import typing import warnings from builtins import * +import importlib.metadata import attr from past.builtins import basestring @@ -46,8 +47,9 @@ import canmatrix.types import canmatrix.utils -if attr.__version__ < '17.4.0': # type: ignore +if importlib.metadata.version("attrs") < '17.4.0': raise RuntimeError("need attrs >= 17.4.0") + logger = logging.getLogger(__name__) defaultFloatFactory = decimal.Decimal # type: typing.Callable[[typing.Any], canmatrix.types.PhysicalValue] @@ -801,6 +803,7 @@ def add_signal(self, signal): """ self.signals.append(signal) return self.signals[len(self.signals) - 1] + def add_signal_group(self, Name, Id, signalNames, e2e_trans=None): # type: (str, int, typing.Sequence[str]) -> None """Add new SignalGroup to the Frame. Add given signals to the group. From 1e0aae02ecc98afb5c85bde25ca952257196e8a7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 20 Jun 2023 15:27:08 +0200 Subject: [PATCH 39/46] bugfix parsing can fibex files (#701) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * bugfix parsing can fibex files --------- Co-authored-by: Eduard Bröcker --- src/canmatrix/formats/fibex.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/canmatrix/formats/fibex.py b/src/canmatrix/formats/fibex.py index 56c6cae3..3041e8df 100644 --- a/src/canmatrix/formats/fibex.py +++ b/src/canmatrix/formats/fibex.py @@ -215,10 +215,10 @@ def load(f, **_options): if len(pdu_triggerings) > 0: cyclic_timing_element = fe.selector(pdu_triggerings[0], "/TIMINGS/CYCLIC-TIMING/REPEATING-TIME-RANGE/VALUE") - if len(cyclic_timing_element) > 0: - time_value_string = cyclic_timing_element[0].text - if time_value_string.startswith("PT") and time_value_string.endswith("S"): - frame.cycle_time = decimal.Decimal(time_value_string[2:-1])*1000 + if len(cyclic_timing_element) > 0: + time_value_string = cyclic_timing_element[0].text + if time_value_string.startswith("PT") and time_value_string.endswith("S"): + frame.cycle_time = decimal.Decimal(time_value_string[2:-1])*1000 frame.transmitters = [fe.sn(a) for a in sending_ecus] for ecu_element in sending_ecus: ecu_name = fe.sn(ecu_element) From 5903fcddbece395ac6e285664d9fd526c3d8ec12 Mon Sep 17 00:00:00 2001 From: Luis Valencia Date: Sun, 25 Jun 2023 02:33:53 -0400 Subject: [PATCH 40/46] Fix importlib.metadata issue on Python 3.7. (#710) --- setup.py | 1 + src/canmatrix/canmatrix.py | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index 29e6edb5..16368ce9 100644 --- a/setup.py +++ b/setup.py @@ -84,6 +84,7 @@ "click", "enum34; python_version < '3.4'", "future", + "importlib-metadata; python_version < '3.8'", "six", "typing; python_version < '3.5'", ], diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index e834dd83..6cd78287 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -34,10 +34,10 @@ import logging import math import struct +import sys import typing import warnings from builtins import * -import importlib.metadata import attr from past.builtins import basestring @@ -47,7 +47,12 @@ import canmatrix.types import canmatrix.utils -if importlib.metadata.version("attrs") < '17.4.0': +if sys.version_info < (3, 8): + from importlib_metadata import version +else: + from importlib.metadata import version + +if version("attrs") < '17.4.0': raise RuntimeError("need attrs >= 17.4.0") logger = logging.getLogger(__name__) From 1713ee2b3b1d2a666723d04e5fe98424b707837d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Wed, 28 Jun 2023 10:27:08 +0200 Subject: [PATCH 41/46] possible fix for #705 (#706) --- src/canmatrix/canmatrix.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 6cd78287..08a02cb2 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -446,7 +446,7 @@ def phys2raw(self, value=None): "Value {} is not valid for {}. Min={} and Max={}".format( value, self, self.min, self.max) ) - raw_value = (value - self.float_factory(self.offset)) / self.float_factory(self.factor) + raw_value = (self.float_factory(value) - self.float_factory(self.offset)) / self.float_factory(self.factor) if not self.is_float: raw_value = int(round(raw_value)) From 3b4bd3196637829c48ff8a237c81a08685be27fd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Mon, 17 Jul 2023 22:40:53 +0200 Subject: [PATCH 42/46] try to fix ci (#715) fix ci --- appveyor.yml | 7 +------ tox.ini | 4 ++-- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/appveyor.yml b/appveyor.yml index 1cdeb6cf..f993c54e 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -7,15 +7,11 @@ skip_branch_with_pr: true environment: matrix: - - TOXENV: py27 - - TOXENV: py35 - - TOXENV: py36 - TOXENV: py37 - TOXENV: py38 - TOXENV: py39 - TOXENV: py310 - - TOXENV: pypy - - TOXENV: pypy3 + - TOXENV: py311 - TOXENV: dist matrix: @@ -28,7 +24,6 @@ init: - ps: if (Get-ChildItem Env:ENABLE_RDP -ErrorAction SilentlyContinue) {iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))} else {echo RDP not enabled} install: - - if "%TOXENV%"=="pypy" choco install python.pypy - if "%TOXENV%"=="pypy3" choco install python.pypy3 - py -m pip install tox - ps: Update-AppveyorBuild -Version "v$(py get_version.py) b$Env:APPVEYOR_BUILD_NUMBER" diff --git a/tox.ini b/tox.ini index 33b2c4d1..b132a6e1 100644 --- a/tox.ini +++ b/tox.ini @@ -1,5 +1,5 @@ [tox] -envlist = py{27,35,36,37,38,39,310}, pypy{,3}, mypy +envlist = py{37,38,39,310,311}, mypy [testenv] extras = @@ -30,7 +30,7 @@ commands= [testenv:codecov] deps= - codecov==2.0.15 + codecov commands= codecov From 7fba45ca5490ecca7572f83cae6dfdf0ddfdbf0d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 18 Jul 2023 07:16:49 +0200 Subject: [PATCH 43/46] Fix ci2 (#716) * remove travis --- .travis.yml | 45 --------------------------------------------- 1 file changed, 45 deletions(-) delete mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index 2d7f7548..00000000 --- a/.travis.yml +++ /dev/null @@ -1,45 +0,0 @@ -language: python - -matrix: - include: - - python: 3.7 - dist: xenial - sudo: true - env: TOXENV=dist - - python: 2.7 - - python: pypy - - python: 3.5 - - python: pypy3.5 - - python: 3.6 - - python: 3.7 - dist: xenial - sudo: true - - python: 3.8 - - python: 3.7 - dist: xenial - sudo: true - env: TOXENV=old_tests - - python: 3.6 - env: TOXENV=mypy - - python: 3.9 - - python: 3.10 - - allow_failures: - - python: 3.6 # until we solve all typing issues - env: TOXENV=mypy - -install: - - pip install tox tox-travis - -script: - - tox - - tox -e codecov - -deploy: - provider: pypi - username: __token__ - skip_existing: true - skip_cleanup: true - on: - tags: true - branch: master From 01bcf663f2f5669ac218a2841fcdb8cafad0f4b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 18 Jul 2023 07:51:43 +0200 Subject: [PATCH 44/46] quickfix for issue #711 (#712) Multiplexer behavior istill nconsistent --- src/canmatrix/canmatrix.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/canmatrix/canmatrix.py b/src/canmatrix/canmatrix.py index 08a02cb2..ac80a9f4 100644 --- a/src/canmatrix/canmatrix.py +++ b/src/canmatrix/canmatrix.py @@ -152,7 +152,7 @@ class Signal(object): * factor, offset, min, max * receivers (ECU Name) * attributes, _values, unit, comment - * _multiplex ('Multiplexor' or Number of Multiplex) + * multiplex ('Multiplexor' or Number of Multiplex) """ name = attr.ib(default="") # type: str @@ -230,6 +230,7 @@ def multiplex_setter(self, value): self.mux_val = int(value) elif value == 'Multiplexor': self.is_multiplexer = True + self.multiplex = 'Multiplexor' ret_multiplex = value return ret_multiplex From 589f5fa33ffeeea287337933712f75ebbfcfec32 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Tue, 18 Jul 2023 08:04:50 +0200 Subject: [PATCH 45/46] possible fix for issue #704 (#707) --- src/canmatrix/formats/dbc.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/src/canmatrix/formats/dbc.py b/src/canmatrix/formats/dbc.py index 4bcc243b..3aa1c157 100644 --- a/src/canmatrix/formats/dbc.py +++ b/src/canmatrix/formats/dbc.py @@ -423,13 +423,7 @@ def dump(in_db, f, **options): # signal-values: for frame in db.frames: - multiplex_written = False for signal in frame.signals: - if signal.multiplex == 'Multiplexor' and multiplex_written: - continue - - multiplex_written = True - if signal.values: f.write( ('VAL_ %d ' % From 3e115f83eaaae501778f0785952081399dcad443 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Eduard=20Br=C3=B6cker?= Date: Wed, 19 Jul 2023 15:40:32 +0200 Subject: [PATCH 46/46] Fibex container pdu (#717) * fibex: support container pdus * fix arxml desc may be none (#711) * remove travis status * fix for #651 * Hope this fixes #650 --- README.md | 1 - docs/contents.rst | 3 + src/canmatrix/formats/arxml.py | 4 +- src/canmatrix/formats/fibex.py | 163 ++++++++++++++++++++------------- tox.ini | 2 - versioneer.py | 3 - 6 files changed, 102 insertions(+), 74 deletions(-) diff --git a/README.md b/README.md index 28213d4c..f0f96321 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ ## **Canmatrix** is a python package to read and write several CAN (Controller Area Network) database formats. ## [![PyPI](https://img.shields.io/pypi/v/canmatrix.svg)](https://pypi.org/project/canmatrix/) [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/canmatrix.svg)](https://pypi.org/project/canmatrix/) -[![Build Status](https://travis-ci.org/ebroecker/canmatrix.svg?branch=development)](https://travis-ci.org/ebroecker/canmatrix) [![Codecov branch](https://img.shields.io/codecov/c/github/ebroecker/canmatrix/development.svg)](https://codecov.io/gh/ebroecker/canmatrix/) [![GitHub issues](https://img.shields.io/github/issues-raw/ebroecker/canmatrix.svg)](https://github.com/ebroecker/canmatrix/issues) diff --git a/docs/contents.rst b/docs/contents.rst index 8b137891..21d7cc02 100644 --- a/docs/contents.rst +++ b/docs/contents.rst @@ -1 +1,4 @@ +Canmatrix Documentation +======================= + diff --git a/src/canmatrix/formats/arxml.py b/src/canmatrix/formats/arxml.py index fe666abf..2501d986 100644 --- a/src/canmatrix/formats/arxml.py +++ b/src/canmatrix/formats/arxml.py @@ -1031,7 +1031,7 @@ def decode_compu_method(compu_method, ea, float_factory): # keyword definition. 06Jun16 ##################################################################################################### - if len(desc) == 0: + if desc is None or len(desc) == 0: vt = ea.get_sub_by_name(compu_scale, 'VT') if vt is not None: desc = vt.text @@ -1248,7 +1248,7 @@ def get_signals(signal_array, frame, ea, multiplex_id, float_factory, bit_offset (is_signed, is_float) = eval_type_of_signal(type_encoding, base_type, ea) - unit_element = ea.get_child(isignal, "UNIT") + unit_element = ea.follow_ref(isignal, "UNIT-REF") display_name = ea.get_child(unit_element, "DISPLAY-NAME") if display_name is not None: signal_unit = display_name.text diff --git a/src/canmatrix/formats/fibex.py b/src/canmatrix/formats/fibex.py index 3041e8df..e8158475 100644 --- a/src/canmatrix/formats/fibex.py +++ b/src/canmatrix/formats/fibex.py @@ -178,9 +178,78 @@ def selector(self, start_element, selector): return sorted(result_list, key=lambda element: element.sourceline) +def get_signals_for_pdu(fe, pdu, overall_startbit = 0): + signals = [] + ecus = [] + for signal_instance in fe.selector(pdu, "//SIGNAL-INSTANCE"): + byte_order_element = fe.selector(signal_instance, "/IS-HIGH-LOW-BYTE-ORDER") + if byte_order_element[0].text == "false": + is_little_endian = True + else: + is_little_endian = False + + start_bit = int(fe.selector(signal_instance, "/BIT-POSITION")[0].text, 0) + overall_startbit + signal = fe.selector(signal_instance, ">SIGNAL-REF")[0] + ecu_instance_refs = fe.selector(signal_instance, "< 0: + ecu_name = fe.sn(fe.get_referencable_parent(ecu_instance_ref)) + receiver_ecus.append(ecu_name) + ecus.append(canmatrix.Ecu(name=ecu_name.strip())) + + signal_name = fe.sn(signal) + coding = fe.selector(signal, ">CODING-REF")[0] + bit_length = int(fe.selector(coding, "/!BIT-LENGTH")[0].text) + compu_methods = fe.selector(coding, "/!COMPU-METHOD") + sig = canmatrix.Signal(name=signal_name) + + for compu_method in compu_methods: + category = fe.selector(compu_method, "/!CATEGORY") + if len(category) > 0 and category[0].text == "LINEAR": + numerator = fe.selector(compu_method, "/!COMPU-NUMERATOR")[0] + denominator = fe.selector(compu_method, "/!COMPU-DENOMINATOR")[0] + teiler = decimal.Decimal(fe.selector(denominator, "/!V")[0].text) + [offset, factor] = [decimal.Decimal(a.text) for a in fe.selector(numerator, "/!V")] + [offset, factor] = [a / teiler for a in [offset, factor]] + sig.offset = offset + sig.factor = factor + try: + sig.min = decimal.Decimal(fe.selector(compu_method, "!PHYS-CONSTRS!LOWER-LIMIT")[0].text) + sig.max = decimal.Decimal(fe.selector(compu_method, "!PHYS-CONSTRS!UPPER-LIMIT")[0].text) + except: + pass + unit = fe.selector(compu_method, ">!UNIT-REF") + if len(unit) > 0: + try: + sig.unit = fe.selector(unit[0], "!DISPLAY-NAME")[0].text + except: + pass + elif len(category) > 0 and category[0].text == "TEXTTABLE": + for compu_scale in fe.selector(compu_method, "/!COMPU-SCALE"): + try: + value_name = fe.selector(compu_scale, "!COMPU-CONST!VT")[0].text + except IndexError: + value_name = fe.get_desc_or_longname(compu_scale) + value_value = fe.selector(compu_scale, "!LOWER-LIMIT")[0].text + sig.add_values(value_value, value_name) + sig.is_little_endian = is_little_endian + if not sig.is_little_endian: + sig.set_startbit(start_bit, bitNumbering=1) + else: + sig.start_bit = start_bit + sig.size = bit_length + sig.receivers = list(set(receiver_ecus)) + + sig.add_comment(fe.get_desc_or_longname(signal)) + signals.append(sig) + return signals, ecus + + def load(f, **_options): fe = Fe(f) result = {} + sig_group_counter = 0 clusters = fe.selector(fe.root, "//CLUSTER") names = [fe.sn(a) for a in clusters] @@ -204,11 +273,35 @@ def load(f, **_options): extended = arbitration_id_element.attrib.get("EXTENDED-ADDRESSING", 'false') == 'true' frame_element = fe.selector(ft, ">FRAME-REF")[0] frame_size = int(fe.selector(frame_element, "/BYTE-LENGTH")[0].text) - pdu = fe.selector(frame_element, ">>PDU-REF")[0] - frame_name = fe.sn(pdu) -# fe.selector(pdu, "< 1: + frame_name = fe.sn(frame_element) + frame = canmatrix.Frame(name=frame_name) + for pdu_instance in pdu_instances: + pdu = fe.selector(pdu_instance, ">PDU-REF")[0] + pdu_startbit_position = int(fe.selector(pdu_instance, "/BIT-POSITION")[0].text, 0) + signals, ecus = get_signals_for_pdu(fe, pdu, pdu_startbit_position) + for sig in signals: + frame.add_signal(sig) + for ecu in ecus: + db.add_ecu(ecu) + + frame.add_signal_group(fe.sn(pdu), sig_group_counter, [sig.name for sig in signals]) + sig_group_counter += 1 + else: + pdu = fe.selector(pdu_instances[0], ">PDU-REF")[0] + frame_name = fe.sn(pdu) + frame = canmatrix.Frame(name=frame_name) + + signals, ecus = get_signals_for_pdu(fe, pdu) + for sig in signals: + frame.add_signal(sig) + for ecu in ecus: + db.add_ecu(ecu) + + # fe.selector(pdu, "< 0: pdu_triggerings = fe.selector(output_ports[0], ">>PDU-TRIGGERING-REF") @@ -231,69 +324,7 @@ def load(f, **_options): if "CAN-FD" in [a.text for a in fe.selector(ft, "//CAN-FRAME-TX-BEHAVIOR") + fe.selector(ft, "//CAN-FRAME-RX-BEHAVIOR")]: frame.is_fd = True - for signal_instance in fe.selector(pdu, "//SIGNAL-INSTANCE"): - byte_order_element = fe.selector(signal_instance, "/IS-HIGH-LOW-BYTE-ORDER") - if byte_order_element[0].text == "false": - is_little_endian = True - else: - is_little_endian = False - - start_bit = int(fe.selector(signal_instance, "/BIT-POSITION")[0].text, 0) - signal = fe.selector(signal_instance, ">SIGNAL-REF")[0] - ecu_instance_refs = fe.selector(signal_instance, "< 0: - ecu_name = fe.sn(fe.get_referencable_parent(ecu_instance_ref)) - receiver_ecus.append(ecu_name) - db.add_ecu(canmatrix.Ecu(ecu_name)) - - signal_name = fe.sn(signal) - coding = fe.selector(signal, ">CODING-REF")[0] - bit_length = int(fe.selector(coding, "/!BIT-LENGTH")[0].text) - compu_methods = fe.selector(coding, "/!COMPU-METHOD") - sig = canmatrix.Signal(name=signal_name) - - for compu_method in compu_methods: - category = fe.selector(compu_method, "/!CATEGORY") - if len(category) > 0 and category[0].text == "LINEAR": - numerator = fe.selector(compu_method, "/!COMPU-NUMERATOR")[0] - denominator = fe.selector(compu_method, "/!COMPU-DENOMINATOR")[0] - teiler = decimal.Decimal(fe.selector(denominator, "/!V")[0].text) - [offset, factor] = [decimal.Decimal(a.text) for a in fe.selector(numerator, "/!V")] - [offset, factor] = [a / teiler for a in [offset, factor]] - sig.offset = offset - sig.factor = factor - try: - sig.min=decimal.Decimal(fe.selector(compu_method, "!PHYS-CONSTRS!LOWER-LIMIT")[0].text) - sig.max = decimal.Decimal(fe.selector(compu_method, "!PHYS-CONSTRS!UPPER-LIMIT")[0].text) - except: - pass - unit = fe.selector(compu_method, ">!UNIT-REF") - if len(unit) > 0: - try: - sig.unit = fe.selector(unit[0],"!DISPLAY-NAME")[0].text - except: - pass - elif len(category) > 0 and category[0].text == "TEXTTABLE": - for compu_scale in fe.selector(compu_method, "/!COMPU-SCALE"): - try: - value_name = fe.selector(compu_scale, "!COMPU-CONST!VT")[0].text - except IndexError: - value_name = fe.get_desc_or_longname(compu_scale) - value_value = fe.selector(compu_scale, "!LOWER-LIMIT")[0].text - sig.add_values(value_value, value_name) - sig.is_little_endian = is_little_endian - if not sig.is_little_endian: - sig.set_startbit(start_bit, bitNumbering=1) - else: - sig.start_bit = start_bit - sig.size = bit_length - sig.receivers = list(set(receiver_ecus)) - - sig.add_comment(fe.get_desc_or_longname(signal)) - frame.add_signal(sig) db.add_frame(frame) return result diff --git a/tox.ini b/tox.ini index b132a6e1..ab77dc1c 100644 --- a/tox.ini +++ b/tox.ini @@ -12,8 +12,6 @@ deps= passenv= TOXENV CI - TRAVIS - TRAVIS_* APPVEYOR APPVEYOR_* commands= diff --git a/versioneer.py b/versioneer.py index 64fea1c8..9da01c2b 100644 --- a/versioneer.py +++ b/versioneer.py @@ -14,9 +14,6 @@ * [![Latest Version] (https://pypip.in/version/versioneer/badge.svg?style=flat) ](https://pypi.python.org/pypi/versioneer/) -* [![Build Status] -(https://travis-ci.org/warner/python-versioneer.png?branch=master) -](https://travis-ci.org/warner/python-versioneer) This is a tool for managing a recorded version number in distutils-based python projects. The goal is to remove the tedious and error-prone "update