Tools: scripts: Update MAVLink parsing for wiki inclusion

[ES-Alexander]

ArduPilot Wiki uses reStructuredText (rst) source files, so supporting rst output is essential for this script to generate wiki files. Ideally this can be run automatically for every vehicle type and version, like the existing parameter lists.

• Add rst support

• Example file for ArduSub 4.1, with all options included:
  ArduSub_Sub-4.1_MAVLink_Messages.rst.zip

• EDIT: Some additional example files from the master branch, for different vehicle types (all message options included):
  MultiVehicle_master_MAVLink_Messages.zip

• EDIT2: Example files for released firmware versions have been added here

---

EDIT3: Wiki inclusion and future planning notes moved to ArduPilot/ardupilot_wiki#6065

[ES-Alexander]

@amilcarlucas where/when are you expecting/wanting yours to run?

As far as I'm aware the parameters repository already generates all the apm.pdef.xml files (with CI checks for new tags to process every 5 days), so I expect it could be reasonably straightforward to add your rsync updates there as just an extra step?

I'm not sure how much sense it would make to combine efforts on a PR in that repo, given your feature is an additional endpoint for the parameters, whereas mine is a new script to run and store the outputs of, but if you're also wanting to include the MAVLink message support in your rsync transfers then I suppose that ties them together somewhat?

[amilcarlucas]

Nevermind, I jumped to conclusions.

[ES-Alexander]

Did some more digging and found that adding a version selection dropdown makes more sense to handle as a patch during generation in the specific applications where it's relevant. Since both of the remaining "planned" components of this PR can be handled in separate repositories, I think this is ready for review / can be merged in 🙂

[ES-Alexander]

Not really sure who should review this PR, but @Hwurzburg might have some ideas or preferences around the formatting of the generated files 🤷‍♂️

[Hwurzburg]

Was all the formatting in the example sub file generated by this script?

[ES-Alexander]

Was all the formatting in the example sub file generated by this script?

Yeah - the .rst example file is just directly from the script output :-)

I generated it from the Sub-4.1 branch with the following invocation:

python mavlink_parse.py -v ArduSub --format rst -b Sub-4.1 --header ardupilot_wiki -gucq

(noting once again that the script requires Python 3.11 to run)

[Hwurzburg]

I've tagged @peterbarker and @IamPete1 to review if the groupings and generated data is correct....with PeterB's help I could add per vehicle pages into the Dev sites wiki under a MAVLink commands topic....

[ES-Alexander]

I've tagged @peterbarker and @IamPete1 to review if the groupings and generated data is correct

I've just rebased over master, and generated some more rst files for different vehicle types, to make that checking a bit more straightforward:

MultiVehicle_master_MAVLink_Messages.zip

with PeterB's help I could add per vehicle pages into the Dev sites wiki under a MAVLink commands topic....

Nice! 😃
I'm happy to make a PR to generate these automatically in what is currently the parameters repository, if that's helpful, but also happy for some other approach to be taken.

I'd note that because the MAVLink protocol updates, and I'm currently including the "unsupported messages" in the output generation, the outputs for each version should ideally be re-generated each time with the latest script version, and the most recent pymavlink release, rather than tying the script to the vehicle branch and only generating once per version. That said, that's not a critical requirement, and generating once per version could be ok if we either accept that newer messages won't show up, or just don't generate the unsupported messages - I just like the idea of being able to search for a message and finding out clearly if it is definitely unsupported 🤷‍♂️

[Hwurzburg]

having to manually update the page generation once in a while is fine for now, IMO....a PR to generate the page would be good, then I can add it to the Dev wiki.....very few users, and only devs, will be using it but it will certainly help them, even if occasionally out of date...

[ES-Alexander]

a PR to generate the page would be good

Initial files are now available for the ≥ 4.0 tags in the parameters repository :-)

They'll still need some re-jigging of the anchors (and insertion of a version selector dropdown) before they can be included in the wiki, but that'll be very similar to the existing process for parameter files, that happens during wiki generation.

[Hwurzburg]

sorry its taken me so long to get back to this, but I wanted to add wiki files for Plane,Copter,Rover messages as of current master....the params repo does not even have files for 4.5...could you generate these three and I will ad to dev wiki...thanks

[ES-Alexander]

sorry its taken me so long to get back to this

I get it - unfortunately the list of interesting/useful things to do seems to grow much faster than the available time to do things 😅

could you generate these three and I will ad to dev wiki

It's been a while since I did those initial generations, so I'll need to figure out the relevant setup/process again. I'm hoping it's mostly covered by the Action I made in the parameters repo, although from memory that's reliant on this PR being merged, so I might need to make some custom branches to do the manual generations for now.

I'll try to sort that out this week :-)

I wanted to add wiki files for Plane,Copter,Rover messages as of current master....the params repo does not even have files for 4.5...could you generate these three

To confirm, did you want files for just the 4.5 release, or current master as well?
If I get to doing this before you respond I'll just generate both - I don't expect it to require much extra effort once I have the process working :-)

[Hwurzburg]

4.6 is about to drop...so master only will be fine