feat(modularisation/docs) New hardware metadata version

[Nick-Munnich]

This is an initial rough draft for a version 2 of the ZMK hardware metadata standard. The goal is to define the metadata such that the metadata files can be collected in a separate repository titled zmk-modules. This draft has not updated the metadata schema, but rather the documentation of it, to be easier to read and encourage discussion. Also note that as the schema has been previously changed without updating the documentation, there are some properties to which I was uncertain of their use - I gave it my best guess.

The assumption is that (almost) all boards and shields currently in the ZMK tree will be moved to external module repositories, all interconnect definitions will be moved to zmk-modules, and only the boards and shields used for testing purposes remain in-tree.

ZMK's setup scripts and related docs would need to be adjusted accordingly, either also in this PR or in a separate one. An initial rough draft zmk-modules can be found here, focusing on folder structure.

As this would obviously break compatibility with a lot of people's setups, I would suggest doing it simultaneously with whatever changes are necessary to move to Zephyr 3.7 LTS. When everything is completed, then I believe we can (finally) close #453.

We could also include some optional properties that would allow for some cool tools to make use of zmk-modules. In particular, the data to enable tools comparing keyboards to one another might be very beneficial, as currently the developers of such tools have to input each new keyboard manually themselves.

TODO:

• Agree on metadata v2
• Create zmk-modules repository
• Adjust ZMK's setup scripts to use zmk-modules
• Document zmk-modules repository
• Fill zmk-modules with existing modules
• Move non-essential boards and shields from ZMK tree into modules

EDIT: Thanks to discussion, it seems that we also need:

• Agree and implement a versioning system for ZMK

[ssbb]

It would be cool to add some physical layout definitions for keyboards. For example, QMK uses a KLE-like JSON format.

We could add some common presets like ANSI60, ANSI104, ISO60, ISO105, etc. Then, we might allow the xxx+x approach for ergonomic boards (e.g., 23332+2 is a Hummingbird layout). Additionally, we could use a KLE-like JSON for custom layouts.

Usage examples:

• Projects such as a keymap-drawer could benefit significantly from this (currently, you have to specify it manually).
• Projects like YellowAfterlife's ergo keyboard list could parse this information to determine the number of columns, rows, and thumbs.
• Some CLI tools could use it to render the proper layout, etc.

[petejohanson]

It would be cool to add some physical layout definitions for keyboards. For example, QMK uses a KLE-like JSON format.

We could add some common presets like ANSI60, ANSI104, ISO60, ISO105, etc. Then, we might allow the xxx+x approach for ergonomic boards (e.g., 23332+2 is a Hummingbird layout). Additionally, we could use a KLE-like JSON for custom layouts.

Usage examples:

• Projects such as a keymap-drawer could benefit significantly from this (currently, you have to specify it manually).
• Projects like YellowAfterlife's ergo keyboard list could parse this information to determine the number of columns, rows, and thumbs.
• Some CLI tools could use it to render the proper layout, etc.

That information is already now possible to set in the devicetree with the new physics layouts refactor so it's available in firmware for the upcoming ZMK Studio work. It would be a net negative to duplicate that in the metadata, IMHO.

[caksoylar]

I think the broad strokes of this organization makes sense. A separate repo where PRs can go into to add "references" from which tooling can discover things is a sensible approach.

One small suggestion I'd have for now would be to have module dict field instead of murl, where url, commit, outdated (also maybe project name, to be used in west.yml) can go as sub-entries.

I think there are a few things that warrant further discussion. I am not listing these because I necessarily have an opinion on them but discussion around them would be good to have.

1. Pinning modules to commits: The proposal mandates it, but there might be advantages to be able to point to a branch/tag.
2. How to handle compatibility with ZMK versions (or even between modules): The proposal has an outdated field which might be useful, but also is not a well-defined concept¹. This is a hard question to answer in general, and having ZMK use some sort of specific versioning scheme might be a prerequisite to a proper solution.
3. Maybe it is better to omit a pull-request field to limit "support" (since these will be used by official tooling) to the main branch/release.

Footnotes

1. Do ZMK committers update these as things break? An automated tool can check for building but can't check for changed/broken functionality ↩

[Nick-Munnich]

One small suggestion I'd have for now would be to have module dict field instead of murl, where url, commit, outdated (also maybe project name, to be used in west.yml) can go as sub-entries.

Great idea, I'll make this change.

1. Pinning modules to commits: The proposal mandates it, but there might be advantages to be able to point to a branch/tag.

The reason for this is specifically a security/malicious actor concern. By specifying a commit, we can prevent harmful images/code/etc from being present in the version of the module that we recommend (unless it is missed on review). I think SHA-1 should suffice here, though cybersecurity/cryptography isn't my specialty.

2. How to handle compatibility with ZMK versions (or even between modules): The proposal has an `outdated` field which might be useful, but also is not a well-defined concept[1](#user-content-fn-1-14bdca0f8be552f5c0b9ba62bc8be843). This is a hard question to answer in general, and having ZMK use some sort of specific versioning scheme might be a prerequisite to a proper solution.

I think I'll change it so that the outdated field points to the most recently working zmk commit and is omitted when up-to-date - then builds with outdated modules could still be attempted, while raising big warnings. A proper versioning scheme would be very nice, but I don't think it's necessarily necessary.

3. Maybe it is better to omit a `pull-request` field to limit "support" (since these will be used by official tooling) to the main branch/release.

I was quite torn on whether to include or remove it. I'll remove it.

Footnotes

1. Do ZMK committers update these as things break? An automated tool can check for building but can't check for changed/broken functionality [↩](#user-content-fnref-1-14bdca0f8be552f5c0b9ba62bc8be843)

My thinking was that we could write a tool that goes through all of the modules in a type category, marks them as outdated, and opens issues on all the remote repos found. The owners/maintainers of the module could comment on the issue to discard it if it doesn't apply to them, causing the tool to automatically remove the outdated flag, or they can apply changes and then open a PR to update the pinned commit prompting manual review. It would be ZMK committers' responsibility to write the issue description if their commit breaks something, and ZMK maintainers' responsibility to run the tool on a breaking merge.

Ideally we could have it auto-accept particular changes (ex. adding wakeup-source), but that seems like a significantly more complex task and could be investigated far into the future. A proper versioning scheme as you mentioned would also help it feel smoother.

[caksoylar]

I think I'll change it so that the outdated field points to the most recently working zmk commit and is omitted when up-to-date - then builds with outdated modules could still be attempted, while raising big warnings. A proper versioning scheme would be very nice, but I don't think it's necessarily necessary.

I like that a bit better, and you may be right on the versioning.

My thinking was that we could write a tool that goes through all of the modules in a type category, marks them as outdated, and opens issues on all the remote repos found. The owners/maintainers of the module could comment on the issue to discard it if it doesn't apply to them, causing the tool to automatically remove the outdated flag, or they can apply changes and then open a PR to update the pinned commit prompting manual review. It would be ZMK committers' responsibility to write the issue description if their commit breaks something, and ZMK maintainers' responsibility to run the tool on a breaking merge.

That makes sense to me.

[urob]

I have been thinking a bit about maintenance costs. Zephyr has a policy in place which only allows changes to modules as mergeable PRs and strictly forbids force-pushes to the main branch. Something similar could help reduce the review burden for the ZMK team.

On the flipside, to reduce maintenance costs for module maintainers, I'd second that some form of ZMK version system (with infrequent api-breaking changes) would be desirable. Even if not strictly necessary, forcing maintainers to check compatibility for every ZMK commit is a huge burden.

Moreover, if modules aren't synchronized, which seems inevitable in a large ecosystem without proper upstream versioning, then the burden ultimately gets passed on to end-users: To update their firmware, they will need to:

1. Find the latest ZMK commit that's supported by all modules they are using
2. For each of their modules, find the corresponding commit that supports this latest-commonly-supported commit

[Nick-Munnich]

Something similar would be an excellent policy to have in zmk-modules.

You raise a good point and argument for versioning as well. If we introduced a versioning system, then the commit and outdated child properties of module could be cleanly replaced by a version property:

module:
  url: https://github.com/module-owner/module-repo-name
  version:
    - 2-1-3: 093dj09sdapasdlkd09285c585e08pjonkjn7y8798
    - 1-0-12: c43a0365183c58f74c285c585e0cbc1ea99f8a12

That would allow tools like zmk-cli to completely remove the burden you described.

[joelspadin]

If I'm understanding this correctly, with this proposal, in order to add or update a module, you would need to manually copy all the .zmk.yml files from the module into the central repository, then update each one to include the module repo's URL and the specific commit from which the data was pulled. Failing to do this correctly could lead to the data in the central repo not matching that of the module repo.

It seems to me like it would be a lot easier to maintain if the central repo contained only module URLs and commit hashes. Then to update a module, you would only need to update the commit hash in one file. This would mean you could not search the central repo directly for specific keyboards/behaviors/etc., but it would be possible to set up a website which is generated by collecting data from every module into a searchable database.

[YellowAfterlife]

On keyboard geometry, that's such a topic - e.g. with Xx+Y notation, how do you describe something like Hillside? Even my current "key count, columns, rows, and extra keys" format doesn't quite work because an extra index finger column and an extra pinky column serve different purposes.

[Nick-Munnich]

On keyboard geometry, that's such a topic - e.g. with Xx+Y notation, how do you describe something like Hillside? Even my current "key count, columns, rows, and extra keys" format doesn't quite work because an extra index finger column and an extra pinky column serve different purposes.

The docs from #2268 found here describe the physical layout system nicely. The current plan is to add tags to each key such as "PINKY_HOME" at some point in the future.

[Nick-Munnich]

Closing this for the time being. While I do still feel like an update to the metadata would be nice, I no longer feel that it is necessary for the module ecosystem to mature. It would be better to revisit this topic at a much later stage and see what would be useful.