WordPress for Docs

[adamziel]

The Markdown Editing Workflow prototype allowed me to create a Gutenberg Handbook Pull Request entirely from my web browser.

Let's expand it into a reusable, free, open-source tool for publishing and collaborating on documentation.

What do we need for Version 1 MVP?

As Version 1 MVP, let's for solve these two use-cases:

• Contributing to WordPress Playground documentation
• Contributing to Theme Developer Handbook

This way we'll explore the basic contribution workflows, publishing changes, and make it all reusable in different repositories. On our way there, we'll need to implement a few vital improvements listed in the Markdown Editing Workflow PR.

Contributing to WordPress Playground documentation

Playground documentation is a custom Docusaurus app. It requires a local build, has annoying quirks around table of contents and Playground blocks, and figuring out the right MDX tends to be challenging. We could replace it with a "WordPress Markdown" pipeline edited through Playground.

Here's what we'd need to get to an MVP:

• Migrate the content to Markdown. That's just removing custom MDX syntax and running the Markdown importer.
• Create a contributor's workflow – same as with theme developer handbook. We could reuse parts of playground-docs-workflow here.
• Create a documentation-looking theme.
• git push -> Publish a static site. We can reuse the prototype from https://github.com/adamziel/playground-docs-workflow.

Contributing to Theme Developer Handbook

@justintadlock mentioned Theme Developer Handbook is a separate site with no easy way to contribute to. Playground could provide that way. Here's what we'd need to get to an MVP:

• Move the content to GitHub
  • Export content as Markdown (or WXR -> Markdown) – or is it already available as Markdown somewhere @justintadlock?
  • Host it in a GitHub repository
• Create a contributor's workflow
  • Add an Edit in Playground link to every page
  • Provide onboarding screen to explain the workflow to new users
  • Enable creating Pull Requests with a single, highly visible button
  • Default to saving changes in the browser to avoid accidental data loss when closing a tab
  • Hook a "Pull Request Preview" GitHub workflow – let's build a reusable one others could easily plug into their repos
• Create a deployment workflow from GitHub to WordPress.org
  • Idea 1: Export Markdown as static HTML and upload it to WordPress.org.
    • Upside: Simple workflow, no database operations involved.
    • Downside: We would need to setup the WP.org theme in Playground. It has many dependencies, .sass files, and I found that task quite challenging. Also, we may need to maintain that setup to keep it in sync with changes on WP.org.
  • Idea 2: git push -> WordPress.org pages workflow.
    • We'd export Markdown as Block markup and create/update/delete WordPress.org pages.
    • Upside: No need to worry about theming setup.
    • Downside: We'd need to build a custom plugin to resolve a delta between a markdown file structure and WordPress page structure. It might be a useful plugin in context of site transfer, but it could also significantly slow down this project.
  • Idea 3: I don't have any – thoughts?
• Single-click Pull Request creation without going through the complex GitHub export form

Does that sound right @justintadlock @juanmaguitar @bph @ndiego @ryanwelcher? Can we remove any of these items? Are we missing any? Also CCing @dd32 @StevenDufresne @tellyworth

Other considerations

As a rough prototype, the current Markdown Editing Workflow is heavily optimized for a single data format: Markdown. I'd like to refactor the code to have a single import/export core where we could plug in other input and output formats. This could mean either reusing the existing import/export infrastructure, or creating a new one – depending on which is more useful:

This way, we'll also unlock the following applications:

• git push -> WordPress deploy workflow.
• "WordPress for documentation" as its own free and open-source product to replace Docusaurus. It would support WordPress plugins, themes, and custom blocks.
• Starting WordPress sites from static files and saving back to those files.
  • Integrating it with VS Studio Code could replace the default markdown editor.
• Exporting content as Markdown and other formats.
• Use WordPress as data converter, e.g. Markdown -> HTML, MediaWiki -> WXR.
• Simplistic site synchronization based on merging Markdown files.
• Starting Playgrounds from GitHub repos with issues as posts and comments as discussions.
• Reuse what we build here for site exports, transfer, and synchronization.

Furthermore, I'd like to approach this in a core-first way and explore the site transfer problem space, e.g. the problem of rewriting URLs between sites.

Let's follow these heuristics where they're useful and diverge when they're not:

• Solid technical foundations over quick and one-off fixes.
• WordPress plugins over custom JavaScript Playground integrations.
• Streaming data over buffering and processing to eventually support huge 100GB+ databases.
• PHP, core-first, zero-dependency libraries like WP_HTML_Tag_Processor, WP_XML_Processor, ZipStreamReader and ZipStreamWriter, and AsyncHttpClient.

Related discussions

• Solving rewriting site URLs in WordPress using the HTML API and URL parser

Related code explorations

• StreamChain: An API for streams-processing data (e.g. HTTP → ZIP → XML → HTML) adamziel/wxr-normalize#1
• https://github.com/adamziel/playground-content-converters
• https://github.com/adamziel/site-transfer-protocol
• https://github.com/adamziel/wxr-normalize
• https://github.com/adamziel/git-sparse-checkout-in-js
• https://github.com/adamziel/playground-docs-workflow
• Edit Visually browser extension – iteration 2 playground-tools#298

Related resources

• WIP: Create Unicode-aware comprehensive sluggify(). wordpress-develop#5466
• https://github.com/TRowbotham/URL-Parser/tree/master
• Experiment: Doodlings on how a Core URL parser could look. wordpress-develop#6666
• https://url.spec.whatwg.org/
• Parsing Millions of URLs per Second

cc @brandonpayton @bgrgicak @jordesign @mtias @zzap @dmsnell

[zzap]

I have questions.

How are we going to deal with media uploads (videos, screenshots, GIFs)?

Add an Edit in Playground link to every page

If the content is going to be in GitHub anyway, I'd love to see another button "Edit in GitHub" as well. For those contributors that like this workflow better.

Provide onboarding screen to explain the workflow to new users

This can be rather annoying if happens every time you click on button. Is there a way to add "Never show this again" option or maybe just add a very visible link to a guideline?

• Enable creating Pull Requests with a single, highly visible button
• Single-click Pull Request creation without going through the complex GitHub export form

This would be very much appreciated. On that note, if the user is logged in to WordPress.org and has GitHub profile connected, is it really necessary to go through GH connection when clicked on "Edit in Playground" button? Can we make it a condition: you don't get to see the button unless your WP.ORG profile is connected to GH. And then, when clicked on button I'm already there, doing PR on my own behalf.

[adamziel]

How are we going to deal with media uploads (videos, screenshots, GIFs)?

Here's a video of how the previous prototype deals with media files. Tl;dr, you upload them into Playground and they get included in the Pull Request. We'd need a way to configure the upload path and adjust the final absolute URLs:

https://github.com/adamziel/playground-docs-workflow

If the content is going to be in GitHub anyway, I'd love to see another button "Edit in GitHub" as well. For those contributors that like this workflow better.

+1

This can be rather annoying if happens every time you click on button. Is there a way to add "Never show this again" option or maybe just add a very visible link to a guideline?

"Never show this again" is the only way we can have an onboarding modal. I don't want to look at it every time I use that flow.

This would be very much appreciated. On that note, if the user is logged in to WordPress.org and has GitHub profile connected, is it really necessary to go through GH connection when clicked on "Edit in Playground" button?

Oh, interesting! I don't know, perhaps there's a way to reuse that GitHub connection – let's explore that. What are your thoughts on contributions from folks without a WordPress.org account?

Also looping in @brandonpayton

[zzap]

Here's a video of how the previous prototype deals with media files.

That looks promising.

What are your thoughts on contributions from folks without a WordPress.org account?

I personally don't like that. Without the wp.org account, they don't have a slack account either. That means a few things:

1. they won't have a place to get credits for their contributions, and
2. they are not communicating with the Docs team in any way, which makes it questionable if they ever read our style guide or follow the standards we're trying to apply (which means that, other than simple typo fixes, I question the quality of edits)
3. this makes spam edits more possible and, even though we would approve/reject proposed changes, it can become very time consuming to go through all of it.

That being said, I'm open to be proved wrong if there's more value than damage in allowing everyone to propose changes.

[estelaris]

Without the wp.org account, they don't have a slack account either.

I spoke to Matt about this and he wants contributors to be logged in into both WP.org and GH accounts. But I do not know which keeps track of which accounts.

[adamziel]

With the recent addition of GitDirectoryResource and the markdown<->blocks conversion tools, this Blueprint loads all Gutenberg markdown files into Playground as WordPress pages using real blocks. It works in local dev env for now. We'll need a CORS proxy to make it work on playground.wordpress.net:

{
	"$schema": "https://playground.wordpress.net/blueprint-schema.json",
	"login": true,
	"landingPage": "/wp-admin/?markdown-file-path=docs/contributors/accessibility-testing.md",
    "constants": {
        "STATIC_FILES_ROOT": "/wordpress/wp-content/static-content"
    },
	"steps": [
		{
			"step": "resetData"
		},
        {
            "step": "writeFiles",
            "writeToPath": "/wordpress/wp-content/static-content",
            "filesTree": {
                "resource": "git:directory",
                "url": "https://github.com/wordpress/gutenberg.git",
                "ref": "trunk",
                "path": "docs"
            }
        },
        {
            "step": "writeFiles",
            "writeToPath": "/wordpress/wp-content/plugins",
            "filesTree": {
                "resource": "git:directory",
                "url": "https://github.com/adamziel/playground-content-converters.git",
                "ref": "trunk",
                "path": "src"
            }
        },
		{
			"step": "activatePlugin",
			"pluginPath": "import-static-files/import-static-files.php"
		},
		{
			"step": "activatePlugin",
			"pluginPath": "store-markdown-as-post-meta/index.php"
		},
		{
			"step": "activatePlugin",
			"pluginPath": "convert-markdown-to-blocks-in-js/convert-markdown-to-blocks-in-js.php"
		},
		{
			"step": "activatePlugin",
			"pluginPath": "save-pages-as-static-files/index.php"
		}
	]
}

The following technical challenges need to be addressed to make this editing workflow practical:

• Converting URLs from markdown to WordPress pages and the other way around – to make working with links easy.
• Avoid cloning static assets, they could require hundreds of megs of downloads. Instead, rewrite the URLs to use the remote assets when possible.
• Importing time – the current Markdown importer uses JavaScript to convert Markdown to blocks. It would be lovely to use PHP instead and save on all the fetch() requests. Also, maybe we don't need to convert everything upfront but could start with the page structure and only convert the actual content when it's accessed?

More and more streams of work in Playground are getting blocked by the URL rewriting issue. We now have good technical foundations to start building a general solution that would also solve for importing WXR, site transfers, etc.