Early draft of storing temp sites in OPFS

[adamziel]

Motivation for the change, related issues

After #1822, Playground forcibly refreshes all the browser tabs after deploying a new webapp version. This fixes version upgrades, but it also causes data loss in the temporary Playgrounds running in those refreshed tabs. Let's explore storing all that temporary data in OPFS so we can restore it after a page refresh.

[brandonpayton]

Some things left to do:

• Stop refreshes with no query params from creating endless, temporarily persisted sites.
• Figure out how to select existing temp-on-OPFS sites based on app URL. Some options:
  • Map name in query API params to existing temp site
  • Be explicit with a dedicated param like temp-slug that uses slug or temp-site-id that uses the UUID. It is good to distinguish between selection of persisted sites and selection of temporary sites because we likely want to handle site-not-found scenarios differently for each:
    • Persistent sites that are not found should probably be shown a warning.
    • Temporary sites that are not found might be shown a warning but could also be automatically redirected to a fresh temp site.
• Make new temp sites actually write WP and data to OPFS. This isn't yet happening.
• Adjust the Save to OPFS code to simply change a site's storage type.

[brandonpayton]

Added one more item to the above list:

• Adjust the Save to OPFS code to simply change a site's storage type.

I also took a look at making sure opfs-temporary sites actually had OPFS mounts, but when I did so, WordPress installation failed because the site's directory already existed with a wp-runtime.json file in it. So we might need to adjust something either about timing or with unzipping WP.

[brandonpayton]

Also changed

• Make default temp site creation update query params. Otherwise, refreshing with no query params will continue to create new temp sites.

to

• Stop refreshes with no query params from creating endless, temporarily persisted sites.

[bgrgicak]

I took some time and added a few things.

There is now a temp-slug that's added to every temp site.
I suggest that we rename it to site-slug later if we don't find a use for having a different name. (There is no use now.)

If a site-slug or temp-slug doesn't exist and the user already has sites, a message will be displayed.
I updated the existing message to include a Start a new Playground button.

[adamziel]

Oh no! Saving fresh sites to OPFS takes a considerable amount of time, like 12 seconds. Such a long waiting time is quite a bad first user experience. Remembering my explorations around zip streaming and OPFS, here's a breakdown of the things that require significant processing time:

• Unzipping
• Moving files through postMessage
• Writing files using an asynchronous OPFS handle – sync handles are much faster

Perhaps we could start with a simple MEMFS site and then run a non-blocking synchronization in the background while somehow postMessage–transmitting files in bulk and not on at a time.

The risk there would be the user closing their browser tab before the sync is complete – we'd need to add an window.onunload modal and track the synchronization state to show the site is corrupted on the next visit.

[adamziel]

Aha, but WordPress is not present in OPFS yet during the very first site boot, therefore we don't have to block loading the site for the 12 seconds it takes to synchronize MEMFS to OPFS – there are no files to synchronize yet.

[adamziel]

I've just pushed archived sites management and the code to write new temp-opfs sites to OPFS.

We'll still need to retain the "in-memory" storage for, e.g., Safari in private mode.

[adamziel]

The experience I went for is this:

• Temp sites are archived after a page refresh
• The currently open temp site isn't getting archived after a page refresh. This is flaky as I might have multiple browser tabs open – we could make it so that requesting a ?temp-slug unarchives a site.
• Archived sites created < 24h ago are deleted

I'm having second thoughts. I wonder how will that feel for the user – sometimes a site disappears, sometimes it doesn't. Also, we'll eventually need something to prevent conflicting writes to the same OPFS site from multiple Playground instances. Alternatively, we could always run those sites in a SharedWorker and avoid any synchronization. That wouldn't work in Safari today, but maybe that's okay.

[brandonpayton]

The experience I went for is this:

• Temp sites are archived after a page refresh
• The currently open temp site isn't getting archived after a page refresh. This is flaky as I might have multiple browser tabs open – we could make it so that requesting a ?temp-slug unarchives a site.
• Archived sites created < 24h ago are deleted

I'm having second thoughts. I wonder how will that feel for the user – sometimes a site disappears, sometimes it doesn't.

Would unarchiving on request be sufficient for reducing user confusion if we wait a week before deleting archived sites?

How practical this is may depend on device storage quotas and availability. But if we run into limits, maybe we could clean up archived sites earlier, starting with the once loaded longest ago.

[brandonpayton]

Perhaps we could start with a simple MEMFS site and then run a non-blocking synchronization in the background while somehow postMessage–transmitting files in bulk and not on at a time.

The risk there would be the user closing their browser tab before the sync is complete – we'd need to add an window.onunload modal and track the synchronization state to show the site is corrupted on the next visit.

@adamziel and I spoke a bit about this, and it might make sense for us to maintain some of this app-specific state separately from the wp-runtime.json metadata which we imagine could be shared along with sites themselves.

We could have a file like <site-dir>/.wp-playground-state.json that has properties like:

• whenLastLoaded
• isArchived
• storage
• initialPersistenceState: not-started | in-progress | completed - to track the process of transitioning temp sites from MEMFS to OPFS after load and hopefully help us identify and address corruption due to incomplete WP installs. The actual property name could be improved. :)

[brandonpayton]

There is now a temp-slug that's added to every temp site.
I suggest that we rename it to site-slug later if we don't find a use for having a different name. (There is no use now.)

Thanks for working on these things, @bgrgicak. Agreed about renaming later if there's no benefit.

[brandonpayton]

Next up:

• Separating Playground app state into a different file. Maybe <site>/.wp-playground-state.json.
• Saving a site's MEMFS files to OPFS in the background
• Handle MEMFS-to-OPFS process being interrupted. This probably means:
  • Not loading a temp site from OPFS unless we've marked that process as complete.
  • Supporting overwriting existing files during the MEMFS-to-OPFS operation
• Adding versions (or maybe schema declarations?) to these metadata and state files to give us an opportunity to be backwards compatible.

I'm hoping to make good progress on these yet today, but please feel free to take any of these if you agree with the direction when you pick up this PR. 🙇

[brandonpayton]

We are discussing backing off of this approach as it is making temporary sites less temporary.

Instead, we need to find a way to coordinate app updates with users.

Let's pause this until we decide what to do.

[adamziel]

Let's discontinue this work:

• This PR adds a lot of complexity: keeping tabs of stale sites, archiving them, deleting them, having a UI to browse them, dealing with storage limits and non-OPFS browsers, just to name some of it.
• Playground users actually want temporary sites that are easy to scrap.
• Even if they didn't, we'd still need to support temporary sites for browsers without OPFS support, e.g. Safari in private mode.
• Refreshless website deployments – load remote.html using the network-first strategy #1849 solves the problem this PR was meant to solve.

This reveals a larger, strategic point: The Playground webapp may never have advanced site management features such as:

• Defaulting to permanent sites
• Archiving and restoring sites
• Synchronizing the application state via cross-tab messaging

Building these features is definitely possible, but it requires spending significant amount of time and effort from building the platform to building a user-facing product. Let's leave that to app authors.

cc @bgrgicak @brandonpayton