Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wasi: allow users to configure a filesystem that can accept new files #390

Closed
codefromthecrypt opened this issue Mar 18, 2022 · 13 comments
Closed
Assignees

Comments

@codefromthecrypt
Copy link
Contributor

codefromthecrypt commented Mar 18, 2022

WASI defines portable system calls to manipulate files. We currently use fs.FS to map to many of these. read-only operations are implicit, but the status quo goes beyond this in the standard library.

For example, fs.FS can write existing files as implementations are free to go beyond the functionality defined on fs.File. Some of this is encouraged. Concretely, os.File goes beyond with mutations such as chmod and write. os.FS

Also, there are standard extensions to fs.FS including fs.ReadDirFS. The point is that in many cases, there is an existing mapping of a go interface to WASI functions, even if it isn't required by fs.FS. By safe casts, you can get most functionality needed.

However, there are gaps, notably creating new files. This issue is solved when the core WASI APIs requested by end users have a corresponding implementation. This is ideally done without introducing any wazero-specific interfaces. As such research is needed and validation is needed prior to introducing an interface not in Go's standard library.

Moreover, as the README suggests, completion of all WASI functions is a non-goal. In other words, if there is a tricky API, and hasn't been asked directly by end-users yet.. wait until that's the case, then solve it. WASI is completely changing its filesystem API so we can conserve energy by only focusing on what users need today.

Solving this issue is a high priority and 1.0 will be held back until it is solved. However, it is likely this will be solved way before 1.0, like in a week or two.

References:

@codefromthecrypt codefromthecrypt changed the title wasi: add writeable alternative to the fs.FS API wasi: add interfaces to create new files (missing in fs.FS API) Mar 18, 2022
@codefromthecrypt codefromthecrypt changed the title wasi: add interfaces to create new files (missing in fs.FS API) wasi: allow users to configure a filesystem that can accept new files Mar 18, 2022
@codefromthecrypt
Copy link
Contributor Author

I completely rewrote the description as the gap here is much smaller than I originally thought. While fs.FS doesn't specify extended APIs it is routine in practice to already support writes (ex os.File).

@mmathys
Copy link

mmathys commented Jul 14, 2022

Does that mean that it is possible to write new files from Wasm?

I have tried using the open syscall in WASI for non-existing files with writing mode, but it fails.

@codefromthecrypt
Copy link
Contributor Author

@mmathys exactly. in fact this it is about time we got back to this one. if you don't mind, as it is usually interesting.. what's the use case you are working with that needs to open for writing? is it a log or config file? or are you sharing data?

@mmathys
Copy link

mmathys commented Jul 15, 2022

We are using files to keep state between function calls when writing functions in C.

@codefromthecrypt
Copy link
Contributor Author

@ncruces noted in #740 that there's a corner case in the io.Reader docs hard to reason with when mapping to io.Reader based implementation of fdread

Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.

https://pkg.go.dev/io#Reader

The "process the n > 0 bytes returned before considering the error err" ends up a bit dodgy as you have a choice to ignore that part or create expense by holding per-fd state to return the error when >n bytes are read.

@ncruces
Copy link
Collaborator

ncruces commented Aug 11, 2022

Looking at the issue and current implementation, I don't think there's anything missing from wazero in this regard.

For instance, if you use os.DirFS you'll get a read-only view into a directory and, from what I could tell, you won't be able to create or modify any files.

But you don't have to use os.DirFS, you can create a copy of it that does OpenFile with O_RDWR here, then with the current implementation you'll have writeable files in your WASI.

Or you can create an implementation that, instead or returning fs.ErrNotExist for a missing file, os.Creates one. With this, you should already able to write new files, no?

Or you don't create an actual os.File, but your own “file” object, that's a thin wrapper around a bytes.Buffer. Then you get memory backed writeable files.

Now, all this has subtleties, but it seems doable, and not something wazero needs to provide out-of-the-box.

As an example, for my usage I wanted to lock my WASM even harder, so I created a file system that can only ever open a single file (not an entire directory).

@mmathys
Copy link

mmathys commented Aug 11, 2022

To fix this, I just replaced instances of

f, err := c.fs.Open(fsOpenPath)

by

f, err := os.OpenFile(fsOpenPath, os.O_APPEND|os.O_CREATE|os.O_RDWR, 0644)

and I get a writable FS.

@codefromthecrypt
Copy link
Contributor Author

The first way we'll implement this is via a new ModuleConfig which allows you to pass a path instead of an fs.FS. When that's the case, we'll use os.OpenFile after coercing WASI flags to go ones. This will happen by our next pre-release, and likely in the next day or two.

@codefromthecrypt codefromthecrypt self-assigned this Dec 13, 2022
@codefromthecrypt
Copy link
Contributor Author

https://github.com/tetratelabs/wazero/releases/tag/v1.0.0-pre.6 adds writefs.NewDirFS, which lets you add new files. We'll track pluggability of this in another issue.

@paralin
Copy link

paralin commented Jan 1, 2023

This is awesome!

@mmathys
Copy link

mmathys commented Jan 1, 2023

Great work!

@karelbilek
Copy link
Contributor

If someone is looking for it now and hits this issue and wonders, where it is; it's in

github.com/tetratelabs/wazero/experimental/sys

https://pkg.go.dev/github.com/tetratelabs/[email protected]/experimental/sys#FS

type FS interface {

@paralin
Copy link

paralin commented Feb 20, 2024

See #2076

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants