Virtual network filesystem

[dashxdr]

Regarding this idea:
#24 (comment)

Since we switched to using the esp-idf's spiffs implementation it is somewhat complex creating a virtual network flash device to imitate the actual flash device. We'd only need to implement 3 functions: read, write, erase. Wrapping that in a network layer so the requests can be transmitted over WIFI would be trivial (both for client + server).

The problem is esp-idf's spiffs implementation hard-codes the read/write/erase functions for its integration of the spiffs library to make use of the esp-idf's spi flash machinery.

It would be straightforward modifying the esp-idf to allow overriding the 3 crucial functions in the register() function, but then we're dealing with convincing esp-idf to accept the change. I haven't dealt with them... and my time is limited. I can't say how responsive they'd be.

So there is a PlanB approach. Don't even bother with spiffs and instead just use the esp-idf's vfs functionality to register a virtual network filesystem. We only need to provide open,close,read,write,fstat functions. https://esp-idf.readthedocs.io/en/v2.0/api/storage/vfs.html

I'm doing some digging to see if someone has already done this. Failing that I'm thinking I'll just go ahead and implement something. Probably do it in javascript on both ends and the duktape-esp32 would just be a bit of 'c' glue. Performance isn't an issue.

[dashxdr]

Ok I got the first attempt working. I did the client in 'c' since mixing javascript + c would require too many back and forth context changes, not sure how it would all workout. The implementation is synchronous only, meaning any network client request will block until the server responds.

Server is implemented in a node.js program. It's all very simple and easy to use. No write support or directory listings or seeks, but the implementation does keep track of the filedescriptors and positions within each file. Currently up to 4 files can be open at once, but likely only 1 would be enough if we're only doing require(). Filenames can't go over 64 characters, including path.

[dashxdr]

New idea occured to me. I ran into a roadblock where I want to include a 100k js file but there is a failure, because the read buffer can't cope with a file that large.

Duktape says http://duktape.org/guide.html#comparisontolua.14 that it doesn't support streaming compilation since it needs multiple passes over the source code, but I bet it wouldn't be hard to modify duktape to support parsing from a callback function with arguments of offset, length. The parser would only need to read a single token at a time.

We don't care if we need to go over a file more than once. We just want to get away from having the whole file sit in memory ready to go. If we integrate a solution of compile-from-file-descriptor where we can rewind the file descriptor in duktape we can get rid of the whole espfs layer and use only spiffs. Moreover whatever RAM we have to keep available for parsing larget javascript files can be devoted to the end application.

[nkolban]

Back in spring of 2017, there was discussion with the Duktape author about being able to run JavaScript directly from flash. He was thinking about that but then we never followed up with him. He is always present on the IRC channel called #duktape. It might be worth our time asking if things have changed (in the year that has passed) about running JavaScript directly from flash.