Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epic Roadmap to new version of Dnotebook #10

Open
3 of 6 tasks
risenW opened this issue Jan 29, 2022 · 1 comment
Open
3 of 6 tasks

Epic Roadmap to new version of Dnotebook #10

risenW opened this issue Jan 29, 2022 · 1 comment
Assignees

Comments

@risenW
Copy link
Member

risenW commented Jan 29, 2022

We decided to rewrite Dnotebook to be more like Jupyter notebooks, where we can write and run code that interacts directly with the user's local OS in a safe and secure way. This overarching goal here is that we want users to have a native experience when using Dnotebook.

The following is a list of proposed features for the new release:
Target release date: March 2022

  • Rewrite core components in Typescript
  • Add a new code editor with syntax highlighting, lining, snippet, and autocomplete.
  • Add support for file system directory: Users can interact with their local file system from the UI so they can create, update and delete files securely
  • Bash command support for managing notebooks: Users can execute bash commands from the UI. This makes the installation of new packages relatively easy and intuitive.
  • Better UI/UX
  • File menu options for managing kernels and notebooks
@risenW
Copy link
Member Author

risenW commented Jan 30, 2022

Some musings on design architecture so far:

Update on the server side

The first week, I started by asking the question, how do I make Dnotebook work like Jupyter, meaning bash command works so I can run yarn add package in the browser, execute it server-side in Nodejs, and get real-time output logs in the browser.

I decided to go with the express HTTP stream. Nodejs has great support for streams, so I would basically start a server, send commands via HTTP request, open a stream response, capture logs from stdout and stderr, and continuously write back a response to the UI. This is basically done at the moment. see https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/routes/code_routes.ts

Now for the next task, I wanted to execute JS code. Executing JS code one time and returning the result is pretty straightforward, we can use an eval for that. But what I needed is a way to execute each cell in the same context. I.e All cells from the notebook will share the same variables-the typical notebook behavior So what do I do here?

Well, interestingly Nodejs has the VM API, which you can use to compile and execute code in a given context. So basically what I did was start a server, initialize a context, then send chunks of code to be executed in that context. Because the server keeps running between execution, the context is kept alive! This was implemented here: https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/server/src/runner/nodejs.ts

So with the core functionality done, I decided to add some more features.

First, I added an inline transpiler, meaning you can write any flavor of JS, and mid-way before it gets executed, it is transpiled using Babel to a compatible version that can run on the VM. See implementation here:

let transformedJsCode = transformSync(code, {

So what next? Well, a lot of things still has to be done on the server-side, some of the important issues I’m yet to figure out is:

  1. How to track the state of cells and know when they finish execution? This is quite tricky given the async nature of JS

  2. Managing multiple child processes-Kernels in Jupyter notebooks-so I can easily start and stop them from the UI

  3. Security of code process

Update on the UI

At the moment, it supports full editing of Markdown, JS, HTML, bash, and JSON. That means, syntax highlighting, code snippets, autocomplete, and execution all work at the moment. I used AceEditor for this. See implementation https://github.com/javascriptdata/dnotebook/blob/ref/typescript/src/frontend/components/CellEditor/index.tsx

So what’s next here? Well, the next major task we want to tackle in the UI is:

I will keep updating this comment as we add new features and encounter weird issues. This will be used to create a detailed architecture document before release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants