[Discuss] ProtoJEP - the evolution of the Classic Notebook v6 and v7

[Zsailer]

ProtoJEP - the evolution of the Classic Notebook v6 and v7

• Notebook 7 is based on the JupyterLab codebase, with "the same" user
  experience. It will achieve this by using today's RetroLab.

• When Jupyter Lab 4 or notebook 7 are installed, it's a single set of
  tools, and both commands start the majority of the same machinery.
  Typing 'jupyter lab' or 'jupyter notebook' will allow users to move
  between the two visual experiences easily. Installing one always
  installs the other so the underlying tools are always available.

• There will be seamless ways for users to move back and forth between
  notebook and lab in those applications.

• The `jupyter lab` and `jupyter notebook` commands always start
  both and only determine the default view.

• Define "the same"

  • Existing documentation works unchanged: old screenshots, videos,
    manuals, keyboard shortcuts, etc, that explained how to do
    something in classic, are not invalidated. Educators who have
    such materials online don't need to change anything.

  • Critical extensions should work on day 1. We need to identify
    those; These extensions will need updates to work with the new
    API, but we pledge to work on porting those core extensions.

    • Nbgrader
    • RISE
    • TOC. Note: can we port it to reuse the current Lab TOC,
      but with its own, notebook-native UX?
    • Collapsible headings. As noted in Keyboard shortcut parity (with notebook) for collapsible headings jupyterlab/jupyterlab#11401, it is related to TOC but separate, and its keyboard behavior is key to its UX (cc @jph00).
    • More domain-specific extensions. These may serve narrower communities, but we will strive to assist them in the transition to the best of our ability:
      • Poliastro CZML viewer - for visualizing geospatial data in CZML format using CesiumJS (cc @astrojuanlu).
    • Any others? We need to do more outreach to find out.

  • Work has been done to document and assist authors in porting
    their own extensions, so they might have updated them to nb7.

• There will be a central location with information on how to port
  extensions for others to do so in the future.

• Performance?

  • What do we want to tell users about or commitment to
    performance.

  • Measure and communicate: most likely, performance will be equal
    or better than in classic, as the bulk of the slowdown in Lab
    comes from multiple tools on the page and activity switching. By
    definition those won't exist in nb7. But we need to have some
    hard data on this to communicate.

• Future goals:

  • Improvements to the APIs to facilitate exploration in JS/CSS.
    Some of the ease of lightweight exploration of the web page that
    is possible under nb6 also creates the maintainability/security
    nightmares we have. How to create most of that experience under
    the new APIs will need development.

  • Potential new high-level APIs for a limited class of extensions,
    akin to how Seaborn provides limited, but powerful and
    easy-to-use plotting in matplotlib for a set of common use
    cases.

User Stories (north star, long term)

• As a user, when I install lab or notebook, I get both experiences,
  so I can choose between them as I work.

• As a user, when I start the application using `jupyter notebook`
  or `jupyter lab` the default view that I see will correspond to
  notebook or lab respectively. The ‘’handler is also updated
  accordingly to point to that application.

• As a user, when I am using the lab or notebook interface, I can
  easily move back and forth within the UI using easily discoverable
  menu items, context menus, buttons, etc.

  • Open this document in the notebook interface

  • Open this document in the lab interface

  • Open this directory in the notebook file browser

  • Etc.

• As a user, when I am using notebook 7, the visual design strongly
  matches what I am used to for the classic notebook.

  • As a user, I want the visual design of the notebook interface to
    get better over time, so my user experience improves over time.

  • As a user, I want the visual design of the notebook and lab to
    be closely aligned, so I know that I am using a single, unified
    experience.

  • As a user I want to UI to not change dramatically so that my
    tutorials, and class material do not get out of date.

• As a user, I want extensions that need a left/right/bottom/status
  area to work in notebook, but I want those areas to be entirely
  hidden by default, so I can have the document oriented notebook
  experience that I am used to.

• As a user, I want to know all of the great new things I am getting
  with notebook 7, so I can make an informed decision to upgrade.

• As a user, I want the most common nbextensions to work in both the
  notebook and lab interfaces, so I can move back and forth without
  missing functionality.

  • RISE
  • Nbgrader
  • TOC
  • Collapsible headings

• As a user, when I install Jupyter Desktop, (the lab4/nb7-based
  native desktop suite, taht consists of the Notebook and *Lab
  *apps), I can double-click on an .ipynb file on my local computer
  (Mac, Win, Linux) and that file opens as a single document that I
  can edit, run code into, etc. The same installation of Jupyter
  Desktop also offers a JupyterLab application with a more IDE-centric
  experience, including file management, terminals and more.

  • From my command-line I can open single files with either
    application (Notebook or Lab), and I can also open Lab pointed
    to a whole directory.

  • In the future (not planned as part of the initial release),

    • Jupyter Desktop will also let me open my Jupyter editor
      (with any extensions I may have installed) as a single
      application, and Jupyter Console.

    • This will let me open otherJupyter-based tools in my native
      desktop experience, such as Consoles attached to Notebooks,
      in their own Native window.

    • Jupyter Desktop will recognize and syncrhonize with any
      web-based Jupyter Servers I may have opened on the same
      system, and can also connect to remote servers and kernels,
      so that I can have the same native Desktop experience
      against remote resources.

Technical details

• Nb7 is the current retrolab code bases.

• Notebook repo is moved (at some point) under jupyterlab org.

  • Keep on the same org to allow to migrate relevant issues

• Port nbgrader and RISE to work with lab4/nb7.

• Question: should the notebook repo get a PR for nb7 with retrolab
  code, or move the retrolab into notebook

  • Likely want to rename “retrolab”repo, to “notebook”and migrate
    relevant issues to it. We can rename "notebook" -> "notebook6"
    and leave it permanently available but with no new work done on
    it after a final release.

  • Will need to sort out details of issue management - technically
    it would fit better under jupyterlab, but the jupyter/notebook
    name/location is great, and moving

Community and communication details

• Find out which other extensions have a critical impact for large
  user bases.

• Gather feedback on github, discourse, etc before turning this into a
  JEP.

• Engage with the Jupyter Extensions community.

• User-facing communication on nb7: it's a brand new version of the
  notebook that has the following features:

  • Theme less is more:

    • Familiar

    • Nostalgic

    • Reviving a beloved project/technology

  • Real-Time Collaboration

  • Dark mode

  • Debugging

  • Internationalization

  • Accessibility. (we won’t have accessible experience, but theres
    less work to do)

  • Access to the Jupyter Extensions Store (made-up name)?

• We should host clinics for some amount of time to help plugin
  authors (nbextensions) to modernize and port their extensions over
  to nb7.

Originally posted by @fperez in #6210 (comment)

[fperez]

As discussed here by @jph00, the collapsible headings extension should be ported too (or the current functionality built-into Lab extended to match those capabilities). In particular, its keyboard-based behavior should be matched.

Note: listing it here so the JEP, when fully fleshed out, has a good accounting of key points raised by users. Eventually these will need to be opened as separate issues elsewhere, but I don't want it lost in the #6210 thread, which is very long.

[fperez]

Minor edits for heading structure and to add the collapsible headings extension to the "day 1" list as per @jph00.

[fperez]

In this issue, @astrojuanlu mentioned that their CZML widget isn't quite Lab-ready. This is a more domain-specific widget, but we should equally consider these use cases and do our best to help these community members with the transition.

I will list it above in the ProtoJEP in a section on slightly more domain-specific tools, that while not as broadly applicable as say nbgrader, TOC or RISE, we still need to keep in mind.

[tonyfast]

we should note zen mode as a new feature, that was an extension before.

[tonyfast]

retrolab in using jupyterlab as a framework, it is an extension. would it be out of scope to have a new extra_requires for other good jupyterlab extensions that enhance the authoring experience like language server protocol or the recently used launcher. i feel like these would be reason for folks to migrate. there is a discussion about shipping a jupyterlab-contrib metapackage, it would be worth tracking this effort to see what features we could hind behind some flags. jupyterlab-contrib/jupyterlab-contrib.github.io#23 (comment)

pip install notebook[cool-stuff] 🤷

[bollwyvl]

I've mentioned this a few places, but i think this is really an opportunity to cut many UIs, (lab, classic, voila, thebe, lite) down to structural and behavioral components, and allowing them to be formally described, visually authored, themed, humanely tested, shipped, and deployed for the browser (initially) but also the terminal and/or native (haha, not electron) platforms.

In essence, this is the client-side equivalent of jupyter-server/team-compass#11, and would likely be influenced by (if not actually implementing concepts from) OpenAPI and OpenAsync.

features we could hind behind some flags

While I do use extras_require for e.g. docs and tests, I don't really trust them for interactive usage, as the pip solver doesn't really take them into account on subsequent uses/upgrades. Put differently, they are great for:

• capturing development lifecycle information
• part of a requirements.txt in a repo2docker setting

But are poor for:

• pip install

Having curated, documented metapackages, with known-compatible ranges is probably a better play.

All this being said: I don't have a lot of skin in the game in using or developing for the classic UI or RetroLab if it takes me any additional effort whatsoever beyond Lab. As a user, I would probably favor something even more minimal but less opinionated by default. Such a thing would likely eschew visual cells divisions, and modal editing entirely, but still could be modeled out of the above data-driven concept.

[rgbkrk]

I love this plan. Thank you @Zsailer

[jtpio]

Closing since the JEP has been accepted:

• Jupyter Notebook version 7 enhancement-proposals#79
• https://jupyter.org/enhancement-proposals/79-notebook-v7/notebook-v7.html

Thanks all!!