Right click no-hold painting, ctrl+left click fast delete

[kephale]

@carsen-stringer ok well I guess I wanted to procrastinate on making my actual presentation material 😁 . This is testable, but I don't think it should be merged until the trigger for disabling the CellPose painting mode.

This adds:

1. Right click to start creating a path with points, such that when the cursor returns to the initial point the path is closed and filled in the Annotations layer.

2. (Fast delete) Ctrl + left click to delete the label under the cursor

A problem:

I'm still missing a proper connection to disable this mode once the widget is closed. This will just take some fiddling.

cellpose_hoverpaint_v03.mp4

[kephale]

@carsen-stringer points out that the initial point needs to be a large open point to work as an end zone. currently the initial point starts large and is getting reset once other points are added.

[kephale]

painting_with_end_zone.mp4

now with working end zone rendering.

[melonora]

Hey sorry for jumping in here. This looks very much like the polygon lasso tool in the shapes layer. Do I understand correctly it is being implemented here but for the labels layer? If there is a current request I might be able to find some time to duplicate the behaviour to the labels layer in napari core itself

[kephale]

@melonora yeah, there is definitely a lot of overlap w/ the polygon lasso. If you can get a labels lasso merged then we would certainly try to reuse as much of it as possible. There are at least 2 nuances about why it would still at least have a wrapper here:

• the mouse bindings here are optimized for annotator efficiency
• an visible "endzone" for completing the region

However, I'm not sure I would block this merge for the upstream change. We can always adjust this code accordingly.

[melonora]

I wouldn't block but will make a mental note

[kephale]

Awesome, thank you @melonora !

[melonora]

Just one question, given that the polygon lasso tool does not render the face of the polygon, is an end zone required?

[kephale]

@carsen-stringer can chime in, but for this PR we implemented the annotation mode that was currently optimized based on annotator experience for the cellpose standalone tool.

I still expect that the napari feature won't match this exactly (at least for keybinds and probably endzone as well). So I do not think it is necessary to try to replicate this PR exactly in napari. napari is hackable and should be hacked. The feature that I would argue is missing from napari is having a lasso for labels. Over time we can converge on the best UX for that.

[melonora]

Yeah ok, I am just wondering if there is experience here with the polygon lasso tool in shapes and whether that on itself fullfills the needs after refactoring of labels / shapes to use the same tooling:)

[kephale]

Yeah, the shapes polygon lasso is fantastic and well implemented. The end zone based trigger for closing the shape definitely accelerates annotation, and is arguably beneficial in other ways (e.g. clicking can jitter your mouse position).

From napari's perspective, I see end zones as part of cellpose's style of annotation. I don't think we need to add end zones to napari at this point in time because they are straightforward to implement (e.g. you currently close the shape on mouse click, but listening to the mouse position and triggering a shape closure is only a little more complicated). However, not everyone will want this interaction for closing shapes. We should add end zones to napari if either:
(1) cellpose becomes the only tool for annotation in napari,
(2) some tools/people other than cellpose start using the same feature, or
(3) users begin to expect this type of annotation in napari.

[kephale]

From some DMs w/ @melonora I wanted to clarify the feature in this PR w.r.t. napari lasso because the video doesn't capture the full UX of cellpose style annotation here.

The current polygon lasso in napari can be used 2 ways:

• click+hold, drag to draw, release to close the shape
• click, hover to draw, click to close the shape

The unique idea that is coming from the cellpose UI is the end zone. Annotating with the end zone is:

• click, hover to draw and close shape when the mouse returns to the end zone (e.g. use mouse position to trigger closing of the shape)

This means the annotator does not need to hold the mouse button (which leads to fatigue), or make a second click. We know the users are drawing closed shapes so they'll need to return to the start. What isn't clear to me is that every user wants this to happen automatically, or if most people still want to be able to click to close the shape.

Otherwise, I think having the napari shapes lasso work for the labels layer would be a strict improvement (e.g. the napari lasso uses an overlay instead of creating a Points layer, and currently this PR doesn't have a line connecting the points).

[carsen-stringer]

thanks so much @kephale this is amazing!

after running the segmentation, I think the output masks should themselves be a labelpainter so they are editable (I think in this case then I shouldn't have cellpose also return the outlines since they will become out-dated with the edits of the user). also the user may want to do labeling without running segmentation. should we have a button to create a labelpainter or is there another alternative?

[psobolewskiPhD]

This is pretty cool!
I'm going to chime in that using the Shapes lasso tool to label -- which lets you correct a bit if needed easily -- and then Layer context menu Convert to Labels is my go-to these days for these types of labeling tasks.

Edit: note that the napari Labels polygon tool has as distance threshold for double-click-to-close. It wouldn't be trivial, but could add similar stopping behavior to Shapes tools to give a landing zone-like effect.

[kephale]

Chatted w/ @Czaki about this. He pointed out the idea that has previously been discussed of using overlays for the Shapes lassos which would also make it easier to reuse the code for Labels.

[kephale]

Following up again, I met with @melonora about this yesterday and we started coordinating on the lasso implementation for the Labels layer.