Implement controlled list manager and reference datatype #10541

[jacobtylerwalls]

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Description of Change

Implement the initial features for the Controlled List Manager.

TODOs:

• wire up last bits of item editor
• new i18n pattern for vue
• refactor to typescript
• make design changes per vue style guide discussions

Run migrations; then, to import some sample test data into your local db:

PYTHONPATH="../arches" python3 manage.py shell -c "from tests.views.controlled_lists_tests import ControlledListTests; ControlledListTests.setUpTestData()"

It's likely you'll need a manage.py updateproject also.

Give your test user the RDM_Administrator group.

Issues Solved

Closes #10541
Closes #10556

Checklist

• Unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Ticket Background

• Sponsored by: GCI

Demo

Component breakdown

[chrabyrd]

Overall looks great. There's a few things to hash out as patterns but a really strong start to laying the Vue groundwork 👍

Most things I flagged once but the request should be considered throughout ( eg no arches translations )

[jacobtylerwalls]

Thanks for the review--grateful to have your eyes on this!

[chrabyrd]

overall looks great, still a few things to work out && likely larger conversations to have. Of course this can be taken care of in smaller chunks in later mockup iterations as well.

[chrabyrd]

Now that #10616 has been integrated this opens up the conversation around theming.

By EOQ1 2024 the PrimeVue team will release v4, which has an updated theming architecture

Currently the roughed-out POC involves importing the stylesheets and manually editing the attrs based on selected theme.

Hopefully we can kick the can long enough for the updated version of PrimeVue so that we don't need to waste cycles on 2 solutions. However if need be we can drop theme-switching in the interim in favor of a css files that imports a primvue theme file and adds/overwrites values, which is then imported into base.htm instead of the the theme css directly.

But while we're on the subject, regardless of architecture, what would populate the stylesheet? Colors for sure. Typography? Font-size? Border-radii? Seems we'd probably want to be on the 'light touch' end of the spectrum but I'm interested to hear your thoughts.

[chrabyrd]

This probably opens up a larger conversation / can of worms but moving forward I'd like to move as much logic out of views as possible, and have them essentially just handle parsing args, calling other functions, and handling responses.

[jacobtylerwalls]

I like it. So is the idea to break this up into three smaller functions and put them under /serializers?

[jacobtylerwalls]

Broke up the serializer into model methods in fb740ae 👍

[jacobtylerwalls]

Thanks, those were really useful nudges re: PUT and re: the file hierarchy. I now have this, LMK what you think!

(UPDATE: I since renamed the types for the arches object from arches.ts to types.ts)

The one thing hanging out in plugins is the gluing code connecting knockout to vue, and ultimately just pointing to the ControlledListsMain. The controlledlists folder (name chosen to mirror the app name in #11072, but I can also imagine "graphdesigner" and whatnot) now contains:

• loose .ts files for api, const, utils etc.
• a components/dir with:
  -- a single highest-order component
  -- subdirs for other components

This is making me think the tests should live side by side with each file.

[chrabyrd]

the API looks good!

3 small thoughts on the folder hierarchy:

• I think it would save us potential headaches to have arches/app/src/theme.ts and arches/app/src/types.ts moved to arches/app/src/arches/(types, theme).ts, because of how webpack overwrites static assets that exist at the same path.

• can src/controlledlists be delineated? eg controlled-lists?

• I'm wondering how well this pattern will expand. Playing it out could result in

└── arches/
    └── app/
        └── src/
            ├── controlled-lists/
            │   └── *stuff*
            ├── foo-report/
            │   └── *stuff*
            ├── bar-widget/
            │   └── *stuff*
            ├── baz-widget/
            │   └── *stuff*
            ├── qux-card/
            │   └── *stuff*
            ├── graph-designer/
            │   └── *stuff*
            ├── arches/
            │   ├── theme.ts
            │   └── types.ts
            ├── plugins/
            │   └── ControlledListManager.vue
            ├── reports/
            │   └── fooReport.vue
            ├── widgets/
            │   ├── barWidget.vue
            │   └── bazWidget.vue
            ├── cards/
            │   └── quxCard.vue
            └── views/
                └── GraphDesigner.vue

Which isn't great as the hierarchy is a breakdown of of application-level components and vue-components.

An alternative could be:

└── arches/
    └── app/
        └── src/
            ├── controlled-lists/
            │   └── *stuff*
            ├── foo-report/
            │   └── *stuff*
            ├── bar-widget/
            │   └── *stuff*
            ├── baz-widget/
            │   └── *stuff*
            ├── qux-card/
            │   └── *stuff*
            ├── graph-designer/
            │   └── *stuff*
            └── arches/
                ├── theme.ts
                ├── types.ts
                ├── plugins/
                │   └── ControlledListManager.vue
                ├── reports/
                │   └── fooReport.vue
                ├── widgets/
                │   ├── barWidget.vue
                │   └── bazWidget.vue
                ├── cards/
                │   └── quxCard.vue
                └── views/
                    └── GraphDesigner.vue

where application-level delineation happens insde the arches/src/arches folder. To me this is a bit better because it comes with de-facto namespacing, so that if someone wanted to write an arches application that overwrites fooReport.vue, they'd need to consciously put it at my_cool_project/src/arches/reports/fooReport.vue instead of my_cool_project/src/reports/fooReport.vue. That could also save some headache of someone out in the wild accidentally overwriting a thing.

Though to me this is still not ideal, in the top level there's mixed breakdown: applications (eg arches) are mixed with componentry (foo-report, bar-widget).

Maybe a better answer is:

└── arches/
    └── app/
        └── src/
            ├── controlled-lists/
            │   ├── plugins/
            │   │   └── ControlledListManager.vue
            │   └── modules/
            │       └── *stuff*
            └── arches/
                ├── theme.ts
                ├── types.ts
                ├── reports/
                │   └── fooReport.vue
                ├── widgets/
                │   ├── barWidget.vue
                │   └── bazWidget.vue
                ├── cards/
                │   └── quxCard.vue
                ├── views/
                │   └── GraphDesigner.vue
                └── modules/
                    ├── foo-report/
                    │   └── *stuff*
                    ├── bar-widget/
                    │   └── *stuff*
                    ├── baz-widget/
                    │   └── *stuff*
                    ├── qux-card/
                    │   └── *stuff*
                    └── graph-designer/
                        └── *stuff*

Where the top-level is delineated by application. The next level is broken down into application structure and a modules folder where all the sub-components live. It's still not ideal because, say, what if I wanted to have an api.ts file for both foo-report and bar-widget? I guess it could go in modules but then where's the delineation between something like types.ts?

But yeah I'm not sure if an ideal solution exists. But I want to hear your thoughts on this and see if we can't land somewhere close.

btw everything else looks good, once folder structure lands this is g2g in my opinion 👍

[chrabyrd]

But then again I'm not sure. The CLM is technically it's own django app but it's still part of arches-core and the arches-core build process/bundle. maybe:

└── arches/
    └── app/
        └── src/
            └── arches/
                ├── theme.ts
                ├── types.ts
                ├── reports/
                │   └── fooReport.vue
                ├── widgets/
                │   ├── barWidget.vue
                │   └── bazWidget.vue
                ├── cards/
                │   └── quxCard.vue
                ├── views/
                │   └── GraphDesigner.vue
                ├── plugins/
                │   └── ControlledListManager.vue
                └── modules/
                    ├── foo-report/
                    │   └── *stuff*
                    ├── bar-widget/
                    │   └── *stuff*
                    ├── baz-widget/
                    │   └── *stuff*
                    ├── qux-card/
                    │   └── *stuff*
                    ├── graph-designer/
                    │   └── *stuff*
                    └── controlled-lists/
                        └── *stuff*

is best? With the idea that all project are now spawned with a src/my_cool_project/declarations.d.ts file instead of src/declarations.d.ts . It makes more sense, but still doesn't really answer the questions raised above 🤔 .But I guess this reinforces my thought that the CLM should be its own arches application.

[jacobtylerwalls]

can src/controlledlists be delineated? eg controlled-lists?

Sure, I think that makes sense for the frontend. I was trying to keep 1:1 parity with controlledlists on the backend, because controlled-lists is not a valid python identifier, but I think it's fine to have controlled-lists on the frontend.

But then again I'm not sure. The CLM is technically it's own django app but it's still part of arches-core and the arches-core build process/bundle.

Maybe in #11072 we need to explore having a /src under arches/controlledlists.

[jacobtylerwalls]

Your last suggestion is really appealing. Some items that occurred to me:

• I like the idea of keeping controlled-lists at the same indentation level as arches, to reinforce the idea that it could be extracted into its own repo later if we wish. I guess I would do the same for big topics like graph-designer.
• baz-widget is probably too small potatoes for such treatment, so perhaps nest the baz-widget components in the same place baz-widget.vue is defined
• I'm not sure where plugins/ControlledListManager.vue should go, but for now, I like the idea of moving all plugins under arches/.

That gives this slight variation on your last suggestion (and all it requires me to do now is move plugins under arches, as you suggested):

└── arches/
    └── app/
        └── src/
            └── arches/
                ├── theme.ts
                ├── types.ts
                ├── reports/
                │   └── foo-report/
                |       └── *stuff*
                ├── widgets/
                │   ├── bar-widget/
                |   |   └── *stuff*
                │   ├── baz-widget/
                |   |   └── *stuff*
                ├── cards/
                │   └── qux-card/
                |       └── *stuff*
                ├── views/
                │   └── GraphDesigner.vue
                ├── plugins/
                │   └── ControlledListManager.vue
            └── controlled-lists/
                └── *stuff*
            └── graph-designer/
                └── *stuff*

What do you think? (We can still explore moving controlledlists up some more levels, outside of arches/app/src on #11072.)

[apeters]

Your last suggestion is really appealing.

That gives this slight variation on your last suggestion (and all it requires me to do now is move plugins under arches, as you suggested):

└── arches/
    └── app/
        └── src/
            └── arches/
                ├── theme.ts
                ├── types.ts
                ├── reports/
                │   └── foo-report/
                |       └── *stuff*
                ├── widgets/
                │   ├── bar-widget/
                |   |   └── *stuff*
                │   ├── baz-widget/
                |   |   └── *stuff*
                ├── cards/
                │   └── qux-card/
                |       └── *stuff*
                ├── views/
                │   └── GraphDesigner.vue
                ├── plugins/
                │   └── ControlledListManager.vue
            └── controlled-lists/
                └── *stuff*
            └── graph-designer/
                └── *stuff*

I like this layout of all the ones discussed. I like that the apps are segregated into their own directories at the level of arches and have their own hierarchies associated with them.

[chrabyrd]

Hmm this is closer but I'm still not sure. With

└── arches/
    └── app/
        └── src/
            └── arches/
                ├── theme.ts
                ├── types.ts
                ├── reports/
                │   └── foo-report/
                |       └── *stuff*
                ├── widgets/
                │   ├── bar-widget/
                |   |   └── *stuff*
                │   ├── baz-widget/
                |   |   └── *stuff*
                ├── cards/
                │   └── qux-card/
                |       └── *stuff*
                ├── views/
                │   └── GraphDesigner.vue
                ├── plugins/
                │   └── ControlledListManager.vue
            └── controlled-lists/
                └── *stuff*
            └── graph-designer/
                └── *stuff*

graph-designer is out-of-place. arches and controlled-lists are both applications, graph-designer is a view. Also what about arches/theme.ts and arches/types.ts? The graph designer is part of Arches and should exist in its hierarchy. In this pattern it would instead have graph-designer/theme.ts and graph-designer/types.ts. But then this what about arches/foo-report? In the current pattern we should move it out to src/foo-report, with only the top-level component existing at arches/reports/FooReport.vue

After giving it some thought I guess I'm leaning towards more closely emulating the folder structure outside of src. eg

.
└── arches/
    └── app/
        └── src/
            └── arches/
                ├── theme.ts
                ├── types.ts
                ├── reports/
                │   ├── fooReport.vue
                │   └── foo-report/
                │       ├── types.ts
                │       ├── api.ts
                │       └── *stuff*
                ├── widgets/
                │   ├── barWidget.vue
                │   ├── bazWidget.vue
                │   ├── bar-widget/
                │   │   ├── types.ts
                │   │   └── *stuff*
                │   └── baz-widget/
                │       └── *stuff*
                ├── cards/
                │   ├── quxCard.vue
                │   └── qux-card/
                │       └── *stuff*
                ├── views/
                │   ├── GraphDesigner.vue
                │   └── graph-designer/
                │       ├── types.ts
                │       ├── api.ts
                │       └── *stuff*
                └── plugins/
                    ├── ControlledListManager.vue
                    └── controlled-list-manager/
                        ├── types.ts
                        ├── api.ts
                        └── *stuff*

Although ControlledListManager is it's own app, it's also just a plugin, and still part of the Arches build/bundle. I admit it's not ideal but it makes sense. An added benefit here is that if someone wanted to, say, overwrite the header of the Graph Designer, they'd replace the file at src/arches/views/graph-designer/Header.vue instead of src/graph-designer/Header.vue.

@jacobtylerwalls with the work to move this out of arches-core, controlled-lists will have its own src folder, making the shoehorning of it here unnecessary, right?

[jacobtylerwalls]

Yeah, my hope is to explore giving it its own /src folder in that PR, but as you know that requires fiddling a bit with webpack.

Re: whether the graph designer is a view, maybe I picked a bad example, but I was trying to pick something that could be an app? What would a "view" be in this context?

[chrabyrd]

Yeah, my hope is to explore giving it its own /src folder in that PR, but as you know that requires fiddling a bit with webpack.

Re: whether the graph designer is a view, maybe I picked a bad example, but I was trying to pick something that could be an app? What would a "view" be in this context?

Yeah that should hopefully be a one-line change, and I'm happy to test that out && get it to you. One thing I hate is that it will de-standardize the webpack logic between projects and core.

IMO nothing should be a Django app, not even this 😂 . But since we're here, I'd say nothing more should be broken out in this fashion.

But for the sake of imagination, let's say search. If it gets broken out in the same fashion as the controlled list manager, it should still hopefully follow the same pattern of having its own src folder (eg: arches/search/src/search/*stuff*, outside of arches/arches), and essentially being out of the Arches folder hierarchy.

[jacobtylerwalls]

I see where you're coming from. Another possibility is to not view "arches/app/src/arches" as "everything having to do with core arches" but more like "arches-router" where it's just a collection of single vue components (plugins, views, reports, whatever) that link out to other folders that sit beside arches. Then, if you take that mental model, you're less bothered by the idea of graph-designer sitting beside arches. That's where I was coming from. Maybe we'd want to rename arches -> arches-routes (or something better) in that case.

In any case, I think we should keep innovating on this the next time we tackle a big vue feature, regardless of whether it's a distinct Django app or not. Sounds like we're talking about where to put future features, and finalizing this feature can happen on #11072.

[chrabyrd]

Yeah while I don't love the pattern, I hear you that it doesn't need to be discussed here in the main body of work. However I do feel we shouldn't punt this until the next large feature, as this is a good opportunity to solidify a pattern.

So yeah I'm happy to approve here and hash out the directory structure in #11072 .

[chrabyrd]

https://c.tenor.com/O-o1GtG_Jy8AAAAC/tenor.gif