Proposal for enabling alternative UI frameworks through initial separation of core and UI

[jsmtux]

tl;dr : I created a branch that has separated the UI from the core code of lmms and would like help to test, improve it and eventually merge.

Hi all,

First of all would like to express my gratitude for such a nice open source application, I always try to use open source software in my day to day, and I feel this is the best music making software available in that category. Because of this, and since I've worked in the past in similar GUI applications, I would like to share how I feel it could be improved, and the steps I already took to make that happen.

I think for any newcomer to the software, the first impression is that the UI is quite dated and inflexible. More important, after having a look at open issues and discussions, there are lots of improvements that cannot be achieved because of the way the UI code is intertwined with the core functionality.

A few weeks ago I started a proof of concept for separating the UI from the rest, with the aim of being able to compile the core of the app without the Widgets dependency. The first step was to make the ui a separate compile unit. If you look at the current CMake scripts, they just gather all sources and compile them together, making it impossible for them to have different dependencies.

Then, I started taking all the UI header files out of the 'include' folder and put them into the UI folder. This way, I could make sure that core was not calling any of them. After this was done, I needed to slightly modify the plugins to indicate they were QWidget plugins, as each one of them contains UI information.

With this, I managed to create a core library that is used by a new target, lmms_headless, that produces an executable that can be invoked from command line and has no QWidget dependencies, and the previous lmms executable. Even if it mostly works, I've only modified files compiled as a default in a Linux system. There are obvious things that don't fully work like the open/save dialogs, but it is already possible to test lots of the functionality.

I'll continue working on this on my spare time, but it would be fantastic to get help testing the software and making sure all targets and options compile and work, so that it could eventually be merged. You can find it here.

My current plans for the future include moving the rest of the headers out of the include folder to keep them close to the cpp files. Possibly only expose interfaces to outside units, like plugins. And finally create a proof of concept for a QML implementation, which would look more modern and work on mobile targets.

Please, let me know your thoughts on this proposal.

[tresf]

The diff: master...jsmtux:separate_ui

I have some initial concerns about naming conventions, but they'd be better suited in a proper PR.

[jsmtux]

I understand your concerns, I should have a look at the style guide. However, how do you feel about the next steps for this proposal? I see we could go two ways:

1. Create a proper PR and fix style and all build targets (windows, osx, all libraries...) in order to merge this release cycle
2. I continue building up in a separate branch towards an initial QML app

Option 1 would mean I don't need to constantly rebase on top of the latest changes, but seeing that it is already quite hard for you to release a new version, maybe it would make matters worse...

[tresf]

Create a proper PR and fix style and all build targets (windows, osx, all libraries...) in order to merge this release cycle

I'm not actively involved these days, so please take my opinion with a grain of salt, but, my thoughts:

• Despite how many files that are touched (446), the refactor isn't outrageous to see merged sooner than later, yet provides an important step towards your goal.
• In my (limited) experience, the UI PRs take forever to merge due to:
  • Getting things pixel perfect
  • Strong (differentiating) opinions over position, placement and design
  • Blockers -- such as artwork, refactors, bloated democratic processes, cross-platform bugs

So, it's my opinion that working towards a separation (of UI and core) -- in steps -- is paramount to making it happen and should be done quickly.

That said, I've been inactive for quite a while now, so my peers' opinion should be weighed over my own.

[tresf]

[QML implementation] ... which would look more modern

Placing this here for inspiration #1911 :)

[tresf]

Tagging @LMMS/developers.

[PhysSong]

First of all, thank you for your interest and efforts on LMMS.
Since the core-UI separation is a very large set of changes, it'd be better if you've started discussions before working on the code to avoid making duplicated works or wasting your time. However, since you already have done a lot of works, could you explain the basic ideas and detailed changes first?
Also, it seems like part of #6472 is about what you're working on, so there might be a good place for further discussion.

[sakertooth]

Hi jsmtux,

I appreciate the feedback you gave about #6472. I listed a few reasons why using Qt in the core may not be a good idea, such as real-time safety. It is a lot of work, I agree, and I also understand that Qt comes with a lot of functionality that may be difficult to replace without bringing in other third-party libraries. In any event, some parts of the code still use Qt versions of functionality provided in the STL, so there should still be an effort to removing Qt as per the guidelines. A STL replacement is like a drop-in replacement, given that we have compiler support for the specific functionality in question. We may not be able to fully remove stuff like QDir and QFile until we get full std::filesystem support for C++17, but classes like QString and QVector should definitely be replaced with their corresponding STL alternative ASAP. My knowledge about QML is rather weak, but the option to use it on the UI side shouldn't be limited by the goal of removing Qt from the core. We will still be using Qt on the UI side, that's perfectly fine, it's just the core where things have to be in real-time and efficient (along with other reasons), which Qt may not fit the criteria for.

There is definitely a lot of opinionated ideas surrounding this topic, so I am also curious to see what you think as well.
Thank you for your contribution, and I am looking forward to seeing a PR by you soon.

[jsmtux]

Hi sakertooth

You are right, a qt-less core and QML on top can perfectly go hand in hand. My biggest reason for keeping qt in core is indeed not needing extra effort.

The part I would need to learn more about is how performance is currently hindered by qt. Things like containers, parsers or file utilities won't be impacting audio playback. Containers are going to be as performant as STL and file operations need to happen asynchronously in any case.

Audio could be indeed affected by Qt's message and slot handling, since the thread running the event loop gets frequently stopped to do bookkeeping. For sure time sensitive operations like audio processing and so on need to be on their own thread. If that's not the case at the moment, I understand the need of rewriting that specific part of the code without relying in signals, slots and so on.

In any case, thank you all for the interesting discussion! I am very happy to see that my current plans fall in line with your thoughts, even if we don't yet agree on the final goal.

I'll keep you posted with my progress.

Cheers!

[tresf]

so there should still be an effort to removing Qt [as per the guidelines]

What this describes is removing Qt as a build dependency altogether, but what the OP aims to achieve is to separate the UI from the headless components. What makes the OP's efforts particularly attractive (to me at least) is that it encourages the latter and can help with unit tests (e.g. in headless environments). I would even go as far as to argue that the headless functionality is a hard-requirement of this effort, quoting:

I managed to create a core library that is used by a new target, lmms_headless, that produces an executable that can be invoked from command line and has no QWidget dependencies

This can and should be part of the PR's tests.

Qt in the core may not be a good idea

I don't think anyone argues with this point, but removing signals/slots should always be weighed against just fixing what we have (expand the comments, please) since the latter is often what plagues the user experience, and mistakes have been made that they live with today. <3

My opinion is still weighed lowest here, since I'm not as active as others commenting, but my vote is to fix formatting, take @jsmtux's PR and use it as a first step into separation of UI from headless.

My knowledge about QML is rather weak, but the option to use it on the UI side shouldn't be limited by the goal of removing Qt from the core.

I have never been a fan of QML, personally, but what @jsmtux proposes (new UI) would likely start as a fork or as an experimental opt-in feature so that it can be developed alongside a moving codebase and I think a QML proof of concept would go a long way to prove viability.

[JohannesLorenz]

First of all, thank you very much for this PR. If it can be integrated, it might be a real breakthrough.

The part I would need to learn more about is how performance is currently hindered by qt. Things like containers, parsers or file utilities won't be impacting audio playback. Containers are going to be as performant as STL and file operations need to happen asynchronously in any case.

I once tried to write down the pitfalls with QT's containers. They might look harmless, but often they are lazy-allocated, so it could mean that if the realtime thread does the first ever writing operation it will do a heap allocation (hope it gets clear from the link example).

On the other hand, it might be OK to do the whole thing in 2 steps

• step 1: Separate GUI and core
• step 2: Rewrite core to be realtime safe

Maybe it's even cleaner to separate these two steps?

[jsmtux]

Thanks for the clarification, Johannes.

On my end, I am focusing on step 1. It will unblock me to try different UI options.

However, step 2 would be also simpler to accomplish after the separation. Having a way of running tests without any ui will hopefully enable better performance analysis and so on.

There's one thing I still don't know if I'll do, which is make all headers private in core and only expose the minimum amount of interfaces to enable the UI. This would simplify making changes in the core, since it will be clear what is used from outside and what can be fully modified. However, it will depend on how long this takes.

Please, let me know your thoughts.

[jsmtux]

Hi LMMS developers,

I've been doing quite some work in the past few days and would like your feedback.

As I've been reading in this and other discussions, there is a lot of interest in heavily refactoring the core library of LMMS. I had a feeling that my initial plans for the GUI separation wouldn't effectively enable you to do this properly. Because of this, I'll extend my scope to provide an interface for core as well as for the GUI part:

As you can see in the diagram above, both GUI and Core are now separated into interface and implementation modules. The Core implementation needs to make use of the GUI interface so that it can for example show messages or request updates. The Core interface will have almost no dependencies, whilst the Core implementation will depend on all the relevant sound libraries.

On the interface side, the division is quite similar. Interface has a very limited set of dependencies and each of the implementations will decide what they need. For the QWidgets implementation we'll depend on Qt Widget library. Of course, both GUI implementations will need to call elements from the Core Interface, but that doesn't mean they can now access all core or care about its dependencies.

As you can imagine, this is quite some more work, but I think it does belong in this package. Thanks to this separation, it will now be possible to change the Core implementation completely, as long as the interface remains the same. Moreover, it will be much clearer when implementing the headless gui to see what I should be calling or doing, since I won't be needing to scan throughout the whole core codebase, just only the relevant interfaces.

Moreover, I'd say this whole exercise will simplify the linking process. Since now we'll be mostly linking against interfaces and not real implementations, many of the fixes in the CMake files around this will hopefully become redundant and the process more robust.

I created the core_header_separation branch on top of the previously shared separate_ui one. As it is, only the lmms_headless target is currently building but it effectively shows the plans I've tried to explain and progress I've been doing.

I'll appreciate any feedback or comment

Cheers!

[Veratil]

Getting into the matters outlined in your suggestions, the dependency from core to ui is not something I added, or even like, it is just something that was already there.

I didn't mean to imply that you added it, I meant that the usage in core needs to be removed. :)

When compiling against the QWidgets implementation, the code will behave as before did. However, when compiling against Headless, it can be as simple as:

std::cerr << "File not found " << file_path << std::endl

This, I think, fulfills @michaelgregorius requirement of the Core not making any assumption about how actions are actually handled. It is also quite similar to @Veratil suggestion of having callbacks; a pure virtual class is almost the same as a structure with callbacks.

We could even try just using a struct without callbacks, just use std::optional and allow the GUI to handle the failure to load however they want:

/* in Core */
struct CoreConfig {
  type opt1,
  type opt2,
  // etc.
};
std::optional<CoreConfig> core::loadConfig() { ... }

/* in GUI */
if (auto ret = core::loadConfig(); ret.has_value()) { /* config valid */ }

Alternatively, if there are multiple ways to fail loading the config we could use a pair/tuple like:

enum class ConfigError { None, SomeError, ... };
std::pair<ConfigError, std::optional<CoreConfig>> core::loadConfig() {
  CoreConfig config;
  // for error
  return {ConfigError::SomeError, {}}
  // good return
  return {ConfigError::None, config}
}

if (auto ret = core::loadConfig(); ret.first == ConfigError::None) { /* config valid */ }

In any case, I feel that by implementing the proposal, any subsequent changes will be made much easier. Having a folder that shows all calls from core to gui will clearly outline what still needs to be done instead of being hidden from everyone's eyes.

This does help a ton in seeing what connections are still around.

What we have there is a class from Core actually creating a gui Dialog. This is fixed in my proposal, with all dialog creation happening on the ui. This example is, I feel, a much worse offender of what @michaelgregorius was taking about when mentioning that core shouldn't be controlling anything in the gui.

Not a worse offender, but the same level of offense to me. Any connect from core to UI is equally bad.

My hopes are that, even if this is not a perfect change, will be a good starting point to fix both the core and the gui in LMMS.

This definitely is a great starting point, and we appreciate all the time you put into doing this so far. Thank you! 🥇

[PhysSong]

Alternatively, if there are multiple ways to fail loading the config we could use a pair/tuple

FYI, we can use std::expected once we can use C++23 in LMMS(maybe far future).

[jsmtux]

Thanks for the conversation! I think we are making good progress by aligning on our differences ahead of the PR creation.

I think the original title was distracting the conversation from my main purpose, which is to enable the creation of a more modern UI for LMMS. This discussion is now called "Proposal for enabling alternative UI frameworks through initial separation of core and UI". If you agree, this will narrow the scope a bit and let us focus in the immediate future.

As you indicated, towards a full separation of UI and Core, my proposal would only go half way. On the other hand you'll probably agree that, for the new objective, both problems (error reporting vs. dialog creation) have different risks and it is worth more focusing on one than the other. As I see it, there are two main problems with the createDialog function:

1. The prototype (virtual gui::ControllerDialog * createDialog( QWidget * _parent );) relies on the existence of QWidgets, making core dependant on the QWidgets library.
2. For createDialog to work, core needs to instantiate the real implementation, which means adding a compile time dependency on the real type. core would need to know if it was instantiating a QWidgetDialog, a QMLDialog or a DoNothingDialog at compile time, instead of leaving that responsability to the UI implementation defined at link time.

Both of those would invalidate the possibility of an alternative framework. On the other hand, just exposing interfaces will allow any alternative (QWidgets, QML or Command line to name a few) to define how they'd like to behave, with the build system only deciding which to use at link time. This means we could compile libLMMSCore.a only once and then reuse that output in the lmms_qwidgets lmms_qml and lmms_headless binaries right away.

The fact that I am limiting this scope will mean that the UI implmentation work will be unblocked faster and that LMMS will be in a better position to integrate your ideas. I feel that if we only aim for a full separation on this initial PR, both objectives will be delayed or even never reach fruition.

Continuing with the example of the saveConfigFile, when the function is called from the UI your solution would work perfectly, core will return an error and let UI deal with it. However, that function is not only exercised by the UI, but also from core itself. In ConfigManager's destructor the saveConfigFile function is also executed. Solving that case would incur in analysing in which moments ConfigManager could be destroyed, find which UI actions start that destruction, and only then rewrite the code accordingly. The same goes, for example for the whole ITimeLineWidget interface, which is a particularily complex case.

Cheers!

[michaelgregorius]

@jsmtux, I agree that having the interfaces from Core to GUI will nicely condense where the leftover problems are that need to be tackled to have a clean separation. The goal would then be to replace all methods in the interfaces by implementations that only go from GUI to Core and remove one method after the other from the interfaces.

With regards to Controller::createDialog: from a quick glance over the code it seems that the specific controller implementations are not really used in an abstract or plugin-like way. The Controller class even has the (ugly) enum ControllerTypes which seems to be used quite a lot through the code. I think it might be possible to implement a specific dialog for each Controller subclass so that if the creation of a specific Controller is triggered from the GUI the corresponding controller GUI can be created from the GUI code as well. The dialog will then be given the Controller instance that it controls.

I hope that your code will really get merged as it would improve the code and make it a lot cleaner than it currently is.

[jsmtux]

@michaelgregorius, thanks for your support.

In the refactoring, I found two different cases for functions inside Core creating GUI elements:

1. All options are known at compile time like Controller or Clip: I created factories that, taking an enum value for the specific type, instantiates the correct instance. ControllerDialogsFactory and ClipViewFactory are examples of this. GUI asks either of these factories when needing a specific controller.
2. Options can be defines at runtime through plugins, like those that inherit from Plugin (Instruments, tools, effects...): I started by removing the gui::PluginView * createView( QWidget * parent ); function for the base class Plugin. That way, Core can use plugins with no extra dependencies. Then, new QWidget specific classes were created inheriting from Instrument, Tool, Effect so that, when needing to call the createView function they cast it like:
   static_cast<QWidgetToolPlugin*>(ToolPlugin::instantiate( desc->name, nullptr )) ->createView(this) );
   Which does work quite well, as any ToolPlugin will be of the QWidgetToolPlugin type when we call it from a source within the QWidgets folder.

So the issues surrounding this should be mostly fixed in my separate_ui branch. It should also build and work for at least linux and macOS, so it would be really appreciated if anyone would like to test it out.

The rest of the work I am doing is under the core_header_separation branch. In that one I am limiting the headers that can be used from the outside of the Core library. This one is, I think, very important to have sorted out before merging anything, but it is still half way done.

[josemanuelventoso-tomtom]

Hi Guys,

It has been a while, partly because of having less time to spend on this project, but mostly because of this starting to get a bit complex whilst trying to separate core and UI.

As explained above, I wanted to expose a series of interfaces from the core to the outside. This way, UI won't need to have more knowledge about core's implementation, allowing for neater API and remove compile time dependencies. However, because of the long inheritance chains we have in LMMS, it was getting increasingly harder to create proper interfaces and only expose what I need. After a lot of failed attempts, I think I found the way to go and wanted to open the conversation about it here.

The biggest change I need to incorporate is to remove the Model from the inheritance, especially on Plugins. The current status is as follows:

As you can see there, Plugin is a Model. This means that, if we were to implement something that derives from Plugin it also needs full knowledge of Model, which ends up being really restrictive. What's more, it gets really complex to expose an interface to let's say, Effect. Users of Effect need to use it as a QObject all over the place, so a possible IEffect would need to inherit from QObject, making Effect inherit from QObject twice, which is not allowed.

The option I am trying to implement, and you can have a look at it in my branch refactor_model_out_2, is that Plugin is not a Model, but it contains one to represent its own lifecycle. The same way it could contain a FloatModel or any other. The class diagram looks as follows:

I was very happy to see that this new implementation makes the code much better. Before, PluginView had a relation with Plugin, so every child like InstrumentView needed to access its Instrument through a Model* pointer, needing in the best cases a cast and in the worst an extra poiner to the instrument, which contained the exact same value, but probably the person writing that code didn't know that. After this, InstrumentView contains a pointer to Instrument, making things much cleaner.

This change, on top of improving the code as exposed above, will allow us to do things like the following:

Which allows creating a proper interface to Effect named IEffect that inherits directly from QObject without introducing any problem. It looks like at the moment LMMS is compiling and workign properly, but I didn't fix the Plugins now. I'll get to it in the following days, since it is quite a small change. And, hopefully, creating the proper interfaces for the Core will get much easier.

Let me know about your opinions or suggestions.

[sakertooth]

Hey @jsmtux, it's been some time since this discussion was taken a look at. It seems you have done some admirable work here, so I just wanted to check on its status. I did have some questions (not fully understanding why inheriting Model would be restrictive and the like), but it seems you have a strong approach going for you. Thanks again.

[josemanuelventoso-tomtom]

Hi @sakertooth , thanks for your interest!

To summarize the progress and my plan a little bit, thelmms_core library is now nicely contained and compiling/linking on its own, but exposes too many of its internals and would be hard to implement an UI to use it. Little by little I am removing files from the include folder, so that GUI implementations only have a reduced, well defined, set of interfaces for all their functionality. It will also mean that a developer working in the core wouldn't impact GUIs as long as the interfaces were kept the same.

In this process, every once in a while I find a block on the road when creating the interfaces. When, for example, there is a bunch of functionality that should be internal from a logical point of view, but because of how it is programmed is hard to make abstract. In those cases I create an extra branch to refactor code a bit, so that I can test changes at smaller increments. One of the blockers was the issue with Model you are referring to, and I used the branch refactor_model_out_2 to make it easier to deal with. That branch should now compile/run with plugins and all just like master does.

I understand the Model stuff might be a bit abstract, hopefully my current task which involves Clips and Tracks will be easier to explain. The main blocker is more or less the same one as before, but the solution I am taking is different. In the case of clips I am not breaking the inheritance as I did between Model and some of its child objects (Composition instead of inheritance), since here the inheritance makes more sense (MidiClip is indeed a Clip).

Changes to Clip structure

In LMMS we have Tracks that contain a vector of Clips. Both of them are inherited by other classes, for example SampleTrack and SampleClip, which are related. In the GUI, a SampleTrackView will hold a pointer to the SampleTrack it contains and will interact with SampleClips.

A naive way of exposing interfaces would be to first create a ISampleTrack, so specific variables and slots like volumeModel or playingChanged in SampleTrack could be accessed. Some functions from Track are also used, like CreateClip, creating the need for an Itrack. And it is here where we find the simplest of this kind problems, the paradigmatic example of Diamond Inheritance Problem as shown in the next diagram:

There would be some ways of dealing with this issue in the simple example above, like Track not inheriting from ITrack. It would force every child class to call manually all the Track functions, but it would be a way out. Unfortunately, some more complex cases do require this link, as I'll explain in a moment. For a while I thought virtual inheritance would get me out of this problem, until I realized QObjects cannot be used this way : ( .

A more convoluted example is the usage of the CreateClip function in Clip. When SampleTrack needs a new clip, it will call that function, which returns a Clip. Of course, being a SampleTrack, the clip it links to will always be a SampleClip that, when CreateClip is called, will be returning an object of SampleClip type. The current code does not explicitly reflect this, so the Clip object is always cast to a SampleClip. Because of it, it wouldn't be possible to just break the inheritance fron IClip to Clip, since CreateClip needs to return a working base interface.

In order to solve this issue I tried to reference the most concrete type I could on each situation. Since we know what kind of Clip will be on each Track, I changed all the places that made reference to the base types to the more specific ones. This had lots of side advantages like no more need for casts all over the place, or giving much more context to whoever is reading the class definition.

With the changes in place, there is no more need to cast from a base class to a child one, making code safer but, more importantly, removing the need for ISampleClip to inherit from IClip. This should work nicely since the GUI layer is quite shallow. Let's say we have a window modifying a PatternClip, it will only be dealing with that kind of Clip, so no need for casting to a different kind. In the case of an elment that needs to show different Clip types, only the common things like names and lenghts will be needed, specifics would be part of another window.

A note on Model refactor

With the context about the changes to Clip, I think it is now simpler to explain previous changes to Model and why it didn't make sense for other classes to inherit from it. In most places where we were inheriting and using Model references like with Plugin, for example, code got much more readable and safe to deal with by referencing the actual child class: a function that accepts Model* and always casts it right away to Plugin* is much less readable than if it just receives Plugin* right away.

There were some classes, of course, where we needed to deal with the more generic part of the Models. An example is when setting the Model's parent. Models are arranged in a tree like structure, so that when a child emits changed parents will forward that change up. Anything using a Model gets notified when any of its children get changed. Up until now, we were just passing this, since the current element did inherit from Model. Now model is just an attribute of the current object, and we can just send the attribute, without the need to expose more.

Thanks

If you read all the way maybe you have some comments or further ideas ; ) . In the following days I'll be uploading the branch I am using for the clip changes, and it will be really helpful if you had a look at it or you testted it. In any case, thanks for the interest.

[jsmtux]

Alright, looks like I answered from my work account 😅

[jsmtux]

Hi guys,

After quite a few months of work, I finally have something meaningful to share. You can have a look at the src/core/interface folder in my core_interfaces_ready to see how the final interfaces could look like.

In its current state, the code is quite useless as I still didn't take a decision on how to handle the plugin separation, and the lmms target is not able to fully link because of this. However, I think it is already useful to share because it shows how all the existing functionality in the QWidgets GUI can be implemented with a minimal set of interfaces exposed in the core library.

I will now try to share a summary of the whole change, and how I think this can be useful to LMMS project.

Process followed to get here

As you can read earlier in this thread, the first step I took was to separate core and gui sources into separate libraries. This way, sources that belonged to each module would be compiled separately, making sure that the folder and logical structures were aligned.

Once this was done, I removed dependencies from core to gui, by only exposing the needed functionality through /src/gui/interface. This was easily achievable, since there are very few calls we make from the core to the gui, and most probably in the future we'll be able to reduce those even more.

The real fun began when I tried removing the dependencies from gui to core. I created the /src/core/interface and made it the only externally accessible directory. Then, I started a cycle of:

1. Compile
2. Add missing functionality to the interface folder
3. repeat

Especially at first, mainly because of long inheritance chains, this was not that straightforward. When I found a big roadblock in the process I stashed the changes and went back to the last working commit. Then made the necessary changes, tested it, and rebased the interface separation on top of it, reducing the number of changes in that commit. As I said, the head of the branch cannot be fully built, but everything until that point can be compiled and tested, with most of the bundled songs and samples working and the interface being quite usable.

Finally, after following the cycle for months, this is the first time that the code QWidgets code compiles. I am quite happy to share this update with as it shows quite well what my vision for the future architecture is. I don't expect it will take me that a long time to fix things enough to run again, but I felt it was a big enough achievement to update you about.

Possibilities enabled by this changes

The exposed interfaces show a vertical slice of all the LMMS functionalities. If we were to tidy them up a bit and document them, we would get a much better understanding of the inner workings of the software, and envision new ways of simplifying and extending it.

A developer working with the core library won't need to care about how their changes would affect the GUI, as long as the external interfaces are kept the same.

Someone implementing a new gui, or extending an existing one, wouldn't need to have any knowledge of the inner workings of LMMS, just know the available interfaces, as with any other library.

This would even allow for different LMMS versions to exist. The same base library can be shared by the main software and from a reduced version for Android devices, for instance. With careful versioning of the interfaces, improvements to the core would arrive to all versions seamlessly.

How to continue from here

I understand that, even if I got everything working again and rebased on top of master, it will be quite a big undertaking to get this integrated into LMMS. Even so, I'd say we'd be able to find a sort of organized, low risk approach.

The first step is, I think, to agree on the end goal. It will be very helpful if you could start having a look at the branch and comment on it. There are countless improvements that can be made to the interfaces, and some of the intermediate steps I took could be simplified or improved.

Once the end goal is agreed upon, we could start integrating one by one each change, and split up the bigger ones to make them manageable, making sure everything works before continuing to the next integration.

The commit where the interfaces get hidden is huge, because at the moment I was developing it it was much simpler to hide all at once. Now that we know what we'll be needing from each class, we can start creating the interfaces in separate testable commits and only restrict the access to core once most of the interfaces are in place.

The other part of the project (the more exciting one, I guess), is to take the exposed interfaces and produce working alternative GUI implementations. Getting the command line one to do useful processing will greatly improve our testing capabilities. Getting a sample QML working for the most basic tasks would probably be a very good demonstration of the new architecture. This can be achieved in parallel to the integration, but it might not be the highest priority.

A less desireable alternative

There is another way of going along with this, that in my opinion is not how we should go about it, but can have some advantages anyway.

The biggest part of the changes is about getting the existing QWidgets interface working. In the future, as we improve the interfaces, keeping the GUI in sync with it will also take most of the effort.

The alternative would be something like an "LMMS 2", where we would start by deleting the QWidgets implementation and from there rebuild the GUI in QML without needing to care about all this legacy. This alternative has two big disadvantages, the first being the constant need to rebase from main LMMS to get the latest improvements to the core and the second being that this "LMMS 2" would be unusable for months, and would take even longer to reach feature parity with the legacy version.

Final words

In any case, please have a look that the changes and let me know your opinion on them and your thoughts about their integration.

Thanks

[jsmtux]

Hi Guys,

After my previous post didn't have much response, I felt I hadn't properly explained the opportunities that were enabled by exposing the interfaces in a clean and separate way from the core application. I hope the demo that follows will better show my intentions 😉

lmms_demo.mp4

As you could expect, it is all pretty bare bones, with only kicker and triple oscillator plugins enabled, defaulting to quite a simple project file, little interaction and only works on macOS. Most of these issues, though could be easily fixed, I'll happily receive your comments on what you'd be more interested on.

Now for the good news, you can read or get the code right now at my qt_6_qml_poc branch, it works using Qt6.6 and the QML code it needs is very small and quite self explanatory.

The reason I am most happy about is that, even though a huge effort remains, most of it is pretty mechanical. In the near future I expect to have an in depth write about how this was achieved and how to contribute, for the moment I'll enjoy my holidays.

Cheers!

[tobydox]

Wow, very promising demo! A new clean QtQuick-based UI implementation would be awesome. Even though I'm no longer involved in the LMMS project, I've been working with QML/QtQuick in my daily business for many years now and simply love it. Keep up the good work and I'm sure it could lead to a next-gen LMMS 2.

[Rossmaxx]

I thought of something some time ago and mentioned it in the discord but forgot to mention it here.

What i would like to say is that it makes some sense for a 3 way split rather than a mere core-gui seperation (atleast according to me).

The first layer, ie the "core" can be just the base audio engine and the bare OS agnostic layer. This should be completely free of qt.

The second layer, ie the "interface" can contain the necessary qt stuff needed to work and also the OS/platform specific layers. This layer can also help with qml integration as there might still need to be some non gui qt for some time.

The third layer shall be for gui.

The core and gui can call into the interface for any communication. This approach might add a middle layer but overall, i feel this approach might improve seperation of concerns.

What do you think of ths?

[jsmtux]

This actually sounds ver similar to the situation I am getting to.
My design would indeed end with a layer that manages sound devices, plugins, tracks songs and little more. Over this core, a layer that, as you say, serves the interface to the qml integration. It sets up the application itself and provides the interfaces from the core in a way qml understands. It would also be dependendant on the os, setting things up a bit differently on Windows or android, for instance. Finally, the gui layer is purely qml, with some ui definitions and javascript interaction.
It is indeed quite a powerful division, since the core part could be used as a pure command line application. In the other hand, changes exclusively to the gui could enable all sorts of different apps only by changing the qml side.
The part where our designs diverge, however, is that in my current implementation the core does contain qt. This is not as bad as it might sound as, since qt5, the framework is very modular and including the qt core module doesn't mean we'd also be including any ui, drawing or OS dependant functionality. You can have a look at the qt modules documentation to see how little is provided by the core module, the only qt dependency we'd have in lmms core. On the other hand it does provide functionality we for sure need there, like message passing which would need to be replaced by something else if we take qt out of the equation.
I hope I'll be able to provide a new build usable from Linux and macos quite soon, that can load/save different Lmms projects and modify dynamically loaded plugin parameters from the gui. With it I will document this layer split I am talking about a bit better and hopefully will allow you or others to investigate this kind of interesting designs, as the core will already be quite self contained.
Thanks for your input!

[TechnoPorg]

Just as some food for thought, this sounds vaguely reminiscent of Domain-Driven Design, which I've worked with quite a bit through the ABP framework for web dev, and in that context I can't say I'm the biggest fan. Your design is subtly different, though, and so I think for LMMS it has a lot of potential and is definitely worth considering.

[mulphys]

Hi all,

First I would like to thank all the LMMS developers for giving me inspiration and many rewarding hours I've spent composing and hacking through your code.

It was a while ago I created a local branch in LMMS version 1.13 and implemented a method of Dynamic Transposition described at tectral.com. Now I decided to re-implemented it in the most recent version of LMMS but found the current version to be quite different from what it was back then and a direct merging of the branch was not possible. And here I found this chat about a possible GUI re-implementation which seems to make sense. So, my question is if this would be more appropriate to fork my project from this new branch.

Also, I was wondering where can I learn about how the micro-tonality option was implemented in LMMS. The reason is that dynamic transposition is essentially a "just intonation" scale with an extra transposition track which determines in what key the music is played in each measure. So, I could use micro-tonality for that. Back then I implemented it directly using fractional frequencies.
Your advice is greatly appreciated.

Andrei

[Veratil]

The fork from jsmtux here is not an official branch, so you would still not be able to merge to the official repo. Your best option is to take the time to migrate to a branch off the latest master commit.

For learning about Microtuner, either @LostRobotMusic can give you some directions here or you can join the Discord server and talk with Lost there.

[mulphys]

Thanks for the advice.
I was on discord but got impression that it's mostly for the users.
What channel there can I use if I have questions to developers? Or is this chat a good place?

Also, I compiled the latest sources on Fedora 39 and noticed that CPU is always running full speed when I start LMMS with a default (empty) project. The fan kicks in and I don't know how to stop it. I looked through the existing issues and found something similar but it did not seem to be my case.
I don't remember having that issue in earlier versions (last time i used it was 1.13). Any idea as to what the reason could be?
Andrei

[TechnoPorg]

May I ask if you are planning to open a PR for your branch any time soon? As a fellow new contributor, I've been following what you're doing with quite a bit of interest, and hope to see your impressive efforts get more attention.

[jsmtux]

Hey! Thanks for the support, always encouraging to get some nice comments!

The PR merging is a bit of a complex topic that I've been giving quite some thought lately. I don't think we could merge my branch as is, it has too much garbage. With all the learnings gathered this last few months, however, there might be a way for getting the work merged in a simpler way.

Current state

The work done up until now consisted of the following steps:

1. Build the core and gui folders into separate compile units
2. Move headers out of the common include folder and closer to the sources
3. Stop gui from accessing the core headers and create a core_interfaces library to provide the access
4. Build a QML gui using the new core_interfaces

The first two need to be done the same way I've done them. They are not too costly in terms of time, and the changes are easy to review (mostly moving files around). The biggest problems were some functions that needed to switch to different files or some global variables that needed some better handling. By far the most time consuming part was the third step, as most of the gui code needed some restructuring to adapt to the new interfaces, this might have taken me 80% of the whole time I've spent in this process. And it doesn't need to be done again.

Merge proposal

The first time around I didn't understand the code as well as I do now and it was impossible to me to know all the functionality gui would need from core. The best way for me to do so was by restricting the header visibility to core_interfaces and fix one by one all the compile errors. That's not needed anymore, as we now know that the functionality now exposed in core_interfaces is all the existing (and any other with the same functionality) gui would need.

So I think work could now be split into two parallel activities:

1. Get the main branch in a state that can support the interfaces from my branch
2. Continue building the QML application on the separate branch

Get the main branch in a state that can support the interfaces from my branch

The first two steps I took need to be replicated, so that the core can be built on its own, producing a static library that gui can use. Once this is done, we could keep the QWidgets gui mostly untouched, continuing to depend on the full core library and all its headers.

Then, a new library would be created core_interfaces, by copying the one are currently provided in my branch. Making core fulfill those interfaces will be quite straightforward, in most cases it will be enough for a class like Song to inherit from ISong, and that's it. For things like AutomatableModel or Clip things will be a bit messier, but not too much.

This would be a great moment for documenting the interfaces, making them more consistent and removing things from them that we think won't be needed by the new gui.

Continue building the QML application on the separate branch

If you were to compile my qt_6_qml_poc branch for Linux or MacOS you'd see that there is quite some functionality already working, with support for a few plugins and some interactivity. But it is far from complete. I think this is quite valuable as the more fleshed out this alternative is, the more help we'll get for getting it mainlined. Also, the more we'll understand about the interfaces and what we want from them.

One of the tasks that I think could be a good starting point for other developers interested in QML development could be porting some of the plugins so they can be used here. At the moment I only have kicker, audiofileprocessor and tripleoscillator working, and only the last of those with a useful QML interface. Once I manage to update the code to allow users to browse through their song libraries I'll also try to write a little bit on how to approach this development. Making another plugin available shouldn't be too much work, and will be a good starting point for helping with other elements in the application.

It might sound inefficient to continue working on this separate branch, but once the previous step is finished it will be trivial to port all this work to mainline. This depends on the core_interfaces, so as long as those are mostly the same everything will continue to work.

What do you think?

Quite soon I'll post another update to the prototype that hopefully will prove the idea is worth it, supporting quite some features and looking decent. But from then on I'll need support driving this forward, as it is not possible for me to put everything in place and much less keep up with the development in master.

So let me know if this plan for the merging makes any sense to you. Once I push the next update I'll try to be present in the Discord to support you building or modifying the code.

[TechnoPorg]

Thanks for the super detailed reply!

I am totally on board with getting steps 1 and 2 done and PR'ed ASAP, and it seems like you've basically done the job already.

The next steps also seem good, although (absolutely nothing personal) I'm not as keen on some of the specifics of those, and think they need more discussion - especially among core maintainers smarter than me - before we actually go ahead with them.
For instance, I'd be more in favour of using C++ 20 modules to control the visibility of core interfaces, rather than the proposed core_interfaces layer. I'm not trying to criticize anything you're doing at all, and I'm by no means a project expert; I just think it's worth pausing for formal discussion following the completion of steps 1 and 2.

And let me know if you need any help with it, because I am happy to lend a hand! I am on the LMMS discord server, or just technoporg for DMs.

[michaelgregorius]

Is it possible to mix classic QWidgets with QML? If that was the case then another approach might be to do the reorganization with regards to the interfaces, merge them into master and then to switch one element after the other to QML. I think something like this might be more feasible than a "big bang" approach where everything would have to be switched/reimplemented before it is merged back into master.

[jsmtux]

Sadly it is really not recommended to mix both. I understand the hesitation about a big bang approach, but the fact that the whole gui needs to change at once wouldn't mean all changes need to happen at once.
A possible alternative could be target LMMS 1.4 to be widgets based but also providing a proper interface as previously described. LMMS 1.5 to improve on such interface but keeping existing widgets. And finally 2.0 building the new guide on top of the new interfaces and removing the old one.
It is just an example, but I am sure together we can find other better alternatives.

[jsmtux]

Hi guys,

I haven't had much time lately to work on LMMS, but still did do quite some progress I wanted to share

2024-04-01.17-06-05.mp4

As you can see, the list of projects is now displayed, the active song can be switched and there is some interaction possible with the pattern tracks. If anyone wants to try it, you'd need to install the base qt6 and qtquick development libraries and execute:
make -j lmms2example tripleoscillator kicker audiofileprocessor

Lots of things crash at the moment, like opening a song with a changeTempo call, closing a project that contains an AutomationTrack, clicking on a track with an instrument without an implemented GUI... But all in all, it starts to resemble a real application.

Let me know what you think

Cheers!

[jsmtux]

Hi LMMS team.

I come bringing some good news 🎉 .

The first is that I feel this exploratory process I am doing is now complete. Once I had a promising separation of core and GUI I continued working on this Qt6 QML prototype to prove its usefulness. As one of the advantages of QML is seamless Android support, I decided I would tackle that to support this exercise. Now that an Android prototype is complete, I think it is time for me to try to formally merge some of my changes. I'll try in the coming weeks to get some PR that at least starts by modifying some of the compilation options.

Now to the fun stuff:

lmms_android_demo.1.mp4

As always, you can find the changes to the code in my branch qt_6_qml_android_poc

As you can see some things do work, more or less the same as in the prototype for desktop. The main missing thing is currently AudioFileProcessor sample loading, somehow it is not finding the files. And of course the UI is too big 😅.

The main step for getting this to work was cross compiling the necessary libraries, for some of them it was a bit painful. The ones I used were:

• fftw 3.3.10
• libsamplerate 0.2.2
• libsndfile 1.2.2
• qt 6.7

Then, for the audio I needed to write an (AAudio)[https://developer.android.com/ndk/guides/audio/aaudio/aaudio] backend. It is a very nice framework that supports high performance and low latency audio. I didn't use it to its fullest yet, but even so it works quite nicely, even in debug builds.

Another painful part had to do with storage permissions, as newer Android are quite strict there. In the end I packed everything as resources and wrote a Java Activity that gets them and writes them to the application folder.

Finally, Android does not support shared memory across processes, which of course means no VST support. I added some dummy VST handler and some ifdefs in the code. It is a bit dirty.

So that's it, let me know what you think

[Rossmaxx]

Great job @jsmtux. You might want to look into the qt6 branch where we got working qt 6 builds and we seem to get stock LMMS working on Qt6. If you don't mind, I would advise you to wait a bit longer for that branch to get merged to master and then work on top of the new master. Once that's done, you can just enable the "WANT_QT6" cmake flag and work on top of that. Best of luck.

[jsmtux]

Thanks for your reply!
We'll, this branch is just a prototype, wouldnt want to merge as is. What I'd like to merge are the changes for supporting the core/gui separation, things like compiling them into separate objects or providing interfaces.
I expect that to take a few months anyway, by that time I am sure the qt6 branch you mention will already be in place.

[Rossmaxx]

The main step for getting this to work was cross compiling the necessary libraries, for some of them it was a bit painful. The ones I used were:

• fftw 3.3.10
• libsamplerate 0.2.2
• libsndfile 1.2.2
• qt 6.7

You can use vcpkg to deal with cross compiling those libraries instead of dealing with the hassle yourself. I hope you will find the necessary resources. Don't hesitate to ask help if stuck.

For Qt, i believe you can use the official installer for Android libraries too.

[jsmtux]

So, as shared in a previous update, I've decided to start to find a way to merge the changes for the prototype. If you have a look at the prototype branch, you'll see it is a huge change we cannot expect anyone to review unless it is split into very small, atomic increments. The one problem that splitting could have is that we lose track of what we're trying to accomplish, so I'll first try to document the end goal and then will expose the first chunk of changes, that I think you'll be able to quickly review so we can at least be on the same page to discuss what is ok and what could be improved.

End goal

The end goal for this separation is:

• Nothing from gui is exposed to any target except for an initialization function to main
• No implementation from core is exposed, except for an initialization function to main or very simple common objects
• A set of interfaces will be created and exposed from core
• Interfaces from core will be sufficient for implementing any kind of interface
• Interfaces from core will be properly documented
• Interfaces from core will be generic, not interface dependant

Current change

The change I'd like now to submit for consideration is, I think, the minimum first step. There are no code changes here at all. The only changes are include files moved between folders. The CMake changes are only for setting new include directories. I don't expect the output binaries to change in any significant way.

What I did is

• Move all the header files corresponding to gui sources to src/gui/include
• Move to src/core/include all the remaining files that are not used by src/common

Even if this change does little, it will enable lots of other small increments. For example, we could now start moving most of the files under src/gui/include to src/gui, as most of them aren't used by either core or common.

You can see the PR at #7341

Thanks in advance for your comments!

[jsmtux]

Hi LMMS team,

Maybe you've already had a chance to look at my PR #7341 . I've been keeping it up to date with the latest master and moved some elements in the src/gui/include folder into src/gui, as suggested by some reviewers.
I've also spent some time in preparing what could be the next step, where I do some minor changes to remove dependencies from core to TrackView classes. This might be interesting so you can see the kind of changes that would go into removing all dependencies from core to gui. As you see it is a bit painful, but not too dangerous. You can have a look at it in this PR https://github.com/jsmtux/lmms/pull/3/files .