SWT and Project Panama aka native interconnect

[laeubi]

The upcoming Project Panama aims at a close integration of java and native code as an alternative for hand-crafted JNI code.

One very interesting part is the jextract tool (you can download a pre build jdk-19-ea with jextract included here. This is based on the JEP 412: Foreign Function & Memory API already part of Java-17.

Even though I don't know if Java 17 is yet an option (@akurtakov ?) and java 19 even a bit far away I'd like to bring the topic to SWT right now so we can investigate and maybe improve and give feedback (e.g. can all relevant SWT native APIs be parsed by jextract?), so I can only encourage every "native swt devloper" to give it a try right now, so hopefully in a glory future we no longer need to build native SWT libraries and can use a pure java interface what will greatly improve the circle of people that could work in that area.

[akurtakov]

Simrel will require Java 17 in 2022-09 release and all EPPs are already shipping with Java 17 JVM so from end user POV this shouldn't be an issue. Some layered products may raise concerns.
For SWT it will be big rewrite to change that so we will probably start with some smaller parts - e.g. webkitgtk on gtk 4.x is not yet ready and porting to it is smth that hopefully we align well - aka swt itself could be fine with Java 11 but browser integration requires Java XX . There are other parts of Eclipse like network proxy and security storage that are done using JNA now and thus would be easier to switch initially. Furthermore they are higher up in the stack and will not enforce everything to move to Java 17 immediately.

[HannesWell]

Java-22 is released today with the Foreign Function & Memory (FFM) API as a final feature through JEP 454.
Compared to the its third preview via JEP 442 in Java-21 only a few changes happened so with some luck the code generated by jextract for Java-22 can also be used on a JDK-21 only with --enable-preview as VM argument.
With Java-21 being the required Java version for the Eclipse SDK and SWT probably in the not too distant future I think this could be considered again.

Using the FFM API would replace the usage of native libaries in SWT and would have several major advantages:

• Java developers would not have to deal with C code anymore, only with pure Java code
• One fragment per OS would be enough since we could save the architecture specific fragments
  • We could even consider that the code for all platforms could go into one bundle, but the generated classes can also be numerous and large so this could become quite big. But I think that's something that can and should be discussed later once the migration is completed and we know exact numbers.
  • No architecture specific content also means that SWT can run on an architecture as long as somebody provides a JDK for it and a user can obtain it. So new architectures (if they arise some day) or exotic ones would be supported without changes.
• No need to build C code anymore and thus a much simpler setup to build SWT from scratch in the CI and locally
  • Probably we would commit the sources generated by jextract into git and just check in the CI if the generated sources are up-to (but that's a detail to be discussed later).

Migrating the entire SWT native code to the FFM API will be a major task, but it can be done gradually. We can just replace native call by native call until all JNI native calls are replaced by FFM calls and no native libraries are necessary anymore.

I assume the most controversial point is if it is ok to require the --enable-preview VM argument to run SWT? For the EPP packages we can add it easily by our own but of course custom products have to be adapted as well. But maybe this is a good opportunity to improve the tooling to simplify the addition of the --enable-preview flag.
Of course the same discussion can be conducted for other parts of the Eclipse platform that use JNI/JNA at the moment too or we have a combined discussion for that.

[laeubi]

I also think one should just check in the generated code, there is no need to generate it over and over again as jextract always requires a special JVM and the native headers and often one wants to adjust something here and there.

And yes one can simply remove the native keyword and redirect the call to the FFM equivalent. I even tested to use the same binding for windows and linux in one example and it worked like a charm, so its not really a question of windows/linux/... but more that the library is the same, e.g. once in the past there was even a swt.win.gtk, now you can use the same code for win/linux as it is all java code, its just a matter of some if / else chains even if some calls diverge.

[akurtakov]

I strongly believe we should start on smaller scale (e.g. https://github.com/eclipse-equinox/equinox/tree/master/bundles/org.eclipse.equinox.security.linux or https://github.com/eclipse-platform/eclipse.platform/tree/master/team/bundles/org.eclipse.core.net.linux) with JNA based bundles so we can get the infrastructure proper (all build JVMs and etc. are still at version 17 and some tools might even fail with Java 21 (it happened during the 11->17 )) and even get more familiar with Panama ourselves.

[laeubi]

We can even start smaller:

• Make Java 21 the primary JVM to execute the maven build eclipse.platform.releng.aggregator#1919

:-)

[tmssngr]

This basically sounds interesting, but I want to emphasize one issue: currently, it is possible to compile a Java/SWT applications to a native executable using GraalVM Native Image. I do not know whether the FFI of the newer Java versions already can be compiled, so I only would consider to change SWT to use the FFI, if it also could be compiled to native executables.

…

-- Thomas Singer

On 2024-03-19 23:31, Hannes Wellmann wrote: Java-22 is released today with the Foreign Function & Memory (FFM) API as a final feature through [JEP 454](https://openjdk.org/jeps/454). Compared to the its third preview via [JEP 442](https://openjdk.org/jeps/442) in Java-21 only a few changes happened so with some luck the code generated by [jextract](https://github.com/openjdk/jextract?tab=readme-ov-file#getting-jextract) for Java-22 can also be used on a JDK-21 only with `--enable-preview` as VM argument. With Java-21 being the required Java version for the Eclipse SDK and SWT probably in the not too distant future I think this could be considered again. Using the FFM API would replace the usage of native libaries in SWT and would have several major advantages: - Java developers would not have to deal with C code anymore, only with pure Java code - One fragment per OS would be enough since we could save the architecture specific fragments - We could even consider that the code for all platforms could go into one bundle, but the generated classes can also be numerous and large so this could become quite big. But I think that's something that can and should be discussed later once the migration is completed and we know exact numbers. - No architecture specific content also means that SWT can run on an architecture as long as somebody provides a JDK for it and a user get it. So new architectures (if they arise some day) or exotic ones would be supported without changes. - No need to build C code anymore and thus a much simpler setup to build SWT from scratch in the CI and locally - Probably we would commit the sources generated by `jextract` into git and just check in the CI if the generated sources are up-to (but that's a detail to be discussed later). Migrating the entire SWT native code to the FFM API will be a major task, but it can be done gradually. We can just replace native call by native call until all JNI native calls are replaced by FFM calls and no native libraries are necessary anymore. I assume the most controversial point is if it is ok to require the `--enable-preview` VM argument to run SWT? For the EPP packages we can add it easily by our own but of course custom products have to be adapted as well. But maybe this is a good opportunity to improve the tooling to simplify the addition of the `--enable-preview` flag. Of course the same discussion can be conducted for other parts of the Eclipse platform that use JNI/JNA at the moment too or we have a combined discussion for that.

[laeubi]

That's more a question for the Graalvm team than something SWT/Platform can solve.

[tmssngr]

Hi Christoph, The first step in the direction of FFI would render SWT useless for me.

…

-- Best regards, Thomas Singer

On 2024-03-20 07:33, Christoph Läubrich wrote: That's more a question for the Graalvm team than something SWT/Platform can solve.

[laeubi]

You can still use older releases of SWT without a problem but FFM is the same as any new Java version / Dependency Upgrade / ... SWT can simply not wait for others to wait as no one is waiting for SWT as well.

So if GraalVM is important for you I suggest to look if there are already discussions and try to bring focus on the topic of Java 21/22 support to it as it will become used sooner or later.

[tmssngr]

Please reconsider your decision as our company has supported SWT in the last years by paying Alexandr Miloslavski.

…

-- Best regards, Thomas Singer

On 2024-03-20 08:31, Christoph Läubrich wrote: You can still use older releases of SWT without a problem but FFM is the same as any new Java version / Dependency Upgrade / ... SWT can simply not wait for others to wait as no one is waiting for SWT as well. So if GraalVM is important for you I suggest to look if there are already discussions and try to bring focus on the topic of Java 21/22 support to it as it will become used sooner or later.

[akurtakov]

I don't see any decision here nor any code change/PR nor a concrete issue.

It's an idea and a good one in terms of reducing maintenance cost and make it easier for more people to contribute to SWT.

If/when/how/what happens is totally up in the air as there is a very long path to be walked before there is even proof of concept, pros vs cons evaluated and so on.
With the above said, we can not shoot an idea that might give fruit in the future because there are problems with it in the moment.

[guw]

@tmssngr The latest GraalVM only supports downcalls on x64. But the current dev version of GraalVM supports downcalls on x64 Linux, Windows and macOS and upcalls on x64 Linux. No Arm64 support, though and I believe SWT also requires upcalls, i.e. callback from OS/native code into SWT.

[laeubi]

@tmssngr The latest GraalVM only supports downcalls on x64. But the current dev version of GraalVM supports downcalls on x64 Linux, Windows and macOS and upcalls on x64 Linux. No Arm64 support, though and I believe SWT also requires upcalls, i.e. callback from OS/native code into SWT.

Thanks for giving an update here a possible mitigation then would be for example to only use down calls for the time being (what should be the was majority).

[HannesWell]

The latest GraalVM only supports downcalls on x64. But the current dev version of GraalVM supports downcalls on x64 Linux, Windows and macOS and upcalls on x64 Linux. No Arm64 support, though and I believe SWT also requires upcalls, i.e. callback from OS/native code into SWT.

Yes it was also mentioned on the Panama-Mailing-List, but I guess/hope that the support in the Graal-VM will improve in the future and since this isn't something that will be applied in the near future of SWT Graal has hopefull catched up until then.

[mickaelistria]

If the idea of using FFI is consensual enough is supported by a clear majority part of the active contributors, then the idea will be implemented anyway.
But SWT can probably start this effort by dealing with 2 branches, or 2 sets of artifacts in the master branch: 1 with FFI (becoming reference/master at some point), and 1 with JNI. The work about FFI could even start right now in a dedicated branch in SWT repo (or again even in the master branch as long as it's isolated enough).
If such an approach for "concurrent" development is chosen, the tricky part is to decide about how to deal with divergence; for that, separated branches might actually be simpler as it's often easy to just rebase.

[iloveeclipse]

I wonder if we can run this SWT experiment with GTK4 port by providing it as a separated fragment, which shouldn't affect anyone as the fragment would not force everyone else to use Java 22+.

In general, a good target for getting rid of native libraries would be next LTS release of Java.

• launcher
• file system access
• SWT + browser

Would be something for the IDE working group to plan for.