Support for custom TestEngines?

[bddckr]

^{Let me prefix this with a big 🎉 Thank you 🎊! This project rocks and works wonderfully. Great job maintaining it over the years and thanks for sharing it with the Android community!}

I'm interested in getting Kotest to work for Android instrumentation tests. While there is https://github.com/LeoColman/kotest-android, it doesn't check all the boxes for me:

• No IDE integration without the Kotest IntelliJ plugin.
  • Can only run the whole instrumentation test class by using the context menu for the class.
• Confusing IDE integration when using the Kotest IntelliJ plugin.
  • The plugin works as expected and adds gutter icons, but when you try to use those, the tests are attempted to be run on the host machine.

It's all open-source, so I wanted to get my hands dirty and see if I can get it to work better.

So now I'm looking into whether this plugin supports using a custom TestEngine for this purpose. I've been wondering whether IDE integration, for local and instrumentation tests alike, could work in a way that relies on the IDE's support for JUnit Jupiter. This would then allow me to use Kotest without their plugin for local tests, but also without anything than android-junit5 for instrumentation tests.

I saw #329 and am now wondering whether one could get Kotest's TestEngine to work with this plugin 🤔 I'm basically asking what is required to get android-junit5 to use a custom TestEngine that's found on the runtime classpath to

• discover tests (and report them to Android Studio)
• execute tests (and report results)

In a first test, I've gotten a custom TestEngine to fire for both the discovery and the execution phase just fine with your plugin! I didn't get it to show up properly in the IDE yet, but that's probably due to my lack of knowing how a custom TestEngine should be implemented. Getting there 😅 This is ultimately just a test - my goal is to get Kotest's engine to be used.

Perhaps the answer to some of my questions is "you won't be able to get Android Studio to give you the gutter icons" for some reason. That's fine and that's why I'm asking here - to gain some knowledge on this topic in case there's already some that could be shared.

tl;dr: Ignoring Kotest, is there a way to get this plugin to use a custom TestEngine? What are the gotchas, if any?

[mannodermaus]

Thanks for bringing up this topic! Essentially, the instrumentation test libraries provided by android-junit5 hook into the Android instrumentation (which is very much hard-wired to running on JUnit 4 only) and then launch a standard discovery process for test cases with the JUnit Platform. With the most recent advancements in #329, this discovery proess can now find any TestEngine implementation on the runtime classpath and ask it to contribute to the overall test plan. This is how it finds JUnit Jupiter tests in the androidTest source set, and it would then also find stuff written with Kotest, for example.

The way to show gutter icons for non-standard test frameworks seems a bit out of scope for this repo, but essentially the idea is that you'd have a pair of dependencies (in this case for Kotest) and that would be enough for the instrumentation to pick up their tests at runtime:

• You add the API to write tests with to the androidTestImplementation configuration
• You add the engine to discover those tests to the androidTestRuntimeOnly configuration

Note that you do need the local branch from #329 at the moment, since the current versions of the instrumentation library still disregard any non-Jupiter test. But still, that's it in a nutshell! I believe that for this specific use case, IDE integration would likely require some tweaking with the corresponding IntelliJ plugin for that framework, making it aware of what an androidTest actually means. But in terms of executing on an Android device, granted that Kotest doesn't rely on any APIs that are unavailable on Android, this should work!

[adecker89]

It looks like #329 was closed without merging. Are there any plans to bring back TestEngine discovery or did it not work as expected?

[mannodermaus]

I closed #329 in favor of two separate PRs, since the migration to JUnit's default test discovery conflated the original intent of that one (it was meant to only deal with the update to JUnit 5.11). The change that you want to look at managed to land as #335 and is currently active in the latest versions. As I wrote in that PR's description, I was unable to find any TestEngine that didn't rely on unsupported-on-Android APIs from Java, so there is not a proven use case for a non-Jupiter TestEngine running with instrumentation tests. However, theoretically it should be okay for the basic case!