Test strategy for 3rd party dependencies

[dhebbeker]

We automatically run tests using the "native" build environment. The native build environment is the test environment available on standard desktop environments and on the continuous integration server.

Some of our code has dependencies to 3rd party libraries/code. And some of those libraries may not be compatible to the native build environment.

It is not necessary to test 3rd party libraries themselves.

It must be noted that the build management tool currently used (PlatformIO) does not support fine tuning the selection of sources to be compiled for test execution via the projects' build configuration (platformio.ini). Libraries can be partially compiled using a library.json file to filter files depending on the target build system. Apart from this libraries are build as a whole.

Mocking dependencies is technically possible but not effortless. Libraries such as ArduinoFake can simplify mocking. But currently we use extensions from the Espressif Arduino framework. Those are not covered by ArduinoFake.

The used libraries do not (always) provide interfaces which are compatible to the test environment. That means, in general, that one needs to implement stubs for the 3rd party dependencies.

A possibility to avoid stubbing would be to avoid compiling code for tests which has dependencies to 3rd party code. The code which is then not tested on the native test environment should be kept to a minimum. These are the 3rd party library adapters. This code should be as simple as possible to reduce the risk of introducing errors. Those adapters are then excluded from testing using the native build environment.

I conclude these are the possibilities:

1. Exclude (private) libraries (especially the 3rd party libraries package) from tests which depend on libraries incompatible to the native build environment. Easiest solution; least code coverage.
2. Define library.json files for the (private) libraries to selectively exclude source files from testing which are incompatible to the build environment.
3. Attempt to mock and stub dependencies which can not be compiled using the native build environment. Highest code coverage; highest effort.

What are your thoughts on this? @FiveTeethless do you have any preferences?

[dhebbeker]

For implementing platform-dependent filters, I think extraScript must be used. Make sure to study the Advanced Scripting manual (in particular Construction Environments).

[dhebbeker]

When excluding private libraries (3rd party libraries) or actually any files from build, one will have to mock those. It does not matter if those files are from 3rd party code, using third party code (adapters) or using adapters.

[dhebbeker]

🤔 "Thinking out loud":

• Mocking those libraries, which I have written adapters for (the 3rd party library adapters) is not a good idea: The mocks would depend on the 3rd party libraries. The removal of this dependency is the actual reason I have written the adapters for. Mocking the 3rd party libraries for testing would mean that I would have to change the mocks in case I choose to use different 3rd party libraries or those change. That necessity for change is something which should be prevented by the adapters.
• Mocking the code which depends on the 3rd party adapters has the disadvantage that I do not test the mocked libraries. The amount of dependent source code may be significant.

👉 In conclusion:

• Only the adapters may depend on 3rd party libraries or frameworks. (I may have to move some direct dependencies to Arduino.h to adapters)
• For testing I "simply" replace the 3rd party library adapters with mocks.

Implementing this means that I would not use ArduinoFake for testing any more. As I would have no dependency to Arduino.h any more. Also that means that I would have to write an adapter for the interfacing the microcontroller's hardware (a HAL). Although Arduino.h is already some kind of adapter. Is this the right thing to do? Maybe I make an exception for Arduino? On the other hand I am not using vanilla Arduino which means I can not mock it using ArduinoFake...

[dhebbeker]

Implementing an adapter for Arduino.h may be over the top. A compromise may be to allow the usage of Arduino.h, use ArduinoFake to stub it while testing and provide an adapter from the Espressif Arduino interface to ArduinoFake.

For the remaining 3rd party dependencies: Create stubs (as test-alternative) for the adapters. Do not test the adapters themselves.

[FiveTeethless]

Oh boy, I'm still not overviewing the total scope of the problem😅
Actually, I'm not sure what extension from Espressif we are using. I thought we are using only Arduino features, that might be provifed by Espressif.

Isn't the library.json approach the best of both worlds?
Of course, the setup of this seems to be a bit complicated.
On the other hand, stubbing also is a lot of effort.

Going down the library.json rabbit hole, I thought: If we spent so much effort in bending PlatformIO to our needs, maybe we should think of leaving it behind and go for other alternatives, like your CMAKE or ESP-IDF?

[dhebbeker]

Oh boy, I'm still not overviewing the total scope of the problem😅

Yeah, I did put together some issues. That is a bit confusing. My apologies. 😅

[dhebbeker]

Actually, I'm not sure what extension from Espressif we are using.

I know of these:

• A version of Wire where one can specify the pins to be used. Other devices often only have I2C1, I2C2, .... But with ESP32 one can freely choose which pins to use for I2C.
• Special Serial.printf(). Although I think it can be substituted using other functions.

The use of those extensions will pose a problem as soon as we attempt to stub their usage using ArduinoFake for testing. Even with the current decision DR003, because not all dependencies to (the extended Arduino Code) are outsourced to the 3rd party adapter package (yet?).

[dhebbeker]

Isn't the library.json approach the best of both worlds? Of course, the setup of this seems to be a bit complicated. On the other hand, stubbing also is a lot of effort.

Unfortunately even with the manifest files extensive stubbing would be required. But it would be at least possible.

[dhebbeker]

If we spent so much effort in bending PlatformIO to our needs, maybe we should think of leaving it behind and go for other alternatives, like your CMAKE or ESP-IDF?

I have been on the verge to look for alternatives as well.

Currently, I think that we profit from plenty of PlatformIO features:

• easy management of platforms
• debugging
• easy configuration (as soon as we have figured out how to use it)

But maybe I just don't know the alternatives well enough.

[FiveTeethless]

Hm well I checked the recent changed file just now 😅 there is already a decission.
I agree to the deciission and it's rationale. 👍

[dhebbeker]

For the record: Most of the issues were resolved by #102. Remaining issues have been transferred to #105.