CI improvements

[kevinrenskers]

It would be nice if our CI workflow...

• would automatically cancel all tests if a new commit is added. Now when you push multiple commits in a row, they all run their tests, even though only the last one really matters.
• would retry failing tests. We need to constantly babysit our PRs to manually rerun failing tests.
• wouldn't fail so often, even when locally everything runs fine.

That last point is obviously the most important one. If tests wouldn't constantly fail, we wouldn't have to keep manually retrying them.

[brustolin]

I agree with everything!!

[philipphofmann]

would retry failing tests. We need to constantly babysit our PRs to manually rerun failing tests.

I disagree with this one. Ideally, our tests are stable and don't need a retry mechanism. The first point is interesting and worth investigating. I think the essential point is that we must put in some effort to stabilize the tests.

[kevinrenskers]

"Ideally" being the main word there :) Until they are stable, having them automatically retry is very very nice and saves us a LOT of manual effort. And we did include a retry mechanism.

[philipphofmann]

The risk of having a retry mechanism is that you don't experience the pain of flaky tests, and your test suite worsens. We have to be careful with that.

[kevinrenskers]

Agreed, but we’d still feel the pain when running them locally which I hope we all do as well. This is for when locally everything is totally fine but on CI they keep randomly failing. But for sure 100% agree that the first priority is to fix the flaky tests.

[armcknight]

I think the most actionable thing left here is to cancel in-progress jobs when new commits are pushed, something I strongly agree with. This can possibly be achieved with cancel-in-progress: https://docs.github.com/en/enterprise-cloud@latest/actions/using-jobs/using-concurrency. Will open an issue for this.